Skip to content

Two-level clustering, Clojure comparison tooling, and test consolidation - #2393

Closed
jucor wants to merge 33 commits into
compdemocracy:jc/pca_test_cleanupfrom
jucor:kmeans_clustering_tooling
Closed

jucor wants to merge 33 commits into
compdemocracy:jc/pca_test_cleanupfrom
jucor:kmeans_clustering_tooling

Conversation

@jucor

@jucor jucor commented Feb 25, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Stacked on #2400 (PCA test cleanup, last PR in the #2311 split). Please review and merge #2396 → #2397 → #2398 → #2399 → #2400 first.

To see only the diff added by this PR (excluding PCA split commits): compare pca_test_cleanup...kmeans_clustering_tooling

⚠️ Important: This PR does NOT achieve parity with Clojure

The Python pipeline still produces significantly different results from the Clojure reference implementation — different cluster counts, empty repness, no comment priorities. This PR lays the groundwork (sklearn K-means, two-level clustering architecture, comparison tooling) for an incoming series of stacked PRs that will fix each discrepancy one-by-one via TDD. The full plan is in docs/PLAN_DISCREPANCY_FIXES.md (see #2394).

Changes

  • Replace custom K-means with sklearn (clusters.py): Uses sklearn.cluster.KMeans and sklearn.metrics.silhouette_score instead of the handwritten implementation ported from Clojure. Initialization uses first-k-distinct-points to match Clojure's deterministic behavior.
  • Implement two-level hierarchical clustering (conversation.py, clusters.py): Matches the Clojure architecture where participants are first grouped into ~100 base clusters, then those are meta-clustered into 2-5 groups using silhouette scoring. Documented in docs/CLOJURE_TWO_LEVEL_CLUSTERING.md.
  • Fix serialization of group-cluster members (conversation.py): to_dict(), get_full_data(), and to_dynamo_dict() were outputting base-cluster IDs instead of participant IDs in group-cluster members. All five serialization sites now call _unfolded_group_clusters() to expand base-cluster IDs to participant IDs, matching Clojure's output format. Added test_serialization_unfolding.py (TDD: 6/8 fail on the bug, 8/8 pass after fix).
  • Consolidate Clojure comparison tests into test_legacy_clojure_regression.py with a shared ClojureComparer library (polismath/regression/clojure_comparer.py). Removes duplicated/scattered legacy test files.
  • Mark known discrepancies as xfail: Tests for Clojure parity (repness, clustering, priorities) are marked xfail with discrepancy IDs (D2, D3, D5, D7, D9, D12) — they will be converted to passing tests as each discrepancy is fixed.
  • Add cold-start Clojure math blob generator (scripts/generate_cold_start_clojure.py): Creates a temporary conversation in Postgres, runs the Clojure poller via Docker, captures the math blob for fair comparison.
  • Add cluster visualization (scripts/visualize_cluster_comparison.py): Visual comparison of Python vs Clojure clustering with PCA sign flip detection.
  • Generate cold-start math blobs for biodiversity and vw datasets.
  • Re-record golden snapshots after two-level clustering and serialization fix.

Test plan

  • 199 passed, 2 skipped, 4 xfailed, 0 failures
  • test_serialization_unfolding.py: 8/8 pass (TDD — 6 fail before fix, 8 pass after)

🤖 Generated with Claude Code

jucor and others added 7 commits February 25, 2026 13:54
Cluster centers are derived from PCA projections, so they inherit the
sign ambiguity. This fix ensures sign flips are detected and corrected
for `.center` paths in addition to `.pca.comps` and `.proj.` paths.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Previously, sign flips were detected per-projection-vector, which fails
when only some components are flipped (e.g., PC1 unchanged, PC2 flipped).

Now detects flips at the component level (.pca.comps[N]) and stores them,
then applies per-dimension correction to projections and cluster centers.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
… tests

TLDR: it works! Differences were just numerical errors, artificially looking
huge on small values. Looking at the whole cloud of projections confirmed the
post sk-learn matches the pre-sklearn results perfectly well.

Problem:
--------
The regression comparer was failing on datasets like FLI and pakistan with
thousands of "differences" showing relative errors up to 874%. Investigation
revealed these were false positives: the high relative errors occurred on
near-zero values (e.g., golden=4.54e-06 vs current=4.42e-05) where even tiny
absolute differences (3e-04) produce huge relative errors.

Diagnosis:
----------
Comparing sklearn SVD-based PCA against power iteration golden snapshots:
- The projection point clouds are visually identical (see scatter plots)
- Important values (Q70-Q100 percentile) match within 0.2%
- Only near-zero values (Q0-Q7 percentile, ~0.0x median) show large rel errors
- These near-zero values represent participants at the origin who don't
  affect visualization or clustering

The element-wise (abs_tol, rel_tol) approach fundamentally cannot handle
this case: it either fails on small values or is too loose for large values.

Solution:
---------
Added projection comparison metrics that measure what actually matters:

| Metric                | Threshold | What it measures                    |
|-----------------------|-----------|-------------------------------------|
| Max |error| / range   | < 1%      | Worst displacement as % of axis    |
| Mean |error| / range  | < 0.1%    | Average displacement as % of axis  |
| R² (all coordinates)  | > 0.9999  | Variance explained (99.99%)        |
| R² (per dimension)    | > 0.999   | Per-PC fit quality (99.9%)         |
| Procrustes disparity  | < 1e-4    | Shape similarity after alignment   |

Results for FLI dataset:
- Max |error| / range: 0.0617% (was flagging 28% rel error on Q1 values)
- R²: 0.9999992
- Procrustes: 8.2e-07

All 7 local datasets now pass regression tests.

Also added:
- Quantile context in error reports (computed from ALL values, not just failures)
- Explanation when element-wise diffs exist but metrics confirm match
@jucor
jucor force-pushed the kmeans_clustering_tooling branch from c282ac1 to a2c92d8 Compare February 25, 2026 13:57
jucor and others added 9 commits February 25, 2026 14:13
- Exclude .pca.center from PCA sign-flip handling (not sign-ambiguous)
- Use AND logic for overall_match (don't let projection metrics override stage failures)
- Guard against division by zero when projection data_range is 0
- Fix return type annotation on _log_projection_metrics

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Note: we will update the golden records in a later commit.
I took care to keep the "sparsity-aware" scaling, computed in a vectorized way.
It's not needed anymore: the PCA is done by Sklearn, and the edge cases
are already handled by wrapped_pca caller, i.e. pca_project_dataframe.

Less code, less complexity, less maintenance.
Re-recorded golden snapshots using the new sklearn SVD-based PCA.
These snapshots now serve as the baseline for regression tests.
Tests that only verify sorting/moderation now use recompute=False to
skip unnecessary PCA computation. Manager tests that need PCA but use
minimal data have targeted filterwarnings with explanations.

Changes:
- Add recompute=False to sorting tests (no PCA needed for ID ordering)
- Add recompute=False to test_moderation (only checks filtering)
- Add @pytest.mark.filterwarnings to TestConversationManager tests
  that use minimal data through the manager interface

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Fix division by zero when participant has no votes (pca.py)
  Mirrors Clojure's (max n-votes 1) approach
- Fix return type annotation for _log_projection_metrics (comparer.py)
- Add explanatory comment for intentional except pass (comparer.py)
- Remove unused variable in test (test_regression.py)
- Add unit test for no-votes edge case (test_pca_unit.py)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@jucor
jucor force-pushed the kmeans_clustering_tooling branch from a2c92d8 to e3fb8a6 Compare February 25, 2026 14:15
@jucor
jucor marked this pull request as ready for review February 25, 2026 14:17
jucor and others added 5 commits February 25, 2026 22:03
- Use pca.mean_ instead of manual centering (sklearn handles centering internally)
- Replace np.random.seed(42) with random_state=42 on PCA constructor
- Update docstring to reflect sklearn PCA + column-mean imputation
- Fix _pca_sign_flips type annotation to Dict[str, Dict[int, int]]
- Scope outlier_fraction to PCA-related paths only (non-PCA data stays strict)
- Add try/except around symlink creation for cross-platform robustness
- Remove unused normalize_vector/proj_vec test helpers

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The sklearn PCA implementation has been validated to match Clojure.
Regressions are now caught by test_regression.py golden snapshots.

This test file will be removed once all other Clojure comparison
tests are similarly validated and covered by regression tests.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…ion.py

- Delete test_golden_data.py: redundant with test_legacy_clojure_regression.py
- Change @Skip to @xfail for test_group_clustering and test_comment_priorities
  so they run and document expected failures rather than being silently skipped

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
jucor and others added 8 commits February 25, 2026 22:10
Consolidate three overlapping test files into a coherent suite:
- Removed: tests/test_legacy_clojure_output.py (structure analysis)
- Removed: tests/legacy_compare_with_clojure.py (direct comparison script)
- Enhanced: tests/test_legacy_clojure_regression.py (main pytest suite)

Created shared utilities and CLI tool:
- polismath/regression/clojure_comparer.py: Comparison utilities with
  ClojureComparer class, cluster distribution/membership comparison,
  PCA projection comparison with transformations
- scripts/clojure_comparer.py: Interactive CLI tool for detailed
  comparison (similar UX to regression_comparer.py)
- docs/CLOJURE_COMPARISON.md: Comprehensive documentation

Organized following codebase patterns:
- Placed utilities in polismath/regression/ alongside other regression
  utilities (comparer.py, recorder.py, datasets.py)
- Updated imports in consuming files
- Added exports to polismath/regression/__init__.py for clean imports

Added pytest marker:
- @pytest.mark.clojure_comparison for selective test execution
- Can exclude with: pytest -m "not clojure_comparison"

Key discovery:
- Clojure uses two-level clustering (participants→base-clusters→groups)
- Python uses single-level clustering (participants→groups directly)
- This architectural difference explains 0% Jaccard similarity

Tests run and detect differences (as expected until initialization is aligned).

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…ture

Level 1: Participants → ~100 base clusters
Level 2: Base clusters → 2-5 group clusters (silhouette-based k selection)

Key changes:
- Add sklearn-based kmeans with first-k initialization (matching Clojure)
- Add silhouette coefficient calculation for optimal k selection
- Implement participant filtering (in-conv threshold logic)
- Add fold/unfold utilities for hierarchical cluster storage
- Update tests to unfold and compare hierarchical structures
- Document Clojure two-level architecture and non-determinism issue

This implements the core two-level clustering but does not yet include
incremental clustering (warm-start from previous state).

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Creates generate_cold_start_clojure.py to generate fresh cold-start Clojure
reference data for fair Python vs Clojure comparison. The script:
- Stops if math worker is running (prevents conflicts)
- Backs up existing math_main row
- Deletes row to force cold-start (load-or-init creates fresh new-conv)
- Runs Clojure computation via Docker
- Extracts cold-start math blob
- Restores original row automatically

Key features:
- Support for --all flag to process all datasets
- Support for --include-local flag for local datasets
- Automatic zid lookup from report_id via reports table
- Loads configuration from polis-kmeans/.env (DATABASE_URL)

Test infrastructure updates:
- datasets.py now prefers cold-start blobs when available
- Added has_cold_start_blob field to DatasetInfo
- get_dataset_files() uses cold-start blob by default

Documentation updates:
- Comprehensive usage guide in SESSION_HANDOFF_KMEANS.md
- Commands to find and verify cold-start blobs
- Configuration requirements and setup instructions

Reference data:
- Generated cold-start blobs for biodiversity and vw datasets
- Tests will now use these for fair cold vs cold comparison

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…add visualization with PCA sign flip detection

Cold-start generation (generate_cold_start_clojure.py):
- Rewrite using "conversation replay" approach that works with Clojure poller design
- Creates temporary conversation with fresh zid, copies votes with fresh timestamps
- Runs poller with MATH_ZID_ALLOWLIST to only process the replayed conversation
- Automatically cleans up all temporary data (math tables, votes, conversation)
- Add bash wrapper script (generate_cold_start.sh) that stops math containers first

Visualization (visualize_cluster_comparison.py):
- Add PCA sign flip detection by comparing component correlations
- Apply sign corrections to base cluster centers before visualization
- Fix convex hull rendering to show outlines for both datasets in overlay view
- Include sign_flips in metrics JSON output

Documentation:
- Update SESSION_HANDOFF_KMEANS.md with new approach and remove "BROKEN" warnings
- Document the conversation replay workflow and cleanup behavior

Regenerate cold-start blobs for biodiversity and vw datasets.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Cold-start generator (generate_cold_start_clojure.py):
- Add --pause-math to pause/resume workers instead of stopping
- Add --verbose/-v for real-time Clojure poller output
- Support multiple datasets as arguments
- Use fast INSERT...SELECT for vote copying (was executemany)
- Handle duplicate votes with DISTINCT ON
- Remove shell wrapper (functionality now in Python script)

Cluster visualizer (visualize_cluster_comparison.py):
- Add --all option for processing all datasets
- Synchronize X/Y axis limits in side-by-side plots
- Print full absolute paths for generated PNGs
- Support multiple datasets as arguments

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- test_basic_outputs: xfail for D9/D5/D7 (wrong z-score thresholds and
  repness formulas produce empty comment_repness)
- test_group_clustering: xfail for D2/D3 (wrong participant threshold
  and missing k-smoother produce different cluster counts)
- test_comment_priorities: update xfail reason to reference D12
- Re-record golden snapshots after two-level clustering changes

Test baseline: 11 passed, 6 xfailed, 0 failures.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR advances Delphi’s alignment work with the Clojure math pipeline by introducing two-level clustering, expanding Clojure/Python comparison tooling, and consolidating/parameterizing the test suite for dataset selection and parallel execution.

Changes:

  • Implement two-level clustering (base clusters → group clusters) and migrate K-means/PCA internals to sklearn-based execution paths.
  • Add/expand Clojure comparison utilities + CLI tooling and generate cold-start Clojure math blobs for fairer comparisons.
  • Consolidate and modernize tests: dataset filtering (--datasets), optional local datasets (--include-local), xdist grouping, and updated regression expectations.

Reviewed changes

Copilot reviewed 35 out of 39 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
delphi/tests/test_repness_smoke.py Switch dataset parametrization to conftest helpers for xdist grouping
delphi/tests/test_regression.py Enable PCA sign-flip tolerance in regression comparer; add comparer unit tests
delphi/tests/test_pipeline_integrity.py Use dataset param helper for grouped parallel execution
delphi/tests/test_pca_unit.py Remove legacy PCA unit tests; focus on sklearn PCA + NaN/edge handling
delphi/tests/test_pca_smoke.py Switch dataset parametrization to conftest helper
delphi/tests/test_pca_edge_cases.py Remove tests for deleted legacy PCA internals; keep dataframe robustness tests
delphi/tests/test_math_pipeline_runs_e2e.py Update mock dataset directory to new biodiversity folder name
delphi/tests/test_legacy_repness_comparison.py Switch dataset parametrization to conftest helper
delphi/tests/test_legacy_clojure_regression.py Consolidate Clojure regression suite; add caching + dataset filtering
delphi/tests/test_legacy_clojure_output.py Remove legacy standalone Clojure output analysis script
delphi/tests/test_golden_data.py Remove redundant golden/Clojure comparison test
delphi/tests/test_conversation_smoke.py Switch dataset parametrization to conftest helper
delphi/tests/test_conversation.py Avoid unnecessary recompute in sorting/moderation tests; suppress sklearn warnings
delphi/tests/legacy_compare_with_clojure.py Remove legacy manual comparison script
delphi/tests/conftest.py Add --datasets, xdist grouping helpers, and collection filtering
delphi/tests/README.md Document dataset selection + parallel execution usage
delphi/scripts/regression_recorder.py Add --include-local support to golden snapshot recorder
delphi/scripts/regression_comparer.py Add ignore-PCA-sign-flip + include-local + outlier-tolerance options
delphi/scripts/generate_cold_start_clojure.py New: generate cold-start Clojure math blobs via DB + docker poller
delphi/scripts/clojure_comparer.py New: CLI to compare Python outputs vs Clojure math_blob
delphi/pyproject.toml Add click dependency; configure pytest --dist=loadgroup; add marker
delphi/polismath/regression/datasets.py Support cold-start math blob detection + preference in dataset files
delphi/polismath/regression/clojure_comparer.py New: shared Clojure comparison utilities (clusters/projections)
delphi/polismath/regression/init.py Export new Clojure comparison utilities
delphi/polismath/pca_kmeans_rep/pca.py Replace legacy PCA + alignment hacks with sklearn PCA + column-mean NaN imputation
delphi/polismath/pca_kmeans_rep/clusters.py Add sklearn KMeans + silhouette helpers; implement first-k-distinct init
delphi/polismath/conversation/conversation.py Implement two-level clustering; emit base-clusters + group-clusters in output
delphi/polismath/benchmarks/bench_pca.py New: PCA benchmark/profile script
delphi/docs/regression_testing.md New: detailed regression testing documentation
delphi/docs/SUBGROUP_CLUSTERING_THIRD_LEVEL.md New: document Clojure subgroup clustering (not implemented)
delphi/docs/CLOJURE_TWO_LEVEL_CLUSTERING.md New: document Clojure clustering architecture and formats
delphi/docs/CLOJURE_COMPARISON.md New: document Clojure comparison test/tooling approach

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines 96 to 132
def init_clusters(data: np.ndarray, k: int) -> List[Cluster]:
"""
Initialize k clusters with centers derived to match Clojure's behavior.

Initialize k clusters with first k distinct data points (matching Clojure).

This uses a simple initialization strategy that takes the first k distinct
points in encounter order. This matches the Clojure implementation's behavior
for deterministic and reproducible clustering results.

Args:
data: Data matrix
data: Data matrix (n_points x n_features)
k: Number of clusters

Returns:
List of initialized clusters
"""
n_points = data.shape[0]

if n_points <= k:
# If fewer points than clusters, make each point its own cluster
return [Cluster(data[i], [i], i) for i in range(n_points)]

# Use deterministic initialization for consistency with Clojure
# Set a fixed random seed
rng = np.random.RandomState(42)

# Prefer points that are far apart for initial centers
# This implements a simplified version of k-means++
centers = []

# Choose the first center randomly
first_idx = rng.randint(0, n_points)
centers.append(data[first_idx])

# Choose the remaining centers
for _ in range(1, k):
# Calculate distances to existing centers
min_dists = []
for i in range(n_points):
point = data[i]
min_dist = min(np.linalg.norm(point - center) for center in centers)
min_dists.append(min_dist)

# Choose the next center with probability proportional to distance
min_dists = np.array(min_dists)

# Handle case where all distances are 0 (prevent divide by zero)
if np.sum(min_dists) == 0:
# If all distances are 0, choose randomly with equal probability
probs = np.ones(n_points) / n_points
else:
probs = min_dists / np.sum(min_dists)

# With fixed seed, this should be deterministic
next_idx = rng.choice(n_points, p=probs)
centers.append(data[next_idx])

# Create clusters with these centers
List of initialized clusters with centers set to first k distinct points
"""
# Get unique rows in order (matches Clojure's distinct + take k)
# np.unique with axis=0 returns sorted unique rows, but we want encounter order
# So we use a different approach to preserve order
seen = set()
unique_indices = []
for i, row in enumerate(data):
row_tuple = tuple(row)
if row_tuple not in seen:
seen.add(row_tuple)
unique_indices.append(i)
if len(unique_indices) >= k:
break

# Take first k distinct points as initial centers
initial_centers = data[unique_indices[:k]]

# Create Cluster objects
clusters = []
for i, center in enumerate(centers):
clusters.append(Cluster(center, [], i))
for i, center in enumerate(initial_centers):
clusters.append(Cluster(center=center, members=[], id=i))

return clusters

Copilot AI Feb 25, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

init_clusters can return fewer than k clusters when the data contains fewer than k distinct points (because it stops after finding k unique rows). Downstream code (e.g., kmeans/clean_start_clusters) assumes exactly k clusters and can misbehave or crash.

Consider explicitly handling the “< k distinct points” case by either (a) reducing k to the number of distinct points found, or (b) padding the remaining centers (e.g., by reusing earlier distinct points) so the returned list length is exactly k and matches the caller’s expectation.

Copilot uses AI. Check for mistakes.
Comment on lines +567 to +651
def _get_first_k_distinct_centers(data: np.ndarray, k: int) -> np.ndarray:
"""
Get first k distinct points as initial centers (matching Clojure's init-clusters).

This uses encounter order (first k unique rows seen) for deterministic initialization.

Args:
data: Data matrix (n_points x n_features)
k: Number of centers needed

Returns:
Array of k initial centers (k x n_features)
"""
seen = set()
unique_indices = []

for i, row in enumerate(data):
row_tuple = tuple(row)
if row_tuple not in seen:
seen.add(row_tuple)
unique_indices.append(i)
if len(unique_indices) >= k:
break

# Return the first k distinct points as centers
return data[unique_indices[:k]]


def kmeans_sklearn(data: np.ndarray,
k: int,
max_iters: int = 100,
init_centers: Optional[np.ndarray] = None,
weights: Optional[np.ndarray] = None,
random_state: int = 42,
use_first_k_init: bool = True) -> Tuple[np.ndarray, np.ndarray, List[np.ndarray]]:
"""
Perform K-means clustering using sklearn with optional weighting.

Args:
data: Data matrix (n_points x n_features)
k: Number of clusters
max_iters: Maximum number of iterations
init_centers: Optional initial cluster centers (k x n_features)
weights: Optional sample weights (n_points,)
random_state: Random seed for reproducibility
use_first_k_init: If True, use first-k initialization (matching Clojure)

Returns:
Tuple of (labels, centers, member_lists) where:
- labels: Cluster assignment for each point (n_points,)
- centers: Final cluster centers (k x n_features)
- member_lists: List of member indices for each cluster
"""
if data.shape[0] == 0:
return np.array([]), np.array([]), []

# Ensure k doesn't exceed number of samples
k = min(k, data.shape[0])

# Set up initialization
if init_centers is not None:
# Use provided centers for initialization
init = init_centers
n_init = 1 # Only one initialization when centers are provided
elif use_first_k_init:
# Use first-k distinct points initialization (matching Clojure)
init = _get_first_k_distinct_centers(data, k)
n_init = 1 # Deterministic initialization, only need one run
else:
# Use k-means++ initialization
init = 'k-means++'
n_init = 10 # Multiple initializations for robustness

# Create and fit the KMeans model
kmeans = KMeans(
n_clusters=k,
init=init,
n_init=n_init,
max_iter=max_iters,
random_state=random_state,
algorithm='lloyd' # Use standard Lloyd's algorithm
)

# Fit with optional sample weights (sklearn handles None gracefully)
kmeans.fit(data, sample_weight=weights)

Copilot AI Feb 25, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kmeans_sklearn uses _get_first_k_distinct_centers(data, k) for deterministic init, but does not ensure the returned centers array has shape (k, n_features). If the data has fewer than k distinct points, _get_first_k_distinct_centers returns fewer rows and sklearn.cluster.KMeans(init=...) will raise because init.shape[0] != n_clusters.

Suggested fix: when use_first_k_init is enabled, compute init_centers = _get_first_k_distinct_centers(...) first and then set k = init_centers.shape[0] (and update member list sizing accordingly), or otherwise pad init_centers to exactly k rows before constructing KMeans.

Copilot uses AI. Check for mistakes.
if len(base_clusters) < 2:
logger.warning(f"Not enough base clusters for group clustering ({len(base_clusters)})")
self.base_clusters = base_clusters
self.group_clusters = base_clusters

Copilot AI Feb 25, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In _compute_clusters, when len(base_clusters) < 2 the code assigns self.group_clusters = base_clusters. But base_clusters[*]['members'] are participant IDs, whereas group_clusters are now intended to have members as base-cluster IDs (and to_dict() also emits group-clusters expecting that schema). This creates an inconsistent output shape in the “single base cluster” case.

Instead, consider emitting a single group cluster whose members is [base_clusters[0]['id']] (or []), keeping the group-cluster schema consistent even for degenerate cases.

Suggested change
self.group_clusters = base_clusters
# Maintain consistent group-cluster schema: members are base-cluster IDs
if len(base_clusters) == 1:
single_base = base_clusters[0]
self.group_clusters = [{
'id': 0,
'center': single_base['center'],
'members': [single_base['id']],
}]
else:
# No base clusters available; no group clusters either
self.group_clusters = []

Copilot uses AI. Check for mistakes.
Comment on lines 95 to 101
@@ -603,53 +100,20 @@ def pca_project_dataframe(df: pd.DataFrame,
'comps': np.zeros((min(n_comps, 2), n_cols))
}

Copilot AI Feb 25, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pca_project_dataframe’s exception fallback builds pca_results['comps'] with shape (min(n_comps, 2), n_cols). If the caller requested more than 2 components and PCA fails, this returns fewer components than requested, which can cause downstream shape assumptions to break.

Consider returning (n_comps, n_cols) (or at least min(n_comps, n_cols)), padding with zeros as needed, to keep the fallback output shape consistent with the function contract.

Copilot uses AI. Check for mistakes.
jucor and others added 2 commits February 25, 2026 23:03
Copilot review fixes:
- clusters.py: adjust k down when fewer distinct points than requested
- conversation.py: group_clusters members are base-cluster IDs (not participants)
- pca.py: fallback comps shape uses min(n_comps, n_cols) not min(n_comps, 2)

Bug fix:
- conversation.py: convert pandas Index to list in repness (JSON serialization)

Test fixes:
- test_clusters: init_clusters returns empty members
- test_conversation: test_recompute uses 10 comments to meet vote threshold
- test_datasets: add missing has_comments arg and cold_start blob fixture
- test_edge_cases: update group_repness expectation for no-clusters case
- test_repness_smoke: mark test_repness_structure as xfail (pre-existing)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…am consumers

Group clusters store base-cluster IDs in 'members' (matching Clojure's
two-level clustering architecture), but downstream functions (conv_repness,
participant_stats, group_votes) need participant IDs to join against the
vote matrix.

Add _unfolded_group_clusters() helper (equivalent to Clojure's
clusters/group-members) and use it in all 5 call sites:
- _compute_repness
- _compute_participant_info
- _compute_group_votes
- to_dict group-votes
- to_dynamo_dict group_votes

Also re-record golden snapshots and remove xfail from test_repness_structure
(now passes with correct unfolding).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- test_group_clustering: unfold Python clusters via _unfolded_group_clusters()
  (both sides now use two-level clustering), tighten thresholds to 0.99
  Jaccard / 0.01 distribution tolerance, require overall_match
- test_comment_priorities: require exact match (1e-6 tolerance) for all
  comment IDs instead of 70% at 20% tolerance
- clojure_comparer: fix docstring to reflect Python also uses two-level
  clustering

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@jucor
jucor force-pushed the kmeans_clustering_tooling branch 2 times, most recently from 0f150d9 to 3c8d039 Compare February 27, 2026 16:03
to_dict(), get_full_data(), and to_dynamo_dict() were writing
self.group_clusters directly, whose 'members' contain base-cluster IDs
(integers 0..N).  Downstream consumers (group_data.py, Clojure compat,
client apps) expect participant IDs.

The internal helper _unfolded_group_clusters() already existed and was
used for repness/participant_info/group_votes computation, but the five
serialization sites were missed.

Fix all five sites to unfold before serializing:
- to_dict():        group-clusters, group_clusters, base-clusters
- get_full_data():  group_clusters
- to_dynamo_dict(): base_clusters / group_clusters

Add test_serialization_unfolding.py (TDD: 6 tests fail on the bug,
8/8 pass after fix) using real recompute() pipeline output.

Re-record golden snapshots to reflect the corrected output.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@jucor jucor closed this Mar 5, 2026
@jucor
jucor deleted the kmeans_clustering_tooling branch March 5, 2026 12:41
@jucor
jucor restored the kmeans_clustering_tooling branch March 5, 2026 12:43
@jucor jucor reopened this Mar 5, 2026
@jucor
jucor changed the base branch from edge to jc/pca_test_cleanup March 5, 2026 12:46
@jucor

jucor commented Mar 5, 2026

Copy link
Copy Markdown
Collaborator Author

Superseded by #2418 (moved head branch to upstream for simpler stacked PR workflow).

@jucor jucor closed this Mar 5, 2026
@jucor
jucor deleted the kmeans_clustering_tooling branch March 5, 2026 14:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants