Repository navigation
Conversation
0b03716 to
86e31a0
Compare
86e31a0 to
c282ac1
Compare
Cluster centers are derived from PCA projections, so they inherit the sign ambiguity. This fix ensures sign flips are detected and corrected for `.center` paths in addition to `.pca.comps` and `.proj.` paths. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Previously, sign flips were detected per-projection-vector, which fails when only some components are flipped (e.g., PC1 unchanged, PC2 flipped). Now detects flips at the component level (.pca.comps[N]) and stores them, then applies per-dimension correction to projections and cluster centers. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
… tests TLDR: it works! Differences were just numerical errors, artificially looking huge on small values. Looking at the whole cloud of projections confirmed the post sk-learn matches the pre-sklearn results perfectly well. Problem: -------- The regression comparer was failing on datasets like FLI and pakistan with thousands of "differences" showing relative errors up to 874%. Investigation revealed these were false positives: the high relative errors occurred on near-zero values (e.g., golden=4.54e-06 vs current=4.42e-05) where even tiny absolute differences (3e-04) produce huge relative errors. Diagnosis: ---------- Comparing sklearn SVD-based PCA against power iteration golden snapshots: - The projection point clouds are visually identical (see scatter plots) - Important values (Q70-Q100 percentile) match within 0.2% - Only near-zero values (Q0-Q7 percentile, ~0.0x median) show large rel errors - These near-zero values represent participants at the origin who don't affect visualization or clustering The element-wise (abs_tol, rel_tol) approach fundamentally cannot handle this case: it either fails on small values or is too loose for large values. Solution: --------- Added projection comparison metrics that measure what actually matters: | Metric | Threshold | What it measures | |-----------------------|-----------|-------------------------------------| | Max |error| / range | < 1% | Worst displacement as % of axis | | Mean |error| / range | < 0.1% | Average displacement as % of axis | | R² (all coordinates) | > 0.9999 | Variance explained (99.99%) | | R² (per dimension) | > 0.999 | Per-PC fit quality (99.9%) | | Procrustes disparity | < 1e-4 | Shape similarity after alignment | Results for FLI dataset: - Max |error| / range: 0.0617% (was flagging 28% rel error on Q1 values) - R²: 0.9999992 - Procrustes: 8.2e-07 All 7 local datasets now pass regression tests. Also added: - Quantile context in error reports (computed from ALL values, not just failures) - Explanation when element-wise diffs exist but metrics confirm match
c282ac1 to
a2c92d8
Compare
- Exclude .pca.center from PCA sign-flip handling (not sign-ambiguous) - Use AND logic for overall_match (don't let projection metrics override stage failures) - Guard against division by zero when projection data_range is 0 - Fix return type annotation on _log_projection_metrics Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Note: we will update the golden records in a later commit.
I took care to keep the "sparsity-aware" scaling, computed in a vectorized way.
It's not needed anymore: the PCA is done by Sklearn, and the edge cases are already handled by wrapped_pca caller, i.e. pca_project_dataframe. Less code, less complexity, less maintenance.
Re-recorded golden snapshots using the new sklearn SVD-based PCA. These snapshots now serve as the baseline for regression tests.
Tests that only verify sorting/moderation now use recompute=False to skip unnecessary PCA computation. Manager tests that need PCA but use minimal data have targeted filterwarnings with explanations. Changes: - Add recompute=False to sorting tests (no PCA needed for ID ordering) - Add recompute=False to test_moderation (only checks filtering) - Add @pytest.mark.filterwarnings to TestConversationManager tests that use minimal data through the manager interface 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Fix division by zero when participant has no votes (pca.py) Mirrors Clojure's (max n-votes 1) approach - Fix return type annotation for _log_projection_metrics (comparer.py) - Add explanatory comment for intentional except pass (comparer.py) - Remove unused variable in test (test_regression.py) - Add unit test for no-votes edge case (test_pca_unit.py) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
a2c92d8 to
e3fb8a6
Compare
- Use pca.mean_ instead of manual centering (sklearn handles centering internally) - Replace np.random.seed(42) with random_state=42 on PCA constructor - Update docstring to reflect sklearn PCA + column-mean imputation - Fix _pca_sign_flips type annotation to Dict[str, Dict[int, int]] - Scope outlier_fraction to PCA-related paths only (non-PCA data stays strict) - Add try/except around symlink creation for cross-platform robustness - Remove unused normalize_vector/proj_vec test helpers Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The sklearn PCA implementation has been validated to match Clojure. Regressions are now caught by test_regression.py golden snapshots. This test file will be removed once all other Clojure comparison tests are similarly validated and covered by regression tests. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…ion.py - Delete test_golden_data.py: redundant with test_legacy_clojure_regression.py - Change @Skip to @xfail for test_group_clustering and test_comment_priorities so they run and document expected failures rather than being silently skipped 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Consolidate three overlapping test files into a coherent suite: - Removed: tests/test_legacy_clojure_output.py (structure analysis) - Removed: tests/legacy_compare_with_clojure.py (direct comparison script) - Enhanced: tests/test_legacy_clojure_regression.py (main pytest suite) Created shared utilities and CLI tool: - polismath/regression/clojure_comparer.py: Comparison utilities with ClojureComparer class, cluster distribution/membership comparison, PCA projection comparison with transformations - scripts/clojure_comparer.py: Interactive CLI tool for detailed comparison (similar UX to regression_comparer.py) - docs/CLOJURE_COMPARISON.md: Comprehensive documentation Organized following codebase patterns: - Placed utilities in polismath/regression/ alongside other regression utilities (comparer.py, recorder.py, datasets.py) - Updated imports in consuming files - Added exports to polismath/regression/__init__.py for clean imports Added pytest marker: - @pytest.mark.clojure_comparison for selective test execution - Can exclude with: pytest -m "not clojure_comparison" Key discovery: - Clojure uses two-level clustering (participants→base-clusters→groups) - Python uses single-level clustering (participants→groups directly) - This architectural difference explains 0% Jaccard similarity Tests run and detect differences (as expected until initialization is aligned). Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…ture Level 1: Participants → ~100 base clusters Level 2: Base clusters → 2-5 group clusters (silhouette-based k selection) Key changes: - Add sklearn-based kmeans with first-k initialization (matching Clojure) - Add silhouette coefficient calculation for optimal k selection - Implement participant filtering (in-conv threshold logic) - Add fold/unfold utilities for hierarchical cluster storage - Update tests to unfold and compare hierarchical structures - Document Clojure two-level architecture and non-determinism issue This implements the core two-level clustering but does not yet include incremental clustering (warm-start from previous state). Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Creates generate_cold_start_clojure.py to generate fresh cold-start Clojure reference data for fair Python vs Clojure comparison. The script: - Stops if math worker is running (prevents conflicts) - Backs up existing math_main row - Deletes row to force cold-start (load-or-init creates fresh new-conv) - Runs Clojure computation via Docker - Extracts cold-start math blob - Restores original row automatically Key features: - Support for --all flag to process all datasets - Support for --include-local flag for local datasets - Automatic zid lookup from report_id via reports table - Loads configuration from polis-kmeans/.env (DATABASE_URL) Test infrastructure updates: - datasets.py now prefers cold-start blobs when available - Added has_cold_start_blob field to DatasetInfo - get_dataset_files() uses cold-start blob by default Documentation updates: - Comprehensive usage guide in SESSION_HANDOFF_KMEANS.md - Commands to find and verify cold-start blobs - Configuration requirements and setup instructions Reference data: - Generated cold-start blobs for biodiversity and vw datasets - Tests will now use these for fair cold vs cold comparison Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…add visualization with PCA sign flip detection Cold-start generation (generate_cold_start_clojure.py): - Rewrite using "conversation replay" approach that works with Clojure poller design - Creates temporary conversation with fresh zid, copies votes with fresh timestamps - Runs poller with MATH_ZID_ALLOWLIST to only process the replayed conversation - Automatically cleans up all temporary data (math tables, votes, conversation) - Add bash wrapper script (generate_cold_start.sh) that stops math containers first Visualization (visualize_cluster_comparison.py): - Add PCA sign flip detection by comparing component correlations - Apply sign corrections to base cluster centers before visualization - Fix convex hull rendering to show outlines for both datasets in overlay view - Include sign_flips in metrics JSON output Documentation: - Update SESSION_HANDOFF_KMEANS.md with new approach and remove "BROKEN" warnings - Document the conversation replay workflow and cleanup behavior Regenerate cold-start blobs for biodiversity and vw datasets. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Cold-start generator (generate_cold_start_clojure.py): - Add --pause-math to pause/resume workers instead of stopping - Add --verbose/-v for real-time Clojure poller output - Support multiple datasets as arguments - Use fast INSERT...SELECT for vote copying (was executemany) - Handle duplicate votes with DISTINCT ON - Remove shell wrapper (functionality now in Python script) Cluster visualizer (visualize_cluster_comparison.py): - Add --all option for processing all datasets - Synchronize X/Y axis limits in side-by-side plots - Print full absolute paths for generated PNGs - Support multiple datasets as arguments Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- test_basic_outputs: xfail for D9/D5/D7 (wrong z-score thresholds and repness formulas produce empty comment_repness) - test_group_clustering: xfail for D2/D3 (wrong participant threshold and missing k-smoother produce different cluster counts) - test_comment_priorities: update xfail reason to reference D12 - Re-record golden snapshots after two-level clustering changes Test baseline: 11 passed, 6 xfailed, 0 failures. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
e3fb8a6 to
0aaf98e
Compare
There was a problem hiding this comment.
Pull request overview
This PR advances Delphi’s alignment work with the Clojure math pipeline by introducing two-level clustering, expanding Clojure/Python comparison tooling, and consolidating/parameterizing the test suite for dataset selection and parallel execution.
Changes:
- Implement two-level clustering (base clusters → group clusters) and migrate K-means/PCA internals to sklearn-based execution paths.
- Add/expand Clojure comparison utilities + CLI tooling and generate cold-start Clojure math blobs for fairer comparisons.
- Consolidate and modernize tests: dataset filtering (
--datasets), optional local datasets (--include-local), xdist grouping, and updated regression expectations.
Reviewed changes
Copilot reviewed 35 out of 39 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| delphi/tests/test_repness_smoke.py | Switch dataset parametrization to conftest helpers for xdist grouping |
| delphi/tests/test_regression.py | Enable PCA sign-flip tolerance in regression comparer; add comparer unit tests |
| delphi/tests/test_pipeline_integrity.py | Use dataset param helper for grouped parallel execution |
| delphi/tests/test_pca_unit.py | Remove legacy PCA unit tests; focus on sklearn PCA + NaN/edge handling |
| delphi/tests/test_pca_smoke.py | Switch dataset parametrization to conftest helper |
| delphi/tests/test_pca_edge_cases.py | Remove tests for deleted legacy PCA internals; keep dataframe robustness tests |
| delphi/tests/test_math_pipeline_runs_e2e.py | Update mock dataset directory to new biodiversity folder name |
| delphi/tests/test_legacy_repness_comparison.py | Switch dataset parametrization to conftest helper |
| delphi/tests/test_legacy_clojure_regression.py | Consolidate Clojure regression suite; add caching + dataset filtering |
| delphi/tests/test_legacy_clojure_output.py | Remove legacy standalone Clojure output analysis script |
| delphi/tests/test_golden_data.py | Remove redundant golden/Clojure comparison test |
| delphi/tests/test_conversation_smoke.py | Switch dataset parametrization to conftest helper |
| delphi/tests/test_conversation.py | Avoid unnecessary recompute in sorting/moderation tests; suppress sklearn warnings |
| delphi/tests/legacy_compare_with_clojure.py | Remove legacy manual comparison script |
| delphi/tests/conftest.py | Add --datasets, xdist grouping helpers, and collection filtering |
| delphi/tests/README.md | Document dataset selection + parallel execution usage |
| delphi/scripts/regression_recorder.py | Add --include-local support to golden snapshot recorder |
| delphi/scripts/regression_comparer.py | Add ignore-PCA-sign-flip + include-local + outlier-tolerance options |
| delphi/scripts/generate_cold_start_clojure.py | New: generate cold-start Clojure math blobs via DB + docker poller |
| delphi/scripts/clojure_comparer.py | New: CLI to compare Python outputs vs Clojure math_blob |
| delphi/pyproject.toml | Add click dependency; configure pytest --dist=loadgroup; add marker |
| delphi/polismath/regression/datasets.py | Support cold-start math blob detection + preference in dataset files |
| delphi/polismath/regression/clojure_comparer.py | New: shared Clojure comparison utilities (clusters/projections) |
| delphi/polismath/regression/init.py | Export new Clojure comparison utilities |
| delphi/polismath/pca_kmeans_rep/pca.py | Replace legacy PCA + alignment hacks with sklearn PCA + column-mean NaN imputation |
| delphi/polismath/pca_kmeans_rep/clusters.py | Add sklearn KMeans + silhouette helpers; implement first-k-distinct init |
| delphi/polismath/conversation/conversation.py | Implement two-level clustering; emit base-clusters + group-clusters in output |
| delphi/polismath/benchmarks/bench_pca.py | New: PCA benchmark/profile script |
| delphi/docs/regression_testing.md | New: detailed regression testing documentation |
| delphi/docs/SUBGROUP_CLUSTERING_THIRD_LEVEL.md | New: document Clojure subgroup clustering (not implemented) |
| delphi/docs/CLOJURE_TWO_LEVEL_CLUSTERING.md | New: document Clojure clustering architecture and formats |
| delphi/docs/CLOJURE_COMPARISON.md | New: document Clojure comparison test/tooling approach |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| def init_clusters(data: np.ndarray, k: int) -> List[Cluster]: | ||
| """ | ||
| Initialize k clusters with centers derived to match Clojure's behavior. | ||
|
|
||
| Initialize k clusters with first k distinct data points (matching Clojure). | ||
|
|
||
| This uses a simple initialization strategy that takes the first k distinct | ||
| points in encounter order. This matches the Clojure implementation's behavior | ||
| for deterministic and reproducible clustering results. | ||
|
|
||
| Args: | ||
| data: Data matrix | ||
| data: Data matrix (n_points x n_features) | ||
| k: Number of clusters | ||
|
|
||
| Returns: | ||
| List of initialized clusters | ||
| """ | ||
| n_points = data.shape[0] | ||
|
|
||
| if n_points <= k: | ||
| # If fewer points than clusters, make each point its own cluster | ||
| return [Cluster(data[i], [i], i) for i in range(n_points)] | ||
|
|
||
| # Use deterministic initialization for consistency with Clojure | ||
| # Set a fixed random seed | ||
| rng = np.random.RandomState(42) | ||
|
|
||
| # Prefer points that are far apart for initial centers | ||
| # This implements a simplified version of k-means++ | ||
| centers = [] | ||
|
|
||
| # Choose the first center randomly | ||
| first_idx = rng.randint(0, n_points) | ||
| centers.append(data[first_idx]) | ||
|
|
||
| # Choose the remaining centers | ||
| for _ in range(1, k): | ||
| # Calculate distances to existing centers | ||
| min_dists = [] | ||
| for i in range(n_points): | ||
| point = data[i] | ||
| min_dist = min(np.linalg.norm(point - center) for center in centers) | ||
| min_dists.append(min_dist) | ||
|
|
||
| # Choose the next center with probability proportional to distance | ||
| min_dists = np.array(min_dists) | ||
|
|
||
| # Handle case where all distances are 0 (prevent divide by zero) | ||
| if np.sum(min_dists) == 0: | ||
| # If all distances are 0, choose randomly with equal probability | ||
| probs = np.ones(n_points) / n_points | ||
| else: | ||
| probs = min_dists / np.sum(min_dists) | ||
|
|
||
| # With fixed seed, this should be deterministic | ||
| next_idx = rng.choice(n_points, p=probs) | ||
| centers.append(data[next_idx]) | ||
|
|
||
| # Create clusters with these centers | ||
| List of initialized clusters with centers set to first k distinct points | ||
| """ | ||
| # Get unique rows in order (matches Clojure's distinct + take k) | ||
| # np.unique with axis=0 returns sorted unique rows, but we want encounter order | ||
| # So we use a different approach to preserve order | ||
| seen = set() | ||
| unique_indices = [] | ||
| for i, row in enumerate(data): | ||
| row_tuple = tuple(row) | ||
| if row_tuple not in seen: | ||
| seen.add(row_tuple) | ||
| unique_indices.append(i) | ||
| if len(unique_indices) >= k: | ||
| break | ||
|
|
||
| # Take first k distinct points as initial centers | ||
| initial_centers = data[unique_indices[:k]] | ||
|
|
||
| # Create Cluster objects | ||
| clusters = [] | ||
| for i, center in enumerate(centers): | ||
| clusters.append(Cluster(center, [], i)) | ||
| for i, center in enumerate(initial_centers): | ||
| clusters.append(Cluster(center=center, members=[], id=i)) | ||
|
|
||
| return clusters |
There was a problem hiding this comment.
init_clusters can return fewer than k clusters when the data contains fewer than k distinct points (because it stops after finding k unique rows). Downstream code (e.g., kmeans/clean_start_clusters) assumes exactly k clusters and can misbehave or crash.
Consider explicitly handling the “< k distinct points” case by either (a) reducing k to the number of distinct points found, or (b) padding the remaining centers (e.g., by reusing earlier distinct points) so the returned list length is exactly k and matches the caller’s expectation.
| def _get_first_k_distinct_centers(data: np.ndarray, k: int) -> np.ndarray: | ||
| """ | ||
| Get first k distinct points as initial centers (matching Clojure's init-clusters). | ||
|
|
||
| This uses encounter order (first k unique rows seen) for deterministic initialization. | ||
|
|
||
| Args: | ||
| data: Data matrix (n_points x n_features) | ||
| k: Number of centers needed | ||
|
|
||
| Returns: | ||
| Array of k initial centers (k x n_features) | ||
| """ | ||
| seen = set() | ||
| unique_indices = [] | ||
|
|
||
| for i, row in enumerate(data): | ||
| row_tuple = tuple(row) | ||
| if row_tuple not in seen: | ||
| seen.add(row_tuple) | ||
| unique_indices.append(i) | ||
| if len(unique_indices) >= k: | ||
| break | ||
|
|
||
| # Return the first k distinct points as centers | ||
| return data[unique_indices[:k]] | ||
|
|
||
|
|
||
| def kmeans_sklearn(data: np.ndarray, | ||
| k: int, | ||
| max_iters: int = 100, | ||
| init_centers: Optional[np.ndarray] = None, | ||
| weights: Optional[np.ndarray] = None, | ||
| random_state: int = 42, | ||
| use_first_k_init: bool = True) -> Tuple[np.ndarray, np.ndarray, List[np.ndarray]]: | ||
| """ | ||
| Perform K-means clustering using sklearn with optional weighting. | ||
|
|
||
| Args: | ||
| data: Data matrix (n_points x n_features) | ||
| k: Number of clusters | ||
| max_iters: Maximum number of iterations | ||
| init_centers: Optional initial cluster centers (k x n_features) | ||
| weights: Optional sample weights (n_points,) | ||
| random_state: Random seed for reproducibility | ||
| use_first_k_init: If True, use first-k initialization (matching Clojure) | ||
|
|
||
| Returns: | ||
| Tuple of (labels, centers, member_lists) where: | ||
| - labels: Cluster assignment for each point (n_points,) | ||
| - centers: Final cluster centers (k x n_features) | ||
| - member_lists: List of member indices for each cluster | ||
| """ | ||
| if data.shape[0] == 0: | ||
| return np.array([]), np.array([]), [] | ||
|
|
||
| # Ensure k doesn't exceed number of samples | ||
| k = min(k, data.shape[0]) | ||
|
|
||
| # Set up initialization | ||
| if init_centers is not None: | ||
| # Use provided centers for initialization | ||
| init = init_centers | ||
| n_init = 1 # Only one initialization when centers are provided | ||
| elif use_first_k_init: | ||
| # Use first-k distinct points initialization (matching Clojure) | ||
| init = _get_first_k_distinct_centers(data, k) | ||
| n_init = 1 # Deterministic initialization, only need one run | ||
| else: | ||
| # Use k-means++ initialization | ||
| init = 'k-means++' | ||
| n_init = 10 # Multiple initializations for robustness | ||
|
|
||
| # Create and fit the KMeans model | ||
| kmeans = KMeans( | ||
| n_clusters=k, | ||
| init=init, | ||
| n_init=n_init, | ||
| max_iter=max_iters, | ||
| random_state=random_state, | ||
| algorithm='lloyd' # Use standard Lloyd's algorithm | ||
| ) | ||
|
|
||
| # Fit with optional sample weights (sklearn handles None gracefully) | ||
| kmeans.fit(data, sample_weight=weights) |
There was a problem hiding this comment.
kmeans_sklearn uses _get_first_k_distinct_centers(data, k) for deterministic init, but does not ensure the returned centers array has shape (k, n_features). If the data has fewer than k distinct points, _get_first_k_distinct_centers returns fewer rows and sklearn.cluster.KMeans(init=...) will raise because init.shape[0] != n_clusters.
Suggested fix: when use_first_k_init is enabled, compute init_centers = _get_first_k_distinct_centers(...) first and then set k = init_centers.shape[0] (and update member list sizing accordingly), or otherwise pad init_centers to exactly k rows before constructing KMeans.
| if len(base_clusters) < 2: | ||
| logger.warning(f"Not enough base clusters for group clustering ({len(base_clusters)})") | ||
| self.base_clusters = base_clusters | ||
| self.group_clusters = base_clusters |
There was a problem hiding this comment.
In _compute_clusters, when len(base_clusters) < 2 the code assigns self.group_clusters = base_clusters. But base_clusters[*]['members'] are participant IDs, whereas group_clusters are now intended to have members as base-cluster IDs (and to_dict() also emits group-clusters expecting that schema). This creates an inconsistent output shape in the “single base cluster” case.
Instead, consider emitting a single group cluster whose members is [base_clusters[0]['id']] (or []), keeping the group-cluster schema consistent even for degenerate cases.
| self.group_clusters = base_clusters | |
| # Maintain consistent group-cluster schema: members are base-cluster IDs | |
| if len(base_clusters) == 1: | |
| single_base = base_clusters[0] | |
| self.group_clusters = [{ | |
| 'id': 0, | |
| 'center': single_base['center'], | |
| 'members': [single_base['id']], | |
| }] | |
| else: | |
| # No base clusters available; no group clusters either | |
| self.group_clusters = [] |
| @@ -603,53 +100,20 @@ def pca_project_dataframe(df: pd.DataFrame, | |||
| 'comps': np.zeros((min(n_comps, 2), n_cols)) | |||
| } | |||
There was a problem hiding this comment.
pca_project_dataframe’s exception fallback builds pca_results['comps'] with shape (min(n_comps, 2), n_cols). If the caller requested more than 2 components and PCA fails, this returns fewer components than requested, which can cause downstream shape assumptions to break.
Consider returning (n_comps, n_cols) (or at least min(n_comps, n_cols)), padding with zeros as needed, to keep the fallback output shape consistent with the function contract.
Copilot review fixes: - clusters.py: adjust k down when fewer distinct points than requested - conversation.py: group_clusters members are base-cluster IDs (not participants) - pca.py: fallback comps shape uses min(n_comps, n_cols) not min(n_comps, 2) Bug fix: - conversation.py: convert pandas Index to list in repness (JSON serialization) Test fixes: - test_clusters: init_clusters returns empty members - test_conversation: test_recompute uses 10 comments to meet vote threshold - test_datasets: add missing has_comments arg and cold_start blob fixture - test_edge_cases: update group_repness expectation for no-clusters case - test_repness_smoke: mark test_repness_structure as xfail (pre-existing) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…am consumers Group clusters store base-cluster IDs in 'members' (matching Clojure's two-level clustering architecture), but downstream functions (conv_repness, participant_stats, group_votes) need participant IDs to join against the vote matrix. Add _unfolded_group_clusters() helper (equivalent to Clojure's clusters/group-members) and use it in all 5 call sites: - _compute_repness - _compute_participant_info - _compute_group_votes - to_dict group-votes - to_dynamo_dict group_votes Also re-record golden snapshots and remove xfail from test_repness_structure (now passes with correct unfolding). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- test_group_clustering: unfold Python clusters via _unfolded_group_clusters() (both sides now use two-level clustering), tighten thresholds to 0.99 Jaccard / 0.01 distribution tolerance, require overall_match - test_comment_priorities: require exact match (1e-6 tolerance) for all comment IDs instead of 70% at 20% tolerance - clojure_comparer: fix docstring to reflect Python also uses two-level clustering Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
0f150d9 to
3c8d039
Compare
to_dict(), get_full_data(), and to_dynamo_dict() were writing self.group_clusters directly, whose 'members' contain base-cluster IDs (integers 0..N). Downstream consumers (group_data.py, Clojure compat, client apps) expect participant IDs. The internal helper _unfolded_group_clusters() already existed and was used for repness/participant_info/group_votes computation, but the five serialization sites were missed. Fix all five sites to unfold before serializing: - to_dict(): group-clusters, group_clusters, base-clusters - get_full_data(): group_clusters - to_dynamo_dict(): base_clusters / group_clusters Add test_serialization_unfolding.py (TDD: 6 tests fail on the bug, 8/8 pass after fix) using real recompute() pipeline output. Re-record golden snapshots to reflect the corrected output. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Superseded by #2418 (moved head branch to upstream for simpler stacked PR workflow). |
Summary
The Python pipeline still produces significantly different results from the Clojure reference implementation — different cluster counts, empty repness, no comment priorities. This PR lays the groundwork (sklearn K-means, two-level clustering architecture, comparison tooling) for an incoming series of stacked PRs that will fix each discrepancy one-by-one via TDD. The full plan is in
docs/PLAN_DISCREPANCY_FIXES.md(see #2394).Changes
clusters.py): Usessklearn.cluster.KMeansandsklearn.metrics.silhouette_scoreinstead of the handwritten implementation ported from Clojure. Initialization uses first-k-distinct-points to match Clojure's deterministic behavior.conversation.py,clusters.py): Matches the Clojure architecture where participants are first grouped into ~100 base clusters, then those are meta-clustered into 2-5 groups using silhouette scoring. Documented indocs/CLOJURE_TWO_LEVEL_CLUSTERING.md.conversation.py):to_dict(),get_full_data(), andto_dynamo_dict()were outputting base-cluster IDs instead of participant IDs in group-clustermembers. All five serialization sites now call_unfolded_group_clusters()to expand base-cluster IDs to participant IDs, matching Clojure's output format. Addedtest_serialization_unfolding.py(TDD: 6/8 fail on the bug, 8/8 pass after fix).test_legacy_clojure_regression.pywith a sharedClojureComparerlibrary (polismath/regression/clojure_comparer.py). Removes duplicated/scattered legacy test files.xfailwith discrepancy IDs (D2, D3, D5, D7, D9, D12) — they will be converted to passing tests as each discrepancy is fixed.scripts/generate_cold_start_clojure.py): Creates a temporary conversation in Postgres, runs the Clojure poller via Docker, captures the math blob for fair comparison.scripts/visualize_cluster_comparison.py): Visual comparison of Python vs Clojure clustering with PCA sign flip detection.Test plan
test_serialization_unfolding.py: 8/8 pass (TDD — 6 fail before fix, 8 pass after)🤖 Generated with Claude Code