Skip to content

Fix PCA NaN handling and remove Clojure alignment hack - #2396

Closed
jucor wants to merge 4 commits into
compdemocracy:edgefrom
jucor:pca_nan_fix
Closed

jucor wants to merge 4 commits into
compdemocracy:edgefrom
jucor:pca_nan_fix

Conversation

@jucor

@jucor jucor commented Feb 25, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Part 1 of the #2311 split (PCA overhaul). This is the foundational fix — subsequent PRs stack on this one.

Stack: This PR → #2397 (test infra) → #2398 (comparer) → #2399 (sklearn PCA) → #2400 (test cleanup)

  • Fix Python PCA to use column means for missing votes (instead of 0), matching the Clojure behavior
  • Add sign-flip-aware comparison in the regression comparer
  • Add tests for NaN handling, PCA regression, and Clojure comparison
  • Remove the "align with Clojure" scaling hack from PCA that was masking the real issue
  • Update golden records to reflect corrected PCA output

Background

Investigation revealed Python filled NaN with 0 while Clojure fills with column mean. The VW dataset had 16° and 52° angle differences in PC1/PC2. After fix, VW matches exactly and biodiversity is within 7° (expected for power iteration on 82% sparse data).

Reviewer note

The golden record JSON diffs (~47k lines) can be skipped — they are the expected output changes from the NaN handling fix. Focus review on:

  • `pca.py` (NaN handling fix)
  • `comparer.py` (sign-flip-aware comparison)
  • `test_pca_unit.py`, `test_legacy_clojure_regression.py` (new tests)

Test plan

  • 226 passed, 2 skipped, 2 xfailed
  • PCA NaN handling unit tests pass
  • Clojure PCA comparison passes

🤖 Generated with Claude Code

jucor and others added 3 commits February 25, 2026 10:31
commit 2024dde
Author: Julien Cornebise <julien@cornebise.com>
Date:   Mon Nov 24 22:14:34 2025 +0000

    Update test_powerit_pca_with_nans to expect ValueError

    The previous commit changed powerit_pca to raise ValueError when given
    NaN values, making NaN handling the caller responsibility. This test
    now verifies that behavior instead of testing graceful NaN handling.

    🤖 Generated with [Claude Code](https://claude.com/claude-code)

    Co-Authored-By: Claude <noreply@anthropic.com>

commit 9839770
Author: Julien Cornebise <julien@cornebise.com>
Date:   Mon Nov 24 22:14:34 2025 +0000

    Fix PCA NaN handling to use column mean (matching Clojure)

    The Python PCA was using 0 to fill missing votes, while Clojure uses
    column means. This caused significant differences in PCA components
    (16 deg and 52 deg angle differences on VW dataset).

    Changes:
    - pca_project_dataframe: Replace NaN with column means instead of 0
    - powerit_pca: Add ValueError if NaN values reach it (caller must preprocess)
    - test_regression.py: Enable ignore_pca_sign_flip for golden comparisons

    New tests:
    - test_pca_unit.py: test_nan_handling_uses_column_mean verifies column
      mean is used, test_nan_handling_differs_from_zero_fill confirms we
      are not using 0
    - test_legacy_clojure_regression.py: test_pca_components_match_clojure
      compares Python vs Clojure PCA components (correlation and angle)

    Results after fix:
    - VW: PC1/PC2 correlation 1.0/-1.0, angle 0 deg (perfect match)
    - Biodiversity: PC1/PC2 correlation 0.99/-1.0, angle 7/4 deg (minor
      numerical differences from power iteration on 82% sparse data)

    🤖 Generated with [Claude Code](https://claude.com/claude-code)

    Co-Authored-By: Claude <noreply@anthropic.com>

commit 4b33297
Author: Julien Cornebise <julien@cornebise.com>
Date:   Mon Nov 24 22:14:34 2025 +0000

    Test that sign flipping propagates

    If the PCA components flip sign, the projections
    must also flip sign.

commit e307a52
Author: Julien Cornebise <julien@cornebise.com>
Date:   Fri Nov 21 09:36:22 2025 +0000

    Flag an unneeded exception

commit 91a1e9c
Author: Julien Cornebise <julien@cornebise.com>
Date:   Wed Nov 19 15:49:34 2025 +0000

    Report any scaling issues in PCA projections

commit 395e636
Author: Julien Cornebise <julien@cornebise.com>
Date:   Wed Nov 19 15:25:46 2025 +0000

    Turn off off the Clojure sign-flip for PCA

    Of course this breaks the comparison to the golden files, as those included
    the sign flip. So we're also giving the option in the regression test to ignore
    PCA sign-flips. That will be handy later when we work on improving the PCA implementation,
    as various PCA implementations have various sign conventions.

commit 0dc1aa7
Author: Julien Cornebise <julien@cornebise.com>
Date:   Mon Nov 24 22:14:34 2025 +0000

    Clean up unused variables and imports

    Address GitHub Copilot review comments:
    - Log superseded votes count in conversation.py instead of leaving unused
    - Remove unused p1_idx/p2_idx index lookups in corr.py
    - Remove unused all_passed variable in regression_comparer.py
    - Remove unused imports (numpy, Path, List, datetime, stats, pca/cluster functions)

    🤖 Generated with [Claude Code](https://claude.com/claude-code)

    Co-Authored-By: Claude <noreply@anthropic.com>
The original angle computation assumed unit-normalized vectors, which
gave misleading results when comparing Python (exact PCA, unit vectors)
with Clojure stochastic PCA for large conversations (non-unit vectors
due to learning rate blending without re-normalization).

Now outputs both the raw angle and the properly normalized angle,
plus the norms of both vectors for diagnostics. The assertion now
checks the normalized angle, which correctly identifies when vectors
point in similar directions regardless of their magnitudes.

Co-Authored-By: Claude <noreply@anthropic.com>
As discussed above, the alignment with clojure was an AI-hallucinated kludge
to pass tests by customized scaling tailored to each dataset, hiding the problems
with the Python PCA implementation (the handling of NaNs). Now that we have fixed
those, we can remove that madness and update the golden records accordingly.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR is the first step of a PCA overhaul stack: it fixes Python PCA NaN handling to match the legacy Clojure behavior, removes prior “alignment” hacks, and updates regression tooling/tests to tolerate PCA sign ambiguity while validating against golden snapshots and Clojure reference output.

Changes:

  • Update PCA preprocessing to fill missing votes (NaN) with per-column means and enforce “no NaNs” at powerit_pca entry.
  • Extend regression comparer + pytest integration to optionally ignore PCA sign flips and provide clearer mismatch diagnostics (incl. scaling detection).
  • Add/adjust PCA-focused tests (unit, edge-case, and Clojure regression comparisons) and add dataset discovery flags for regression CLIs.

Reviewed changes

Copilot reviewed 9 out of 11 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
delphi/polismath/pca_kmeans_rep/pca.py Switch NaN fill to column means; remove Clojure alignment/scaling hack; make powerit_pca reject NaNs.
delphi/polismath/regression/comparer.py Add ignore_pca_sign_flip option and PCA-path heuristics; detect sign flips/scaling for PCA-related arrays.
delphi/polismath/regression/datasets.py Clarify dataset lookup semantics (explicit name lookup searches local + committed).
delphi/scripts/regression_comparer.py CLI flags for ignoring PCA sign flips and including local datasets; pass through to dataset listing.
delphi/scripts/regression_recorder.py CLI flag to include local datasets; pass through to dataset listing.
delphi/tests/test_regression.py Run regression comparisons with PCA sign-flip tolerance enabled; add comparer-focused unit tests.
delphi/tests/test_pca_unit.py Add tests around projection sign behavior and NaN mean-filling behavior.
delphi/tests/test_pca_edge_cases.py Update expectation: powerit_pca should raise on NaNs.
delphi/tests/test_legacy_clojure_regression.py Add test comparing Python PCA components vs Clojure (sign-invariant).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread delphi/polismath/regression/comparer.py Resolved
Comment thread delphi/scripts/regression_comparer.py Resolved
Comment thread delphi/scripts/regression_recorder.py Resolved
Comment thread delphi/polismath/regression/comparer.py Resolved
Comment thread delphi/polismath/pca_kmeans_rep/pca.py Resolved
- Fix all-NaN column edge case in PCA: replace NaN col_means with 0.0
- Lower per-occurrence sign-flip log level to DEBUG (summary stays WARNING)
- Fix dataset validation: explicit names always search local datasets

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@whilo whilo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall: Approve. This is a solid fix. I traced the bug back to the original PCA port (c520b6af) — the Clojure code at conversation.clj:355-377 explicitly fills nil with per-column averages before PCA, and the Python port replaced that with np.nan_to_num(data, nan=0.0). The align_with_clojure hack (hardcoded sign flips and scaling by dataset size) was layered on top to mask the symptoms. Good to see both the root cause fixed and the hack removed.

Minor feedback:

  1. comparer.py: _detect_scaling_factor uses float | None return type (Python 3.10+ syntax). The project pins >=3.12 in pyproject.toml so this works, but the rest of the codebase uses Optional[float] from typing — worth being consistent.

  2. pca.py line ~508: When col_means contains NaN (all-NaN column), you silently fall back to 0.0. Consider logging a warning — an entirely unvoted column is unusual and could indicate a data pipeline issue upstream.

  3. test_legacy_clojure_regression.py lines 333-337: The unnormalized angle computation is dead code (printed but never asserted on), and the # NOTE: This assumes both vectors are unit-normalized comment is misleading since the code right below it computes the normalized version that's actually used. Either remove it or label it clearly as diagnostic.

  4. Sign convention suggestion for a future PR: Rather than using ignore_pca_sign_flip=True in regression tests (which loosens PCA sensitivity), consider a deterministic sign-fixing convention (e.g., force the largest-magnitude element of each component to be positive). This would make golden records fully reproducible without needing sign flip tolerance. Not a blocker for this PR.

@whilo whilo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall: Approve. This is a solid fix. I traced the bug back to the original PCA port (c520b6af) — the Clojure code at conversation.clj:355-377 explicitly fills nil with per-column averages before PCA, and the Python port replaced that with np.nan_to_num(data, nan=0.0). The align_with_clojure hack (hardcoded sign flips and scaling by dataset size) was layered on top to mask the symptoms. Good to see both the root cause fixed and the hack removed.

Minor feedback:

  1. comparer.py: _detect_scaling_factor uses float | None return type (Python 3.10+ syntax). The project pins >=3.12 in pyproject.toml so this works, but the rest of the codebase uses Optional[float] from typing — worth being consistent.

  2. pca.py line ~508: When col_means contains NaN (all-NaN column), you silently fall back to 0.0. Consider logging a warning — an entirely unvoted column is unusual and could indicate a data pipeline issue upstream.

  3. test_legacy_clojure_regression.py lines 333-337: The unnormalized angle computation is dead code (printed but never asserted on), and the # NOTE: This assumes both vectors are unit-normalized comment is misleading since the code right below it computes the normalized version that's actually used. Either remove it or label it clearly as diagnostic.

  4. Sign convention suggestion for a future PR: Rather than using ignore_pca_sign_flip=True in regression tests (which loosens PCA sensitivity), consider a deterministic sign-fixing convention (e.g., force the largest-magnitude element of each component to be positive). This would make golden records fully reproducible without needing sign flip tolerance. Not a blocker for this PR.

@jucor

jucor commented Mar 5, 2026

Copy link
Copy Markdown
Collaborator Author

Superseded by #2413 (moved head branch to upstream for simpler stacked PR workflow).

@jucor jucor closed this Mar 5, 2026
@jucor
jucor deleted the pca_nan_fix branch March 5, 2026 14:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants