diff --git a/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md b/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md index d4f23505f3..46f89bb8d7 100644 --- a/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md +++ b/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md @@ -542,8 +542,123 @@ has no defensible behavior (doubly wrong disagree, weighted sum vs product). No ### What's Next -1. **PR 7 — Fix D8 (Finalize comment stats)**: Change repful classification from - `pa > 0.5 AND ra > 1.0` to `rat > rdt` (Clojure's simpler logic). +1. **PR 7 — Fix D8 (Finalize comment stats logic)**: Switch the `repful` classification + to match Clojure's `rat > rdt` rule (next section below). + +--- + +## PR 7: Fix D8 — Finalize Comment Stats Logic + +### TDD steps +1. **Baseline**: 5 xfailed (D8 tests), 289 passed, 3 skipped, 60 xfailed (public) +2. **Red**: Removed xfail from D8 formula tests, added 4 new edge case tests → 5 failures + (test_repful_uses_rat_vs_rdt, test_rat_greater_than_rdt_is_agree, + test_equal_rat_rdt_is_disagree, test_both_negative, test_both_zero) +3. **Fix**: Replaced both scalar and vectorized repful classification: + - Old: `if pa > 0.5 and ra > 1.0 → 'agree'; elif pd > 0.5 and rd > 1.0 → 'disagree'; else: higher metric` + - New: `if rat > rdt → 'agree'; else → 'disagree'` (repness.clj:175-177) +4. **Green**: All 5 D8 formula tests pass, 4 blob comparison tests xfail (D10 selection) +5. **Full suite (public)**: 4 regression failures, all in `comment_repness[*].repness` — the + repness value changes direction when repful flips (ra↔rd). No other fields affected. +6. **Re-recorded golden snapshots** for all 7 datasets (public + private) +7. **Final (with --include-local)**: 19/19 regression tests pass, 3 pre-existing failures + (pakistan-incremental D2, bg2050/pakistan incremental PCA dimensions) + +### Changes +- `repness.py`: `finalize_cmt_stats()` — replaced 3-branch threshold logic with `rat > rdt` +- `repness.py`: Vectorized repful in `compute_group_comment_stats_df()` — replaced `np.select` + with `np.where(rat > rdt, 'agree', 'disagree')` +- `test_discrepancy_fixes.py`: Expanded `TestD8FinalizeStats` from 2 to 7 tests (5 formula + + 1 blob xfail + edge cases for equal/negative/zero rat/rdt) + +### Key insight: the old logic was unnecessarily complex +Clojure simply compares `rat > rdt` — the two-proportion z-test scores already encode group +significance. The old Python logic added redundant probability/ratio thresholds (`pa > 0.5`, +`ra > 1.0`) with a metric-based fallback, but these gates are unnecessary given that rat/rdt +already capture the relative group difference. + +### Session 10 (2026-03-14) + +- Created branch `jc/clj-parity-d8-finalize-stats` on top of `jc/clj-parity-d7-repness-metric` +- Read Clojure source (repness.clj:170-185) to verify formula +- TDD cycle: red (5 failures) → fix → green (5 pass, 4 xfail) +- Verified regression diffs are all in `comment_repness[*].repness` — repful direction change +- Re-recorded golden snapshots for all 7 datasets +- Final validation: 19/19 regression tests pass + +### What's Next + +1. **PR 14 refactor**: Extract stats computation from `compute_group_comment_stats_df` for + testability, then add vectorized blob injection tests. +2. Review remaining VM fixes: D15, D10, D11, D3, D12, D1, PR15. +3. K-divergence investigation (separate session off D15 branch). + +--- + +## Review Session (2026-03-17) + +### What was done + +**Reviewed and created draft PRs for D5, D6, D7, D8:** +- #2448 (D5), #2449 (D6), #2450 (D7), #2451 (D8) — all draft +- Reviewed production code, tests, docs, golden snapshots for each +- D6: restored dropped Clojure XXX comment about pi_hat==1 edge case +- D7: confirmed no feature flag needed (old formula doubly wrong) +- D8: noted (1-pd) = pa identity, confirming old disagree formula was backwards + +**Added blob injection tests (scalar path):** +- D5: `prop_test(n_success, n_trials)` vs blob `p-test` — RED→GREEN verified +- D6: `two_prop_test(group counts)` vs blob `repness-test` — RED→GREEN verified +- D4: `(n_success+1)/(n_trials+2)` vs blob `p-success` — passes (already fixed) +- D8: `rat > rdt` vs blob `repful-for` — passes (D6 fix makes rat/rdt correct) + +**Key discovery: formula-only tests are tautological.** They verify our code matches our +reading of the Clojure source — if we misread it, the test passes and the code is wrong. +Only blob injection tests (comparing against real Clojure output) catch real mismatches. +Every fix PR must now include blob comparison tests. + +**Key discovery: Python and Clojure pick different k (vw: Python=4, Clojure=2).** +Both use silhouette. The divergence comes from upstream PCA/clustering differences +(sklearn SVD vs Clojure power iteration). This is independent of all repness fixes +(D4-D11). Investigation planned off D15 branch. See +`delphi/docs/HANDOFF_K_DIVERGENCE_INVESTIGATION.md`. + +**Key discovery: `n-trials` in Clojure blob = `S` (total seen, including passes),** +not `A+D` (agrees + disagrees). Verified: `prop_test(11, 14)` = blob `p-test` for +tid=49 group 0 in vw, where A=2, D=11, S=14, A+D=13. + +**Infrastructure fixes:** +- CI: added `jc/**` to `pull_request.branches` in `python-ci.yml` (bottom of stack) +- CI: `find_dotenv()` + `DATABASE_*` fallback in `test_postgres_real_data.py` +- CI: removed dead `POSTGRES_*` exec vars from workflow +- All stack PRs now have passing CI (Delphi Python Tests) + +**Stack reordering:** +- D15 moved before D10 (was after D3) — prerequisite for k-divergence investigation +- New order: D8 → D15 → D10 → D11 → D3 → D12 → D1 → PR15 +- D15 only touches `conversation.py`, independent of repness fixes + +**Vectorized blob injection tests: deferred.** The `compute_group_comment_stats_df` +function is too monolithic to test in isolation — it builds its own DataFrame from +raw inputs. PR 14 refactor (extract stats computation) is needed first to make the +vectorized path testable. Plan: refactor at base of repness chain, then re-climb +adding vectorized blob tests at each stage. + +### Session 11 (2026-03-17) + +- Fetched 6 new VM branches (D10, D11, D3, D15, D12, D1) + PR15 +- Reviewed D5, D6, D7, D8 code, tests, docs — all approved with minor fixes +- Created draft PRs #2448-#2451 on GitHub +- Cleaned macOS resource fork artifacts from VM private data repo +- Fetched private data snapshots from VM (D5+D6 committed, D7+D8 uncommitted on VM) +- Updated stack titles (17 PRs) +- Discovered CI wasn't running Python tests on `jc/**` PRs — fixed +- Discovered `jc/fix-test-db-connection` broke CI (hardcoded .env path + no DATABASE_URL fallback) — fixed +- Added blob injection tests to D5 (RED→GREEN), D6 (RED→GREEN), D8 branches +- Investigated k divergence: Python=4, Clojure=2 on vw cold-start +- Created handoff doc and plan section for k-divergence investigation +- Reordered stack: D15 before D10 for k-investigation prereq +- Rebased full chain (D15 → D10 → D11 → D3 → D12 → D1 → PR15) onto new D8 --- @@ -605,3 +720,13 @@ to create a new worktree. If yes, provide a prompt they can use to start that se - `strict=False` on xfail means xpass (unexpected pass) is reported but not a failure. Used when some datasets pass by coincidence. - After rebase, D9 `test_repness_not_empty` started xpassing — the `comment_repness` list is populated (all pairs), but `group_repness` (selected reps) may still be affected by wrong thresholds. Consider tightening this test when fixing D9. - To rebase when base is updated: `git fetch origin kmeans_analysis_docs && git rebase --onto origin/kmeans_analysis_docs series-of-fixes-base series-of-fixes && git tag -f series-of-fixes-base origin/kmeans_analysis_docs` +- **PR 14 readability goal**: Wherever PR 14 removes an unvectorized (scalar) code path in + favor of its vectorized replacement, the vectorized version must be made **at least as + readable** as the scalar one it replaces. Example: the scalar functions (`comment_stats`, + `add_comparative_stats`, `repness_metric`, `finalize_cmt_stats`) read like a step-by-step + recipe, while their vectorized replacement (`compute_group_comment_stats_df`) buries the + same logic in 150 lines of DataFrame plumbing. Fix: extract the statistics computation + (probabilities → tests → ratios → metrics → classify) into its own function, then delete + the scalar functions, then update tests. Apply this principle to every scalar/vectorized + pair removed in PR 14. + Found during D5 review (2026-03-16). diff --git a/delphi/docs/PLAN_DISCREPANCY_FIXES.md b/delphi/docs/PLAN_DISCREPANCY_FIXES.md index fa2f04ceaf..e2b74f2062 100644 --- a/delphi/docs/PLAN_DISCREPANCY_FIXES.md +++ b/delphi/docs/PLAN_DISCREPANCY_FIXES.md @@ -471,7 +471,7 @@ By this point, we should have good test coverage from all the per-discrepancy te | D6 | Two-proportion test | **PR 5** | — | **DONE** ✓ | | D7 | Repness metric | PR 6 | — | **DONE** ✓ | | D8 | Finalize cmt stats | PR 7 | — | Fix | -| D9 | Z-score thresholds | **PR 3** | **#2518** | **DONE** ✓ | +| D9 | Z-score thresholds | **PR 3** | **#2446** | **DONE** ✓ | | D10 | Rep comment selection | PR 8 | — | Fix (with legacy env var) | | D11 | Consensus selection | PR 9 | — | Fix (with legacy env var) | | D12 | Comment priorities | PR 11 | — | Fix (implement from scratch) | @@ -490,7 +490,7 @@ By this point, we should have good test coverage from all the per-discrepancy te ## Tasks parallelization -D9 is done (PR #2518). The remaining fixes have the following dependency structure: +D9 is done (PR #2446). The remaining fixes have the following dependency structure: ### Repness chain dependency graph (all in `repness.py`) diff --git a/delphi/polismath/pca_kmeans_rep/repness.py b/delphi/polismath/pca_kmeans_rep/repness.py index 8bb15eadcf..61aba2a658 100644 --- a/delphi/polismath/pca_kmeans_rep/repness.py +++ b/delphi/polismath/pca_kmeans_rep/repness.py @@ -272,34 +272,31 @@ def repness_metric(stats: Dict[str, Any], key_prefix: str) -> float: def finalize_cmt_stats(stats: Dict[str, Any]) -> Dict[str, Any]: """ - Finalize comment statistics and determine if agree or disagree is more representative. - + Finalize comment stats and classify as agree/disagree, matching Clojure. + + Clojure (math/src/polismath/math/repness.clj:173-180): + (defn finalize-cmt-stats + [tid {:keys [... rat rdt ...]}] + (let [[...] (if (> rat rdt) + [na ns pa pat ra rat :agree] + [nd ns pd pdt rd rdt :disagree])] + ...)) + + Pure comparison of the two two-prop z-scores. No probability/ratio + threshold logic — strict `rat > rdt` (rat == rdt falls through to disagree). + Always populates `agree_metric` / `disagree_metric` (used downstream by + selection routines that rank candidates). + Args: stats: Statistics for a comment/group - + Returns: - Finalized statistics with best representativeness + Finalized statistics with `repful`, `agree_metric`, `disagree_metric`. """ result = deepcopy(stats) - - # Calculate agree and disagree metrics result['agree_metric'] = repness_metric(stats, 'a') result['disagree_metric'] = repness_metric(stats, 'd') - - # Determine whether agree or disagree is more representative - if result['pa'] > 0.5 and result['ra'] > 1.0: - # More agree than disagree, and more than other groups - result['repful'] = 'agree' - elif result['pd'] > 0.5 and result['rd'] > 1.0: - # More disagree than agree, and more than other groups - result['repful'] = 'disagree' - else: - # Use the higher metric - if result['agree_metric'] >= result['disagree_metric']: - result['repful'] = 'agree' - else: - result['repful'] = 'disagree' - + result['repful'] = 'agree' if stats['rat'] > stats['rdt'] else 'disagree' return result @@ -731,18 +728,9 @@ def compute_group_comment_stats_df(votes_long: pd.DataFrame, stats_df['disagree_metric'] = (stats_df['rd'] * stats_df['rdt'] * stats_df['pd'] * stats_df['pdt']) - # Determine repful ('agree' or 'disagree') - # Logic: if pa > 0.5 and ra > 1.0 -> 'agree' - # elif pd > 0.5 and rd > 1.0 -> 'disagree' - # else: use higher metric - conditions = [ - (stats_df['pa'] > 0.5) & (stats_df['ra'] > 1.0), - (stats_df['pd'] > 0.5) & (stats_df['rd'] > 1.0), - ] - choices = ['agree', 'disagree'] - stats_df['repful'] = np.select(conditions, choices, - default=np.where(stats_df['agree_metric'] >= stats_df['disagree_metric'], - 'agree', 'disagree')) + # Clojure (repness.clj:178): (if (> rat rdt) ... :agree ... :disagree) + # Pure comparison of rat vs rdt — no probability/ratio thresholds. + stats_df['repful'] = np.where(stats_df['rat'] > stats_df['rdt'], 'agree', 'disagree') return stats_df diff --git a/delphi/tests/test_discrepancy_fixes.py b/delphi/tests/test_discrepancy_fixes.py index f7aeeaa507..4a6df498d8 100644 --- a/delphi/tests/test_discrepancy_fixes.py +++ b/delphi/tests/test_discrepancy_fixes.py @@ -978,26 +978,95 @@ class TestD8FinalizeStats: Clojure uses simple rat > rdt → 'agree'; else → 'disagree' """ - @pytest.mark.xfail(reason="D8: Python uses pa/ra thresholds, target is rat>rdt comparison") def test_repful_uses_rat_vs_rdt(self): - """repful classification should use rat > rdt (Clojure logic).""" - # Case where Python and Clojure disagree: - # pa > 0.5 and ra > 1.0 → Python says 'agree' - # but rat < rdt → Clojure says 'disagree' + """repful classification should use rat > rdt (Clojure logic). + + Case where the OLD Python 3-branch logic disagrees with Clojure: + pa > 0.5 AND ra > 1.0 → old Python says 'agree', + but rat < rdt → Clojure says 'disagree'. + """ stats = { 'pa': 0.6, 'pat': 1.0, 'ra': 1.2, 'rat': 0.5, 'pd': 0.4, 'pdt': -0.5, 'rd': 0.8, 'rdt': 1.5, - 'agree_metric': 0.0, # placeholder + 'agree_metric': 0.0, 'disagree_metric': 0.0, } - result = finalize_cmt_stats(stats) - # Clojure: rat (0.5) < rdt (1.5) → 'disagree' - # Python: pa (0.6) > 0.5 and ra (1.2) > 1.0 → 'agree' check.equal(result['repful'], 'disagree', f"repful should be 'disagree' when rat < rdt, got '{result['repful']}'") + def test_repful_uses_rat_vs_rdt_inverse(self): + """Inverse case: Clojure says 'agree' where old Python 3-branch said 'disagree'. + + pd > 0.5 AND rd > 1.0 → old Python says 'disagree', but rat > rdt → Clojure 'agree'. + """ + stats = { + 'pa': 0.4, 'pat': -0.5, 'ra': 0.8, 'rat': 1.5, + 'pd': 0.6, 'pdt': 1.0, 'rd': 1.2, 'rdt': 0.5, + 'agree_metric': 0.0, + 'disagree_metric': 0.0, + } + result = finalize_cmt_stats(stats) + check.equal(result['repful'], 'agree', + f"repful should be 'agree' when rat > rdt, got '{result['repful']}'") + + def test_repful_strict_greater_than(self): + """Clojure uses strict (> rat rdt) — when rat == rdt, falls through to disagree.""" + stats = { + 'pa': 0.5, 'pat': 0.0, 'ra': 1.0, 'rat': 1.5, + 'pd': 0.5, 'pdt': 0.0, 'rd': 1.0, 'rdt': 1.5, + 'agree_metric': 0.0, + 'disagree_metric': 0.0, + } + result = finalize_cmt_stats(stats) + # Clojure: (> 1.5 1.5) is false → :disagree branch + check.equal(result['repful'], 'disagree', + f"rat == rdt should yield 'disagree' (strict >), got '{result['repful']}'") + + def test_repful_negative_z_scores(self): + """Comparison works with negative z-scores: e.g. rat=-0.5 > rdt=-2.0 → 'agree'.""" + stats = { + 'pa': 0.3, 'pat': -1.0, 'ra': 0.5, 'rat': -0.5, + 'pd': 0.7, 'pdt': -2.0, 'rd': 1.5, 'rdt': -2.0, + 'agree_metric': 0.0, + 'disagree_metric': 0.0, + } + result = finalize_cmt_stats(stats) + # -0.5 > -2.0 → 'agree' + check.equal(result['repful'], 'agree', + f"rat=-0.5 > rdt=-2.0 should yield 'agree', got '{result['repful']}'") + + def test_finalize_cmt_stats_keeps_metrics(self): + """Regression: finalize_cmt_stats must still populate agree_metric / disagree_metric.""" + stats = { + 'pa': 0.8, 'pat': 3.0, 'ra': 1.5, 'rat': 2.0, + 'pd': 0.2, 'pdt': -1.0, 'rd': 0.5, 'rdt': -0.5, + } + result = finalize_cmt_stats(stats) + check.is_in('agree_metric', result) + check.is_in('disagree_metric', result) + check.is_in('repful', result) + # Sanity: with rat=2.0 > rdt=-0.5, repful is 'agree' + check.equal(result['repful'], 'agree') + + def test_repful_both_zero(self): + """Boundary: rat == rdt == 0 should fall through to 'disagree' (strict >). + + Distinct from `test_repful_strict_greater_than` (rat==rdt==1.5): + this case pins the all-zero boundary specifically. + """ + stats = { + 'pa': 0.5, 'pat': 0.0, 'ra': 1.0, 'rat': 0.0, + 'pd': 0.5, 'pdt': 0.0, 'rd': 1.0, 'rdt': 0.0, + 'agree_metric': 0.0, + 'disagree_metric': 0.0, + } + result = finalize_cmt_stats(stats) + # Clojure: (> 0 0) is false → :disagree branch + check.equal(result['repful'], 'disagree', + f"rat == rdt == 0 should yield 'disagree' (strict >), got '{result['repful']}'") + @pytest.mark.xfail(reason="D8/D10: repful logic differs + no shared comments") def test_repful_matches_clojure_blob(self, conv, clojure_blob, dataset_name): """repful-for (Clojure) vs repful (Python) for shared rep comments."""