diff --git a/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md b/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md index 8c5d6797cb..d4f23505f3 100644 --- a/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md +++ b/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md @@ -495,6 +495,58 @@ not a Beta distribution posterior. --- +## PR 6: Fix D7 — Repness Metric Formula + +### TDD steps +1. **Baseline**: 1 failed (pakistan-incremental D2, pre-existing), 102 passed, 5 skipped, 143 xfailed, 2 xpassed +2. **Red**: Wrote 5 new D7 tests (agree product, disagree product, zero-kills-metric, sign preservation, + multiple known values) + updated unit tests in test_repness_unit.py and test_old_format_repness.py → 7 failures +3. **Fix**: Changed both scalar `repness_metric()` and vectorized `compute_group_comment_stats_df()`: + - agree_metric: `pa * (|pat| + |rat|)` → `ra * rat * pa * pat` + - disagree_metric: `(1 - pd) * (|pdt| + |rdt|)` → `rd * rdt * pd * pdt` +4. **Green**: All 5 D7 formula tests pass + 4 blob comparison xfailed (D10 selection differs) +5. **Full suite (public)**: 4 regression failures, all in `repness.group_repness` fields (expected — + metric reranking changes which comments are selected) +6. **Re-recorded golden snapshots** for all 7 datasets (public + private) +7. **Final (with --include-local)**: 1 failed (pakistan-incremental D2, pre-existing), 98 passed, + 5 skipped, 131 xfailed, 3 xpassed — 19/19 regression tests pass + +### Changes +- `repness.py`: `repness_metric()` changed from `p_factor * (abs(p_test) + abs(r_test))` to + `r * r_test * p * p_test`. The old disagree formula was doubly wrong: used `(1-pd)` instead of + `pd`, and used a weighted sum instead of a product. +- `repness.py`: Vectorized metric in `compute_group_comment_stats_df()` updated to match. +- `test_discrepancy_fixes.py`: Expanded `TestD7RepnessMetric` from 2 to 6 tests (5 formula + 1 blob). +- `test_repness_unit.py`, `test_old_format_repness.py`: Updated expected values to match product formula. + +### Key insight: the old disagree formula was doubly wrong +Python's old disagree_metric used `(1 - pd) * (|pdt| + |rdt|)`. This was wrong in two ways: +1. Used `(1 - pd)` instead of `pd` — Clojure uses `pd` directly as the probability factor +2. Used a weighted sum of absolutes instead of a signed product + +The old formula could produce high disagree metrics for comments with low disagree probability +(since `1 - pd` is high when `pd` is low), which is the opposite of the intended behavior. + +### Decision: no feature flag for old formula +The plan suggested keeping the old formula behind a flag. After investigation, the old formula +has no defensible behavior (doubly wrong disagree, weighted sum vs product). No flag needed. + +### Session 9 (2026-03-13) + +- Created branch `jc/clj-parity-d7-repness-metric` on top of `jc/clj-parity-d6-two-prop-test` +- Read Clojure source (repness.clj:188-190, finalize-cmt-stats:170-185) to verify formula +- TDD cycle: red (7 failures across 3 test files) → fix → green (all pass) +- Verified regression diffs are all in `repness.group_repness` — no non-repness changes +- Re-recorded golden snapshots for all 7 datasets +- Final validation: 19/19 regression tests pass, 1 pre-existing failure (pakistan-incremental D2) + +### What's Next + +1. **PR 7 — Fix D8 (Finalize comment stats)**: Change repful classification from + `pa > 0.5 AND ra > 1.0` to `rat > rdt` (Clojure's simpler logic). + +--- + ## TDD Discipline **CRITICAL: For every fix, ALWAYS follow this order:** diff --git a/delphi/docs/PLAN_DISCREPANCY_FIXES.md b/delphi/docs/PLAN_DISCREPANCY_FIXES.md index 620253a493..fa2f04ceaf 100644 --- a/delphi/docs/PLAN_DISCREPANCY_FIXES.md +++ b/delphi/docs/PLAN_DISCREPANCY_FIXES.md @@ -469,7 +469,7 @@ By this point, we should have good test coverage from all the per-discrepancy te | D4 | Pseudocount formula | **PR 2** | **#2435** | **DONE** ✓ | | D5 | Proportion test | **PR 4** | — | **DONE** ✓ | | D6 | Two-proportion test | **PR 5** | — | **DONE** ✓ | -| D7 | Repness metric | PR 6 | — | Fix (with flag for old formula) | +| D7 | Repness metric | PR 6 | — | **DONE** ✓ | | D8 | Finalize cmt stats | PR 7 | — | Fix | | D9 | Z-score thresholds | **PR 3** | **#2518** | **DONE** ✓ | | D10 | Rep comment selection | PR 8 | — | Fix (with legacy env var) | diff --git a/delphi/polismath/pca_kmeans_rep/repness.py b/delphi/polismath/pca_kmeans_rep/repness.py index 7f75877fc2..8bb15eadcf 100644 --- a/delphi/polismath/pca_kmeans_rep/repness.py +++ b/delphi/polismath/pca_kmeans_rep/repness.py @@ -236,26 +236,38 @@ def add_comparative_stats(comment_stats: Dict[str, Any], def repness_metric(stats: Dict[str, Any], key_prefix: str) -> float: """ - Calculate a representativeness metric for ranking. - + Composite representativeness score, matching Clojure's repness-metric. + + Clojure (math/src/polismath/math/repness.clj:191-193): + (defn repness-metric + [{:keys [repness repness-test p-success p-test]}] + (* repness repness-test p-success p-test)) + + For Python the keys are looked up via key_prefix: + 'a' (agree) → ra * rat * pa * pat + 'd' (disagree) → rd * rdt * pd * pdt + + This is a *signed* product of 4 values — there is no abs(). A negative + z-score (pat / rat / pdt / rdt) flips the sign of the metric, exactly as + in Clojure. Downstream `select_rep_comments` sorts candidates by this + metric in descending order and keeps the top N, so negative metrics rank + at the bottom of the candidate pool. They are not actively *filtered* + here, though — fallback paths (e.g. fewer than the requested N candidates + pass significance) can still surface a negative-metric comment. Callers + that need strict positive-metric semantics should gate at the call site. + Args: stats: Statistics for a comment/group key_prefix: 'a' for agreement, 'd' for disagreement - + Returns: - Composite representativeness score + Composite representativeness score (signed product of 4 values). """ - # Get the relevant probability and test values p = stats[f'p{key_prefix}'] p_test = stats[f'p{key_prefix}t'] r = stats[f'r{key_prefix}'] r_test = stats[f'r{key_prefix}t'] - - # Take probability into account - p_factor = p if key_prefix == 'a' else (1 - p) - - # Calculate composite score - return p_factor * (abs(p_test) + abs(r_test)) + return r * r_test * p * p_test def finalize_cmt_stats(stats: Dict[str, Any]) -> Dict[str, Any]: @@ -711,10 +723,13 @@ def compute_group_comment_stats_df(votes_long: pd.DataFrame, ) # Compute metrics - # agree_metric = pa * (|pat| + |rat|) - # disagree_metric = (1 - pd) * (|pdt| + |rdt|) - stats_df['agree_metric'] = stats_df['pa'] * (stats_df['pat'].abs() + stats_df['rat'].abs()) - stats_df['disagree_metric'] = (1 - stats_df['pd']) * (stats_df['pdt'].abs() + stats_df['rdt'].abs()) + # Clojure (repness.clj:191-193): (* repness repness-test p-success p-test) + # agree_metric = ra * rat * pa * pat + # disagree_metric = rd * rdt * pd * pdt (signed product — see scalar repness_metric) + stats_df['agree_metric'] = (stats_df['ra'] * stats_df['rat'] + * stats_df['pa'] * stats_df['pat']) + stats_df['disagree_metric'] = (stats_df['rd'] * stats_df['rdt'] + * stats_df['pd'] * stats_df['pdt']) # Determine repful ('agree' or 'disagree') # Logic: if pa > 0.5 and ra > 1.0 -> 'agree' diff --git a/delphi/tests/test_discrepancy_fixes.py b/delphi/tests/test_discrepancy_fixes.py index ad764d52d6..f7aeeaa507 100644 --- a/delphi/tests/test_discrepancy_fixes.py +++ b/delphi/tests/test_discrepancy_fixes.py @@ -914,7 +914,6 @@ class TestD7RepnessMetric: kills the whole metric. """ - @pytest.mark.xfail(reason="D7: Python uses pa*(|pat|+|rat|), target is ra*rat*pa*pat") def test_metric_formula_is_product(self): """repness_metric should use product formula (ra * rat * pa * pat).""" stats = { @@ -1215,10 +1214,19 @@ def test_prop_test_matches_clojure_formula_synthetic(self): assert abs(result - expected) < 1e-10, f"prop_test({succ}, {n})={result}, expected {expected}" def test_clojure_repness_metric_product(self): - """Verify Clojure's repness metric is a product: ra * rat * pa * pat.""" - ra, rat, pa, pat = 1.5, 2.0, 0.8, 3.0 - expected = ra * rat * pa * pat # = 7.2 - assert expected == pytest.approx(7.2) + """Python's repness_metric matches Clojure (* repness repness-test p-success p-test). + + Verifies the actual production function, not a re-implementation of the formula. + """ + stats = { + 'pa': 0.8, 'pat': 3.0, 'ra': 1.5, 'rat': 2.0, + 'pd': 0.2, 'pdt': -1.0, 'rd': 0.5, 'rdt': -0.5, + } + # Agree: (* ra rat pa pat) = 1.5 * 2.0 * 0.8 * 3.0 = 7.2 + assert repness_metric(stats, 'a') == pytest.approx(7.2) + # Disagree (same product, no (1-pd) trick): (* rd rdt pd pdt) + # = 0.5 * -0.5 * 0.2 * -1.0 = 0.05 (two negatives cancel — signed product) + assert repness_metric(stats, 'd') == pytest.approx(0.05) def test_clojure_repful_uses_rat_vs_rdt(self): """Clojure determines repful by comparing rat vs rdt.""" diff --git a/delphi/tests/test_old_format_repness.py b/delphi/tests/test_old_format_repness.py index e56a18c267..94459e8bd3 100644 --- a/delphi/tests/test_old_format_repness.py +++ b/delphi/tests/test_old_format_repness.py @@ -166,12 +166,14 @@ def test_repness_metric(self): # Calculate agree metric agree_metric = repness_metric(stats, 'a') - expected_agree = 0.8 * (abs(3.0) + abs(2.5)) + # Clojure (repness.clj:191-193): (* ra rat pa pat) + expected_agree = 2.0 * 2.5 * 0.8 * 3.0 # = 12.0 assert np.isclose(agree_metric, expected_agree) # Calculate disagree metric disagree_metric = repness_metric(stats, 'd') - expected_disagree = (1 - 0.2) * (abs(-3.0) + abs(-2.5)) + # Clojure: (* rd rdt pd pdt) — signed product, two negatives cancel + expected_disagree = 0.33 * (-2.5) * 0.2 * (-3.0) # = 0.495 assert np.isclose(disagree_metric, expected_disagree) def test_finalize_cmt_stats(self): diff --git a/delphi/tests/test_repness_unit.py b/delphi/tests/test_repness_unit.py index fd41d1f118..094a839b05 100644 --- a/delphi/tests/test_repness_unit.py +++ b/delphi/tests/test_repness_unit.py @@ -176,14 +176,15 @@ def test_repness_metric(self): 'rdt': -2.5 } - # Calculate agree metric + # Clojure formula (repness.clj:191-193): (* repness repness-test p-success p-test) + # → agree: ra * rat * pa * pat + # → disagree: rd * rdt * pd * pdt (signed product — two negatives cancel) agree_metric = repness_metric(stats, 'a') - expected_agree = 0.8 * (abs(3.0) + abs(2.5)) + expected_agree = 2.0 * 2.5 * 0.8 * 3.0 # = 12.0 assert np.isclose(agree_metric, expected_agree) - - # Calculate disagree metric + disagree_metric = repness_metric(stats, 'd') - expected_disagree = (1 - 0.2) * (abs(-3.0) + abs(-2.5)) + expected_disagree = 0.33 * (-2.5) * 0.2 * (-3.0) # = 0.495 assert np.isclose(disagree_metric, expected_disagree) def test_finalize_cmt_stats(self):