Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md
Original file line number Diff line number Diff line change
Expand Up @@ -495,6 +495,58 @@ not a Beta distribution posterior.

---

## PR 6: Fix D7 — Repness Metric Formula

### TDD steps
1. **Baseline**: 1 failed (pakistan-incremental D2, pre-existing), 102 passed, 5 skipped, 143 xfailed, 2 xpassed
2. **Red**: Wrote 5 new D7 tests (agree product, disagree product, zero-kills-metric, sign preservation,
multiple known values) + updated unit tests in test_repness_unit.py and test_old_format_repness.py → 7 failures
3. **Fix**: Changed both scalar `repness_metric()` and vectorized `compute_group_comment_stats_df()`:
- agree_metric: `pa * (|pat| + |rat|)` → `ra * rat * pa * pat`
- disagree_metric: `(1 - pd) * (|pdt| + |rdt|)` → `rd * rdt * pd * pdt`
4. **Green**: All 5 D7 formula tests pass + 4 blob comparison xfailed (D10 selection differs)
5. **Full suite (public)**: 4 regression failures, all in `repness.group_repness` fields (expected —
metric reranking changes which comments are selected)
6. **Re-recorded golden snapshots** for all 7 datasets (public + private)
7. **Final (with --include-local)**: 1 failed (pakistan-incremental D2, pre-existing), 98 passed,
5 skipped, 131 xfailed, 3 xpassed — 19/19 regression tests pass

### Changes
- `repness.py`: `repness_metric()` changed from `p_factor * (abs(p_test) + abs(r_test))` to
`r * r_test * p * p_test`. The old disagree formula was doubly wrong: used `(1-pd)` instead of
`pd`, and used a weighted sum instead of a product.
- `repness.py`: Vectorized metric in `compute_group_comment_stats_df()` updated to match.
- `test_discrepancy_fixes.py`: Expanded `TestD7RepnessMetric` from 2 to 6 tests (5 formula + 1 blob).
- `test_repness_unit.py`, `test_old_format_repness.py`: Updated expected values to match product formula.
Comment thread
jucor marked this conversation as resolved.
Resolved

### Key insight: the old disagree formula was doubly wrong
Python's old disagree_metric used `(1 - pd) * (|pdt| + |rdt|)`. This was wrong in two ways:
1. Used `(1 - pd)` instead of `pd` — Clojure uses `pd` directly as the probability factor
2. Used a weighted sum of absolutes instead of a signed product

The old formula could produce high disagree metrics for comments with low disagree probability
(since `1 - pd` is high when `pd` is low), which is the opposite of the intended behavior.

### Decision: no feature flag for old formula
The plan suggested keeping the old formula behind a flag. After investigation, the old formula
has no defensible behavior (doubly wrong disagree, weighted sum vs product). No flag needed.

### Session 9 (2026-03-13)

- Created branch `jc/clj-parity-d7-repness-metric` on top of `jc/clj-parity-d6-two-prop-test`
- Read Clojure source (repness.clj:188-190, finalize-cmt-stats:170-185) to verify formula
- TDD cycle: red (7 failures across 3 test files) → fix → green (all pass)
- Verified regression diffs are all in `repness.group_repness` — no non-repness changes
- Re-recorded golden snapshots for all 7 datasets
- Final validation: 19/19 regression tests pass, 1 pre-existing failure (pakistan-incremental D2)

### What's Next

1. **PR 7 — Fix D8 (Finalize comment stats)**: Change repful classification from
`pa > 0.5 AND ra > 1.0` to `rat > rdt` (Clojure's simpler logic).

---

## TDD Discipline

**CRITICAL: For every fix, ALWAYS follow this order:**
Expand Down
2 changes: 1 addition & 1 deletion delphi/docs/PLAN_DISCREPANCY_FIXES.md
Original file line number Diff line number Diff line change
Expand Up @@ -469,7 +469,7 @@ By this point, we should have good test coverage from all the per-discrepancy te
| D4 | Pseudocount formula | **PR 2** | **#2435** | **DONE** ✓ |
| D5 | Proportion test | **PR 4** | — | **DONE** ✓ |
| D6 | Two-proportion test | **PR 5** | — | **DONE** ✓ |
| D7 | Repness metric | PR 6 | — | Fix (with flag for old formula) |
| D7 | Repness metric | PR 6 | — | **DONE** ✓ |
| D8 | Finalize cmt stats | PR 7 | — | Fix |
| D9 | Z-score thresholds | **PR 3** | **#2518** | **DONE** ✓ |
| D10 | Rep comment selection | PR 8 | — | Fix (with legacy env var) |
Expand Down
45 changes: 30 additions & 15 deletions delphi/polismath/pca_kmeans_rep/repness.py
Original file line number Diff line number Diff line change
Expand Up @@ -236,26 +236,38 @@ def add_comparative_stats(comment_stats: Dict[str, Any],

def repness_metric(stats: Dict[str, Any], key_prefix: str) -> float:
"""
Calculate a representativeness metric for ranking.

Composite representativeness score, matching Clojure's repness-metric.

Clojure (math/src/polismath/math/repness.clj:191-193):
(defn repness-metric
[{:keys [repness repness-test p-success p-test]}]
(* repness repness-test p-success p-test))

For Python the keys are looked up via key_prefix:
'a' (agree) → ra * rat * pa * pat
'd' (disagree) → rd * rdt * pd * pdt

This is a *signed* product of 4 values — there is no abs(). A negative
z-score (pat / rat / pdt / rdt) flips the sign of the metric, exactly as
in Clojure. Downstream `select_rep_comments` sorts candidates by this
metric in descending order and keeps the top N, so negative metrics rank
at the bottom of the candidate pool. They are not actively *filtered*
here, though — fallback paths (e.g. fewer than the requested N candidates
pass significance) can still surface a negative-metric comment. Callers
that need strict positive-metric semantics should gate at the call site.

Args:
stats: Statistics for a comment/group
key_prefix: 'a' for agreement, 'd' for disagreement

Returns:
Composite representativeness score
Composite representativeness score (signed product of 4 values).
"""
# Get the relevant probability and test values
p = stats[f'p{key_prefix}']
p_test = stats[f'p{key_prefix}t']
r = stats[f'r{key_prefix}']
r_test = stats[f'r{key_prefix}t']

# Take probability into account
p_factor = p if key_prefix == 'a' else (1 - p)

# Calculate composite score
return p_factor * (abs(p_test) + abs(r_test))
return r * r_test * p * p_test


def finalize_cmt_stats(stats: Dict[str, Any]) -> Dict[str, Any]:
Expand Down Expand Up @@ -711,10 +723,13 @@ def compute_group_comment_stats_df(votes_long: pd.DataFrame,
)

# Compute metrics
# agree_metric = pa * (|pat| + |rat|)
# disagree_metric = (1 - pd) * (|pdt| + |rdt|)
stats_df['agree_metric'] = stats_df['pa'] * (stats_df['pat'].abs() + stats_df['rat'].abs())
stats_df['disagree_metric'] = (1 - stats_df['pd']) * (stats_df['pdt'].abs() + stats_df['rdt'].abs())
# Clojure (repness.clj:191-193): (* repness repness-test p-success p-test)
# agree_metric = ra * rat * pa * pat
# disagree_metric = rd * rdt * pd * pdt (signed product — see scalar repness_metric)
stats_df['agree_metric'] = (stats_df['ra'] * stats_df['rat']
* stats_df['pa'] * stats_df['pat'])
stats_df['disagree_metric'] = (stats_df['rd'] * stats_df['rdt']
* stats_df['pd'] * stats_df['pdt'])

# Determine repful ('agree' or 'disagree')
# Logic: if pa > 0.5 and ra > 1.0 -> 'agree'
Expand Down
18 changes: 13 additions & 5 deletions delphi/tests/test_discrepancy_fixes.py
Original file line number Diff line number Diff line change
Expand Up @@ -914,7 +914,6 @@ class TestD7RepnessMetric:
kills the whole metric.
"""

@pytest.mark.xfail(reason="D7: Python uses pa*(|pat|+|rat|), target is ra*rat*pa*pat")
def test_metric_formula_is_product(self):
"""repness_metric should use product formula (ra * rat * pa * pat)."""
stats = {
Expand Down Expand Up @@ -1215,10 +1214,19 @@ def test_prop_test_matches_clojure_formula_synthetic(self):
assert abs(result - expected) < 1e-10, f"prop_test({succ}, {n})={result}, expected {expected}"

def test_clojure_repness_metric_product(self):
"""Verify Clojure's repness metric is a product: ra * rat * pa * pat."""
ra, rat, pa, pat = 1.5, 2.0, 0.8, 3.0
expected = ra * rat * pa * pat # = 7.2
assert expected == pytest.approx(7.2)
"""Python's repness_metric matches Clojure (* repness repness-test p-success p-test).

Verifies the actual production function, not a re-implementation of the formula.
"""
stats = {
'pa': 0.8, 'pat': 3.0, 'ra': 1.5, 'rat': 2.0,
'pd': 0.2, 'pdt': -1.0, 'rd': 0.5, 'rdt': -0.5,
}
# Agree: (* ra rat pa pat) = 1.5 * 2.0 * 0.8 * 3.0 = 7.2
assert repness_metric(stats, 'a') == pytest.approx(7.2)
# Disagree (same product, no (1-pd) trick): (* rd rdt pd pdt)
# = 0.5 * -0.5 * 0.2 * -1.0 = 0.05 (two negatives cancel — signed product)
assert repness_metric(stats, 'd') == pytest.approx(0.05)

def test_clojure_repful_uses_rat_vs_rdt(self):
"""Clojure determines repful by comparing rat vs rdt."""
Expand Down
6 changes: 4 additions & 2 deletions delphi/tests/test_old_format_repness.py
Original file line number Diff line number Diff line change
Expand Up @@ -166,12 +166,14 @@ def test_repness_metric(self):

# Calculate agree metric
agree_metric = repness_metric(stats, 'a')
expected_agree = 0.8 * (abs(3.0) + abs(2.5))
# Clojure (repness.clj:191-193): (* ra rat pa pat)
expected_agree = 2.0 * 2.5 * 0.8 * 3.0 # = 12.0
assert np.isclose(agree_metric, expected_agree)

# Calculate disagree metric
disagree_metric = repness_metric(stats, 'd')
expected_disagree = (1 - 0.2) * (abs(-3.0) + abs(-2.5))
# Clojure: (* rd rdt pd pdt) — signed product, two negatives cancel
expected_disagree = 0.33 * (-2.5) * 0.2 * (-3.0) # = 0.495
assert np.isclose(disagree_metric, expected_disagree)

def test_finalize_cmt_stats(self):
Expand Down
11 changes: 6 additions & 5 deletions delphi/tests/test_repness_unit.py
Original file line number Diff line number Diff line change
Expand Up @@ -176,14 +176,15 @@ def test_repness_metric(self):
'rdt': -2.5
}

# Calculate agree metric
# Clojure formula (repness.clj:191-193): (* repness repness-test p-success p-test)
# → agree: ra * rat * pa * pat
# → disagree: rd * rdt * pd * pdt (signed product — two negatives cancel)
agree_metric = repness_metric(stats, 'a')
expected_agree = 0.8 * (abs(3.0) + abs(2.5))
expected_agree = 2.0 * 2.5 * 0.8 * 3.0 # = 12.0
assert np.isclose(agree_metric, expected_agree)

# Calculate disagree metric

disagree_metric = repness_metric(stats, 'd')
expected_disagree = (1 - 0.2) * (abs(-3.0) + abs(-2.5))
expected_disagree = 0.33 * (-2.5) * 0.2 * (-3.0) # = 0.495
assert np.isclose(disagree_metric, expected_disagree)

def test_finalize_cmt_stats(self):
Expand Down
Loading