Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 1 addition & 3 deletions .github/workflows/python-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -95,10 +95,8 @@ jobs:
-e AWS_REGION=us-east-1 \
-e AWS_ACCESS_KEY_ID=dummy \
-e AWS_SECRET_ACCESS_KEY=dummy \
-e POSTGRES_HOST=postgres \
-e POSTGRES_PASSWORD=PdwPNS2mDN73Vfbc \
-e POSTGRES_DB=polis-test \
-e SKIP_GOLDEN=1 \
-e POSTGRES_CONNECT_TIMEOUT=5 \
delphi \
bash -c " \
set -e; \
Expand Down
15 changes: 15 additions & 0 deletions delphi/CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,21 @@ Always use the commands above to determine the most substantial conversation whe
- `POSTGRES_USER`: Database username
- `POSTGRES_PASSWORD`: Database password
- `POSTGRES_HOST`: Database host
- `POSTGRES_CONNECT_TIMEOUT`: Seconds before the initial TCP `connect()`
gives up. **Default 30s** (conservative for production: transient
slowness, scale-up, network blips). CI and `example.env` override to **5s**
so tests and local dev fail fast when Postgres isn't running — without
this, an unreachable DB causes the process to hang for the kernel default
(~60–120s+). Honored by:
- `polismath/database/postgres.py` — SQLAlchemy `PostgresClient`.
- `polismath/run_math_pipeline.py` — psycopg2 `connect()` (the production
math worker invoked from `run_delphi.py`).

Note that SQLAlchemy's `pool_pre_ping` does NOT replace this: pre-ping
only acts on already-pooled connections, not on the initial socket connect.
Other psycopg2 callsites (`tests/`, `scripts/regression_download.py`) still
hardcode their own timeouts (typically 5s) — flag as a future cleanup if
you change anything in their neighborhood.

- **Docker Configuration**:

Expand Down
770 changes: 768 additions & 2 deletions delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md

Large diffs are not rendered by default.

134 changes: 134 additions & 0 deletions delphi/docs/HANDOFF_PR14_VECTORIZED_REFACTOR.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
# Handoff: PR 14 — Make Vectorized Code Readable + Blob Injection Tests

## Goal

The scalar functions (`comment_stats`, `add_comparative_stats`, `repness_metric`,
`finalize_cmt_stats`) read like a step-by-step recipe. Their vectorized replacement
(`compute_group_comment_stats_df`) buries the same logic in 150 lines of DataFrame
plumbing. The vectorized path is the ONLY production code path — the scalar functions
are dead code, called only from tests and benchmarks.

**PR 14 must make the vectorized code at least as readable as the scalar code, then
delete the scalar functions.** This also enables vectorized blob injection tests —
the only non-tautological way to verify correctness against the Clojure math blob.

## Starting point

Branch off `jc/clj-parity-d9-fix` (Stack 13). This is BELOW D5-D8 in the stack,
so the refactor will be inherited by all formula fix PRs.

## Current stack (as of 2026-03-17)

```
Stack 13: jc/clj-parity-d9-fix ← branch off HERE for PR 14
Stack 14: jc/clj-parity-d5-prop-test (draft PR #2448)
Stack 15: jc/clj-parity-d6-two-prop-test (draft PR #2449)
Stack 16: jc/clj-parity-d7-repness-metric (draft PR #2450)
Stack 17: jc/clj-parity-d8-finalize-stats (draft PR #2451)
Stack 18: jc/clj-parity-d15-moderation-handling-zeros-vs-removes (PR #2452)
Stack 19: jc/clj-parity-kmeans-k-divergence (k-divergence fix)
Stack 20: jc/clj-parity-d10-rep-comment-selection
Stack 21: jc/clj-parity-d11-consensus-comment-selection
Stack 22: jc/clj-parity-d3-k-smoother-buffer
Stack 23: jc/clj-parity-d12-comment-priorities
Stack 24: jc/clj-parity-d1-pca-sign-flip-prevention
Stack 25: jc/clj-parity-pr15-load-votes-sort
```

D5-D8 are draft PRs waiting for vectorized blob tests before being marked ready.

## Task sequence

### 1. Refactor `compute_group_comment_stats_df`

Split into two phases:

**(a) DataFrame construction** (the plumbing):
- Build participant→group mapping
- Compute total counts per comment
- Create full (group, comment) cross-product index
- Join total counts, compute "other" counts

**(b) Statistics computation** (the readable part):
A new function with clean DataFrame inputs (columns: `na`, `nd`, `ns`,
`other_agree`, `other_disagree`, `other_votes`) → outputs (columns: `pa`, `pd`,
`pat`, `pdt`, `ra`, `rd`, `rat`, `rdt`, `agree_metric`, `disagree_metric`, `repful`).

This function should read like the scalar recipe:
```python
# Probabilities with pseudocounts
df['pa'] = (df['na'] + PSEUDO_COUNT/2) / (df['ns'] + PSEUDO_COUNT)
# Proportion tests
df['pat'] = prop_test_vectorized(df['na'], df['ns'])
# Representativeness ratios
df['ra'] = df['pa'] / df['other_pa']
# ...etc
```

### 2. Write vectorized blob injection tests

Inject Clojure group memberships + votes into the new statistics function,
compare output to blob values. Test the PRODUCTION code path.

For each `(group, tid)` in the blob's `repness`:
- Compare `pat` (or `pdt`) to blob `p-test`
- Compare `rat` (or `rdt`) to blob `repness-test`
- Compare `pa` (or `pd`) to blob `p-success`
- Compare `repful` to blob `repful-for`

The blob only stores the winning side's values. `repful-for` tells you which
side won: if `agree`, then `n-success`=na, `p-test`=pat, `repness-test`=rat.
If `disagree`, then `n-success`=nd, `p-test`=pdt, `repness-test`=rdt.

### 3. Verify scalar ≡ vectorized

Run both paths on all datasets, compare outputs field by field. Must be
identical (not just close — exact match) since they implement the same formulas.

### 4. Delete scalar functions

Remove: `comment_stats`, `add_comparative_stats`, `repness_metric`,
`finalize_cmt_stats`, and scalar `prop_test`/`two_prop_test` if no longer
needed (the vectorized versions remain).

Update all tests that called the scalar API.

### 5. Cascade rebase

Rebase D5→D6→D7→D8→D15→k-divergence→D10→D11→D3→D12→D1→PR15 on top.
Use `.claude/skills/pr-stack/rebase-stack.sh` for the cascade.

### 6. Add per-PR vectorized blob injection tests

For each of D5, D6, D8: insert a RED blob test commit before the fix,
verify it fails, then verify the fix makes it pass. Same TDD pattern used
for the scalar blob tests already in those branches.

Then mark #2448-#2451 as ready (un-draft).

## Key blob context

- `n-trials` in the Clojure blob = `S` (total seen, INCLUDING passes), not
`A+D` (agrees + disagrees). Verified: `prop_test(11, 14)` matches blob
`p-test` for tid=49 group 0 in vw (A=2, D=11, S=14, A+D=13).
- `group-votes[gid].votes[tid]` = `{A: agrees, D: disagrees, S: total_seen}`
- Blob `repness[gid][i]` fields: `n-success`, `n-trials`, `p-success`,
`p-test`, `repness` (=ra or rd), `repness-test` (=rat or rdt),
`repful-for`, `tid`, `best-agree` (optional).

## Files to modify

- `delphi/polismath/pca_kmeans_rep/repness.py` — the refactor
- `delphi/tests/test_discrepancy_fixes.py` — vectorized blob tests
- `delphi/tests/test_repness_unit.py` — update for new API
- `delphi/tests/test_old_format_repness.py` — update for new API
- `delphi/polismath/benchmarks/bench_repness.py` — update imports

## Reference

- Journal: `delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md` — Session 11 entry,
PR 14 readability goal in Notes for Future Sessions
- Plan: `delphi/docs/PLAN_DISCREPANCY_FIXES.md` — mandatory blob comparison
section in Testing Principles
- Clojure source: `math/src/polismath/math/stats.clj` (prop-test, two-prop-test),
`math/src/polismath/math/repness.clj` (finalize-cmt-stats, repness-metric)
118 changes: 118 additions & 0 deletions delphi/docs/INVESTIGATION_K_DIVERGENCE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
# K-Divergence Investigation — RESOLVED

## Problem (was)

After all cold-start-relevant formula fixes (D2-D15), Python and Clojure
selected different k values on cold-start blobs. On vw: Python=4, Clojure=2.

## Root Cause: Participant Row Ordering

The k divergence was caused by **different participant ordering in the rating
matrix**, which cascades through base-cluster IDs into group-level k-means
initialization via first-k-distinct.

### The chain

```
rating_mat row order
→ PCA projection order
→ base-cluster ID assignment (map-indexed on input rows)
→ group-level k-means first-k-distinct init (first k base-cluster centers)
→ different local optima → different silhouette scores → different k
```

### Clojure ordering

Clojure's NamedMatrix preserves **insertion order** (backed by
`java.util.Vector`). When `rowname-subset` filters to `in-conv` participants,
`filter-by-index` (utils.clj:128-133) preserves the **original matrix row
order** (iterates source, checks membership in filter set). So the base-cluster
ordering is the vote-encounter order of participants in the rating matrix.

### Python ordering (before fix)

Python used `natsorted()` (conversation.py:232) to sort rating matrix rows
by PID. This gave ascending PID order `[1, 2, 3, 4, 5, ...]` instead of
the vote-encounter order `[2, 3, 4, 6, 8, ...]` that Clojure produces.

### Impact

With first-k-distinct initialization, different ordering → different initial
centers → different k-means local optima → different silhouette landscape:

| k | Python (PID order) | Clojure (encounter order) |
|---|-------------------|--------------------------|
| 2 | sil=0.457 | **sil=0.487 (wins)** |
| 3 | sil=0.481 | sil=0.329 |
| 4 | **sil=0.508 (wins)** | sil=0.362 |

## Fix

Changed `update_votes()` and `_apply_moderation()` to preserve vote-encounter
order for participant rows instead of natsort:

1. `update_votes()`: track first-appearance order from `vote_updates`, append
new PIDs in encounter order (not `natsorted`)
2. `_apply_moderation()`: filter `raw_rating_mat.index` preserving order
(list comprehension instead of `natsorted`)

Column (comment ID) ordering remains `natsorted` — column permutation doesn't
affect PCA eigenvalues/vectors, only reorders component loadings.

## Results after fix

| Dataset | CS blob | Clj k | Py k | Sizes match? |
|---------|---------|-------|------|--------------|
| vw | ✓ | 2 | **2** | [50,17] exact |
| biodiversity | ✓ | 2 | **2** | [81,19] exact |
| bg2018 | ✓ | 2 | **2** | close ([51,49] vs [52,48]) |
| FLI | ✓ | 2 | 3 | **still diverges** |
| engage | empty | — | — | — |
| bg2050 | empty | — | — | — |
| pakistan | empty | — | — | — |

### FLI: inherent PCA divergence (not fixable)

FLI has 94.5% NaN sparsity. The PCA components are nearly but not exactly
identical (|cos|≈0.9997 vs 1.000000 for vw). This produces a silhouette
landscape where k=2 and k=3 differ by only 0.001. The tiny PCA difference
tips the balance. Injection test confirms: with Clojure projections injected,
Python picks k=2. This is inherent to the PCA algorithm difference (sklearn
full SVD vs Clojure power iteration) and not fixable without replicating
Clojure's PCA exactly.

## Investigation Findings (for the record)

### PCA is NOT the primary cause for most datasets

- vw: PCA components have cosine similarity = 1.000000 (identical!)
- Projections are exactly negated (sign flip, irrelevant for clustering)
- Silhouette scores are identical for both projection sets

### Silhouette implementation matches

- Both use (b-a)/max(a,b) formula, unweighted mean
- Both compute on base-cluster centers (not raw participants)
- Clojure's `weighted-mean` without weights = unweighted mean

### K-means initialization matches

- Both use first-k-distinct (Clojure: `init-clusters`, Python: `_get_first_k_distinct_centers`)
- Both sort base clusters by ID
- The only difference was the DATA ORDER feeding into first-k-distinct

## Files modified

- `delphi/polismath/conversation/conversation.py` — `update_votes()` and `_apply_moderation()`
- `delphi/tests/test_conversation.py` — updated ordering tests
- `delphi/tests/test_legacy_clojure_regression.py` — removed xfail on `test_group_clustering`

## Future investigation

- **FLI k divergence**: Could be resolved by implementing Clojure's power
iteration PCA. Low priority — the silhouette gap is 0.001.
- **Column ordering**: Currently natsorted, Clojure uses insertion order.
Doesn't affect clustering but could affect other comparisons.
- **Multiple k-means restarts**: Using k-means++ with n_init=10 finds the
global optimum (k=4 for vw) regardless of ordering. This would be more
robust than first-k-distinct but would NOT match Clojure.
Loading
Loading