Skip to content

[Stack 9/27] Per-discrepancy test infrastructure - #2420

Closed
jucor wants to merge 10 commits into
jc/kmeans_analysis_docsfrom
jc/series-of-fixes
Closed

jucor wants to merge 10 commits into
jc/kmeans_analysis_docsfrom
jc/series-of-fixes

Conversation

@jucor

@jucor jucor commented Mar 5, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Stacked on #2419 (Deep analysis of Python-Clojure discrepancies and fix plan). Please review and merge #2419 first.
Next in stack: #2421 (Fix D2: in-conv participant threshold + D2c vote count source)

Per-discrepancy test infrastructure for TDD fixing of Python-Clojure differences.

Changes

  • Add per-discrepancy test markers and parametrized test infrastructure
  • Cold-start recorder: coordinate parallel runs with marker file, auto-pause math workers
  • Update journal with xpassed test breakdown across all datasets
  • Address Copilot review: remove unused import, fix script issues
  • Add naming convention documentation

Test plan

  • 223 passed, 4 skipped, 22 xfailed, 7 xpassed, 0 failures
    🤖 Generated with Claude Code

@jucor
jucor force-pushed the jc/kmeans_analysis_docs branch from 74d276e to cda5015 Compare March 6, 2026 15:34
@jucor
jucor force-pushed the jc/series-of-fixes branch from e43b0f9 to 5e2a0de Compare March 6, 2026 15:34
@jucor
jucor force-pushed the jc/kmeans_analysis_docs branch from cda5015 to f4136dd Compare March 10, 2026 11:12
@jucor
jucor force-pushed the jc/series-of-fixes branch from 5e2a0de to 33ecc1c Compare March 10, 2026 11:12
@jucor
jucor force-pushed the jc/kmeans_analysis_docs branch from f4136dd to d965abb Compare March 10, 2026 12:29
@jucor
jucor force-pushed the jc/series-of-fixes branch from 33ecc1c to d7b7c34 Compare March 10, 2026 12:29
@jucor
jucor force-pushed the jc/kmeans_analysis_docs branch from d965abb to 3886fb5 Compare March 10, 2026 14:13
@jucor
jucor force-pushed the jc/series-of-fixes branch from d7b7c34 to d2b434e Compare March 10, 2026 14:14
@jucor
jucor force-pushed the jc/kmeans_analysis_docs branch from 3886fb5 to 000bb0a Compare March 10, 2026 15:39
@jucor
jucor force-pushed the jc/series-of-fixes branch from d2b434e to 4220632 Compare March 10, 2026 15:40
@jucor
jucor requested review from ballPointPenguin and whilo March 10, 2026 16:08
@jucor jucor changed the title [Clj parity PR 0] Per-discrepancy test infrastructure [Stack 7/8] Per-discrepancy test infrastructure Mar 10, 2026
@jucor jucor mentioned this pull request Mar 11, 2026
5 tasks done
@jucor jucor changed the title [Stack 7/8] Per-discrepancy test infrastructure [Stack 7/9] Per-discrepancy test infrastructure Mar 11, 2026
@jucor jucor changed the title [Stack 7/9] Per-discrepancy test infrastructure [Stack 7/10] Per-discrepancy test infrastructure Mar 11, 2026
@jucor jucor changed the title [Stack 7/10] Per-discrepancy test infrastructure [Stack 7/11] Per-discrepancy test infrastructure Mar 11, 2026
@jucor
jucor requested a review from Copilot March 13, 2026 12:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds per-discrepancy (Python vs Clojure) test scaffolding to support TDD parity work, plus tooling updates to generate/compare Clojure cold-start blobs and document the fix workflow.

Changes:

  • Introduces tests/test_discrepancy_fixes.py with per-discrepancy, per-dataset parametrized tests (mostly xfail-guarded) and shared dataset/Conversation fixtures.
  • Updates cold-start blob generator to coordinate parallel runs (pause/unpause workers via marker file) and improve fake conversation replay.
  • Refreshes comparer/tooling/docs (vote loader swap, remove unused import, add PR naming convention + journal).

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 8 comments.

Show a summary per file
File Description
delphi/tests/test_repness_smoke.py Marks an existing flaky/unknown repness structure validation as xfail.
delphi/tests/test_discrepancy_fixes.py New per-discrepancy test module with dataset discovery + xfail-based parity targets.
delphi/scripts/generate_cold_start_clojure.py Adds pause-marker coordination and expands replay to copy participants/comments before votes.
delphi/scripts/clojure_comparer.py Switches vote-loading helper and updates failure guidance message.
delphi/polismath/pca_kmeans_rep/clusters.py Removes unused sklearn import.
delphi/docs/PLAN_DISCREPANCY_FIXES.md Adds PR naming convention documentation.
delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md Adds a running journal for parity work and test status tracking.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread delphi/tests/test_discrepancy_fixes.py Resolved
Comment thread delphi/tests/test_discrepancy_fixes.py Resolved
Comment thread delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md Resolved
Comment thread delphi/tests/test_discrepancy_fixes.py Resolved
@jucor jucor changed the title [Stack 7/11] Per-discrepancy test infrastructure [Stack 7/12] Per-discrepancy test infrastructure Mar 13, 2026
@jucor jucor changed the title [Stack 7/12] Per-discrepancy test infrastructure [Stack 7/13] Per-discrepancy test infrastructure Mar 13, 2026
@jucor
jucor requested a review from Copilot March 13, 2026 16:12

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds per-discrepancy (Python vs Clojure) test infrastructure and supporting dataset/blob tooling to enable TDD-style parity fixes across multiple reference blob variants (incremental vs cold-start), plus updates to cold-start blob generation and legacy comparison tests.

Changes:

  • Introduces test_discrepancy_fixes.py with discrepancy-scoped, dataset+blob parametrized tests (mostly xfailed to document known gaps).
  • Extends regression dataset utilities to support explicit blob variants (incremental / cold_start) and “filled blob” discovery.
  • Updates cold-start Clojure blob generation script to coordinate parallel runs and reduce worker conflicts.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 11 comments.

Show a summary per file
File Description
delphi/tests/test_repness_smoke.py Marks a pre-existing repness structure test as xfail.
delphi/tests/test_legacy_repness_comparison.py Switches discovered dataset parametrization to include blob variants and parses composite ids.
delphi/tests/test_legacy_clojure_regression.py Parametrizes legacy Clojure regression tests over blob variants; adds caching for shared Conversations.
delphi/tests/test_discrepancy_fixes.py New per-discrepancy parity test suite with dataset+blob parametrization and targeted xfails.
delphi/tests/conftest.py Adds composite-id parsing and extends use_discovered_datasets marker to support blob variants.
delphi/scripts/generate_cold_start_clojure.py Adds pause/unpause coordination for math workers and improves cold-start replay reliability (participants/comments copy).
delphi/scripts/clojure_comparer.py Switches vote loading helper and updates messaging to point to discrepancy plan.
delphi/polismath/regression/datasets.py Adds explicit blob_type selection and get_blob_variants() discovery of filled blobs.
delphi/polismath/regression/init.py Exports get_blob_variants.
delphi/polismath/pca_kmeans_rep/clusters.py Removes an unused sklearn import.
delphi/docs/PLAN_DISCREPANCY_FIXES.md Documents PR naming convention for the parity-fix series.
delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md Adds a new journal tracking baseline + xfail/xpass status across datasets.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread delphi/tests/test_repness_smoke.py Resolved
Comment thread delphi/tests/test_discrepancy_fixes.py Resolved
Comment thread delphi/tests/test_discrepancy_fixes.py Resolved
Comment thread delphi/polismath/regression/datasets.py Resolved
Comment thread delphi/tests/conftest.py Resolved
Comment thread delphi/tests/test_discrepancy_fixes.py Resolved
@jucor jucor changed the title [Stack 7/13] Per-discrepancy test infrastructure [Stack 7/15] Per-discrepancy test infrastructure Mar 16, 2026
@jucor
jucor force-pushed the jc/kmeans_analysis_docs branch from 000bb0a to f323584 Compare March 16, 2026 16:04
@jucor
jucor force-pushed the jc/kmeans_analysis_docs branch from 11831d4 to 2d235be Compare March 30, 2026 12:48
@jucor
jucor force-pushed the jc/series-of-fixes branch from 679694b to 1439005 Compare March 30, 2026 12:48
@jucor jucor changed the title [Stack 8/26] Per-discrepancy test infrastructure [Stack 9/27] Per-discrepancy test infrastructure Mar 30, 2026
@jucor
jucor force-pushed the jc/kmeans_analysis_docs branch from 2d235be to b979d12 Compare March 30, 2026 12:54
@jucor
jucor force-pushed the jc/series-of-fixes branch from 1439005 to 689ed20 Compare March 30, 2026 12:54
@jucor
jucor requested a review from Copilot March 30, 2026 16:25

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 8 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md Resolved
Comment thread delphi/tests/test_discrepancy_fixes.py Resolved
Comment thread delphi/tests/conftest.py Resolved
Comment thread delphi/tests/conftest.py Resolved
Comment thread delphi/tests/test_discrepancy_fixes.py Resolved
Comment thread delphi/tests/test_discrepancy_fixes.py Resolved
jucor and others added 9 commits March 30, 2026 17:44
Copilot review fixes:
- clusters.py: adjust k down when fewer distinct points than requested
- conversation.py: group_clusters members are base-cluster IDs (not participants)
- pca.py: fallback comps shape uses min(n_comps, n_cols) not min(n_comps, 2)

Bug fix:
- conversation.py: convert pandas Index to list in repness (JSON serialization)

Test fixes:
- test_clusters: init_clusters returns empty members
- test_conversation: test_recompute uses 10 comments to meet vote threshold
- test_datasets: add missing has_comments arg and cold_start blob fixture
- test_edge_cases: update group_repness expectation for no-clusters case
- test_repness_smoke: mark test_repness_structure as xfail (pre-existing)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Create tests/test_discrepancy_fixes.py with 30 tests across 12 classes,
one per discrepancy (D2, D4, D5, D6, D7, D8, D9, D10, D11, D12, D15)
plus synthetic edge cases. All discrepancy tests are xfailed — designed
to fail before each fix and pass after. Parametrized by all datasets
with Clojure reference blobs.

Current results: 5 passed, 2 skipped, 18 xfailed, 5 xpassed.

Create docs/JOURNAL_OF_DISCREPANCIES_FIX.md with initial baseline,
test results, Clojure blob structure notes, and design decisions.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Instead of erroring when the math worker is running, automatically pause
it during generation and resume it afterwards. This prevents the race
condition (existing worker picking up fake conversation votes) without
requiring manual intervention. Also give each poller container a
predictable name and force-remove it on exit to prevent orphans.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move pause/unpause into per-dataset processing so each run handles it
independently. Use a marker file (/tmp/polis-math-coldstart-paused) to
track whether we caused the pause, and check for sibling coldstart
containers before unpausing. Only the last concurrent run to finish
unpauses the math worker, and only if it was paused by us (not manually
by the user).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove unused `pairwise_distances` import from clusters.py
- Move `load_votes` import from test code to `benchmark_utils` module
  so the clojure_comparer script doesn't depend on test-only code
- Fix outdated "K-means++" references — Python now uses first-k-distinct
  initialization matching Clojure

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Document the 10 xpassed tests (all strict=False):
- D2 in-conv × 2 on vw (thresholds coincide on small dataset)
- D6 two_prop_test × 1 (pseudocount diff too small)
- D9 repness_not_empty × 7 (test too weak — checks non-empty,
  not correct count. TODO: tighten when fixing D9)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Clojure comparison tests now run against both blob variants when available:
- incremental: result of progressive refinement as votes trickled in
- cold_start: computed from scratch in one pass on full dataset

Each dataset generates separate test IDs (e.g., biodiversity-incremental,
biodiversity-cold_start). Only blobs with meaningful content (PCA data or
non-empty clusters) are included.

Key changes:
- Add get_blob_variants() to discover filled blob variants per dataset
- Add _is_blob_filled() to check if a blob has meaningful content
- Extend get_dataset_files() with explicit blob_type parameter
- Add use_blobs=True option to @pytest.mark.use_discovered_datasets
- Add parse_dataset_blob_id() helper for composite ID parsing
- Update test_legacy_clojure_regression, test_discrepancy_fixes, and
  test_legacy_repness_comparison to parametrize by blob variant
- Conversation computation is shared across blob variants of same dataset

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Legacy repness comparison: replace visibility-only test with asserting
tests (xfail for group coverage and set matching).

Legacy clojure regression: add test_repness_matches_clojure comparing
selected comment sets and z-values against the math blob (xfail).

Fix int/str tid mismatch that caused all shared-comment lookups to find
zero matches, making blob comparison tests pass vacuously.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@jucor
jucor force-pushed the jc/kmeans_analysis_docs branch from b979d12 to fa0395b Compare March 30, 2026 16:48
@jucor
jucor force-pushed the jc/series-of-fixes branch from 689ed20 to 35e84d5 Compare March 30, 2026 16:49
… imports

- Fix doc reference: CLJ-PARITY-FIXES-PLAN.md → PLAN_DISCREPANCY_FIXES.md (test + journal)
- Remove unused imports: json, ClojureComparer, unfold_clojure_group_clusters
- Fix docstring examples: -full → -incremental (matching actual composite ID format)
- Fix error message: 'No full blob' → 'No incremental blob'
- Fix _extract_dataset_from_test: use endswith() instead of in/replace
- Add strict=True to xfail in test_repness_smoke.py
@jucor
jucor force-pushed the jc/series-of-fixes branch from 35e84d5 to 9585ffa Compare March 30, 2026 17:05
@github-actions

Copy link
Copy Markdown
Contributor

Delphi Coverage Report

File Stmts Miss Cover
init.py 2 0 100%
benchmarks/bench_pca.py 76 76 0%
benchmarks/bench_repness.py 81 81 0%
benchmarks/bench_update_votes.py 38 38 0%
benchmarks/benchmark_utils.py 34 34 0%
components/init.py 1 0 100%
components/config.py 165 133 19%
conversation/init.py 2 0 100%
conversation/conversation.py 1118 336 70%
conversation/manager.py 131 42 68%
database/init.py 1 0 100%
database/dynamodb.py 387 233 40%
database/postgres.py 305 205 33%
pca_kmeans_rep/init.py 5 0 100%
pca_kmeans_rep/clusters.py 257 22 91%
pca_kmeans_rep/corr.py 98 17 83%
pca_kmeans_rep/pca.py 52 16 69%
pca_kmeans_rep/repness.py 361 48 87%
pca_kmeans_rep/stats.py 107 22 79%
regression/init.py 4 0 100%
regression/clojure_comparer.py 188 17 91%
regression/comparer.py 887 720 19%
regression/datasets.py 135 27 80%
regression/recorder.py 36 27 25%
regression/utils.py 137 118 14%
run_math_pipeline.py 260 114 56%
umap_narrative/500_generate_embedding_umap_cluster.py 210 109 48%
umap_narrative/501_calculate_comment_extremity.py 112 54 52%
umap_narrative/502_calculate_priorities.py 135 135 0%
umap_narrative/700_datamapplot_for_layer.py 502 502 0%
umap_narrative/701_static_datamapplot_for_layer.py 310 310 0%
umap_narrative/702_consensus_divisive_datamapplot.py 432 432 0%
umap_narrative/801_narrative_report_batch.py 785 785 0%
umap_narrative/802_process_batch_results.py 265 265 0%
umap_narrative/803_check_batch_status.py 175 175 0%
umap_narrative/llm_factory_constructor/init.py 2 2 0%
umap_narrative/llm_factory_constructor/model_provider.py 157 157 0%
umap_narrative/polismath_commentgraph/init.py 1 0 100%
umap_narrative/polismath_commentgraph/cli.py 270 270 0%
umap_narrative/polismath_commentgraph/core/init.py 3 3 0%
umap_narrative/polismath_commentgraph/core/clustering.py 108 108 0%
umap_narrative/polismath_commentgraph/core/embedding.py 104 104 0%
umap_narrative/polismath_commentgraph/lambda_handler.py 219 219 0%
umap_narrative/polismath_commentgraph/schemas/init.py 2 0 100%
umap_narrative/polismath_commentgraph/schemas/dynamo_models.py 160 9 94%
umap_narrative/polismath_commentgraph/tests/conftest.py 17 17 0%
umap_narrative/polismath_commentgraph/tests/test_clustering.py 74 74 0%
umap_narrative/polismath_commentgraph/tests/test_embedding.py 55 55 0%
umap_narrative/polismath_commentgraph/tests/test_storage.py 87 87 0%
umap_narrative/polismath_commentgraph/utils/init.py 3 0 100%
umap_narrative/polismath_commentgraph/utils/converter.py 283 237 16%
umap_narrative/polismath_commentgraph/utils/group_data.py 354 336 5%
umap_narrative/polismath_commentgraph/utils/storage.py 584 477 18%
umap_narrative/reset_conversation.py 159 50 69%
umap_narrative/run_pipeline.py 453 312 31%
utils/general.py 62 41 34%
Total 10951 7651 30%

@jucor

jucor commented Mar 30, 2026

Copy link
Copy Markdown
Collaborator Author

Superseded by spr-managed PR stack. See the new stack starting at #2508.

@jucor jucor closed this Mar 30, 2026
jucor added a commit that referenced this pull request Jun 2, 2026
* CI: add concurrency group to E2E workflow; run CI on spr/ branches

When spr force-pushes all stack branches at once, each branch triggers
a new E2E run while the previous one is still queued. The concurrency
group (keyed on branch ref) cancels the stale run, cutting the number
of concurrent E2E jobs roughly in half.

Also add `spr/**` to the branch filters in python-ci, jest-server-test,
and cypress-tests workflows so spr-managed PRs get the same CI coverage
as `jc/**` branches.

Additionally, update `.spr.yml` to align stack configuration with our
team's workflow: switch the stack's base branch from `main` to `edge`,
change `mergeMethod` from `rebase` to `squash` (matching our team's
merge style), and remove obsolete options (`multiCommitPRs`,
`concatCommitMessages`, `showStackNumberInTitle`) that no longer apply
to our spr fork's stack-of-PRs workflow.

commit-id:1a8eb714

* Add SKIP_GOLDEN env var to disable golden snapshot tests

## Summary


Add `SKIP_GOLDEN=1` environment variable to disable golden snapshot regression tests.

During stacked PR development, golden snapshots become stale as computation changes cascade through the stack. Rather than re-recording snapshots at every rebase (which causes conflict cascades in jj/git), we skip them until the stack is merged into `edge`.

### Changes

- **`test_regression.py`**: Add `@_skip_golden` decorator to `test_conversation_regression` and `test_conversation_stages_individually` — the only two tests that compare against golden snapshots. Other dataset-using tests (Clojure comparison, smoke tests) are unaffected.
- **`python-ci.yml`**: Set `SKIP_GOLDEN=1` in CI so the stacked PRs don't fail on stale snapshots.

### Usage

```bash
SKIP_GOLDEN=1 pytest tests/          # skip golden snapshot tests
pytest tests/                         # run everything (default)
```

## Test plan

- [x] `SKIP_GOLDEN=1 pytest tests/test_regression.py -v`: 4 skipped, 5 passed
- [x] `pytest tests/test_regression.py -v`: all 9 collected (golden tests run normally)


## Squashed commits

- Add SKIP_GOLDEN env var to disable golden snapshot regression tests

commit-id:d39cf65d

* Speed up CI: replace pip with uv pip in Dockerfile (~2x faster installs)

## Summary


Replace `pip install` with `uv pip install` in the delphi Dockerfile for faster dependency installation. `uv pip` is a drop-in replacement — same `requirements.lock`, same `pyproject.toml`, same installed packages in `site-packages`.

### Changes

- **`delphi/Dockerfile`**:
  - Copy `uv` binary from `ghcr.io/astral-sh/uv:0.11.2` (pinned version, single static binary)
  - Place in `/opt/uv/` in builder to avoid leaking into production image via `COPY --from=builder /usr/local/bin`
  - Set `UV_SYSTEM_PYTHON=1` (install into system Python, not a venv)
  - Replace all `pip install` with `uv pip install` in builder and test stages
  - Update BuildKit cache mount targets from `/root/.cache/pip` to `/root/.cache/uv`
  - Test stage copies `uv` from builder (single source of truth)

### What's NOT changed

- **Makefile** — untouched, `make rebuild-delphi` works as before
- **docker-compose.yml / docker-compose.test.yml** — untouched
- **pyproject.toml / requirements.lock** — untouched, same format
- **pip-compile workflow** — untouched, still used for lock file generation
- **Final/production image** — no `uv` added, stays lean

### CI Benchmark (GitHub Actions, `ubuntu-latest`, `--no-cache`)

5 pip runs (Mar 27 stack push) vs 3 uv pip runs (this PR):

| Step | pip (n=5) | uv pip (n=3) | Speedup |
|------|-----------|--------------|---------|
| **Docker build** | **264s** (sd=6) | **169s** (sd=2) | **1.56x (-94s)** |
| Pytest run | 223s (sd=4) | 227s (sd=4) | ~same |

**~94 seconds saved per CI run** on the Docker build step. Pytest runtime is unchanged (same packages, same tests). Low variance in both groups confirms this is a real improvement, not noise.

### Local Benchmark (Apple M1 Max, 64GB, Docker Desktop, `--no-cache`)

| Step | pip | uv pip | Speedup |
|------|-----|--------|---------|
| **Dependencies install** | **149.3s** | **80.4s** | **1.9x** |
| Dev deps install | 10.0s | 2.2s | **4.5x** |

## Test plan

- [x] `docker compose -f docker-compose.test.yml build --no-cache delphi` succeeds locally
- [x] Built image has all expected packages (`pip show` diagnostic passes in build log)
- [x] CI passes (3 successful runs)
- [x] `make rebuild-delphi` works (Makefile untouched)


## Squashed commits

- Speed up CI by using `uv pip` instead of `pip`

commit-id:0c448343

* Deep analysis of Python-Clojure discrepancies and fix plan

## Summary


Documentation-only PR: deep analysis of Python vs Clojure discrepancies and a TDD fix plan.

### Changes

- Deep analysis documents (`deep-analysis-for-julien/`) comparing Python and Clojure implementations statement-by-statement
- Consolidate CLAUDE.md documentation for the delphi project
- Discrepancy fix plan (`docs/PLAN_DISCREPANCY_FIXES.md`) with prioritized list of fixes

## Test plan

- [x] Documentation only — no code changes
🤖 Generated with [Claude Code](https://claude.com/claude-code)


## Squashed commits

- Adapt agents background and support Gemini
- Add sections
- Add deep analysis of Delphi vs Clojure math pipeline
- Fix inaccuracies found during statement-by-statement verification
- Rename deep analysis folder
- Replace AGENTS.md/GEMINI.md symlinks with proper delphi/CLAUDE.md
- Address Copilot review: fix doc references, move deep-analysis-for-julien into delphi/docs/

commit-id:d2f65026

* Per-discrepancy test infrastructure

## Summary


Per-discrepancy test infrastructure for TDD fixing of Python-Clojure differences.

### Changes

- Add per-discrepancy test markers and parametrized test infrastructure
- Cold-start recorder: coordinate parallel runs with marker file, auto-pause math workers
- Update journal with xpassed test breakdown across all datasets
- Address Copilot review: remove unused import, fix script issues
- Add naming convention documentation

## Test plan

- [x] 223 passed, 4 skipped, 22 xfailed, 7 xpassed, 0 failures
🤖 Generated with [Claude Code](https://claude.com/claude-code)


## Squashed commits

- Address Copilot review feedback and fix pre-existing test failures
- Add per-discrepancy test infrastructure and fix journal
- Fix cold-start recorder: auto-pause math workers and clean up containers
- Cold-start recorder: coordinate parallel runs with marker file
- Address Copilot review: remove unused import, fix script issues
- Update journal: xpassed test breakdown with all 7 datasets
- Add naming convention
- Test both incremental and cold-start Clojure blobs
- Add repness blob comparison tests and fix tid type mismatches
- Address Copilot review on PR #2420: fix stale references and clean up imports

commit-id:bdc830db

* Fix D2: in-conv participant threshold + D2c vote count source

## Summary


Fixes the in-conv participant threshold (D2), vote count source (D2c), and base-cluster sort order (D2b) to match Clojure. Adds monotonicity guard tests (D2d).

### D2: In-conv threshold

- **Before**: `threshold = 7 + sqrt(n_cmts) * 0.1` — increasingly restrictive for larger conversations (e.g., 8.8 for biodiversity's 314 comments)
- **After**: `threshold = min(7, n_cmts)` — matches Clojure exactly

### D2b: Base-cluster sort order (from Copilot review)

- **Before**: Base clusters sorted by size (descending) with IDs reassigned — changes encounter order of centers fed into group-level k-means
- **After**: Keep k-means ID order, matching Clojure's `(sort-by :id ...)`

### D2c: Vote count source (raw vs filtered matrix)

- **Before**: `_compute_user_vote_counts` and `n_cmts` used `self.rating_mat` (filtered — moderated-out comment columns removed). A participant who voted on 8 comments could drop to 5 visible votes after 3 comments were moderated-out, falling below threshold.
- **After**: Both use `self.raw_rating_mat` (includes all votes, even on moderated-out comments), matching Clojure's `user-vote-counts` (conversation.clj:217-225) which reads from `raw-rating-mat`.

### D2d: In-conv monotonicity (design decision)

Python does full recompute from `raw_rating_mat` every time, so monotonicity ("once in, always in") is guaranteed without persistence — votes are immutable in PostgreSQL, so a participant's count never decreases. This is **strictly better** than Clojure's approach (which persists in-conv to `math_main` because it uses delta vote processing).

5 guard tests (T1-T5) document this invariant and warn that switching to delta processing would require persisting in-conv to DynamoDB (ref: #2358).

### Impact

- biodiversity: 428 → 441 in-conv participants (now matches Clojure)
- Verified on 4 datasets with complete Clojure cold-start blobs

### Incremental vs cold-start blob testing

D2 tests run against both **cold-start** and **incremental** Clojure blobs (infrastructure from #2420):

- **Cold-start blobs** are computed in one pass on the full dataset. The in-conv threshold `min(7, n_cmts)` is evaluated once with the final `n_cmts`. Python matches these exactly.
- **Incremental blobs** were built progressively as votes trickled in over the conversation's lifetime. The threshold was evaluated at each iteration with a smaller `n_cmts`, admitting a few extra participants during earlier iterations. The difference is tiny (1–2 participants).

D2 tests on incremental blobs are currently **xfailed** with an explanatory comment. Matching incremental behaviour exactly would require simulating the progressive threshold — tracked as future work under Replay Infrastructure.

### Test results

```
253 passed, 5 skipped, 36 xfailed (0 failures)
```

## Test plan

- [x] D2 tests pass on all datasets with complete Clojure cold-start blobs
- [x] D2c: 3 synthetic tests verify vote counts include moderated-out votes, n_cmts includes moderated-out comments, participants stay in-conv after moderation
- [x] D2d: 5 monotonicity tests (basic across updates, survives moderation, worker restart + moderation, restart without new votes, mixed participants)
- [x] D2 tests xfail on incremental blobs (with explanatory comments)
- [x] Full test suite: 253 passed, 0 failures
- [x] Golden snapshots re-recorded for affected datasets

🤖 Generated with [Claude Code](https://claude.com/claude-code)


## Squashed commits

- Fix D2: in-conv threshold min(7, n_cmts) to match Clojure
- Skip D2 tests on datasets with incomplete Clojure blobs
- Address Copilot review: fix base-cluster sort order (D2b) and stale comment
- Add PR 1 test results to journal
- Plan: add D2c (vote count source) and D2d (in-conv monotonicity) to fix plan
- Journal: add session 3 findings (D2c vote count source, D2d monotonicity)
- Re-record golden snapshots and remove passing xfail markers
- xfail D2 in-conv tests on incremental blobs
- Journal: add session 4, update plan with D2 incremental in Replay PR B
- Fix D2c: use raw_rating_mat for vote counts and n_cmts threshold

commit-id:c0a682ec

* Fix D4: pseudocount formula

## Summary


- Change `PSEUDO_COUNT` from 1.5 to 2.0, matching Clojure's Beta(2,2) prior
- This changes probability smoothing from `pa = (na + 0.75)/(ns + 1.5)` to `pa = (na + 1)/(ns + 2)`
- All `pa`/`pd` values now match Clojure's `p-success` exactly (verified on all datasets with Clojure blobs)

## Changes

- `repness.py`: `PSEUDO_COUNT = 2.0` with updated comment
- `test_discrepancy_fixes.py`: remove xfail from 3 D4 tests (constant check, pa values per dataset, synthetic)
- `test_repness_unit.py`, `test_old_format_repness.py`: import `PSEUDO_COUNT` instead of hardcoding 1.5
- `simplified_repness_test.py`: update hardcoded constant
- Golden snapshots re-recorded for public datasets (vw, biodiversity)

## Test plan

- [x] TDD red: 6 D4 tests fail before fix
- [x] TDD green: all 6 D4 tests pass after fix
- [x] Full public suite: 258 passed, 0 failures
- [x] Private datasets (--include-local): 60 passed, 0 failures (discrepancy tests)
- [x] Regression tests pass on public + FLI + bg2018

🤖 Generated with [Claude Code](https://claude.com/claude-code)


## Squashed commits

- Fix D4: PSEUDO_COUNT 1.5 → 2.0 to match Clojure's Beta(2,2) prior
- Journal: add session 6 (D4 fix), update plan marking D4 done

commit-id:6ae3ee43

* Speed up regression tests

## Summary


- Default `benchmark=False` in `compare_with_golden()` — benchmark mode ran the pipeline 3x for timing statistics, unnecessary for correctness checks. The `regression_comparer.py` script already had `--benchmark` as opt-in, so this aligns the default.
- Add `skip_intermediate_stages` parameter to `compute_all_stages()` — `test_conversation_regression` now skips stages 1-4 (empty, load-only, PCA-only, PCA+clustering) since it only checks `overall_match`. `test_conversation_stages_individually` still runs all stages for granular failure detection.

### Measured speedup on one of the large private test conversations

| Test | Before | After | Speedup |
|------|--------|-------|---------|
| `test_conversation_regression` | 317s | 23s | **13.9x** |
| `test_conversation_stages_individually` | 60s | 32s | **1.9x** |

The regression test's ~14x speedup comes from two combined effects: no longer running the pipeline 3x (benchmark), and skipping 4 redundant intermediate stages.

## Test plan

- [x] All 9 public regression tests pass (vw + biodiversity)
- [x] Private dataset tests pass (`--include-local`)
- [x] Timing verified on large private dataset

🤖 Generated with [Claude Code](https://claude.com/claude-code)


## Squashed commits

- Address Copilot review: fix stale terminology, hardcoded blob_type, and synthetic test tid range
- Speed up regression tests: disable benchmark, skip intermediate stages

commit-id:f39f3218
@jucor
jucor deleted the jc/series-of-fixes branch July 12, 2026 19:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants