Ryzen phase-9 and benchmark work — 53 commits, first run through CI

.planning/NEXT-STEPS.md

@@ -0,0 +1,122 @@
+# NEXT-STEPS — What to do after Phase 7
+
+**Status:** Phase 7 (IAM Data Ingestion + Sweep Infrastructure) shipped 2026-05-06.
+The infrastructure is ready. The remaining unblockers are **manual data acquisition** and **one sweep run**, after which Phases 8-9 + the trained-corrector real-data retrain are all unblocked.
+
+---
+
+## Why this exists
+
+Phase 7 ended with all 17 RED stubs GREEN, all 4 plans landed, and `benchmark sweep` / `benchmark report --per-writer` callable from the CLI. But two downstream goals remain blocked on **user-side work that can't be automated:**
+
+1. **Phase 8 (Statistics Layer)** needs ≥10-sample multi-strategy runs in the benchmark DB. That requires running the sweep against real IAM data.
+2. **Trained-corrector real-data retrain** needs (vlm_text, ground_truth) pairs from real VLM runs. Today the corrector is gated OFF (`HE_USE_TRAINED_CORRECTOR=0`) because synthetic-only training causes hallucinations on hard cases — see `handwriting_engine/trained_correction/EVAL-RESULTS.md`. The sweep produces exactly those pairs.
+
+Both unblock from a single sweep run.
+
+---
+
+## Step 1 — Download the IAM Handwriting Database
+
+The IAM dataset is registration-gated, so this is manual.
+
+1. Register and download from one of:
+   - [HEIA-FR mirror](https://fki.tic.heia-fr.ch/databases/iam-handwriting-database)
+   - Original Univ. Bern site (legacy)
+2. Extract `lines.tgz` (line-level images) and `ascii.tgz` (transcriptions). Layout expected:
+   ```
+   <iam-root>/
+   ├── ascii/
+   │   └── lines.txt
+   └── lines/
+       └── <writer>/<form>/<form>-<line>.png
+   ```
+3. **Optional but recommended:** also grab the `largeWriterIndependentTextLineRecognitionTask/` partition files. They split IAM into `trainset.txt`, `validationset1.txt`, `testset.txt`. The sweep should run against `testset.txt` only — never on training data, or the baseline isn't a true generalization measure.
+
+---
+
+## Step 2 — Ingest IAM into the benchmark DB
+
+```bash
+cd "/Users/user/Documents/Work & Projects/VSCode Projects/handwriting-engine"
+
+# Test partition only (safer, ~2k samples):
+python3 -m handwriting_engine.cli benchmark ingest-iam \
+    --ascii-dir <iam-root>/ascii \
+    --lines-dir <iam-root>/lines \
+    --partition-file <iam-root>/largeWriterIndependentTextLineRecognitionTask/testset.txt
+
+# Verify ingestion:
+python3 -m handwriting_engine.cli benchmark list --show-samples | head -20
+```
+
+Expected: rows with `category='iam'`, `student='iam-writer-XXX'`. The CLI prints `{ingested, skipped_dup, skipped_missing}` counts.
+
+> **Cost note:** The next step runs ALL FIVE strategies against every ingested sample. If you ingest the full test set (~2k lines), one sweep can run ~$3-5 in API calls (Gemini Flash is the cheap default). If unsure, ingest ~50 samples first using a partition subset, do a smoke sweep, then scale up.
+
+---
+
+## Step 3 — Run the multi-strategy sweep
+
+```bash
+python3 -m handwriting_engine.cli benchmark sweep --provider gemini
+# Confirms cost projection, then executes all 5 strategies:
+#   baseline, self_correct, line_level, prompt_adapted, zoomed_verify
+# Returns one run_id per strategy.
+```
+
+Add `--yes` to skip the confirmation prompt (useful in CI / headless runs).
+
+---
+
+## Step 4 — Inspect per-writer breakdown
+
+```bash
+# Replace <run_id> with one of the run_ids the sweep printed:
+python3 -m handwriting_engine.cli benchmark report --run-id <run_id> --per-writer
+```
+
+This is the IAM-03 deliverable: shows whether a strategy's CER gain is consistent across writers or driven by a few easy ones.
+
+---
+
+## Step 5 — Retrain the trained corrector on real data
+
+Now the (vlm_output, ground_truth) pairs from the sweep can fine-tune the FLAN-T5 corrector that's currently gated off:
+
+```bash
+# Continue from the v1 synthetic checkpoint (don't start from scratch — preserves
+# the easy-error fixes the synthetic data already taught it):
+python3 -m handwriting_engine.trained_correction.train \
+    --from-benchmark-db ~/.handwriting-engine/benchmark.db \
+    --continue-from ~/.handwriting-engine/models/trained-corrector-v1 \
+    --output-dir ~/.handwriting-engine/models/trained-corrector-v2 \
+    --epochs 3
+
+# A/B eval: v2 vs v1 vs heuristic-only
+python3 -m handwriting_engine.cli trained-correction eval --n-pairs 200 --seed 9999
+```
+
+Expected: hallucinations on hard cases (the 3/10 spot-check failures documented in `trained_correction/EVAL-RESULTS.md`) drop substantially — because the model now sees the actual VLM error distribution rather than a guessed-at synthetic one. Once the gated A/B passes, flip `HE_USE_TRAINED_CORRECTOR=1` to default-on the combined heuristic→trained pipeline.
+
+---
+
+## Step 6 — Plan Phase 8
+
+Once the sweep run lives in the DB, Phase 8 (Statistics Layer) is unblocked:
+- Wilcoxon signed-rank p-values on `benchmark compare`
+- 95% bootstrap CIs on CER estimates
+- Cohen's r effect size
+
+Run `/gsd:plan-phase 8` from inside the engine directory when ready.
+
+---
+
+## Out-of-band side projects flagged in the broader strategy
+
+These do not block Phase 8 but are worth queuing for follow-up sessions:
+
+- **S2 — Per-writer few-shot exemplars.** Currently `writer_profile_store.build_calibration_block()` injects writer-specific text hints into prompts. Stronger: pull 2-3 already-labeled images of the same writer from the benchmark DB and pass them as multi-image prompts (Gemini and Claude both support it). Likely the biggest single CER gain on returning writers (lab notebook semester scenarios). **Spec drafted: `.planning/S2-SPEC-per-writer-few-shot.md` (2026-05-06).**
+- **S3 — Wire `~/.claude/skills/handwriting-reader/` skill to call the engine library directly.** The skill currently does its own multi-pass workflow. One source of truth = engine improvements propagate immediately. **Spec drafted: `.planning/S3-SPEC-skill-engine-bridge.md` (2026-05-06).**
+- **S4 — Professor OS feedback loop.** `professor/LabNoteBookGrader/` graders should surface low-confidence reads, capture corrections, and write them back to the benchmark DB as per-writer ground truth. Per-writer accuracy then compounds over a semester. **Spec drafted: `.planning/S4-SPEC-professor-feedback-loop.md` (2026-05-06).**
+- **S5 — Char-level consensus + confusion-pair-aware postprocess.** Word-level voting catches obvious disagreements; char-level catches single-character swaps (`rn↔m`, `cl↔d`). The drill-down report already tracks confusion pairs; postprocess can consume them. **Spec drafted: `.planning/S5-SPEC-char-consensus-confusion-postprocess.md` (2026-05-06).**

.planning/REQUIREMENTS.md

@@ -27,9 +27,9 @@ Requirements for the benchmarking milestone. Each maps to a roadmap phase.
 
 ### Reporting
 
-- [ ] **RPT-01**: Schema v4 adds `is_baseline` flag to runs table; `benchmark set-baseline RUN_ID` pins a run as the regression anchor; `detect_regressions()` compares against the pinned baseline, not runs[-2].
-- [ ] **RPT-02**: `benchmark recommend` outputs the best strategy+provider configuration with a weighted composite score (70% CER / 15% cost / 15% stability across runs).
-- [ ] **RPT-03**: Developer can collect and store ground-truth transcriptions from real student lab notebooks using `benchmark ingest-lab` with a guided annotation workflow, enabling production-distribution benchmarks distinct from IAM.
+- [x] **RPT-01**: Schema v6 adds `is_baseline` flag to runs table; `benchmark set-baseline RUN_ID` pins a run as the regression anchor; `detect_regressions()` compares against the pinned baseline, not runs[-2]. (shipped 2026-05-06)
+- [x] **RPT-02**: `benchmark recommend` outputs the best strategy+provider configuration with a weighted composite score (70% CER / 15% cost / 15% stability across runs). (shipped 2026-05-06; end-to-end on multi-strategy sweep gated on IAM data)
+- [x] **RPT-03**: Developer can collect and store ground-truth transcriptions from real student lab notebooks using `benchmark ingest-lab` with a guided annotation workflow, enabling production-distribution benchmarks distinct from IAM. (shipped 2026-05-06)
 
 ## v4.0 Requirements (Deferred)
 
@@ -63,11 +63,11 @@ Requirements for the benchmarking milestone. Each maps to a roadmap phase.
 | IAM-01 | Phase 7 | Complete |
 | IAM-02 | Phase 7 | Pending |
 | IAM-03 | Phase 7 | Pending |
-| STAT-01 | Phase 8 | Pending |
-| STAT-02 | Phase 8 | Pending |
-| RPT-01 | Phase 9 | Pending |
-| RPT-02 | Phase 9 | Pending |
-| RPT-03 | Phase 9 | Pending |
+| STAT-01 | Phase 8 | Implemented (verification gated on IAM data) |
+| STAT-02 | Phase 8 | Implemented (verification gated on IAM data) |
+| RPT-01 | Phase 9 | Implemented |
+| RPT-02 | Phase 9 | Implemented (verification gated on IAM sweep) |
+| RPT-03 | Phase 9 | Implemented |
 
 **Coverage:**
 - v3.0 requirements: 12 total

.planning/ROADMAP.md

@@ -23,9 +23,9 @@ Full details: `.planning/milestones/v2.0-ROADMAP.md`
 ### v3.0 — Verified Accuracy
 
 - [x] **Phase 6: Measurement Foundation** — Reproducible baseline + variance floor + cost guardrails (completed 2026-04-11)
-- [ ] **Phase 7: IAM Data Ingestion + Sweep Infrastructure** — Full IAM benchmark pipeline
-- [ ] **Phase 8: Statistics Layer** — Statistical defensibility for all comparisons
-- [ ] **Phase 9: Final Sweep, Recommendation, and Baseline Lock** — Best config identified, regression anchor committed
+- [x] **Phase 7: IAM Data Ingestion + Sweep Infrastructure** — Full IAM benchmark pipeline (completed 2026-05-06)
+- [ ] **Phase 8: Statistics Layer** — Statistical defensibility for all comparisons (implementation shipped 2026-05-06; criterion verification gated on user IAM download + first sweep run; see `.planning/NEXT-STEPS.md`)
+- [ ] **Phase 9: Final Sweep, Recommendation, and Baseline Lock** — Best config identified, regression anchor committed (implementation shipped 2026-05-06; verification of "best config recommendation" gated on user IAM download + first multi-strategy sweep)
 
 ## Phase Details
 
@@ -42,9 +42,9 @@ Full details: `.planning/milestones/v2.0-ROADMAP.md`
 
 Plans:
 - [x] 06-01-PLAN.md — Wave 0 test stubs: failing tests for all Phase 6 behaviors (FOUND-01 through FOUND-04) (completed 2026-04-11)
-- [ ] 06-02-PLAN.md — v4 schema migration + dataclass extensions (db.py, models.py)
-- [x] 06-03-PLAN.md — Provenance capture + marker rate computation + report display (evaluate.py, report.py)
-- [ ] 06-04-PLAN.md — CLI surface: benchmark calibrate subcommand + cost guardrail + provenance flags (cli.py)
+- [x] 06-02-PLAN.md — v4 schema migration + dataclass extensions (db.py, models.py) (completed 2026-04-11)
+- [x] 06-03-PLAN.md — Provenance capture + marker rate computation + report display (evaluate.py, report.py) (completed 2026-04-11)
+- [x] 06-04-PLAN.md — CLI surface: benchmark calibrate subcommand + cost guardrail + provenance flags (cli.py) (completed 2026-04-11)
 
 ### Phase 7: IAM Data Ingestion + Sweep Infrastructure
 **Goal**: The developer can load the IAM Handwriting Database into the benchmark system and execute a full multi-strategy sweep against it, with per-writer variance visible in reports.
@@ -57,10 +57,10 @@ Plans:
 **Plans**: 4 plans
 
 Plans:
-- [ ] 07-01-PLAN.md — Wave 0 RED test stubs: 16 failing tests across TestIAMIngest, TestSweep, TestPerWriterReport (IAM-01, IAM-02, IAM-03)
-- [ ] 07-02-PLAN.md — IAM ingest infrastructure: parse_iam_lines(), ingest_iam(), benchmark ingest-iam CLI (IAM-01)
-- [ ] 07-03-PLAN.md — Sweep infrastructure: line_level/auto_retry threading, run_sweep(), benchmark sweep CLI (IAM-02)
-- [ ] 07-04-PLAN.md — Per-writer report: generate_per_writer_report(), benchmark report --per-writer flag (IAM-03)
+- [x] 07-01-PLAN.md — Wave 0 RED test stubs: 17 failing tests across TestIAMIngest, TestSweep, TestPerWriterReport (IAM-01, IAM-02, IAM-03) (completed 2026-04-11)
+- [x] 07-02-PLAN.md — IAM ingest infrastructure: parse_iam_lines(), ingest_iam(), benchmark ingest-iam CLI (IAM-01) (completed 2026-04-12)
+- [x] 07-03-PLAN.md — Sweep infrastructure: line_level/auto_retry threading, run_sweep(), benchmark sweep CLI (IAM-02) (completed 2026-05-06)
+- [x] 07-04-PLAN.md — Per-writer report: generate_per_writer_report(), benchmark report --per-writer flag (IAM-03) (completed 2026-05-06)
 
 ### Phase 8: Statistics Layer
 **Goal**: CER comparisons between strategies are statistically defensible — not just raw delta numbers — so the developer can assert with confidence that a measured improvement is real.
@@ -69,7 +69,10 @@ Plans:
 **Success Criteria** (what must be TRUE when this phase completes):
   1. Running `benchmark compare RUN_A RUN_B` on any two runs with n >= 10 samples automatically appends a Wilcoxon signed-rank p-value and Cohen's r effect size to the output, with no extra flags needed.
   2. The same `benchmark compare` output includes 95% bootstrap confidence intervals on both CER estimates, so the developer can see whether the CI bands overlap and judge whether the difference is distinguishable from sampling noise.
-**Plans**: TBD
+**Plans**: 1 plan
+
+Plans:
+- [x] 08-01-PLAN.md — Stats module + compare_runs wire-up: paired Wilcoxon, percentile bootstrap CI, Cohen's r (no scipy dep) (completed 2026-05-06; criterion verification gated on IAM data)
 
 ### Phase 9: Final Sweep, Recommendation, and Baseline Lock
 **Goal**: The developer knows which strategy+provider configuration is best for lab notebook grading, and a regression baseline is pinned so any future code change that silently degrades accuracy is immediately detectable.
@@ -79,7 +82,12 @@ Plans:
   1. Developer runs `benchmark set-baseline RUN_ID` to pin any run as the regression anchor; `detect_regressions()` then compares future runs against that pinned run (not the penultimate run), and the schema tracks the `is_baseline` flag durably across sessions.
   2. `benchmark recommend` outputs a single ranked recommendation with a composite score (70% CER / 15% cost / 15% stability) and the winning strategy+provider combination is unambiguous.
   3. Developer can run `benchmark ingest-lab` against real student lab notebook images and store ground-truth transcriptions via a guided annotation workflow, producing a production-distribution test set that is separate from IAM.
-**Plans**: TBD
+**Plans**: 3 plans
+
+Plans:
+- [x] 09-01-PLAN.md — Schema v6 is_baseline + set_baseline/get_baseline + detect_regressions retarget + CLI (RPT-01) (completed 2026-05-06)
+- [x] 09-02-PLAN.md — recommend_strategy() composite 70/15/15 score + CLI (RPT-02) (completed 2026-05-06; verification gated on multi-strategy IAM sweep)
+- [x] 09-03-PLAN.md — ingest_lab() guided annotation + CLI (RPT-03) (completed 2026-05-06)
 
 ## Progress
 
@@ -91,6 +99,6 @@ Plans:
 | 4. Preprocessing + Writer Adaptation | v2.0 | 1/1 | ✅ Complete | 2026-04-09 |
 | 5. Post-Processing + Benchmark Suite | v2.0 | 1/1 | ✅ Complete | 2026-04-09 |
 | 6. Measurement Foundation | 4/4 | Complete    | 2026-04-11 | - |
-| 7. IAM Data Ingestion + Sweep Infrastructure | 1/4 | In Progress|  | - |
-| 8. Statistics Layer | v3.0 | 0/? | Not started | - |
+| 7. IAM Data Ingestion + Sweep Infrastructure | v3.0 | 4/4 | ✅ Complete | 2026-05-06 |
+| 8. Statistics Layer | v3.0 | 0/? | Blocked on user IAM download + first sweep | - |
 | 9. Final Sweep, Recommendation, and Baseline Lock | v3.0 | 0/? | Not started | - |