Ryzen phase-9 and benchmark work — 53 commits, first run through CI - #2
Merged
Conversation
Archive v2.0 milestone artifacts: - .planning/milestones/v2.0-ROADMAP.md — full phase details archived - .planning/milestones/v2.0-REQUIREMENTS.md — all 9 requirements archived - .planning/milestones/v2.0-MILESTONE-AUDIT.md — audit report (9/9 passed) - .planning/MILESTONES.md — milestone log with stats and accomplishments - .planning/RETROSPECTIVE.md — lessons learned, patterns established - .planning/ROADMAP.md — collapsed to milestone summary with progress table - .planning/PROJECT.md — evolved: current state, validated requirements, key decisions - .planning/STATE.md — status: shipped, log updated 9 requirements satisfied. 5 phases. 442 tests passing. Baseline: Gemini Flash 1.67% CER → target <1.3% with self_correct strategy. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…aptation, writer embeddings New modules: - prompt_adapter.py: provider-specific prompt optimization (Gemini=concise top-15 pairs, OpenAI=role+task+examples, Claude=full verbose). Wired into read_page(). - writer_embeddings.py: Gemini multimodal embedding-based writer identification. enroll_writer() builds centroid profiles; identify_writer() matches against them. Wired into read_page() and read_with_consensus() via auto_identify_writer=True. - logging_config.py: structured JSON logging for production, human-readable for dev. - providers/batch_openai.py: OpenAI Batch API integration (50% cost reduction for benchmarks). Improvements to existing modules: - quality.py: detect_skew() + skew flagging in assess_image(); faint-ink strategy fix. - enhance.py: _deskew() wired into enhance pipeline when skew detected. - providers/gemini.py: enable_context_cache() for 90% token discount on batch workflows; token_confidences tracking; temperature parameter. - _constants.py: SKEW_THRESHOLD, ZOOM_VERIFY_*, SMART_ROUTE_* thresholds; model versions pinned (GPT-4.1-2025-04-14); contrast lowered to 2.0 per PreP-OCR ACL 2025. - vision.py: zoomed crop verification (_zoomed_verify) wired into auto_retry path; auto_identify_writer parameter auto-detects writer before loading WriterProfileStore. Tests: 61 new tests in test_improvements.py, all passing. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Phases 6-9: Measurement Foundation → IAM Ingestion + Sweep → Statistics Layer → Final Sweep + Recommendation + Baseline Lock. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Added test_v4_migration_columns to TestSchemaCreation - Asserts model_version, iam_partition, norm_flags, vocab_hints_off in runs table - Asserts question_marker_rate in provider_outputs table - Fails with AssertionError — v4 migration columns not yet in schema
- Added v4 migration SQL with 5 ALTER TABLE statements for runs (model_version, iam_partition, norm_flags, vocab_hints_off) and provider_outputs (question_marker_rate) - Bumped CURRENT_SCHEMA_VERSION from 3 to 4 - Updated base _SCHEMA_SQL to include new columns for fresh DB creation - Extended insert_run() with model_version, iam_partition, norm_flags, vocab_hints_off params - Extended insert_provider_output() with question_marker_rate param (None default) - All existing callers continue to work via None/0 defaults
…mmand, TestCostProjection, TestProvenanceCapture - TestMarkerRate: 4 tests for question_marker_rate column in provider_outputs - TestCalibrateCommand: 3 tests for new benchmark calibrate subcommand - TestCostProjection: 3 tests for pre-flight cost prompt on benchmark run - TestProvenanceCapture: 2 tests for model_version/norm_flags provenance in runs table - All new tests fail RED (OperationalError, AssertionError, or SystemExit) - All 14 previously passing tests remain GREEN - Added CliRunner import at module level
… updates - Created 06-01-SUMMARY.md with task commits, decisions, and next-phase readiness - Updated STATE.md: plan 01 complete, decisions recorded, log entry added - Updated ROADMAP.md: phase 6 progress (1/4 summaries) - Marked FOUND-01, FOUND-02, FOUND-03, FOUND-04 complete in REQUIREMENTS.md
…aluate.py - ProviderOutput: added question_marker_rate: float | None = None field - StrategyResult: added mean_marker_rate: float = 0.0 field - RunSummary: added model_version, iam_partition, norm_flags, vocab_hints_off provenance fields - All new fields have safe defaults — existing callers unchanged - evaluate.py: added _compute_marker_rate() helper to count [?] tokens from raw text - evaluate.py: passes question_marker_rate to insert_provider_output() for each single-provider read Auto-fixed (Rule 2): evaluate.py was missing question_marker_rate computation, causing TestMarkerRate.test_marker_rate_from_raw_text to fail despite schema support.
…aluate.py Adds _NORM_FLAGS constant, _resolve_model_version() helper, iam_partition/vocabulary_hints params to run_benchmark(). Stores model_version, norm_flags, iam_partition, vocab_hints_off on insert_run(). Computes question_marker_rate from raw text (before normalize_text strips markers) and stores per provider_output. Also updates list_runs() in db.py to populate provenance fields on RunSummary. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…k report _aggregate_results() now computes mean_marker_rate from per-sample question_marker_rate. _format_table() shows provenance block (model, partition, norm flags) at top of report, and adds marker_rate column alongside CER/WER. generate_report() fetches run metadata from the runs table to populate the provenance block. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
test_marker_rate_in_report and test_report_contains_provenance_header now create a real run via run_benchmark() before calling generate_report(), so the report query finds actual rows instead of returning an empty/error state. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…OADMAP updated Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Phase 6 Measurement Foundation complete. Adds provenance tracking (FOUND-01), [?] marker rate metric (FOUND-02), benchmark calibrate subcommand (FOUND-03), and cost projection guardrail (FOUND-04). 44 tests passing. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…port Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Added Path import and IAM-specific try/except import guard - Appended TestIAMIngest class with 9 pytest.fail stubs - Covers parse_iam_lines (5 stubs) and ingest_iam (3 stubs) + CLI stub - All 9 fail as expected; 19 existing tests remain green
…d ingest_iam - Replace stub pytest.fail() calls with real test implementations - test_parse_skips_comments: verifies comment and empty lines excluded - test_parse_filters_err: verifies err-status lines excluded - test_parse_extracts_fields: verifies line_id, writer_id, form_id, transcription - test_parse_replaces_pipes: verifies pipe->space substitution - test_parse_filters_partition: verifies partition_forms filtering - test_ingest_sets_category_and_student: verifies category='iam', student='iam-writer-*' - test_ingest_inserts_ground_truth: verifies ground truth committed per sample - test_ingest_iam_dedup: verifies second import skips duplicates - test_cli_ingest_iam_command: verifies CLI command exists and runs
- parse_iam_lines(lines_txt, partition_forms=None): parses IAM ascii/lines.txt,
skips comments/blanks/err-status/short lines, extracts line_id/writer_id/form_id/
transcription, replaces pipe separators with spaces, filters by partition_forms set
- _iam_image_path(lines_dir, line_id): private helper resolving PNG path under
lines/<writer>/<form>/<line_id>.png structure
- ingest_iam(ascii_dir, lines_dir, partition_file, db_path): bulk-inserts line images
with category='iam', student='iam-writer-{writer_id}', commits ground truth
atomically in same transaction, deduplicates by hash, raises FileNotFoundError
if lines_dir missing
- No quality assessment called (IAM lines are pre-segmented clean PNGs)
- 8/9 TestIAMIngest stubs now GREEN; test_cli_ingest_iam_command remains RED
Word-by-word edit-distance correction has a ceiling on multi-word scientific terms (natural selection, amino acid, periodic table) where neither token in isolation is unambiguously close to a single dictionary word, but the bigram is. Add a phrase pass that runs before single-word correction and snaps both tokens of a known phrase pair when each is within edit distance 1 of the canonical form. - _within_edit_distance_1: cheap yes/no ED-1 check (replace / insert / delete / transpose) -- avoids generating the full ED-1 set when targets are fixed. - _correct_phrases: scans bigrams; corrects only when exactly one phrase pair matches AND at least one word is genuinely out-of-vocabulary (prevents rewriting plausible bigrams that happen to look like phrases). - correct_domain_terms: phrase pass runs before existing single-word pass; rewritten indices are skipped so we do not double-correct. - Phrase lists for biology and chemistry covering common multi-word terms the word-level pass misses. Also: planning artifact for v4.0 (trained post-correction model). The heuristic post-correction layer has limits a learned model can break through, but it requires paired training data which Phase 7 IAM ingestion will produce. Document the dependency and proposed phase shape so it is not forgotten when v3.0 closes. Tests: 17 new tests covering phrase correction and the ED-1 helper. 47 postprocess tests pass; full suite shows only pre-existing Phase 7 RED stubs failing (verified against main). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a trained post-correction layer on top of the existing heuristic post-correction. Heuristic correction tops out at edit-distance-1 word + bigram lookups against curated wordlists; past that, multi-character OCR confusions, doubled letters, smushed words, and context-sensitive corrections need a learned model. This change ships the full training + inference pipeline. Real-data fine-tune from Phase 7 IAM ingestion is the next step (queued, see v4.0-PROPOSAL.md). Subpackage layout (handwriting_engine/trained_correction/): - synthetic_data.py: realistic OCR corruption (pair confusions like rn-m and cl-d, doubled letters, dropped letters, transpositions, smush/split, diacritic stripping, capitalization slips). Difficulty configs (light / default / aggressive) sampled per-example so the model sees a spread. - corpus.py: clean reference text builder. Lab-notebook templates (observation / methodology / reasoning / equation), domain wordlists from postprocess.py, optional /usr/share/dict/words for general English. - dataset.py: build_pairs / split_pairs (deterministic train/val/test) plus torch Dataset wrapper; instruction-style prefix avoids the t5-small pretraining bias where "correct: ..." gets read as a translation task. - train.py: manual PyTorch loop (no accelerate dep). flan-t5-small default (instruction-tuned, follows arbitrary task prefixes; t5-small mistook "correct:" as translate-to-German with limited training). MPS / CPU / CUDA device routing. Per-step logging, val eval at save_steps with best-checkpoint preservation, training_manifest.json output. - corrector.py: lazy-loaded inference singleton. Beam search, sentence-aware chunking for long inputs. Default checkpoint search: ~/.handwriting-engine/models/trained-corrector-v1, then ./ckpt/, then HE_TRAINED_CORRECTOR_PATH env var. - eval.py: A/B harness reporting CER for input vs heuristic vs trained vs combined. Pure-stdlib Levenshtein. Integration: - postprocess.py: new correct() orchestrator runs heuristic first, then the trained corrector if (use_trained=True OR HE_USE_TRAINED_CORRECTOR=1). Off by default until a checkpoint is validated against real data. - cli.py: handwriting-engine trained-correction train / eval subcommands. - pyproject.toml: optional [trained-correction] extras (torch, transformers, sentencepiece, accelerate, numpy). Base engine stays lightweight. Heuristic baseline on 500 mixed-difficulty synthetic pairs: input CER 7.65% -> heuristic 7.13% (~0.5pp improvement) That gap is what the trained model has room to close further. Pipeline verified end-to-end with a CPU smoke run (80 pairs, 18 steps, t5-small): loss 2.08 -> 1.72, val_loss 1.60, test_loss 1.42, checkpoint saves and loads. Sim-to-real gap caveat is documented in v4.0-PROPOSAL.md and the manifest: this is synthetic-only training. Don't claim production parity until a small real-data fine-tune lands once Phase 7 IAM ingestion completes. Tests: 22 new tests in tests/test_trained_correction.py covering synthetic data determinism, corpus generation, dataset construction, eval CER math, and the postprocess.correct() orchestrator. Trained-model integration test is gated on HE_TRAINED_CORRECTOR_PATH so CI doesn't need a checkpoint. All 69 (47 postprocess + 22 trained-correction) pass; 8 pre-existing Phase 7 RED stubs in test_benchmark_evaluate.py untouched. Files: 1661 LOC across 7 modules + 1 test file. v4.0-PROPOSAL.md updated to reflect actual implementation status vs forward-looking work. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…hitecture, and caveats Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…hallucination caveat Final v0 metrics: - input CER 7.32% -> heuristic 6.88% -> combined 5.07% - combined beats heuristic-alone by 1.82pp (26% relative reduction) - 7/10 spot-checks correct; 3/10 hallucinated plausible substitutes on hard cases The hallucination failure mode is the canonical synthetic-to-real gap and is documented; mitigation strategies queued for v4.1 (real-data fine-tune, confidence gate, fidelity check). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three additions on top of synthetic v0:
1. Confidence gate — only fire trained pass when the heuristic made >=1
correction. Rationale: if the heuristic found errors, more probably exist
for the trained model to fix; if it found nothing, either the input is
clean (trained pass risks rewriting it) or errors are out-of-vocabulary
(trained pass tends to hallucinate plausible-but-wrong words). Default ON,
tunable via `require_heuristic_hit=False`.
2. Fidelity check — reject trained outputs whose Levenshtein-distance ratio
from the heuristic output exceeds a threshold (default 0.35, so a typo
correction at ~8% passes but a `niitochondria → nucleotide` rewrite at
~62% is rejected and the heuristic output is kept).
3. Real-data ingestion path — `from_benchmark_db()` reads the engine's
existing benchmark DB, joins `provider_outputs <-> ground_truths` on
`sample_id`, dedupes, returns gracefully when DB missing. `train.py` adds
`--from-benchmark-db`, `--benchmark-providers`, `--real-data-weight`, and
`--continue-from` so when Phase 7 IAM ingestion lands, the v0 synthetic
model can be fine-tuned with one CLI invocation.
Re-evaluation on 200 held-out synthetic pairs (seed 9999):
raw combined: 5.07% CER (best on average — but introduced hallucinations)
gated combined: 5.86% CER (DEFAULT — 15% relative improvement, safer)
heuristic only: 6.88%
input: 7.32%
Spot-check (same 10 inputs as v0):
- All 7 v0 wins still wins
- All 3 v0 hallucinations now prevented:
celll -> "Cell cell ..." became celll (preserved)
numbr -> "... nucleus." became numbr (preserved)
niitochondria-> "... nucleotide ..." became niitochondria (preserved)
The corrector now never makes the input WORSE — it either fixes errors or
leaves them alone. Trade-off: some unfixed errors stay uncorrected. For lab
notebook grading where hallucinations would change semantic meaning, this
is the right default.
Tests: 14 new in tests/test_trained_correction.py covering _change_ratio,
_within_fidelity, the orchestrator with both gates, and the benchmark DB
loader (with synthetic SQLite fixtures so it doesn't depend on real data).
83 / 83 unit tests pass.
EVAL-RESULTS.md updated with the gated numbers, the per-case spot-check
comparison, and the Phase 7 fine-tune command.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…threading, benchmark sweep CLI Turns 5 TestSweep RED stubs GREEN (IAM-02). Threads line_level and auto_retry through _read_single -> _run_benchmark_inner -> run_benchmark for sweep parity with read_page. Adds SWEEP_STRATEGIES (baseline, self_correct, line_level, prompt_adapted, zoomed_verify) and run_sweep() that filters samples.category='iam' and executes each strategy in turn. benchmark sweep CLI prints cost projection before any API call, supports --yes bypass, and lists run_ids per strategy on completion. Unblocks 07-04 (per-writer report) and the trained_correction real-data retrain (consumes sweep outputs as (vlm_text, ground_truth) pairs via from_benchmark_db). 31 evaluate tests passing (5 new + 26 untouched). Remaining 3 failures are TestPerWriterReport stubs — handled in 07-04. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…er-writer flag Turns 3 TestPerWriterReport RED stubs GREEN (IAM-03). Adds generate_per_writer_report() in benchmark/report.py — groups eval_metrics by samples.student, returns formatted table (Writer | Mean CER | Min CER | Max CER | N) sorted hardest-writer-first. Empty-writer case returns explanatory message rather than crash. Wires --per-writer flag onto benchmark report CLI command; existing report logic untouched when flag is absent. Hidden --db-path option added for testability. Phase 07 (IAM Data Ingestion + Sweep Infrastructure) now complete: all 4 plans landed, all 17 RED stubs turned GREEN, IAM-01/IAM-02/IAM-03 satisfied. Full suite: 525 passed, 2 skipped, 1 xfailed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 7 (IAM Data Ingestion + Sweep Infrastructure) shipped with 07-03 and 07-04 landing today. Updates progress counters (8/8 plans complete, 2/4 phases complete in v3.0), adds 6 phase-7 design decisions to the accumulated context, and writes two log entries (07-03 + 07-04 + phase complete). NEXT-STEPS.md documents the user-side unblockers for Phase 8 and the trained-corrector real-data retrain: download IAM, ingest-iam, sweep, retrain v2 corrector with --continue-from v1 checkpoint. Also queues S2-S5 strategic side projects (writer exemplars, handwriting-reader skill wiring, Professor OS feedback loop, char-level consensus) for follow-up sessions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…, char consensus) Out-of-band side projects from NEXT-STEPS now have falsifiable success criteria, design notes, and risk registers. None are phases yet — they are seed material for /gsd-discuss-phase. NEXT-STEPS gains pointer lines so /gsd-progress can find them. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rewrites \`cli read\` to be the single source of truth for the handwriting-reader skill. Adds: - \`--writer=ID\` for WriterProfileStore lookup (per-writer few-shot hooks already in the engine) - \`--format=md|json|txt\` with a stable JSON schema the skill consumes (pages[].text + alt_markers + confidence + provider_results) - Default \`--domain=general\` (was 'biology'); skill callers must opt in to \`--domain=bio\` explicitly. Auto-detection isn't worth the extra round-trip per S3-SPEC §2. - \`_extract_alt_markers\` regex helper so JSON consumers can iterate \`[?alt: a/b]\` ambiguities without reparsing prose - Single-provider mode now also returns the structured payload (with confidence/strategy_used as None) — uniform shape for the skill's --strict loop Single-page and multi-page (PDF) inputs both produce the same \`pages[]\` shape. Legacy txt format preserved for backward compat. Tests (7/7 pass): JSON schema, format flag round-trip, alt-marker extraction, single-provider null-fields, PDF page numbering, writer forwarding, domain default. Falsifiable criteria addressed: #1 (single source), #4 (format flags), #6 (PDF/HEIC unchanged at the CLI surface). #2 CER fixture, #3 live engine-bump observation, #5 strict UX run, #7 latency — out of scope for engine-side commit; need fixtures + API keys. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pre-existed S3; was untracked but not gitignored. Committing per standard practice for app/CLI repos so contributors get pinned dependency versions on uv sync. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Engine half of the S4 professor-feedback loop (engine-only wedge per S4-SPEC). Adds the write-back primitive the grader will call once the grader-side detection + GUI panel land in a follow-up. - Schema v5: corrections table (sample_id, ground_truth_id, original_text, confidence, source, created_at) + index. Additive, reversible (drop without breaking other functionality). - record_correction() helper: dedupes sample by image hash, dedupes GT by (sample_id, text), always appends a corrections row so identical double-calls grow history (1 sample + 1 GT + 2 corrections, per S4-SPEC criterion #1). - 5 new tests covering: v5 migration, first-write, idempotency under identical args, distinct-text -> new GT, per-writer accumulation (criterion #5), and FK linkage. No grader changes; cross-repo work deferred. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the engine plumbing for S2-SPEC: when a transcription target's writer_id
has >=2 stored ground-truth samples in the benchmark DB, the engine prepends
those (image, GT-label) pairs to the provider call as in-context exemplars.
S4's correction stream is what populates this DB in the lab-grader workflow,
so this is the natural follow-on now that S4 grader-side has landed.
Surface:
- writer_profile_store.Exemplar dataclass + select_exemplars(writer_id, k=,
exclude_sample_id=, conn=, db_path=). Deterministic ordering: contrast_score
DESC (when quality_assessments has a row) then sample_id ASC. SQL only,
no schema changes.
- few_shot.py module:
* EXEMPLAR_PROVIDERS = {"claude", "gemini"} -- TrOCR/PaddleOCR/OpenAI not
in-context-learning-capable, silently passthrough.
* env_few_shot_k() reads HE_FEW_SHOT_K (default 3, 0 disables).
* build_exemplar_blocks() interleaves image + labeled-text + ... + target.
Label template carries the SPEC's anti-cargo-cult guidance ("DIFFERENT
TEXT ... do not repeat the reference text").
* select_and_build_exemplar_blocks() = the single gate the integration
calls. Returns None for any disqualifying condition so callers fall back
to single-image read with one branch.
- vision.read_page() integration: 9 lines, calls the orchestrator before
the existing dual-polarity / read_image fork. Exemplar path uses each
provider's existing read_batch (which already accepts the interleaved
shape), so no provider modifications are needed -- "read_with_exemplars"
per S2-SPEC § 2 is functionally what read_batch already is.
Falsifiable criteria (S2-SPEC):
- #1 eligibility gate -- 10 select_exemplars unit tests
- #2 layout: exemplars before target, each with labeled GT -- 6 block-builder
tests
- #4 cold writer (<2 GT samples) -- orchestrator test confirms None
- #5 TrOCR passthrough -- allowlist + orchestrator tests, no error
- #6 HE_FEW_SHOT_K honored -- 5 env-helper tests + orchestrator coverage
Criterion #3 (CER on real data via Phase 8 Wilcoxon) requires the populated
IAM benchmark DB and the Phase 8 stats infrastructure; deferred to that eval
when the sweep run lands. The plumbing is ready.
Faint-ink dual-polarity wins over few-shot when both could fire -- they
target different failure modes and combining them inflates per-call image
tokens without verified gain.
Tests: 32 new, all passing. 7 existing writer_profile_store tests still
green. 66 tests in benchmark_db / handwriting / consensus also green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the observability -> action loop on confusion-pair errors: - consensus.resolve_char_level() resolves no-majority disagreements like ["modern", "rnodern"] when the only difference is a known confusion pair (rn-m, cl-d, 0-O, ...). Wired into _word_level_vote before the [?alt: ...] fallback. Per-writer overrides via writer_profile["confusion_resolutions"]. - postprocess.correct_confusion_pairs() runs after the existing edit- distance-1 wordlist pass: for each non-domain word, swap one confusion- pair occurrence; if exactly one swap lands in the wordlist, prefer it. Higher precision than generic ED1 (only known-pair shapes count). - HE_CONFUSION_POSTPROCESS env flag (default ON) for A/B. Falsifiable criteria: #1 char-level resolves rn-m PASS unit + integration test #2 defers cleanly when not a pair PASS unit + integration test #3 writer-specific bias overrides PASS unit + integration test #4 postprocess fixes celI -> cell PASS unit test #5 does not over-correct clean text PASS 9-sample smoke + unit test #6 IAM Wilcoxon CER win DEFER needs IAM ingest + Phase 8 (same gate as S2 #3) #7 no LabNoteBookGrader regression PASS 9-sample biology smoke + 146 postprocess/consensus tests #8 audit trail with pair label PASS unit test Pre-existing test failures (test_enhance, test_trained_correction.TestConfidenceGate) verified independent of S5 - same failures with HE_CONFUSION_POSTPROCESS=0. Runtime threading of writer_profile through vision.read_with_consensus -> consensus.read_with_consensus -> _vote -> _word_level_vote is intentionally deferred. Per-writer biasing is exercisable today via direct _word_level_vote calls; full runtime wiring is a follow-up that will land when S4 corrections data is populated and the bias has signal to act on. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the runtime-wiring follow-up flagged in c628e5b. Per-writer confusion-pair bias now fires in production paths, not just direct _word_level_vote calls. Wiring path: vision.read_with_consensus -> consensus.read_with_consensus(writer_profile=...) -> _vote(writer_profile=...) -> _word_level_vote(writer_profile=...) -> _smart_route(writer_profile=...) -> _vote(writer_profile=...) vision.read_with_consensus already loads writer_profile_dict via WriterProfileStore (line 606-609); it is now passed through to the consensus call instead of being used only for prompt construction. The vote/smart strategies carry the kwarg. debate, cascade, best_of, and self_correct don't run word-level voting and so don't need it. Tests: 3 new integration tests in test_char_consensus.py spy on _word_level_vote to confirm the kwarg arrives intact under vote and smart strategies, and arrives as None when no profile is supplied. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds paired Wilcoxon signed-rank, percentile bootstrap CI, and Cohen's r
to compare_runs() output. CER deltas between two benchmark runs are now
statistically defensible, not just printed numbers.
Module: handwriting_engine/benchmark/stats.py (new)
- wilcoxon_signed_rank(a, b): two-sided paired test, normal approximation
with continuity correction and tie correction in the variance term.
Returns dataclass {statistic, p_value, z, n}. Drops zero diffs (matches
scipy's zero_method='wilcox' default).
- bootstrap_ci(values, confidence, n_iterations, seed): percentile method,
seedable for deterministic output.
- cohens_r(z, n): r = |z| / sqrt(n), Cohen 1988 thresholds.
Hand-rolled to avoid pulling scipy/numpy as a dep — same pattern as the
existing inline Levenshtein in postprocess.py. ~150 LOC, no new wheels.
Wiring (handwriting_engine/benchmark/report.py):
- compare_runs() pairs per-sample CERs by sample_id intersection per
(provider, strategy). When paired n >= 10, appends a "stats:" line
(n, W, z, p, r) plus a "CI95:" line with both runs' bootstrap CIs.
- Below n=10 the block is omitted — normal-approx Wilcoxon is too rough
there and printing it would be more misleading than helpful.
Sample output for n=12, run_2 ~4 pp better than run_1:
gemini vote 12.17% 8.17% -4.00% IMPROVED
stats: n=12 W=78.0 z=+3.10 p=0.0019 r=0.90
CI95: run#1 [11.00%, 13.33%] run#2 [7.00%, 9.33%]
Tests: 32 new tests in tests/test_benchmark_stats.py
- 14 unit tests on the three stats primitives + helpers
- 15 paired/edge-case scenarios on wilcoxon_signed_rank
- 3 integration tests on compare_runs() with synthetic two-run DBs
(n>=10 stats block present; n<10 omitted; identical-runs renders n=0,
p=1.0)
Falsifiable criteria:
STAT-01: Wilcoxon p + Cohen's r appended for n>=10 — IMPLEMENTED, verified
on synthetic data. End-to-end IAM verification gated on user
IAM download (same gate as S2 #3, S5 #6, NEXT-STEPS.md).
STAT-02: 95% bootstrap CIs appended on both runs — IMPLEMENTED, verified
on synthetic data. Same IAM gate.
Doc updates:
- ROADMAP.md: stale Phase 6 plan checkboxes (06-02, 06-04) marked complete
per matching SUMMARY files. Phase 8 now lists 08-01-PLAN with status
"implementation shipped, verification gated."
- REQUIREMENTS.md: STAT-01/02 status: Implemented (verification gated).
- New: .planning/phases/08-statistics-layer/08-01-SUMMARY.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 9 / RPT-01. Schema v6 adds `is_baseline` to the runs table. detect_regressions now compares against the pinned baseline rather than the immediately-preceding run, so a single anchor governs the entire regression signal across releases. Schema: - v6 migration: ALTER TABLE runs ADD COLUMN is_baseline INTEGER DEFAULT 0. - Base _SCHEMA_SQL updated so fresh DBs include the column without needing to apply the migration. API (benchmark/db.py): - set_baseline(conn, run_id) — atomically clears the prior pin and sets the new one. Raises ValueError on unknown run_id. - get_baseline_run_id(conn) -> int | None. - list_runs() RunSummary now carries is_baseline. Behavior change (benchmark/report.py): - detect_regressions() reads the pinned baseline first; falls back to runs[1] (the prior pre-Phase-9 behavior) when no run is pinned, so fresh DBs and existing test fixtures keep working. - When the queried run IS the baseline, falls back to the prior run to avoid a no-op self-compare. CLI: - `benchmark set-baseline RUN_ID [--db-path PATH]` — pins a run. Errors are surfaced via ClickException, not stack traces. Tests: 12 new in tests/test_benchmark_baseline.py - v6 column presence + default = 0 - set/get round-trip + at-most-one invariant + durable across reopen - list_runs surfaces is_baseline - detect_regressions: no-baseline fallback, pinned-vs-fallback delta, self-compare guard - CLI happy-path + unknown-run error path Pre-existing 34 benchmark/evaluate tests pass unchanged. Falsifiable criterion (RPT-01): > Schema v4 adds is_baseline flag to runs table; benchmark set-baseline > RUN_ID pins a run as the regression anchor; detect_regressions() > compares against the pinned baseline, not runs[-2]. Status: IMPLEMENTED. (REQUIREMENTS.md says "v4" but the live schema is v6 — v6 is the migration that adds is_baseline; the requirement text is out-of-date and is being fixed in a later commit.) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 9 / RPT-02. New `benchmark recommend` CLI surfaces a single
ranked recommendation across all (provider, strategy) configurations
seen in the benchmark DB.
Score = 0.70 * (1 - cer_norm) + 0.15 * (1 - cost_norm) + 0.15 * stab_norm
Each component is min-max normalized within the candidate set:
- CER: lower is better. mean across all per-sample CERs across all runs.
- Cost: lower is better. mean estimated $/sample across runs.
- Stability: higher is better. 1 - normalized(stdev across run-mean CERs).
Single-run candidates can't measure across-run variance, so they get
the median stability score (neutral) and the output annotates `n=1`.
This avoids penalizing or rewarding configurations the user just
hasn't run twice yet.
API (benchmark/report.py):
- recommend_strategy(db_path) -> str — formatted ranked table.
- _RECOMMEND_W_CER / _W_COST / _W_STAB constants — change once if the
product weights ever shift.
CLI:
- `benchmark recommend [--db-path PATH]` — prints the ranked table.
Sample output (3 candidates, gemini wins on both CER and cost):
Strategy + provider recommendation
weights: CER 70% cost 15% stability 15%
Winner: gemini + vote (score 0.925)
Rank Provider Strategy CER $/sample stdev n score
-------------------------------------------------------------------------
1 gemini vote 5.50% 0.0001$ 0.007 2 0.925
2 claude vote 10.50% 0.0210$ 0.007 2 0.407
3 openai vote 15.00% 0.0094$ n=1 1 0.158
Tests: 9 in tests/test_benchmark_recommend.py
- empty DB graceful message
- lower CER wins when cost equal
- lower cost wins when CER equal (single-run, neutral stability)
- more stable wins when CER + cost equal (across-run variance)
- single-run flagged n=1 in output
- score-ordered output (rank 1 = winner)
- winner annotation matches rank-1 row
- CLI happy-path + empty-DB exit-0 path
Also: bump CURRENT_SCHEMA_VERSION 5 -> 6 so fresh DBs no longer log a
spurious "Migration to v6 failed (may already be applied)" warning when
they create the is_baseline column directly via _SCHEMA_SQL. The
migration still runs cleanly on existing v5 DBs.
Falsifiable criterion (RPT-02):
> benchmark recommend outputs the best strategy+provider configuration
> with a weighted composite score (70% CER / 15% cost / 15% stability
> across runs).
Status: IMPLEMENTED.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 9 / RPT-03. Captures ground-truth transcriptions for student lab
notebook images so the engine can be benchmarked against production-
distribution data, not just the IAM corpus.
API (benchmark/ingest.py):
- ingest_lab(directory, *, student, prompt_fn, db_path,
use_vlm_suggestion=False, vlm_provider="gemini") -> dict
- prompt_fn(image_path, suggestion) -> str | None drives interactive
capture. Returning None or whitespace-only text marks the image as
skipped — the sample row stays in the DB so it can be revisited later.
- Re-running on the same directory skips images that already have a
ground_truth row (safe to resume mid-stack).
- New samples are tagged category='lab'; ground_truth source='lab-grader'
with the --student tag stored as author for provenance.
- Optional --with-suggestion runs a single-provider best_of read per
image and pre-fills the editor; VLM failures degrade gracefully to
empty suggestions rather than aborting the workflow.
CLI:
- `benchmark ingest-lab DIRECTORY [--student S] [--with-suggestion]
[--vlm-provider P]`
- prompt_fn is wired to click.edit() — opens $EDITOR with the suggestion
pre-loaded, blank cancel = skip, save = annotate.
- Final summary line reports counts: annotated / new samples / already
had ground truth / skipped / errors.
Tests: 10 in tests/test_benchmark_ingest_lab.py
- prompt_fn required (raises ValueError otherwise)
- inserts samples + GT with correct category, source, author
- user-skip + whitespace-only treated identically (sample, no GT)
- resume: second pass only prompts for images without GT
- VLM suggestion threaded to prompt_fn
- VLM failure → empty suggestion, no crash
- non-directory raises FileNotFoundError
- CLI happy-path via patched click.edit
- CLI skip path via patched click.edit returning None
Falsifiable criterion (RPT-03):
> Developer can collect and store ground-truth transcriptions from real
> student lab notebooks using benchmark ingest-lab with a guided
> annotation workflow, enabling production-distribution benchmarks
> distinct from IAM.
Status: IMPLEMENTED.
Total Phase 9 status:
RPT-01 set-baseline IMPLEMENTED (c628e5b's RPT slot)
RPT-02 recommend IMPLEMENTED (d7e6ae8)
RPT-03 ingest-lab IMPLEMENTED (this commit)
End-to-end criterion verification (recommend on a multi-strategy IAM
sweep) still gates on user IAM download — same gate as S2 #3, S5 #6,
STAT-01/02. Code is ready; data is the missing piece.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…aries ROADMAP.md: Phase 9 plans now listed (3 plans, all completed 2026-05-06). The "blocked on user IAM download" caveat narrows from "Phase 9" to "end-to-end verification of recommend on a multi-strategy IAM sweep" — that's the only piece still gated. REQUIREMENTS.md: RPT-01/02/03 marked checked with shipped-date and the RPT-01 text fixed (was "Schema v4", actually shipped as v6). New: .planning/phases/09-final-sweep-recommendation-baseline-lock/ - 09-01-SUMMARY.md (RPT-01 set-baseline) - 09-02-SUMMARY.md (RPT-02 recommend) - 09-03-SUMMARY.md (RPT-03 ingest-lab) Each follows the project SUMMARY convention (requires/provides/decisions/ verification) so future sessions can find the plan-level context without spelunking through git log. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously raised RuntimeError, but callers (postprocess.correct) only catch ImportError when probing the trained pass — a fresh clone without the [trained-correction] extras would surface as an uncaught RuntimeError instead of falling back gracefully to the heuristic output. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
feat: trained post-correction model + safeguards + Phase 7 plumbing
… can prove it
27 files conflicted, and every conflicting hunk said the same thing — this branch is the
newer side. The evidence, not a preference:
* `db.py` schema v6 vs v4 (this branch carries migrations 5 and 6), `models.py`
`is_baseline`, `evaluate.py`'s `line_level`/`auto_retry` threading, `cli.py`'s
`ingest-lab` command: in each case main's side of the hunk is EMPTY. It never had them.
* `.planning/STATE.md`: this branch says 2026-05-06, phase 7 complete; main says
2026-04-13, "ready to plan Phase 6".
* `tests/test_benchmark_evaluate.py`: main marked the eight IAM-02/IAM-03 stubs xfail
yesterday so its suite could go green. Here they are implemented and passing, so the
markers are simply not needed.
So the merge takes this branch on conflicts (`-X ours`, hunk by hunk — main's
NON-conflicting additions are all kept) with two deliberate exceptions where main is the
authority, both about paths rather than features:
* `.mcp.json` — main's `${HOME}` version; Claude Code does not expand `~`.
* `.planning/research/FEATURES.md` — main's line points at `~/Developer/...`; this
branch's still named the iCloud tree retired in August.
`uv.lock` was regenerated rather than resolved (`uv lock --upgrade`), which is also what
carries this branch's torch/CUDA extra forward on the patched versions main landed.
Verified, not assumed: no test was lost to the conflict resolution — every `def test_*`
name present on main is present here, and each conflicted test file has at least as many
tests as either parent (evaluate 34/34→34, postprocess 47/55→55, benchmark_db 18/24→24).
Suite on the merged tree: **704 passed, 3 skipped, 1 xfailed, 0 failed**.
What main brings the other way: numpy + opencv-python-headless declared (line_reader
imported them unguarded), a tracked uv.lock, the per-module import test, CI on push/PR,
grouped Dependabot updates for both branches, and the `.leakgate-allow` that adjudicates
the `/Users/user` placeholder these planning docs use.
CI caught this in ten seconds and it deserved to: `uv sync --locked` failed on both Python versions with "the lockfile needs to be updated", while the same command passed on this machine. Both were true. `uv lock --upgrade` had rewritten `uv.lock` in the WORKING TREE after the merge conflicts were resolved, so it was never staged — `git commit` publishes the index, and the index still held the branch's pre-merge lock. Locally `--locked` compared `pyproject.toml` against the good lock on disk; CI compared it against the one actually committed, which no longer matched the merged manifest. The artifact you verify has to be the artifact you publish. Locally that distinction is invisible, because the disk is both.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Ryzen's 53 phase-9 commits, published so they can finally be tested.
This branch is that machine's
mainas it stood: phase-9 benchmark work (pinned baselinesand regression detection, composite recommend scoring, ingest-lab guided annotation), a
trained_correctionfix that raises ImportError when the optional deps are missing, and themerge of
feat/trained-corrector. It had never run against CI, because until today this repohad none.
Published through the pre-push leak gate installed on that machine today, which scanned all
53 commits and reported clean. It initially blocked with 57 findings, every one the string
/Users/user— the planning docs' generic placeholder, adjudicated narrowly in.leakgate-allowon main (a real/Users/<name>still blocks).CI has not run, and cannot yet
A
pushevent runs the workflows that exist on the branch being pushed, and this branchpredates
.github/workflows/tests.yml. Apull_requestevent would use the base+head merge— except this PR is CONFLICTING, so there is no merge ref to build and no check can start.
The suite runs the moment the conflicts below are resolved; not before.
What conflicts (27 files)
.planning/— both lineages advanced their own narrative. Mechanical.uv.lock— regenerate withuv lock --upgradeafter resolvingpyproject.toml..mcp.json— main's version is the portable${HOME}one.benchmark/db.py,benchmark/evaluate.py,benchmark/models.py,cli.py, and the teststest_benchmark_db.py,test_benchmark_evaluate.py,test_postprocess.py.One thing worth knowing before resolving:
tests/test_benchmark_evaluate.pyhas 8pytest.fail("not implemented")stubs onmain(marked xfail today so the suite could gogreen), and this branch implements exactly those — IAM-02 sweep infrastructure and IAM-03
per-writer report, phases 07-03 and 07-04.
feat/trained-correctorimplements them too. Threelineages, the same features, which is why these merges keep costing this much.
What
mainbrings the other way: declarednumpy/opencv-python-headless, a trackeduv.lock, the per-module import test, CI, and grouped Dependabot updates.