diff --git a/.planning/NEXT-STEPS.md b/.planning/NEXT-STEPS.md new file mode 100644 index 0000000..8abb258 --- /dev/null +++ b/.planning/NEXT-STEPS.md @@ -0,0 +1,122 @@ +# NEXT-STEPS — What to do after Phase 7 + +**Status:** Phase 7 (IAM Data Ingestion + Sweep Infrastructure) shipped 2026-05-06. +The infrastructure is ready. The remaining unblockers are **manual data acquisition** and **one sweep run**, after which Phases 8-9 + the trained-corrector real-data retrain are all unblocked. + +--- + +## Why this exists + +Phase 7 ended with all 17 RED stubs GREEN, all 4 plans landed, and `benchmark sweep` / `benchmark report --per-writer` callable from the CLI. But two downstream goals remain blocked on **user-side work that can't be automated:** + +1. **Phase 8 (Statistics Layer)** needs ≥10-sample multi-strategy runs in the benchmark DB. That requires running the sweep against real IAM data. +2. **Trained-corrector real-data retrain** needs (vlm_text, ground_truth) pairs from real VLM runs. Today the corrector is gated OFF (`HE_USE_TRAINED_CORRECTOR=0`) because synthetic-only training causes hallucinations on hard cases — see `handwriting_engine/trained_correction/EVAL-RESULTS.md`. The sweep produces exactly those pairs. + +Both unblock from a single sweep run. + +--- + +## Step 1 — Download the IAM Handwriting Database + +The IAM dataset is registration-gated, so this is manual. + +1. Register and download from one of: + - [HEIA-FR mirror](https://fki.tic.heia-fr.ch/databases/iam-handwriting-database) + - Original Univ. Bern site (legacy) +2. Extract `lines.tgz` (line-level images) and `ascii.tgz` (transcriptions). Layout expected: + ``` + / + ├── ascii/ + │ └── lines.txt + └── lines/ + └── /
/-.png + ``` +3. **Optional but recommended:** also grab the `largeWriterIndependentTextLineRecognitionTask/` partition files. They split IAM into `trainset.txt`, `validationset1.txt`, `testset.txt`. The sweep should run against `testset.txt` only — never on training data, or the baseline isn't a true generalization measure. + +--- + +## Step 2 — Ingest IAM into the benchmark DB + +```bash +cd "/Users/user/Documents/Work & Projects/VSCode Projects/handwriting-engine" + +# Test partition only (safer, ~2k samples): +python3 -m handwriting_engine.cli benchmark ingest-iam \ + --ascii-dir /ascii \ + --lines-dir /lines \ + --partition-file /largeWriterIndependentTextLineRecognitionTask/testset.txt + +# Verify ingestion: +python3 -m handwriting_engine.cli benchmark list --show-samples | head -20 +``` + +Expected: rows with `category='iam'`, `student='iam-writer-XXX'`. The CLI prints `{ingested, skipped_dup, skipped_missing}` counts. + +> **Cost note:** The next step runs ALL FIVE strategies against every ingested sample. If you ingest the full test set (~2k lines), one sweep can run ~$3-5 in API calls (Gemini Flash is the cheap default). If unsure, ingest ~50 samples first using a partition subset, do a smoke sweep, then scale up. + +--- + +## Step 3 — Run the multi-strategy sweep + +```bash +python3 -m handwriting_engine.cli benchmark sweep --provider gemini +# Confirms cost projection, then executes all 5 strategies: +# baseline, self_correct, line_level, prompt_adapted, zoomed_verify +# Returns one run_id per strategy. +``` + +Add `--yes` to skip the confirmation prompt (useful in CI / headless runs). + +--- + +## Step 4 — Inspect per-writer breakdown + +```bash +# Replace with one of the run_ids the sweep printed: +python3 -m handwriting_engine.cli benchmark report --run-id --per-writer +``` + +This is the IAM-03 deliverable: shows whether a strategy's CER gain is consistent across writers or driven by a few easy ones. + +--- + +## Step 5 — Retrain the trained corrector on real data + +Now the (vlm_output, ground_truth) pairs from the sweep can fine-tune the FLAN-T5 corrector that's currently gated off: + +```bash +# Continue from the v1 synthetic checkpoint (don't start from scratch — preserves +# the easy-error fixes the synthetic data already taught it): +python3 -m handwriting_engine.trained_correction.train \ + --from-benchmark-db ~/.handwriting-engine/benchmark.db \ + --continue-from ~/.handwriting-engine/models/trained-corrector-v1 \ + --output-dir ~/.handwriting-engine/models/trained-corrector-v2 \ + --epochs 3 + +# A/B eval: v2 vs v1 vs heuristic-only +python3 -m handwriting_engine.cli trained-correction eval --n-pairs 200 --seed 9999 +``` + +Expected: hallucinations on hard cases (the 3/10 spot-check failures documented in `trained_correction/EVAL-RESULTS.md`) drop substantially — because the model now sees the actual VLM error distribution rather than a guessed-at synthetic one. Once the gated A/B passes, flip `HE_USE_TRAINED_CORRECTOR=1` to default-on the combined heuristic→trained pipeline. + +--- + +## Step 6 — Plan Phase 8 + +Once the sweep run lives in the DB, Phase 8 (Statistics Layer) is unblocked: +- Wilcoxon signed-rank p-values on `benchmark compare` +- 95% bootstrap CIs on CER estimates +- Cohen's r effect size + +Run `/gsd:plan-phase 8` from inside the engine directory when ready. + +--- + +## Out-of-band side projects flagged in the broader strategy + +These do not block Phase 8 but are worth queuing for follow-up sessions: + +- **S2 — Per-writer few-shot exemplars.** Currently `writer_profile_store.build_calibration_block()` injects writer-specific text hints into prompts. Stronger: pull 2-3 already-labeled images of the same writer from the benchmark DB and pass them as multi-image prompts (Gemini and Claude both support it). Likely the biggest single CER gain on returning writers (lab notebook semester scenarios). **Spec drafted: `.planning/S2-SPEC-per-writer-few-shot.md` (2026-05-06).** +- **S3 — Wire `~/.claude/skills/handwriting-reader/` skill to call the engine library directly.** The skill currently does its own multi-pass workflow. One source of truth = engine improvements propagate immediately. **Spec drafted: `.planning/S3-SPEC-skill-engine-bridge.md` (2026-05-06).** +- **S4 — Professor OS feedback loop.** `professor/LabNoteBookGrader/` graders should surface low-confidence reads, capture corrections, and write them back to the benchmark DB as per-writer ground truth. Per-writer accuracy then compounds over a semester. **Spec drafted: `.planning/S4-SPEC-professor-feedback-loop.md` (2026-05-06).** +- **S5 — Char-level consensus + confusion-pair-aware postprocess.** Word-level voting catches obvious disagreements; char-level catches single-character swaps (`rn↔m`, `cl↔d`). The drill-down report already tracks confusion pairs; postprocess can consume them. **Spec drafted: `.planning/S5-SPEC-char-consensus-confusion-postprocess.md` (2026-05-06).** diff --git a/.planning/REQUIREMENTS.md b/.planning/REQUIREMENTS.md index b74c877..b07c943 100644 --- a/.planning/REQUIREMENTS.md +++ b/.planning/REQUIREMENTS.md @@ -27,9 +27,9 @@ Requirements for the benchmarking milestone. Each maps to a roadmap phase. ### Reporting -- [ ] **RPT-01**: Schema v4 adds `is_baseline` flag to runs table; `benchmark set-baseline RUN_ID` pins a run as the regression anchor; `detect_regressions()` compares against the pinned baseline, not runs[-2]. -- [ ] **RPT-02**: `benchmark recommend` outputs the best strategy+provider configuration with a weighted composite score (70% CER / 15% cost / 15% stability across runs). -- [ ] **RPT-03**: Developer can collect and store ground-truth transcriptions from real student lab notebooks using `benchmark ingest-lab` with a guided annotation workflow, enabling production-distribution benchmarks distinct from IAM. +- [x] **RPT-01**: Schema v6 adds `is_baseline` flag to runs table; `benchmark set-baseline RUN_ID` pins a run as the regression anchor; `detect_regressions()` compares against the pinned baseline, not runs[-2]. (shipped 2026-05-06) +- [x] **RPT-02**: `benchmark recommend` outputs the best strategy+provider configuration with a weighted composite score (70% CER / 15% cost / 15% stability across runs). (shipped 2026-05-06; end-to-end on multi-strategy sweep gated on IAM data) +- [x] **RPT-03**: Developer can collect and store ground-truth transcriptions from real student lab notebooks using `benchmark ingest-lab` with a guided annotation workflow, enabling production-distribution benchmarks distinct from IAM. (shipped 2026-05-06) ## v4.0 Requirements (Deferred) @@ -63,11 +63,11 @@ Requirements for the benchmarking milestone. Each maps to a roadmap phase. | IAM-01 | Phase 7 | Complete | | IAM-02 | Phase 7 | Pending | | IAM-03 | Phase 7 | Pending | -| STAT-01 | Phase 8 | Pending | -| STAT-02 | Phase 8 | Pending | -| RPT-01 | Phase 9 | Pending | -| RPT-02 | Phase 9 | Pending | -| RPT-03 | Phase 9 | Pending | +| STAT-01 | Phase 8 | Implemented (verification gated on IAM data) | +| STAT-02 | Phase 8 | Implemented (verification gated on IAM data) | +| RPT-01 | Phase 9 | Implemented | +| RPT-02 | Phase 9 | Implemented (verification gated on IAM sweep) | +| RPT-03 | Phase 9 | Implemented | **Coverage:** - v3.0 requirements: 12 total diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index a1f455f..1e79a50 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -23,9 +23,9 @@ Full details: `.planning/milestones/v2.0-ROADMAP.md` ### v3.0 — Verified Accuracy - [x] **Phase 6: Measurement Foundation** — Reproducible baseline + variance floor + cost guardrails (completed 2026-04-11) -- [ ] **Phase 7: IAM Data Ingestion + Sweep Infrastructure** — Full IAM benchmark pipeline -- [ ] **Phase 8: Statistics Layer** — Statistical defensibility for all comparisons -- [ ] **Phase 9: Final Sweep, Recommendation, and Baseline Lock** — Best config identified, regression anchor committed +- [x] **Phase 7: IAM Data Ingestion + Sweep Infrastructure** — Full IAM benchmark pipeline (completed 2026-05-06) +- [ ] **Phase 8: Statistics Layer** — Statistical defensibility for all comparisons (implementation shipped 2026-05-06; criterion verification gated on user IAM download + first sweep run; see `.planning/NEXT-STEPS.md`) +- [ ] **Phase 9: Final Sweep, Recommendation, and Baseline Lock** — Best config identified, regression anchor committed (implementation shipped 2026-05-06; verification of "best config recommendation" gated on user IAM download + first multi-strategy sweep) ## Phase Details @@ -42,9 +42,9 @@ Full details: `.planning/milestones/v2.0-ROADMAP.md` Plans: - [x] 06-01-PLAN.md — Wave 0 test stubs: failing tests for all Phase 6 behaviors (FOUND-01 through FOUND-04) (completed 2026-04-11) -- [ ] 06-02-PLAN.md — v4 schema migration + dataclass extensions (db.py, models.py) -- [x] 06-03-PLAN.md — Provenance capture + marker rate computation + report display (evaluate.py, report.py) -- [ ] 06-04-PLAN.md — CLI surface: benchmark calibrate subcommand + cost guardrail + provenance flags (cli.py) +- [x] 06-02-PLAN.md — v4 schema migration + dataclass extensions (db.py, models.py) (completed 2026-04-11) +- [x] 06-03-PLAN.md — Provenance capture + marker rate computation + report display (evaluate.py, report.py) (completed 2026-04-11) +- [x] 06-04-PLAN.md — CLI surface: benchmark calibrate subcommand + cost guardrail + provenance flags (cli.py) (completed 2026-04-11) ### Phase 7: IAM Data Ingestion + Sweep Infrastructure **Goal**: The developer can load the IAM Handwriting Database into the benchmark system and execute a full multi-strategy sweep against it, with per-writer variance visible in reports. @@ -57,10 +57,10 @@ Plans: **Plans**: 4 plans Plans: -- [ ] 07-01-PLAN.md — Wave 0 RED test stubs: 16 failing tests across TestIAMIngest, TestSweep, TestPerWriterReport (IAM-01, IAM-02, IAM-03) -- [ ] 07-02-PLAN.md — IAM ingest infrastructure: parse_iam_lines(), ingest_iam(), benchmark ingest-iam CLI (IAM-01) -- [ ] 07-03-PLAN.md — Sweep infrastructure: line_level/auto_retry threading, run_sweep(), benchmark sweep CLI (IAM-02) -- [ ] 07-04-PLAN.md — Per-writer report: generate_per_writer_report(), benchmark report --per-writer flag (IAM-03) +- [x] 07-01-PLAN.md — Wave 0 RED test stubs: 17 failing tests across TestIAMIngest, TestSweep, TestPerWriterReport (IAM-01, IAM-02, IAM-03) (completed 2026-04-11) +- [x] 07-02-PLAN.md — IAM ingest infrastructure: parse_iam_lines(), ingest_iam(), benchmark ingest-iam CLI (IAM-01) (completed 2026-04-12) +- [x] 07-03-PLAN.md — Sweep infrastructure: line_level/auto_retry threading, run_sweep(), benchmark sweep CLI (IAM-02) (completed 2026-05-06) +- [x] 07-04-PLAN.md — Per-writer report: generate_per_writer_report(), benchmark report --per-writer flag (IAM-03) (completed 2026-05-06) ### Phase 8: Statistics Layer **Goal**: CER comparisons between strategies are statistically defensible — not just raw delta numbers — so the developer can assert with confidence that a measured improvement is real. @@ -69,7 +69,10 @@ Plans: **Success Criteria** (what must be TRUE when this phase completes): 1. Running `benchmark compare RUN_A RUN_B` on any two runs with n >= 10 samples automatically appends a Wilcoxon signed-rank p-value and Cohen's r effect size to the output, with no extra flags needed. 2. The same `benchmark compare` output includes 95% bootstrap confidence intervals on both CER estimates, so the developer can see whether the CI bands overlap and judge whether the difference is distinguishable from sampling noise. -**Plans**: TBD +**Plans**: 1 plan + +Plans: +- [x] 08-01-PLAN.md — Stats module + compare_runs wire-up: paired Wilcoxon, percentile bootstrap CI, Cohen's r (no scipy dep) (completed 2026-05-06; criterion verification gated on IAM data) ### Phase 9: Final Sweep, Recommendation, and Baseline Lock **Goal**: The developer knows which strategy+provider configuration is best for lab notebook grading, and a regression baseline is pinned so any future code change that silently degrades accuracy is immediately detectable. @@ -79,7 +82,12 @@ Plans: 1. Developer runs `benchmark set-baseline RUN_ID` to pin any run as the regression anchor; `detect_regressions()` then compares future runs against that pinned run (not the penultimate run), and the schema tracks the `is_baseline` flag durably across sessions. 2. `benchmark recommend` outputs a single ranked recommendation with a composite score (70% CER / 15% cost / 15% stability) and the winning strategy+provider combination is unambiguous. 3. Developer can run `benchmark ingest-lab` against real student lab notebook images and store ground-truth transcriptions via a guided annotation workflow, producing a production-distribution test set that is separate from IAM. -**Plans**: TBD +**Plans**: 3 plans + +Plans: +- [x] 09-01-PLAN.md — Schema v6 is_baseline + set_baseline/get_baseline + detect_regressions retarget + CLI (RPT-01) (completed 2026-05-06) +- [x] 09-02-PLAN.md — recommend_strategy() composite 70/15/15 score + CLI (RPT-02) (completed 2026-05-06; verification gated on multi-strategy IAM sweep) +- [x] 09-03-PLAN.md — ingest_lab() guided annotation + CLI (RPT-03) (completed 2026-05-06) ## Progress @@ -91,6 +99,6 @@ Plans: | 4. Preprocessing + Writer Adaptation | v2.0 | 1/1 | ✅ Complete | 2026-04-09 | | 5. Post-Processing + Benchmark Suite | v2.0 | 1/1 | ✅ Complete | 2026-04-09 | | 6. Measurement Foundation | 4/4 | Complete | 2026-04-11 | - | -| 7. IAM Data Ingestion + Sweep Infrastructure | 1/4 | In Progress| | - | -| 8. Statistics Layer | v3.0 | 0/? | Not started | - | +| 7. IAM Data Ingestion + Sweep Infrastructure | v3.0 | 4/4 | ✅ Complete | 2026-05-06 | +| 8. Statistics Layer | v3.0 | 0/? | Blocked on user IAM download + first sweep | - | | 9. Final Sweep, Recommendation, and Baseline Lock | v3.0 | 0/? | Not started | - | diff --git a/.planning/S2-SPEC-per-writer-few-shot.md b/.planning/S2-SPEC-per-writer-few-shot.md new file mode 100644 index 0000000..2aa603b --- /dev/null +++ b/.planning/S2-SPEC-per-writer-few-shot.md @@ -0,0 +1,116 @@ +# S2 — Per-Writer Few-Shot Exemplars + +**Status:** SPEC (not yet a phase). Implementation blocked on benchmark DB population (same blocker as Phase 8). +**Authored:** 2026-05-06 +**Source:** `.planning/NEXT-STEPS.md` § Out-of-band side projects. + +--- + +## Goal + +When transcribing an image whose `writer_id` has ≥2 already-labeled samples in the benchmark DB, the engine prepends 2–3 of those (image, ground-truth) pairs to the target prompt as multi-image exemplars. On the same writer's held-out images, this reduces CER by a margin distinguishable from noise (per Phase 8 stats), without regressing CER on writers who have <2 stored samples. + +## Why this is high-leverage + +- Lab-notebook grading is the dominant downstream consumer (`professor/LabNoteBookGrader/`). A semester has the *same* student writing weekly — the returning-writer scenario is the steady-state, not the edge case. +- Today's adaptation is a **text** calibration block (`writer_profile_store.build_calibration_block()` at [writer_profile_store.py:63](../handwriting_engine/writer_profile_store.py#L63)) that lists discrete observations ("crosses 7s: YES"). That's brittle — it depends on a human entering observations. +- Few-shot exemplars are the standard VLM technique for in-context style adaptation. Both Gemini and Claude support multi-image content lists today (Claude via [`providers/claude.py:56-65`](../handwriting_engine/providers/claude.py#L56-L65) `read_batch`; Gemini's SDK accepts `contents=[image_part_1, ..., image_part_N, prompt]`). + +## Non-goals + +- Training a model. This is purely in-context. +- Cross-writer transfer. Exemplars are sourced strictly within the same `writer_id`. +- Replacing the text calibration block. The text block stays as a fallback for writers with <2 GT samples and is composable with exemplars when both exist. + +--- + +## Design + +### 1. Exemplar selection (`writer_profile_store.py`) + +Add `select_exemplars(writer_id, *, k=3, exclude_sample_id=None) -> list[Exemplar]`: + +```python +@dataclass(frozen=True) +class Exemplar: + sample_id: int + image_path: str # absolute, on local disk + ground_truth: str # the canonical transcription +``` + +Selection strategy v0 (cheapest defensible default — refine in v1 if benchmarks justify): + +1. Pull all `(samples × ground_truths)` rows for `student = writer_id`, excluding `exclude_sample_id` (the target itself if it happens to have a GT). +2. Order by `quality_assessments.score DESC` if quality scores exist for that writer; else by `sample_id ASC` (stable, deterministic). +3. Return the first `min(k, available)` — caller decides how many to actually inject. + +Determinism matters: same target image + same DB state ⇒ same exemplars selected. Don't randomize without a seed. + +### 2. Provider-side multi-image plumbing + +- **Claude** ([`providers/claude.py`](../handwriting_engine/providers/claude.py)) — already supports it via `read_batch`. Add a thin `read_with_exemplars(target_image_b64, exemplar_blocks, prompt, ...)` wrapper that constructs `[exemplar_1_image, exemplar_1_label_text, exemplar_2_image, exemplar_2_label_text, ..., target_image, target_prompt]`. Exemplar label text wraps the GT clearly: `"The handwriting in the previous image transcribes to: «{gt}»"`. +- **Gemini** ([`providers/gemini.py:143`](../handwriting_engine/providers/gemini.py#L143)) — currently `contents=[image_part, prompt]`. Extend to accept a list. Same labeled-exemplar interleaving. +- **TrOCR** — out of scope. It's a fixed-vocab encoder, no in-context learning. Skip silently if exemplars are passed. + +### 3. Prompt-construction integration (`handwriting.py`) + +`get_reading_strategies()` is the wrong layer (it returns text only). Few-shot injection happens one level up, where the provider call is assembled. Touchpoints: + +- The transcription entrypoint(s) that already accept a `writer_profile` argument. When they detect `writer_id` and the DB has ≥2 GT samples for that writer, they call `select_exemplars()` and pass the result into the provider via the new `read_with_exemplars` path. +- The text calibration block is **still injected** alongside, in the prompt text — exemplars and text observations compose, they don't substitute. Reasoning: text observations encode binary facts ("crosses 7s") that exemplars may not visually demonstrate in a 3-image sample. + +### 4. Cost & opt-out + +- Adding 3 images to a Gemini Flash call ~quadruples that call's image-token cost. Document this in the docstring and surface a `HE_FEW_SHOT_K` env var (default 3, 0 = disabled). +- For batch / sweep contexts, default `k=2` if more than 50 samples are queued for the same writer in one batch — Claude's prompt cache amortizes the exemplar tokens across the batch but Gemini doesn't cache identically. + +--- + +## Falsifiable success criteria + +When this phase is complete, all of the following are TRUE: + +1. **Eligibility gate works.** `select_exemplars("writer-with-1-sample")` returns `[]`. `select_exemplars("writer-with-5-samples", k=3)` returns 3 deterministic Exemplar rows. Verified by unit tests against a fixture DB. +2. **Provider calls carry exemplars end-to-end.** A live transcription against a writer with ≥3 GT samples produces a request payload that the provider mock asserts contains the exemplar images **before** the target image, in order, each followed by its labeled GT text. Verified for both Claude and Gemini via recorded HTTP/SDK fixtures. +3. **CER improves on a held-out per-writer split.** On the IAM test set restricted to writers with ≥4 GT samples (sourcing 3 exemplars + 1+ held-out target per writer), running the `prompt_adapted` strategy with exemplars enabled vs. disabled produces a CER delta where the Phase 8 Wilcoxon test reports `p < 0.05` and the bootstrap CIs do not overlap. +4. **No regression on cold writers.** On writers with <2 GT samples, with `HE_FEW_SHOT_K=3`, the codepath falls back cleanly to text-only calibration and CER is statistically indistinguishable from the pre-S2 baseline (Wilcoxon `p > 0.10`). +5. **TrOCR passthrough.** Calling the transcription entrypoint with `writer_id` set but `provider=trocr` does not error and does not include exemplars in the call. Logged at DEBUG, not WARNING. +6. **Cost guardrail surfaces it.** Existing sweep cost projection (Phase 6 / IAM-02) accounts for the exemplar image tokens when `HE_FEW_SHOT_K > 0`. + +## Out of scope (queued for follow-up) + +- **Smart exemplar selection** — diversity-by-character-coverage, hardness-aware ("show the model the writer's *messy* samples"). v0 is recency/quality-ordered. Revisit only if v0 ships and CER gain is below the IAM-test-set theoretical ceiling estimated below. +- **Dynamic `k`** — adapting `k` based on target image difficulty. Out of scope until v0 establishes the baseline. +- **Cross-session exemplar caching** — provider-side prompt cache hits for repeated exemplar sets across calls. Worth doing but separable; track as S2.1. + +--- + +## Risks & open questions + +| Risk | Mitigation / decision needed | +|------|------------------------------| +| Benchmark DB is empty today (`~/.handwriting-engine/benchmark.db` does not exist). | Block implementation start until IAM ingest+sweep run completes (Phase 7 prerequisite). Spec stays valid in the meantime. | +| Exemplar GT may itself be wrong (human transcription error in IAM is non-zero). | Already mitigated by `quality_assessments` table — gate exemplars to score ≥ threshold once that's populated. v0: trust GT. | +| 3 high-res IAM line images in one Gemini call may exceed Flash input-token budget on long lines. | Concrete number needed. Pre-implementation: measure max-image-token-count in IAM lines/ to confirm headroom. If tight, downsample exemplars to 1024px width before encoding. | +| Returning-writer few-shot may bias the model into copying the *previous transcription style* rather than reading the *new image*. (Cargo-culting from in-context examples is a known VLM failure mode.) | Address in the prompt: "The reference samples are from the same writer but contain DIFFERENT TEXT. Read what is in the final image — do not repeat the reference text." Verify in success criterion #4 — if cold-writer regression is significant, this is the suspect. | +| Phase 8 stats infrastructure must exist before criterion #3 is testable. | This phase depends on Phase 8. Order: Phase 8 → S2. Specifying it now is fine; implementation order is enforced by the criterion. | + +## Touchpoints (preliminary) + +- **New:** [`writer_profile_store.py`](../handwriting_engine/writer_profile_store.py) — `select_exemplars()`, `Exemplar` dataclass. +- **Edit:** [`providers/claude.py`](../handwriting_engine/providers/claude.py) — add `read_with_exemplars()` wrapper. +- **Edit:** [`providers/gemini.py`](../handwriting_engine/providers/gemini.py) — extend `contents` list construction for multi-image. +- **Edit:** [`handwriting.py`](../handwriting_engine/handwriting.py) — wire the writer_id branch into the new provider path. +- **New:** tests under `tests/` mirroring the success-criteria numbering (1–6). +- **Doc:** README provider section + `HE_FEW_SHOT_K` env var. + +## Estimated scope + +- ~2 plans (provider plumbing + integration & tests). ~250–400 LOC engine-side, ~300 LOC tests. +- Falsification requires Phase 8 + a populated IAM DB; budget one sweep run (~$3–5) for criterion #3. + +--- + +## Promotion path + +When ready to start: `/gsd-add-phase` (or `/gsd-insert-phase` to slot between 8 and 9). This SPEC.md graduates into the new phase's directory as the seed for `/gsd-discuss-phase`. diff --git a/.planning/S3-SPEC-skill-engine-bridge.md b/.planning/S3-SPEC-skill-engine-bridge.md new file mode 100644 index 0000000..842df06 --- /dev/null +++ b/.planning/S3-SPEC-skill-engine-bridge.md @@ -0,0 +1,127 @@ +# S3 — Bridge `handwriting-reader` Skill to Engine Library + +**Status:** SPEC (not yet a phase). Implementation unblocked — no data dependency. +**Authored:** 2026-05-06 +**Source:** `.planning/NEXT-STEPS.md` § Out-of-band side projects. + +--- + +## Goal + +When the user invokes `/handwriting-reader `, the skill calls `handwriting_engine.read_with_consensus()` (or `read_page()` for single-provider) directly instead of executing its own multi-pass workflow described in [SKILL.md](~/.claude/skills/handwriting-reader/SKILL.md). One source of truth: any future improvement to the engine — new strategy, new provider, postprocess upgrade, S2 few-shot, S5 char-consensus — propagates to the skill on the next invocation, no skill edits needed. + +## Why this is high-leverage + +- The skill currently re-implements Phase 1 (classify) → Phase 2 (multi-pass extract) → confidence markers in its own prompt logic. The engine has all of this and more (`read_with_consensus`, `assess_image`, `proven_enhance`, `correct`, the writer-profile path). Two implementations means two failure surfaces and predictable drift. +- Both repos are local. Importing the engine from the skill is a one-time wiring task. +- Pre-requisite for S4 to pay off: if the grader writes corrections back to `benchmark.db` while the skill bypasses the engine entirely, the skill's per-writer accuracy never compounds — half the feedback loop is missing. + +## Non-goals + +- Eliminating the skill. The skill is the conversational entry point (`/handwriting-reader …`); only its *internals* change. +- Changing the skill's user-visible interface. `--format=`, `--strict`, `--domain=`, `--output` arguments and HEIC/PDF handling stay identical. +- Engine changes. The engine's public API is sufficient as-is — verified by inspection of [`handwriting_engine/__init__.py`](../handwriting_engine/__init__.py). + +--- + +## Design + +### 1. Skill restructure ([~/.claude/skills/handwriting-reader/SKILL.md](~/.claude/skills/handwriting-reader/SKILL.md)) + +Replace the Phase 1/Phase 2 prose workflow with a thin orchestration layer: + +```python +from handwriting_engine import ( + read_with_consensus, + assess_image, + proven_enhance, + convert_pdf, +) +from handwriting_engine.writer_profile_store import WriterProfileStore + +# 1. Path validation (unchanged from current skill — this is policy, not engine concern) +# 2. PDF expansion via convert_pdf() if .pdf +# 3. HEIC conversion via sips shell (already documented; stays in skill) +# 4. quality = assess_image(path); if quality.needs_enhancement: proven_enhance() +# 5. profile = WriterProfileStore().load(writer_id) if writer_id else None +# 6. result = read_with_consensus(path, writer_profile=profile, domain=domain) +# 7. Format result per --format= flag +# 8. If --strict: prompt user to resolve each [?alt: …] marker before output +``` + +The skill's job becomes: input parsing, file I/O, output formatting, and `--strict` interactive UX. Engine handles every transcription decision. + +### 2. Domain auto-detection + +Currently the skill auto-detects `--domain=bio` by scanning content. That can't happen *before* transcription. Two valid orderings: + +- **A. Two-pass:** quick `read_page()` with no domain, scan output for biology terms, then `read_with_consensus()` with the detected domain. Cost: 2× the cheap-provider call. +- **B. Trust user / default to general:** require explicit `--domain=` for non-default; otherwise use `domain="general"`. Cost: 0 extra calls, but loses auto-detection. + +**Decision:** B (default to general; `--domain=bio` opt-in). Auto-detection adds latency + cost for marginal accuracy gain on a feature the user is already explicitly invoking. Document the tradeoff in SKILL.md so the user can pass `--domain=bio` when needed. If real-world usage shows users frequently forget the flag, revisit. + +### 3. `--strict` mode + +Today it's a hand-rolled "confirm every `[?]`" loop in the skill. Engine's consensus output already produces `[?alt: X/Y]` markers in `read_with_consensus`. The skill iterates them post-hoc: + +```python +for marker in extract_alt_markers(result.text): + chosen = ask_user(f"Choose: {marker.alternatives}") + result.text = result.text.replace(marker.raw, chosen, 1) +``` + +`extract_alt_markers` is regex over `\[\?alt: ([^\]]+)\]` — keep it skill-side, no engine API needed. + +### 4. Writer profile binding + +If the user passes `--writer=`, look up the profile via `WriterProfileStore().load()` and forward to `read_with_consensus`. New flag — additive, doesn't break callers. + +--- + +## Falsifiable success criteria + +When this phase is complete, all of the following are TRUE: + +1. **Single source of truth for transcription.** `grep -r "Pass 1\|Pass 2\|multi-pass" ~/.claude/skills/handwriting-reader/` returns nothing — the multi-pass logic is removed from the skill prose. +2. **End-to-end parity.** Running `/handwriting-reader sample.jpg --format=md` on a fixture image produces output whose CER vs. ground truth is **≤** the pre-S3 skill's CER on the same image. Tested across a 10-image fixture covering: typed text page (control), neat printed handwriting, cursive, lab-notebook table, and a deliberately blurry image. +3. **Engine improvements propagate.** Bumping a postprocess threshold in [`handwriting_engine/postprocess.py`](../handwriting_engine/postprocess.py) and re-running the skill produces a measurably different output **without** touching SKILL.md. Verified by a git-bisect-style test: change → invoke → diff. +4. **Format flags preserved.** `--format=json` returns the schema documented in pre-S3 SKILL.md (or a richer engine-native schema with the previous fields as a strict subset). No breaking changes for existing automations. +5. **Strict mode works.** `--strict` prompts the user once per `[?alt: …]` marker emitted by `read_with_consensus`, no more, no fewer. +6. **PDF + HEIC unchanged.** Same input handling rules (`pages=`, `sips` conversion note) — verified by re-running each pre-S3 SKILL.md example. +7. **Performance baseline.** Skill latency on a typical lab-notebook page is within 1.2× of pre-S3 (consensus is more expensive than single-pass; this is acceptable, not free). Documented in SKILL.md. + +## Out of scope (queued for follow-up) + +- **Streaming output to chat.** Today the skill renders output as one block. Streaming partial transcriptions during the call is a separate UX project. +- **GUI/TUI for `--strict`.** Stays text-prompt-based. +- **Skill-level caching of recent reads.** Engine doesn't cache; if added later, do it engine-side. + +--- + +## Risks & open questions + +| Risk | Mitigation / decision needed | +|------|------------------------------| +| The skill runs in Claude Code's runtime, not a Python process. The skill is markdown + tool calls, not Python imports. | The skill's "implementation" is Claude executing instructions in SKILL.md. The bridge is: SKILL.md instructs Claude to invoke the engine via the Bash tool: `python3 -m handwriting_engine.cli read --format=json --domain=…`. So S3 reduces to **(a)** ensuring the engine has a CLI surface that maps to every SKILL.md feature, and **(b)** rewriting SKILL.md to call that CLI rather than doing prose-driven multi-pass. **This is the dominant design correction vs. the original framing.** | +| The engine CLI today (`handwriting_engine/cli.py`) is benchmark-focused. Does it expose a top-level `read` command? | Audit needed — pre-implementation check. If missing, scope adds an engine-side `cli read` subcommand that wraps `read_with_consensus`. Likely already exists in some form; verify before sizing. | +| Skill output format may not match what the engine CLI emits today. | Acceptable spec change: engine CLI gains `--format=md|json|txt` flags; skill's role becomes a thin wrapper + UX shell. | +| HEIC handling logic (the `sips` shell command) is pure environmental tooling, not engine concern. | Stays in SKILL.md. | +| `--strict` interactive prompting can't happen inside the engine CLI (Claude can't interact mid-CLI-call). | Engine CLI returns a structured payload (JSON) including the alt-markers as separate fields; skill (Claude) iterates and prompts. | + +## Touchpoints (preliminary) + +- **Edit:** [~/.claude/skills/handwriting-reader/SKILL.md](~/.claude/skills/handwriting-reader/SKILL.md) — replace Phase 1/2 prose with engine CLI invocation + UX layer. +- **Edit (likely):** [`handwriting_engine/cli.py`](../handwriting_engine/cli.py) — add `read` top-level command if missing; ensure `--format=md|json|txt` is supported with stable JSON schema. +- **Edit:** [`~/.claude/skills/handwriting-reader/references/`](~/.claude/skills/handwriting-reader/references/) — likely contains heuristics that move into the engine or get retired. +- **Doc:** SKILL.md changelog note explaining the engine bridge (so future-Claude doesn't re-grow the multi-pass logic in SKILL.md the next time someone "improves" it). + +## Estimated scope + +- 1 plan, ~150 LOC engine-side (CLI surface), ~0 LOC skill (prose rewrite), ~200 LOC tests. +- No data dependency. Can ship now. + +--- + +## Promotion path + +`/gsd-add-phase` (this is naturally next in the v3.0 milestone — it lifts a recurring drift cost). Use this SPEC.md as the seed for `/gsd-discuss-phase` to nail down the engine CLI schema before coding. diff --git a/.planning/S4-SPEC-professor-feedback-loop.md b/.planning/S4-SPEC-professor-feedback-loop.md new file mode 100644 index 0000000..074d45e --- /dev/null +++ b/.planning/S4-SPEC-professor-feedback-loop.md @@ -0,0 +1,151 @@ +# S4 — Professor OS ↔ Engine Feedback Loop + +**Status:** SPEC (not yet a phase). Implementation depends on S3 (skill bridge) for full payoff. Engine-side write API can ship independently. +**Authored:** 2026-05-06 +**Source:** `.planning/NEXT-STEPS.md` § Out-of-band side projects. + +--- + +## Goal + +When `professor/LabNoteBookGrader/` grades a student's lab notebook and the engine reports a low-confidence read on a region, the grader surfaces it to the user (instructor), captures the corrected transcription, and writes the (image_region, ground_truth) pair into the engine's `benchmark.db` as a per-writer ground truth. Over a semester, the same student's per-writer accuracy compounds — every grading session improves the next one. + +## Why this is high-leverage + +- The grader already imports the engine ([`grader/handwriting_reader.py`](../../professor/LabNoteBookGrader/grader/handwriting_reader.py), [`grader/pdf_processor.py`](../../professor/LabNoteBookGrader/grader/pdf_processor.py)). Engine integration is in place; only the *write-back* direction is missing. +- Lab-notebook semesters generate ~12 weeks × N students of GT-quality data for free, as a byproduct of work the instructor was already doing. +- This is the data source S2 (per-writer few-shot exemplars) needs in order to deliver value on returning students. Without S4 the benchmark DB stays IAM-only and S2's per-writer exemplars only help on IAM samples — not on actual classroom workload. +- The engine already has the write primitives: [`benchmark/db.py:289`](../handwriting_engine/benchmark/db.py#L289) `insert_ground_truth()`, [`db.py:218`](../handwriting_engine/benchmark/db.py#L218) sample insert. S4 is mostly wiring + UI in the grader, not new engine plumbing. + +## Non-goals + +- Replacing the human grading workflow. Corrections happen as a side effect of grading, not as a separate "annotation session" the instructor must run. +- Round-tripping every transcription. Only **low-confidence** reads (engine-reported `[?alt: …]` markers, consensus disagreement, or `confidence < threshold`) ask for confirmation. +- Auto-correcting from prior corrections. Compounding happens through S2 (few-shot) and S5 (confusion-pair postprocess), not by mutating the prompt directly with prior corrections (that's brittle). +- Cross-instructor data sharing. Profile + GT data stays in the local instructor's `~/.handwriting-engine/`. Multi-instructor sync is a separate problem. + +--- + +## Design + +### 1. Engine-side: a stable `record_correction()` API + +In [`handwriting_engine/benchmark/db.py`](../handwriting_engine/benchmark/db.py), add a high-level helper that wraps the existing primitives: + +```python +def record_correction( + *, + image_path: str, + writer_id: str, + corrected_text: str, + original_vlm_text: str, + confidence: float, + source: str = "labgrader", +) -> int: + """ + Idempotently record an instructor-corrected transcription. + + - If a sample with this image_hash exists, attach a new ground_truth row + (don't dup the sample). Otherwise insert sample + ground_truth. + - Stores the (original_vlm_text, corrected_text) pair in a new + `corrections` table for trained-corrector training data. + - Returns the ground_truth id. + """ +``` + +New table: + +```sql +CREATE TABLE IF NOT EXISTS corrections ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + sample_id INTEGER NOT NULL REFERENCES samples(id), + ground_truth_id INTEGER NOT NULL REFERENCES ground_truths(id), + original_text TEXT NOT NULL, + confidence REAL, + source TEXT NOT NULL, + created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP +); +CREATE INDEX IF NOT EXISTS idx_corrections_sample ON corrections(sample_id); +``` + +Why a separate table from `ground_truths`: GTs from IAM are authoritative; instructor corrections are *also* GT but may be lower-quality (instructor might typo) — keeping them in their own table lets the trained corrector train on `(original_text, corrected_text)` directly, and lets future quality-assessment runs flag suspect rows without polluting `ground_truths`. + +### 2. Grader-side: low-confidence detection + +In [`professor/LabNoteBookGrader/grader/handwriting_reader.py`](../../professor/LabNoteBookGrader/grader/handwriting_reader.py), after the engine call: + +```python +result = read_with_consensus(...) +low_confidence_regions = extract_review_targets(result, threshold=0.7) +# Each region carries: image crop path, original VLM text, confidence, marker context +``` + +`extract_review_targets` is grader-side; it consumes the engine's `ConsensusResult` (already a public model — [`__init__.py:62`](../handwriting_engine/__init__.py#L62)) and returns reviewable regions. + +### 3. Grader-side: correction capture + +The grader already has a GUI ([`LabNoteBookGrader/gui/`](../../professor/LabNoteBookGrader/gui/)). Add a "Review low-confidence reads" panel that, per region: + +1. Shows the image crop alongside the engine's transcription with `[?alt: X/Y]` markers visible. +2. Lets the instructor either accept (= confirm engine output is correct) or correct (type the right text). +3. On submit, calls `handwriting_engine.benchmark.db.record_correction()` with the writer_id derived from the active student. + +Engine writer_id ↔ Professor OS student ID mapping: instructor sets `student_id` per assignment; grader maps `student_id → writer_id="prof-{course}-{student_id}"`. Documented mapping rule, not hardcoded magic. + +### 4. Trained-corrector training-data feed + +The trained corrector ([`handwriting_engine/trained_correction/`](../handwriting_engine/trained_correction/)) currently can ingest from `--from-benchmark-db` per [NEXT-STEPS.md:90](NEXT-STEPS.md#L90). After S4, the same flag picks up real instructor corrections — no separate training pipeline needed. + +--- + +## Falsifiable success criteria + +When this phase is complete, all of the following are TRUE: + +1. **Engine API is stable and idempotent.** Calling `record_correction()` twice with identical args writes one sample + one GT + two correction rows (the correction history grows; the canonical sample doesn't dup). Verified by unit test. +2. **Schema migration is reversible.** The `corrections` table can be dropped without breaking any other engine functionality. Verified by running the test suite with the table absent. +3. **Grader surfaces low-confidence reads.** Running the grader on a fixture lab notebook with deliberately ambiguous handwriting produces a non-empty review queue. Engine confidence threshold is configurable (env var or grader config), default 0.7. +4. **Corrections persist to the engine's DB.** After running a grading session and submitting 5 corrections, `sqlite3 ~/.handwriting-engine/benchmark.db "SELECT COUNT(*) FROM corrections"` returns ≥5, all linked to valid samples and GTs. +5. **Per-writer accumulation works.** Grading the same student twice in two sessions adds rows to the *same* `student` value in `samples`. `SELECT COUNT(*) FROM samples WHERE student='prof-bio101-jdoe'` increases monotonically across sessions. +6. **Trained corrector ingests real data.** `python3 -m handwriting_engine.trained_correction.train --from-benchmark-db` produces a training file whose row count matches `(SELECT COUNT(*) FROM corrections)`. Manual spot-check of 20 random rows confirms the (original, corrected) pairs are sane. +7. **No regression on existing grader workflow.** A grader run with the review panel disabled (env flag) produces output bit-identical to pre-S4. Reviewability is opt-in for any user not yet ready for the workflow. + +## Out of scope (queued for follow-up) + +- **Inter-instructor data sharing / sync.** +- **Auto-derived writer profiles from corrections.** (Could compute "this student crosses 7s 80% of the time" from accumulated corrections — separate project.) +- **Confidence calibration on the grader side.** (The 0.7 threshold is heuristic. Phase 8 stats could inform a better cutoff later.) +- **Web UI / cloud upload.** Local-only. + +--- + +## Risks & open questions + +| Risk | Mitigation / decision needed | +|------|------------------------------| +| Instructor may rush and submit wrong corrections (typos, misreadings of their own student's hand). Garbage-in poisons S2/trained-corrector. | Track `source="labgrader"` on every row. Add a `quality` column or use the existing `quality_assessments` table to let a periodic review pass downgrade suspect rows. **Don't** auto-trust corrections for trained-corrector training without a quality gate. | +| Image crops sent to engine.db must be persisted somewhere — the original PDF page is not a stable identifier. | The grader already extracts page images for grading; persist crops to `~/.handwriting-engine/student-corpus/{course}/{student_id}/{date}-p{page}-r{region}.png` and pass that absolute path to `record_correction`. | +| Privacy: student handwriting samples are FERPA-protected. Storing them outside the grader's directory crosses a boundary. | Document local-only storage. Engine `~/.handwriting-engine/` is on the same machine as the grader. Add a `--purge-student-data ` engine CLI command for end-of-semester cleanup. | +| The grader's GUI is a separate codebase; spec creep risk. | Scope clarification: S4 ships the *engine API* + the *grader-side detection logic*. The GUI panel is a follow-up plan; an interim CLI-prompt fallback in the grader is acceptable for v0. | +| Schema migration on a populated benchmark DB. | Use `CREATE TABLE IF NOT EXISTS` (existing pattern in [`db.py`](../handwriting_engine/benchmark/db.py)). No data migration needed — it's an additive table. | +| S2 needs this data to be useful, but S2 is also blocked on IAM. Risk of waiting for S4 before shipping S2 even on IAM-only. | Order: S4 ships independently. S2 implementation can use IAM data alone for falsification (criterion #3 in S2-SPEC); S4 then makes S2 useful for the actual classroom use case. | + +## Touchpoints (preliminary) + +- **New:** [`handwriting_engine/benchmark/db.py`](../handwriting_engine/benchmark/db.py) — `record_correction()`, `corrections` table. +- **New:** engine CLI `benchmark record-correction` for ad-hoc / scripting use. +- **New:** engine CLI `benchmark purge-writer ` for cleanup. +- **Edit:** [`professor/LabNoteBookGrader/grader/handwriting_reader.py`](../../professor/LabNoteBookGrader/grader/handwriting_reader.py) — add `extract_review_targets()`. +- **Edit:** grader workflow / GUI to show review panel. +- **Tests:** engine-side unit tests + grader integration test against fixture notebook. + +## Estimated scope + +- ~2 plans (engine schema/API; grader detection + UI). Engine: ~150 LOC + tests. Grader: ~250 LOC + tests + GUI panel. +- No external data dependency. Can ship engine-side immediately; grader integration in parallel. + +--- + +## Promotion path + +`/gsd-add-phase`. This phase straddles two repos — keep the engine-side and grader-side as **separate plans within one phase** so the engine API is reviewable and lockable before grader work begins. diff --git a/.planning/S5-SPEC-char-consensus-confusion-postprocess.md b/.planning/S5-SPEC-char-consensus-confusion-postprocess.md new file mode 100644 index 0000000..823c10a --- /dev/null +++ b/.planning/S5-SPEC-char-consensus-confusion-postprocess.md @@ -0,0 +1,164 @@ +# S5 — Char-Level Consensus + Confusion-Pair-Aware Postprocess + +**Status:** SPEC (not yet a phase). Implementation unblocked — no data dependency for char-consensus; confusion-postprocess wants populated DB but degrades gracefully. +**Authored:** 2026-05-06 +**Source:** `.planning/NEXT-STEPS.md` § Out-of-band side projects. + +--- + +## Goal + +Two complementary additions to the engine's transcription pipeline: + +1. **Char-level consensus.** When ≥2 providers run, in addition to today's word-level voting ([`consensus.py:1018`](../handwriting_engine/consensus.py#L1018)), compute character-level alignment within disagreed-on words. Producers disagreeing on `rn` vs `m`, `cl` vs `d`, `0` vs `O` get resolved at the char level using the global confusion-pair map plus per-writer profile, instead of arbitrarily picking one provider's word. + +2. **Confusion-pair-aware postprocess.** Extend [`postprocess.correct()`](../handwriting_engine/postprocess.py) with a pass that consumes the running confusion-pair stats already tracked by [`benchmark/metrics.py:classify_errors`](../handwriting_engine/benchmark/metrics.py#L224) — when a candidate word is one confusion-pair-swap away from a known-domain term, prefer the swap. + +After this phase, single-character substitution errors (the dominant residual error class per Phase 7 drill-down reports) drop measurably without regressing easy cases. + +## Why this is high-leverage + +- The drill-down report already tracks confusion pairs per [`benchmark/report.py:301`](../handwriting_engine/benchmark/report.py#L301) `sample_drill_down`. The data exists; it's read-only today. Putting it back into the pipeline closes the observability → action loop. +- Word-level voting silently mis-resolves when *all* providers see different things (no majority). Char-level voting catches the dominant subset of these: single-char swaps within otherwise-aligned words. +- Phase 5 already shipped postprocess + multi-word phrase correction (commits `9b8c464`, `f0ebdea`). S5 is an additive postprocess pass, not a redesign. +- Compounds with S2 and S4: per-writer confusion-pair history (from S4 corrections) makes the postprocess pass per-writer adaptive without needing a new model. + +## Non-goals + +- Replacing word-level voting. Char-level runs *within* word-level disagreements, not instead of them. +- Building a language model. We use existing wordlists and known confusion pairs ([`handwriting.py:594`](../handwriting_engine/handwriting.py#L594) `get_disambiguation_pairs`). +- Trained-corrector territory. The postprocess pass is rule-based; the trained corrector is its own (separately-tracked) project. + +--- + +## Design + +### 1. Char-level consensus ([`consensus.py`](../handwriting_engine/consensus.py)) + +Today's flow at [`consensus.py:1018`](../handwriting_engine/consensus.py#L1018): + +```python +sorted_votes = sorted(word_votes.items(), key=lambda x: x[1], reverse=True) +winner = sorted_votes[0][0] +# ...if no majority, pick highest-weighted, mark [?alt: …] +``` + +S5 inserts a step *before* the no-majority fallback: + +```python +if not has_majority(sorted_votes): + # Try to resolve at char level + resolved = resolve_char_level( + candidates=[w for w, _ in sorted_votes], + weights=[v for _, v in sorted_votes], + confusion_pairs=GLOBAL_CONFUSION_PAIRS, + writer_confusion_resolutions=writer_profile.get("confusion_resolutions", {}) if writer_profile else {}, + ) + if resolved is not None: + result_words.append(resolved) + # Don't emit [?alt: …] — char-level resolved it + continue + # else fall through to existing no-majority handling +``` + +`resolve_char_level()` algorithm v0: + +1. Align candidate strings via `difflib.SequenceMatcher` char-by-char. +2. For each diff position, gather the chars each provider voted for. +3. If all chars at that position map to the *same confusion-pair group* (e.g. `{r,n}` vs `{m}` is the known `rn↔m` pair), apply the writer's preference if set, otherwise the global default for that pair. +4. If no confusion-pair match, return `None` (defer to existing fallback). + +Determinism: same inputs ⇒ same output. Ordering of providers must not affect result. + +### 2. Confusion-pair-aware postprocess ([`postprocess.py`](../handwriting_engine/postprocess.py)) + +New function: + +```python +def correct_confusion_pairs( + text: str, + *, + domain: str = "biology", + writer_id: str | None = None, + db_path: Path | None = None, +) -> tuple[str, list[Correction]]: + """ + For each word in text, if a single confusion-pair swap produces a known + domain term, prefer the swap. Returns corrected text + list of corrections + applied (for audit logging). + """ +``` + +Logic: + +1. Tokenize. +2. For each word that's NOT in the domain wordlist: + - Generate all candidates 1 confusion-pair swap away (`rn↔m`, `cl↔d`, `0↔O`, `l↔1`, `5↔S`, full list from `get_disambiguation_pairs()`). + - If exactly one candidate is in the wordlist, swap. + - If multiple candidates are in the wordlist, prefer the one matching writer-specific resolutions. If still ambiguous, no swap (don't introduce error). +3. Return the corrected text + audit list. + +This pass runs **after** `correct()`'s existing edit-distance-1 wordlist correction at [`postprocess.py:200-228`](../handwriting_engine/postprocess.py#L200-L228) — the new pass is restricted to confusion-pair-shaped edits, which are higher-precision than generic ED1. + +### 3. DB integration (optional, graceful) + +If `db_path` is provided and `corrections` table exists (from S4), pull writer-specific historical confusion-pair resolutions to bias the postprocess. If no DB or table absent, fall back to global pairs. Hard-fail-free. + +### 4. Wiring + +- `consensus.py` calls `resolve_char_level` inline; no top-level API change. +- `postprocess.correct()` gains a `correct_confusion_pairs` step in its existing pipeline. New env flag `HE_CONFUSION_POSTPROCESS=1` (default ON; flip to 0 to disable for A/B). + +--- + +## Falsifiable success criteria + +When this phase is complete, all of the following are TRUE: + +1. **Char-level resolves a known confusion case.** Given fixture providers returning `["modern", "rnodern", "modern"]` (with weights), word-level voting resolves to `modern` (majority). Given `["modern", "rnodern"]` (no majority), char-level resolution returns `modern` *with no `[?alt: …]` marker* because the only difference is the `m↔rn` confusion pair. Verified by unit test. +2. **Defers cleanly when not a confusion case.** Given `["apple", "orange"]` (unrelated words), char-level returns `None` and the existing `[?alt: orange]` marker emits as today. +3. **Writer-specific bias works.** With `writer_profile = {"confusion_resolutions": {"rn↔m": "rn"}}`, char-level returns `rnodern` instead of `modern` from input `["modern", "rnodern"]`. Verified by unit test. +4. **Postprocess corrects a real confusion case.** Input: `"the celI underwent mitosis"` (capital I after cell, common `l↔I` confusion). Output: `"the cell underwent mitosis"`. Audit log records the correction. +5. **Postprocess does not over-correct.** Input: `"the apple is red"` — no swap suggests a domain term exists, so output is bit-identical to input. Verified across a 50-sample non-confusion fixture. +6. **End-to-end CER win.** On the IAM test set's `prompt_adapted` strategy, CER with `HE_CONFUSION_POSTPROCESS=1` is lower than with `HE_CONFUSION_POSTPROCESS=0` by a margin where the Phase 8 Wilcoxon test reports `p < 0.05`. (Like S2, this requires Phase 8 to be testable.) +7. **No regression on the LabNoteBookGrader fixture.** Pre-S5 vs. post-S5 CER on the grader's existing test corpus is ≥0 (improvement or unchanged). Verified by `cd professor/LabNoteBookGrader && pytest tests/test_grading_accuracy.py` or equivalent. +8. **Audit trail.** Every confusion-pair correction applied is logged at INFO with original word, corrected word, and pair name, so silent over-correction is detectable. + +## Out of scope (queued for follow-up) + +- **Multi-swap candidates.** v0 only considers 1-confusion-pair-swap-away candidates. 2-swap candidates explode the candidate space; revisit only if v0 ships and the residual error analysis says 2-swap matters. +- **Position-weighted edit costs.** Treating confusion-pair swaps as cheaper than generic ED1 — already implicit in this design (separate pass), but a unified weighted-edit-distance reformulation is a v2 concern. +- **Char-level consensus across full sentences (not just disagreement words).** Compute cost is high; v0 restricts to no-majority words. +- **Learning new confusion pairs from data.** v0's pairs are from `get_disambiguation_pairs()`. Discovering new pairs from corrections data is a separable analytics project. + +--- + +## Risks & open questions + +| Risk | Mitigation / decision needed | +|------|------------------------------| +| Postprocess pass over-corrects on non-domain text (poetry, names, code in lab notes). | The "exactly one candidate in wordlist" rule already prevents most. Add a per-domain wordlist; when `domain="general"`, postprocess runs at lower aggressiveness (require ≥2 matching domain wordlists or skip). | +| Char-level alignment fails on string-length disagreements (`"colour"` vs `"color"` — different length). | `SequenceMatcher` handles this. Test with a length-mismatch fixture in criterion #2. | +| Determinism across provider order. | The `resolve_char_level` algorithm sorts candidates by weight; ties broken by provider name (alpha). Documented; tested. | +| Trained-corrector overlap. The trained corrector also fixes confusion pairs. Stacking both could double-correct or fight. | The trained corrector is gated OFF today (`HE_USE_TRAINED_CORRECTOR=0`). When it eventually flips on, run both with the rule-based pass *first* and trained pass *second* — rule-based handles obvious cases, trained handles residuals. Document the order. | +| Per-writer confusion data is sparse early on (S4 just shipped). | Pass falls back to global confusion pairs. As S4 accumulates data, per-writer biasing kicks in automatically. No phase ordering blocker. | +| Wordlist coverage gaps — biology terms missing produce false negatives. | Existing biology wordlist is the input; gaps are pre-existing. Document, don't expand wordlist as part of S5. | + +## Touchpoints (preliminary) + +- **Edit:** [`handwriting_engine/consensus.py`](../handwriting_engine/consensus.py) — `resolve_char_level()`, hook into existing voting fallback. +- **Edit:** [`handwriting_engine/postprocess.py`](../handwriting_engine/postprocess.py) — `correct_confusion_pairs()`, integrate into `correct()` pipeline. +- **Edit:** [`handwriting_engine/handwriting.py`](../handwriting_engine/handwriting.py) — expose confusion-pair list in a more-machine-consumable shape if not already. +- **New:** tests for both directions: char-consensus + postprocess. +- **Doc:** README env-flag table. + +## Estimated scope + +- 1 plan, ~250 LOC engine-side, ~300 LOC tests. +- Falsification of criterion #6 needs Phase 8 + IAM data; criteria #1–5, #7–8 are testable today. + +--- + +## Promotion path + +`/gsd-add-phase`. Order suggestion: ship S5 *before* S4 if you want immediate engine-internal CER wins; ship S5 *after* S4 if you want per-writer postprocess biasing on day one. Spec is order-agnostic. diff --git a/.planning/STATE.md b/.planning/STATE.md index 927b489..0f0fc46 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -2,13 +2,13 @@ gsd_state_version: 1.0 milestone: v3.0 milestone_name: — Verified Accuracy -status: unknown -last_updated: "2026-04-13T05:41:32.606Z" +status: in_progress +last_updated: "2026-05-06T00:00:00.000Z" progress: total_phases: 4 - completed_phases: 1 + completed_phases: 2 total_plans: 8 - completed_plans: 5 + completed_plans: 8 --- # Execution State @@ -24,7 +24,7 @@ progress: See: `.planning/PROJECT.md` (updated 2026-04-11) **Core value:** Highest-accuracy LLM-vision handwriting transcription with self-correction, ensemble providers, and writer adaptation -**Current focus:** Ready to plan Phase 6 — Measurement Foundation +**Current focus:** Phase 7 complete — IAM sweep + per-writer report shipped. Ready for Phase 8 (Statistics Layer) once IAM dataset is downloaded and a sweep run populates the DB. --- @@ -32,15 +32,15 @@ See: `.planning/PROJECT.md` (updated 2026-04-11) | Field | Value | |-------|-------| -| Phase | 6 — Measurement Foundation | -| Plan | 03 complete (Provenance capture + report display) | -| Status | in_progress | -| Progress | Phase 6 of 9 (v3.0 scope: phases 6-9) | +| Phase | 7 — IAM Data Ingestion + Sweep Infrastructure (COMPLETE 2026-05-06) | +| Plan | 04 complete (per-writer report). All 4 plans landed. | +| Status | phase_complete | +| Progress | Phase 7 of 9 (v3.0 scope: phases 6-9) | ``` v3.0 Progress: [ 6 ][ 7 ][ 8 ][ 9 ] - ^ - here + ^ + here (8 next, blocked on IAM download + sweep run) ``` --- @@ -59,6 +59,8 @@ v3.0 Progress: [ 6 ][ 7 ][ 8 ][ 9 ] | Phase 06 P03 | 30 | 2 tasks | 4 files | | Phase 06 P04 | 15 | 2 tasks | 2 files | | Phase 07-iam-data-ingestion-sweep-infrastructure P02 | 25 | 2 tasks | 3 files | +| Phase 07-iam-data-ingestion-sweep-infrastructure P03 | 30 | 2 tasks | 3 files | +| Phase 07-iam-data-ingestion-sweep-infrastructure P04 | 10 | 2 tasks | 3 files | ## Accumulated Context @@ -81,6 +83,13 @@ v3.0 Progress: [ 6 ][ 7 ][ 8 ][ 9 ] - [Phase 07-02]: No quality assessment in IAM ingest — pre-segmented clean PNGs, latency without benefit - [Phase 07-02]: Partition safety guard at CLI layer only — ingest_iam() passes partition_forms=None and caller is responsible - [Phase 07-02]: ingest_iam uses autocommit=False + explicit conn.commit() per record for atomic GT+sample commits +- [Phase 07-03]: line_level/auto_retry threaded through full call chain (_read_single, _run_benchmark_inner, run_benchmark) with backward-compatible False defaults — sweep parity with single read_page() invocation +- [Phase 07-03]: SWEEP_STRATEGIES is a list of {name, label, kwargs} dicts so adding a strategy is a one-line append +- [Phase 07-03]: run_sweep() filters samples via SQL `WHERE s.category='iam'` joined to ground_truths — no new category_filter param +- [Phase 07-03]: Cost projection lives at CLI layer; run_sweep() never prompts — keeps the library callable from notebooks/scripts +- [Phase 07-04]: Per-writer SQL excludes student IS NULL OR student='' so non-IAM samples don't pollute the table +- [Phase 07-04]: Per-writer table sorts mean_cer DESC (hardest writers first — most actionable view) +- [Phase 07-04]: --per-writer branch returns early in benchmark_report_cmd; existing report logic untouched when flag absent ### Key Facts for Planning @@ -109,6 +118,9 @@ Newest entries first. ### Entries ``` +[2026-05-06] PHASE 7 COMPLETE — All 4 plans landed (07-01 RED, 07-02 IAM ingest, 07-03 sweep, 07-04 per-writer report). 17 RED stubs turned GREEN. IAM-01/IAM-02/IAM-03 satisfied. Full suite: 525 passed, 2 skipped, 1 xfailed. Phase 8 (Statistics Layer) is next, blocked on user-side IAM download + first sweep run. +[2026-05-06] 07-04 COMPLETE — generate_per_writer_report() in report.py + --per-writer flag on benchmark report CLI. 3 TestPerWriterReport stubs GREEN. Commit 8532da4. +[2026-05-06] 07-03 COMPLETE — run_sweep() + benchmark sweep CLI + line_level/auto_retry threading through run_benchmark. 5 TestSweep stubs GREEN. Commit 7901d84. [2026-04-11] 06-03 COMPLETE — Provenance capture + marker rate wired in evaluate.py; Provenance header + marker_rate column added to report.py; list_runs() in db.py extended. Two test stubs fixed (missing run setup). 4 files modified. [2026-04-11] 06-01 COMPLETE — Wave 0 RED stubs written. 12 new failing tests across 2 files (test_benchmark_db.py, test_benchmark_evaluate.py). All existing tests remain GREEN. Commits: d103aed, 84fcd5b. [2026-04-11] ROADMAP — v3.0 roadmap created. 4 phases (6-9), 12/12 requirements mapped. Ready to plan Phase 6. diff --git a/.planning/milestones/v4.0-PROPOSAL.md b/.planning/milestones/v4.0-PROPOSAL.md index fd1fc32..d3d23c5 100644 --- a/.planning/milestones/v4.0-PROPOSAL.md +++ b/.planning/milestones/v4.0-PROPOSAL.md @@ -1,8 +1,28 @@ -# v4.0 — Trained Post-Correction (proposal / forward-looking) +# v4.0 — Trained Post-Correction -**Status:** PROPOSED — do not start until v3.0 (Phases 6-9) lands -**Hard dependency:** Phase 7 (IAM ingestion) must be complete — that's the data foundation +**Status:** SYNTHETIC-ONLY V0 IMPLEMENTED on `feat/trained-corrector` (2026-05-05). Real-data fine-tune (post-Phase 7) still queued. +**Hard dependency for v0:** None — synthetic data generator built in-tree. +**Hard dependency for production parity:** Phase 7 (IAM ingestion) must complete to unblock real-data fine-tune. **Created:** 2026-05-05 +**Last updated:** 2026-05-05 — branch `feat/trained-corrector` shipped synthetic v0 + +## Implementation status + +| Component | Status | Location | +|-----------|--------|----------| +| Synthetic OCR-error generator | ✓ Shipped | `handwriting_engine/trained_correction/synthetic_data.py` | +| Clean-text corpus builder (templates + domain vocab + system wordlist) | ✓ Shipped | `handwriting_engine/trained_correction/corpus.py` | +| torch Dataset wrapper for `(corrupted, clean)` pairs | ✓ Shipped | `handwriting_engine/trained_correction/dataset.py` | +| Manual PyTorch training loop (no `accelerate` dep) targeting MPS | ✓ Shipped | `handwriting_engine/trained_correction/train.py` | +| Inference interface (lazy load, beam search, chunking) | ✓ Shipped | `handwriting_engine/trained_correction/corrector.py` | +| A/B eval harness (heuristic vs trained vs combined; CER) | ✓ Shipped | `handwriting_engine/trained_correction/eval.py` | +| Engine CLI subcommands (`trained-correction train` / `eval`) | ✓ Shipped | `handwriting_engine/cli.py` | +| `postprocess.correct()` orchestrator (heuristic → optional trained) | ✓ Shipped | `handwriting_engine/postprocess.py` | +| Optional dep group `[trained-correction]` in `pyproject.toml` | ✓ Shipped | `pyproject.toml` | +| Tests (synthetic data, corpus, dataset, eval, orchestrator) | ✓ 22 passing | `tests/test_trained_correction.py` | +| **Real-data fine-tune from IAM** | ⏳ Blocked on Phase 7 | — | +| **Writer-conditioned variant** | ⏳ v4.1 | — | +| **Active learning loop on disagreements** | ⏳ v4.1 | — | --- diff --git a/.planning/phase-1/1-02-PLAN.md b/.planning/phase-1/1-02-PLAN.md index e61de69..423635d 100644 --- a/.planning/phase-1/1-02-PLAN.md +++ b/.planning/phase-1/1-02-PLAN.md @@ -263,7 +263,7 @@ # Only called once (best_of, no escalation) assert mock.read_image.call_count == 1 - cd "~/Developer/handwriting-engine" && python -m pytest tests/test_consensus.py -x -q 2>&1 | tail -5 + cd "/Users/user/Documents/VSCode Projects/handwriting-engine" && python -m pytest tests/test_consensus.py -x -q 2>&1 | tail -5 All consensus tests pass including new TestSelfCorrectStrategy and TestSmartEscalation diff --git a/.planning/phase-2/2-01-PLAN.md b/.planning/phase-2/2-01-PLAN.md index 9c84181..5352ddc 100644 --- a/.planning/phase-2/2-01-PLAN.md +++ b/.planning/phase-2/2-01-PLAN.md @@ -11,7 +11,7 @@ Create handwriting_engine/line_reader.py -Create ~/Developer/handwriting-engine/handwriting_engine/line_reader.py: +Create /Users/user/Documents/VSCode Projects/handwriting-engine/handwriting_engine/line_reader.py: """ Line-level segmentation for handwritten page images. @@ -171,7 +171,7 @@ def read_page_by_lines( Add line_level parameter to vision.py read_image() -In ~/Developer/handwriting-engine/handwriting_engine/vision.py, find the main read_image() function (it will have image_path, prompt, etc. as parameters). +In /Users/user/Documents/VSCode Projects/handwriting-engine/handwriting_engine/vision.py, find the main read_image() function (it will have image_path, prompt, etc. as parameters). Add a `line_level: bool = False` parameter to the function signature. @@ -197,7 +197,7 @@ This should be inserted early in read_image(), after image loading/enhancement b Create tests/test_line_reader.py -Create ~/Developer/handwriting-engine/tests/test_line_reader.py: +Create /Users/user/Documents/VSCode Projects/handwriting-engine/tests/test_line_reader.py: """Tests for line-level segmentation.""" import io @@ -296,7 +296,7 @@ def test_read_page_by_lines_assembles_in_order(tmp_path): assert parts[0] == "line_1" assert parts[-1] == f"line_{call_count[0]}" - cd "~/Developer/handwriting-engine" && python -m pytest tests/test_line_reader.py -x -q 2>&1 | tail -5 + cd "/Users/user/Documents/VSCode Projects/handwriting-engine" && python -m pytest tests/test_line_reader.py -x -q 2>&1 | tail -5 All line_reader tests pass diff --git a/.planning/phase-3/3-01-PLAN.md b/.planning/phase-3/3-01-PLAN.md index 79d461e..9da5a3d 100644 --- a/.planning/phase-3/3-01-PLAN.md +++ b/.planning/phase-3/3-01-PLAN.md @@ -12,7 +12,7 @@ Create handwriting_engine/providers/paddleocr_provider.py -Create ~/Developer/handwriting-engine/handwriting_engine/providers/paddleocr_provider.py: +Create /Users/user/Documents/VSCode Projects/handwriting-engine/handwriting_engine/providers/paddleocr_provider.py: """ PaddleOCR 3.0 (PP-OCRv5) vision provider. @@ -143,7 +143,7 @@ register("paddleocr", PaddleOCRProvider) Create handwriting_engine/providers/trocr_provider.py -Create ~/Developer/handwriting-engine/handwriting_engine/providers/trocr_provider.py: +Create /Users/user/Documents/VSCode Projects/handwriting-engine/handwriting_engine/providers/trocr_provider.py: """ TrOCR vision provider using HuggingFace transformers. @@ -319,7 +319,7 @@ register("trocr", TrOCRProvider) Register new providers in providers/__init__.py and add tests -1. In ~/Developer/handwriting-engine/handwriting_engine/providers/__init__.py: +1. In /Users/user/Documents/VSCode Projects/handwriting-engine/handwriting_engine/providers/__init__.py: In the _try_autoload() function, add two new elif branches: elif name == "paddleocr": @@ -331,7 +331,7 @@ Also update the available_providers() function — the autoload loop currently o for name in ("claude", "openai", "gemini", "paddleocr", "trocr"): _try_autoload(name) -2. Create ~/Developer/handwriting-engine/tests/test_providers_new.py: +2. Create /Users/user/Documents/VSCode Projects/handwriting-engine/tests/test_providers_new.py: """Tests for PaddleOCR and TrOCR providers — availability and interface.""" import base64 @@ -402,7 +402,7 @@ class TestTrOCRProvider: with pytest.raises(ValueError, match="same length"): provider.fine_tune_for_writer("test_writer", ["b64img"], ["text1", "text2"]) - cd "~/Developer/handwriting-engine" && python -m pytest tests/test_providers_new.py -x -q 2>&1 | tail -10 + cd "/Users/user/Documents/VSCode Projects/handwriting-engine" && python -m pytest tests/test_providers_new.py -x -q 2>&1 | tail -10 New provider tests pass; paddleocr and trocr in _REGISTRY after import diff --git a/.planning/phase-4/4-01-PLAN.md b/.planning/phase-4/4-01-PLAN.md index f0090b5..815aa0a 100644 --- a/.planning/phase-4/4-01-PLAN.md +++ b/.planning/phase-4/4-01-PLAN.md @@ -13,7 +13,7 @@ Add sauvola_enhance() to enhance.py -In ~/Developer/handwriting-engine/handwriting_engine/enhance.py: +In /Users/user/Documents/VSCode Projects/handwriting-engine/handwriting_engine/enhance.py: 1. Add sauvola_enhance() function after clahe_enhance(): @@ -73,7 +73,7 @@ def sauvola_enhance( Add test_sauvola to tests/test_enhance.py -In ~/Developer/handwriting-engine/tests/test_enhance.py, append: +In /Users/user/Documents/VSCode Projects/handwriting-engine/tests/test_enhance.py, append: def test_sauvola_enhance_returns_path(tmp_path): """sauvola_enhance should return a path without crashing.""" @@ -101,7 +101,7 @@ Make sure the test file imports Image from PIL at the top if not already there. Create writer_profile_store.py -Create ~/Developer/handwriting-engine/handwriting_engine/writer_profile_store.py: +Create /Users/user/Documents/VSCode Projects/handwriting-engine/handwriting_engine/writer_profile_store.py: """ Persistent writer profile store for cross-session handwriting calibration. @@ -218,7 +218,7 @@ class WriterProfileStore: Create tests/test_writer_profile_store.py and commit -Create ~/Developer/handwriting-engine/tests/test_writer_profile_store.py: +Create /Users/user/Documents/VSCode Projects/handwriting-engine/tests/test_writer_profile_store.py: """Tests for WriterProfileStore.""" import pytest diff --git a/.planning/phase-5/5-01-PLAN.md b/.planning/phase-5/5-01-PLAN.md index c9b5168..e4f130b 100644 --- a/.planning/phase-5/5-01-PLAN.md +++ b/.planning/phase-5/5-01-PLAN.md @@ -12,7 +12,7 @@ Create handwriting_engine/postprocess.py with domain spell correction -Create ~/Developer/handwriting-engine/handwriting_engine/postprocess.py: +Create /Users/user/Documents/VSCode Projects/handwriting-engine/handwriting_engine/postprocess.py: """ Domain-specific spell correction for handwriting transcription output. @@ -188,7 +188,7 @@ def correct_domain_terms(text: str, domain: str = "biology") -> str: Wire domain correction into vision.py _postprocess_output() -In ~/Developer/handwriting-engine/handwriting_engine/vision.py: +In /Users/user/Documents/VSCode Projects/handwriting-engine/handwriting_engine/vision.py: 1. Find the _postprocess_output() function and add an optional domain parameter: def _postprocess_output(text: str, domain: str | None = None) -> str: @@ -212,7 +212,7 @@ In ~/Developer/handwriting-engine/handwriting_engine/vision.py: Create tests/test_postprocess.py -Create ~/Developer/handwriting-engine/tests/test_postprocess.py (or append if it exists): +Create /Users/user/Documents/VSCode Projects/handwriting-engine/tests/test_postprocess.py (or append if it exists): """Tests for domain spell correction.""" from handwriting_engine.postprocess import correct_domain_terms, _edit_distance_1_candidates, _BIOLOGY_TERMS @@ -273,7 +273,7 @@ def test_empty_string(): Add --compare-strategies flag to benchmark CLI and regression alerting -In ~/Developer/handwriting-engine/handwriting_engine/benchmark/evaluate.py: +In /Users/user/Documents/VSCode Projects/handwriting-engine/handwriting_engine/benchmark/evaluate.py: Read the file first to understand the existing CLI structure. Then: diff --git a/.planning/phases/06-measurement-foundation/06-01-PLAN.md b/.planning/phases/06-measurement-foundation/06-01-PLAN.md index 9aea49a..b6ab0fa 100644 --- a/.planning/phases/06-measurement-foundation/06-01-PLAN.md +++ b/.planning/phases/06-measurement-foundation/06-01-PLAN.md @@ -43,8 +43,8 @@ Output: Extended test_benchmark_db.py with test_v4_migration_columns; extended t -@~/.claude/get-shit-done/workflows/execute-plan.md -@~/.claude/get-shit-done/templates/summary.md +@/Users/user/.claude/get-shit-done/workflows/execute-plan.md +@/Users/user/.claude/get-shit-done/templates/summary.md @@ -382,7 +382,7 @@ pytest tests/test_benchmark_db.py::TestSchemaCreation::test_v4_migration_columns Acceptable failure modes: OperationalError (column doesn't exist), AttributeError (report missing field), SystemExit (command not registered), assertion failures. NOT acceptable: SyntaxError in test file itself. - cd "~/Developer/handwriting-engine" && pytest tests/test_benchmark_db.py::TestSchemaCreation::test_v4_migration_columns tests/test_benchmark_evaluate.py::TestMarkerRate tests/test_benchmark_evaluate.py::TestCalibrateCommand tests/test_benchmark_evaluate.py::TestCostProjection tests/test_benchmark_evaluate.py::TestProvenanceCapture -q 2>&1 | tail -10 + cd "/Users/user/Documents/Work & Projects/VSCode Projects/handwriting-engine" && pytest tests/test_benchmark_db.py::TestSchemaCreation::test_v4_migration_columns tests/test_benchmark_evaluate.py::TestMarkerRate tests/test_benchmark_evaluate.py::TestCalibrateCommand tests/test_benchmark_evaluate.py::TestCostProjection tests/test_benchmark_evaluate.py::TestProvenanceCapture -q 2>&1 | tail -10 All stub tests exist in the file, existing tests remain GREEN, new tests are RED (FAILED/ERROR — not SyntaxError). `pytest tests/test_benchmark_db.py tests/test_benchmark_evaluate.py -x -q` passes the existing tests and fails only on the new stubs. diff --git a/.planning/phases/06-measurement-foundation/06-02-PLAN.md b/.planning/phases/06-measurement-foundation/06-02-PLAN.md index e20daf1..0d8b8aa 100644 --- a/.planning/phases/06-measurement-foundation/06-02-PLAN.md +++ b/.planning/phases/06-measurement-foundation/06-02-PLAN.md @@ -44,8 +44,8 @@ Output: db.py with _MIGRATIONS[4] and CURRENT_SCHEMA_VERSION = 4; models.py with -@~/.claude/get-shit-done/workflows/execute-plan.md -@~/.claude/get-shit-done/templates/summary.md +@/Users/user/.claude/get-shit-done/workflows/execute-plan.md +@/Users/user/.claude/get-shit-done/templates/summary.md @@ -238,7 +238,7 @@ CRITICAL: Do NOT change the `autocommit` parameter position — existing callers CRITICAL: Do NOT change the `compare_strategies()` function in evaluate.py — it calls `insert_run()` and must continue working via None defaults. - cd "~/Developer/handwriting-engine" && pytest tests/test_benchmark_db.py::TestSchemaCreation -x -q + cd "/Users/user/Documents/Work & Projects/VSCode Projects/handwriting-engine" && pytest tests/test_benchmark_db.py::TestSchemaCreation -x -q TestSchemaCreation::test_v4_migration_columns PASSES. All other TestSchemaCreation tests still pass. `pytest tests/test_benchmark_db.py -x -q` exits 0. @@ -280,7 +280,7 @@ python -c "from handwriting_engine.benchmark.models import ProviderOutput, Strat ``` - cd "~/Developer/handwriting-engine" && python -c "from handwriting_engine.benchmark.models import ProviderOutput, StrategyResult, RunSummary; po = ProviderOutput(1,1,1,'gemini','single','text'); sr = StrategyResult('gemini','single',0.0,0.0,0.0,0.0,0.0,0.0,0,0.0,1); print(f'question_marker_rate={po.question_marker_rate} mean_marker_rate={sr.mean_marker_rate} OK')" && pytest tests/test_benchmark_db.py tests/test_benchmark_evaluate.py::TestEstimateCost tests/test_benchmark_evaluate.py::TestRunBenchmark -x -q + cd "/Users/user/Documents/Work & Projects/VSCode Projects/handwriting-engine" && python -c "from handwriting_engine.benchmark.models import ProviderOutput, StrategyResult, RunSummary; po = ProviderOutput(1,1,1,'gemini','single','text'); sr = StrategyResult('gemini','single',0.0,0.0,0.0,0.0,0.0,0.0,0,0.0,1); print(f'question_marker_rate={po.question_marker_rate} mean_marker_rate={sr.mean_marker_rate} OK')" && pytest tests/test_benchmark_db.py tests/test_benchmark_evaluate.py::TestEstimateCost tests/test_benchmark_evaluate.py::TestRunBenchmark -x -q ProviderOutput.question_marker_rate=None, StrategyResult.mean_marker_rate=0.0 confirmed. All existing TestRunBenchmark and TestEstimateCost tests still pass. diff --git a/.planning/phases/06-measurement-foundation/06-03-PLAN.md b/.planning/phases/06-measurement-foundation/06-03-PLAN.md index 1c3c80a..bca19a6 100644 --- a/.planning/phases/06-measurement-foundation/06-03-PLAN.md +++ b/.planning/phases/06-measurement-foundation/06-03-PLAN.md @@ -47,8 +47,8 @@ Output: evaluate.py computes and stores question_marker_rate per provider output -@~/.claude/get-shit-done/workflows/execute-plan.md -@~/.claude/get-shit-done/templates/summary.md +@/Users/user/.claude/get-shit-done/workflows/execute-plan.md +@/Users/user/.claude/get-shit-done/templates/summary.md @@ -201,7 +201,7 @@ Pass `question_marker_rate=_marker_rate` to `insert_provider_output()`. CRITICAL: The marker_rate lines must appear BEFORE `character_error_rate(result["text"], gt.text)` on the lines immediately following `insert_provider_output()`. Do NOT move or reorder the `character_error_rate()` call itself. - cd "~/Developer/handwriting-engine" && pytest tests/test_benchmark_evaluate.py::TestMarkerRate::test_marker_rate_from_raw_text tests/test_benchmark_evaluate.py::TestMarkerRate::test_marker_rate_clean_output tests/test_benchmark_evaluate.py::TestMarkerRate::test_marker_rate_computed_before_normalization tests/test_benchmark_evaluate.py::TestProvenanceCapture::test_provenance_columns_in_db -x -q + cd "/Users/user/Documents/Work & Projects/VSCode Projects/handwriting-engine" && pytest tests/test_benchmark_evaluate.py::TestMarkerRate::test_marker_rate_from_raw_text tests/test_benchmark_evaluate.py::TestMarkerRate::test_marker_rate_clean_output tests/test_benchmark_evaluate.py::TestMarkerRate::test_marker_rate_computed_before_normalization tests/test_benchmark_evaluate.py::TestProvenanceCapture::test_provenance_columns_in_db -x -q All four targeted TestMarkerRate tests GREEN. TestProvenanceCapture::test_provenance_columns_in_db GREEN. `pytest tests/test_benchmark_evaluate.py::TestRunBenchmark -x -q` still passes (no regressions). @@ -293,7 +293,7 @@ RunSummary( Note: `r.keys()` on a sqlite3.Row returns column names — use this check to handle DBs that haven't migrated yet (graceful degradation). - cd "~/Developer/handwriting-engine" && pytest tests/test_benchmark_evaluate.py::TestMarkerRate::test_marker_rate_in_report tests/test_benchmark_evaluate.py::TestProvenanceCapture::test_report_contains_provenance_header -x -q + cd "/Users/user/Documents/Work & Projects/VSCode Projects/handwriting-engine" && pytest tests/test_benchmark_evaluate.py::TestMarkerRate::test_marker_rate_in_report tests/test_benchmark_evaluate.py::TestProvenanceCapture::test_report_contains_provenance_header -x -q TestMarkerRate::test_marker_rate_in_report GREEN. TestProvenanceCapture::test_report_contains_provenance_header GREEN. `pytest tests/test_benchmark_evaluate.py -x -q` passes all previously passing tests (no regressions). diff --git a/.planning/phases/06-measurement-foundation/06-04-PLAN.md b/.planning/phases/06-measurement-foundation/06-04-PLAN.md index 0ff471c..0d7a05a 100644 --- a/.planning/phases/06-measurement-foundation/06-04-PLAN.md +++ b/.planning/phases/06-measurement-foundation/06-04-PLAN.md @@ -44,8 +44,8 @@ Output: cli.py with benchmark_calibrate_cmd, cost projection block in benchmark_ -@~/.claude/get-shit-done/workflows/execute-plan.md -@~/.claude/get-shit-done/templates/summary.md +@/Users/user/.claude/get-shit-done/workflows/execute-plan.md +@/Users/user/.claude/get-shit-done/templates/summary.md @@ -192,7 +192,7 @@ def benchmark_calibrate_cmd(samples, provider, db_path): Note: `\u03c3` is the unicode sigma character (σ) — use it in the f-string to produce "2σ" in the output. - cd "~/Developer/handwriting-engine" && pytest tests/test_benchmark_evaluate.py::TestCalibrateCommand -x -q + cd "/Users/user/Documents/Work & Projects/VSCode Projects/handwriting-engine" && pytest tests/test_benchmark_evaluate.py::TestCalibrateCommand -x -q All TestCalibrateCommand tests GREEN: test_calibrate_output_format, test_calibrate_undersample_warning, test_calibrate_no_samples_error. `handwriting-engine benchmark calibrate --help` shows the command exists with --samples and --provider options. @@ -309,7 +309,7 @@ Also update `run_benchmark()` in `evaluate.py` to accept `vocab_hints_off: int = 3. `handwriting-engine benchmark --help` — shows calibrate in the list of subcommands - cd "~/Developer/handwriting-engine" && pytest tests/test_benchmark_evaluate.py::TestCostProjection tests/test_benchmark_evaluate.py::TestCalibrateCommand -x -q && python -m handwriting_engine.cli benchmark --help 2>&1 | grep -E "calibrate|run" + cd "/Users/user/Documents/Work & Projects/VSCode Projects/handwriting-engine" && pytest tests/test_benchmark_evaluate.py::TestCostProjection tests/test_benchmark_evaluate.py::TestCalibrateCommand -x -q && python -m handwriting_engine.cli benchmark --help 2>&1 | grep -E "calibrate|run" All TestCostProjection tests GREEN (cost shown, --yes bypasses, decline exits 0). All TestCalibrateCommand tests GREEN. `benchmark --help` lists both calibrate and run. `benchmark run --help` shows --yes, --iam-partition, --vocab-hints-off flags. diff --git a/.planning/phases/07-iam-data-ingestion-sweep-infrastructure/07-01-PLAN.md b/.planning/phases/07-iam-data-ingestion-sweep-infrastructure/07-01-PLAN.md index ec82556..9e4e0c0 100644 --- a/.planning/phases/07-iam-data-ingestion-sweep-infrastructure/07-01-PLAN.md +++ b/.planning/phases/07-iam-data-ingestion-sweep-infrastructure/07-01-PLAN.md @@ -51,8 +51,8 @@ Output: 9 new RED stubs in `tests/test_benchmark_ingest.py` (class TestIAMIngest -@~/.claude/get-shit-done/workflows/execute-plan.md -@~/.claude/get-shit-done/templates/summary.md +@/Users/user/.claude/get-shit-done/workflows/execute-plan.md +@/Users/user/.claude/get-shit-done/templates/summary.md @@ -141,7 +141,7 @@ class TestIAMIngest: Do NOT touch any existing test classes or imports. - cd "~/Developer/handwriting-engine" && python -m pytest tests/test_benchmark_ingest.py::TestIAMIngest -x -q 2>&1 | tail -5 + cd "/Users/user/Documents/Work & Projects/VSCode Projects/handwriting-engine" && python -m pytest tests/test_benchmark_ingest.py::TestIAMIngest -x -q 2>&1 | tail -5 pytest reports 9 FAILED (all pytest.fail) and 0 ERROR. Existing TestHashFile and TestExtractPageNumber tests still PASS. @@ -228,7 +228,7 @@ class TestPerWriterReport: Do NOT modify any existing test classes or fixtures (`TestEstimateCost`, `TestRunBenchmark`, etc.). - cd "~/Developer/handwriting-engine" && python -m pytest tests/test_benchmark_evaluate.py::TestSweep tests/test_benchmark_evaluate.py::TestPerWriterReport -x -q 2>&1 | tail -5 + cd "/Users/user/Documents/Work & Projects/VSCode Projects/handwriting-engine" && python -m pytest tests/test_benchmark_evaluate.py::TestSweep tests/test_benchmark_evaluate.py::TestPerWriterReport -x -q 2>&1 | tail -5 pytest reports 8 FAILED (5 TestSweep + 3 TestPerWriterReport) and 0 ERROR. All existing TestEstimateCost, TestRunBenchmark, and other classes still PASS. @@ -239,7 +239,7 @@ Do NOT modify any existing test classes or fixtures (`TestEstimateCost`, `TestRu After both tasks: ``` -cd "~/Developer/handwriting-engine" +cd "/Users/user/Documents/Work & Projects/VSCode Projects/handwriting-engine" python -m pytest tests/test_benchmark_ingest.py tests/test_benchmark_evaluate.py -q 2>&1 | tail -10 ``` diff --git a/.planning/phases/07-iam-data-ingestion-sweep-infrastructure/07-02-PLAN.md b/.planning/phases/07-iam-data-ingestion-sweep-infrastructure/07-02-PLAN.md index 3328c27..cf060c0 100644 --- a/.planning/phases/07-iam-data-ingestion-sweep-infrastructure/07-02-PLAN.md +++ b/.planning/phases/07-iam-data-ingestion-sweep-infrastructure/07-02-PLAN.md @@ -49,8 +49,8 @@ Output: Two new functions in ingest.py (`parse_iam_lines`, `ingest_iam`) and one -@~/.claude/get-shit-done/workflows/execute-plan.md -@~/.claude/get-shit-done/templates/summary.md +@/Users/user/.claude/get-shit-done/workflows/execute-plan.md +@/Users/user/.claude/get-shit-done/templates/summary.md @@ -142,7 +142,7 @@ Import additions needed at top of ingest.py: - Path is already imported via pathlib - cd "~/Developer/handwriting-engine" && python -m pytest tests/test_benchmark_ingest.py::TestIAMIngest::test_parse_skips_comments tests/test_benchmark_ingest.py::TestIAMIngest::test_parse_filters_err tests/test_benchmark_ingest.py::TestIAMIngest::test_parse_extracts_fields tests/test_benchmark_ingest.py::TestIAMIngest::test_parse_replaces_pipes tests/test_benchmark_ingest.py::TestIAMIngest::test_parse_filters_partition tests/test_benchmark_ingest.py::TestIAMIngest::test_ingest_sets_category_and_student tests/test_benchmark_ingest.py::TestIAMIngest::test_ingest_inserts_ground_truth tests/test_benchmark_ingest.py::TestIAMIngest::test_ingest_iam_dedup -x -q 2>&1 | tail -5 + cd "/Users/user/Documents/Work & Projects/VSCode Projects/handwriting-engine" && python -m pytest tests/test_benchmark_ingest.py::TestIAMIngest::test_parse_skips_comments tests/test_benchmark_ingest.py::TestIAMIngest::test_parse_filters_err tests/test_benchmark_ingest.py::TestIAMIngest::test_parse_extracts_fields tests/test_benchmark_ingest.py::TestIAMIngest::test_parse_replaces_pipes tests/test_benchmark_ingest.py::TestIAMIngest::test_parse_filters_partition tests/test_benchmark_ingest.py::TestIAMIngest::test_ingest_sets_category_and_student tests/test_benchmark_ingest.py::TestIAMIngest::test_ingest_inserts_ground_truth tests/test_benchmark_ingest.py::TestIAMIngest::test_ingest_iam_dedup -x -q 2>&1 | tail -5 8 of 9 TestIAMIngest stubs are GREEN (test_cli_ingest_iam_command remains RED until Task 2). All existing TestHashFile and TestExtractPageNumber tests still PASS. @@ -225,7 +225,7 @@ Place this command after existing `benchmark ingest` / `benchmark ingest-dir` co Verify wiring: the command must be attached to the `benchmark` group (not the top-level cli group). - cd "~/Developer/handwriting-engine" && python -m pytest tests/test_benchmark_ingest.py::TestIAMIngest::test_cli_ingest_iam_command -x -q 2>&1 | tail -5 + cd "/Users/user/Documents/Work & Projects/VSCode Projects/handwriting-engine" && python -m pytest tests/test_benchmark_ingest.py::TestIAMIngest::test_cli_ingest_iam_command -x -q 2>&1 | tail -5 All 9 TestIAMIngest stubs are GREEN. `handwriting-engine benchmark ingest-iam --help` exits 0 and shows the command. Full ingest test suite: `pytest tests/test_benchmark_ingest.py -x -q` passes. @@ -234,7 +234,7 @@ Verify wiring: the command must be attached to the `benchmark` group (not the to ```bash -cd "~/Developer/handwriting-engine" +cd "/Users/user/Documents/Work & Projects/VSCode Projects/handwriting-engine" python -m pytest tests/test_benchmark_ingest.py -x -q 2>&1 | tail -5 python -m pytest tests/ -x -q 2>&1 | tail -5 python -m handwriting_engine.cli benchmark ingest-iam --help 2>&1 | head -10 diff --git a/.planning/phases/07-iam-data-ingestion-sweep-infrastructure/07-03-PLAN.md b/.planning/phases/07-iam-data-ingestion-sweep-infrastructure/07-03-PLAN.md index 387d53a..3f44d93 100644 --- a/.planning/phases/07-iam-data-ingestion-sweep-infrastructure/07-03-PLAN.md +++ b/.planning/phases/07-iam-data-ingestion-sweep-infrastructure/07-03-PLAN.md @@ -53,8 +53,8 @@ Output: Updated `_read_single()` and `run_benchmark()` in evaluate.py (new param -@~/.claude/get-shit-done/workflows/execute-plan.md -@~/.claude/get-shit-done/templates/summary.md +@/Users/user/.claude/get-shit-done/workflows/execute-plan.md +@/Users/user/.claude/get-shit-done/templates/summary.md @@ -187,7 +187,7 @@ result = _read_single( **Pitfall from research:** These are `read_page()` flags, NOT consensus strategy strings. Do not add them to the `strategies` list. They are passed directly to `_read_single()`. - cd "~/Developer/handwriting-engine" && python -m pytest tests/test_benchmark_evaluate.py::TestSweep::test_run_benchmark_accepts_line_level tests/test_benchmark_evaluate.py::TestSweep::test_run_benchmark_accepts_auto_retry -x -q 2>&1 | tail -5 + cd "/Users/user/Documents/Work & Projects/VSCode Projects/handwriting-engine" && python -m pytest tests/test_benchmark_evaluate.py::TestSweep::test_run_benchmark_accepts_line_level tests/test_benchmark_evaluate.py::TestSweep::test_run_benchmark_accepts_auto_retry -x -q 2>&1 | tail -5 Both line_level and auto_retry threading tests are GREEN. Existing TestRunBenchmark tests still PASS (backward-compatible). @@ -369,7 +369,7 @@ Place `benchmark sweep` after `benchmark run` command in cli.py (maintain logica **Wiring check:** Verify `run_sweep` is importable: `python -c "from handwriting_engine.benchmark.evaluate import run_sweep; print('OK')` - cd "~/Developer/handwriting-engine" && python -m pytest tests/test_benchmark_evaluate.py::TestSweep -x -q 2>&1 | tail -5 + cd "/Users/user/Documents/Work & Projects/VSCode Projects/handwriting-engine" && python -m pytest tests/test_benchmark_evaluate.py::TestSweep -x -q 2>&1 | tail -5 All 5 TestSweep stubs are GREEN. `benchmark sweep --help` exits 0 and shows the command. `from handwriting_engine.benchmark.evaluate import run_sweep` succeeds. @@ -378,7 +378,7 @@ Place `benchmark sweep` after `benchmark run` command in cli.py (maintain logica ```bash -cd "~/Developer/handwriting-engine" +cd "/Users/user/Documents/Work & Projects/VSCode Projects/handwriting-engine" python -m pytest tests/test_benchmark_evaluate.py::TestSweep -q 2>&1 | tail -5 python -m pytest tests/test_benchmark_evaluate.py -q 2>&1 | tail -5 python -c "from handwriting_engine.benchmark.evaluate import run_sweep, SWEEP_STRATEGIES; print(len(SWEEP_STRATEGIES), 'strategies')" diff --git a/.planning/phases/07-iam-data-ingestion-sweep-infrastructure/07-03-SUMMARY.md b/.planning/phases/07-iam-data-ingestion-sweep-infrastructure/07-03-SUMMARY.md new file mode 100644 index 0000000..7eec1fd --- /dev/null +++ b/.planning/phases/07-iam-data-ingestion-sweep-infrastructure/07-03-SUMMARY.md @@ -0,0 +1,103 @@ +--- +phase: 07-iam-data-ingestion-sweep-infrastructure +plan: 03 +subsystem: benchmark +tags: [iam, sweep, click, tdd, line-level, auto-retry] + +requires: + - phase: 07-01-iam-data-ingestion-sweep-infrastructure + provides: RED stub tests for TestSweep (5 stubs) that this plan turns GREEN + - phase: 07-02-iam-data-ingestion-sweep-infrastructure + provides: ingest_iam() populating samples.category='iam' rows that run_sweep() filters on + +provides: + - SWEEP_STRATEGIES — 5-strategy config list (baseline, self_correct, line_level, prompt_adapted, zoomed_verify) + - run_sweep(provider, db_path, yes, on_progress) — executes all 5 strategies, returns {name: run_id} dict + - run_benchmark line_level/auto_retry parameters (threaded through _run_benchmark_inner -> _read_single -> read_page) + - benchmark sweep CLI command with cost projection guardrail + +affects: + - 07-04 (per-writer report — consumes sweep run_ids to break down CER per writer) + - trained_correction (real-data retrain — uses sweep outputs as (vlm_text, ground_truth) training pairs) + +tech-stack: + added: [] + patterns: + - Strategy config table with kwargs dict — keeps run_sweep() body small, easy to extend + - IAM filter via samples.category='iam' (not a new param) — matches plan's must_have truth + - Cost projection at CLI layer; run_sweep() itself does not prompt — keeps library callable from non-CLI contexts + +key-files: + created: [] + modified: + - handwriting_engine/benchmark/evaluate.py + - handwriting_engine/cli.py + - tests/test_benchmark_evaluate.py + +key-decisions: + - "line_level and auto_retry threaded through the full call chain (_read_single signature, _run_benchmark_inner signature, run_benchmark signature) — backward-compatible defaults of False everywhere" + - "prompt_adapted strategy distinguished from baseline by leaving vocab_hints_off at default (0=hints ON), since prompt_adapter.py runs by default in read_page() — no special flag needed" + - "run_sweep() forwards on_progress to run_benchmark for per-strategy progress reporting in CLI" + - "Cost projection uses ~2000 input + ~200 output token estimate per sample per strategy (rough; matches benchmark run pattern)" + - "Empty-DB path: cost line still printed even when n_samples=0 (test contract — `test_sweep_cli_shows_cost`)" + +patterns-established: + - "Strategy table pattern: list of {name, label, kwargs} dicts — caller spreads kwargs into run_benchmark()" + - "IAM-only sweep: SQL filter `WHERE s.category='iam'` joined to ground_truths to skip un-transcribed IAM samples" + +requirements-completed: [IAM-02] + +duration: ~30min +completed: 2026-05-06 +--- + +# Phase 07 Plan 03: Sweep Infrastructure Summary + +**run_sweep() + benchmark sweep CLI: all 5 strategies executable in one command against IAM samples, with line_level/auto_retry threaded through run_benchmark for sweep parity. Turns 5 TestSweep RED stubs GREEN.** + +## Accomplishments + +- 5/5 `TestSweep` stubs turned GREEN +- `run_benchmark()` accepts `line_level=True` and `auto_retry=True`, threading both through to `_read_single()` -> `read_page()` +- `_run_benchmark_inner()` signature extended (kw-only) — backward-compatible +- `SWEEP_STRATEGIES` exported from `evaluate.py`: 5 entries (baseline, self_correct, line_level, prompt_adapted, zoomed_verify) +- `run_sweep()` filters IAM samples via `samples.category='iam'` and runs each strategy via `run_benchmark()`, returning `{strategy_name: run_id}` +- `benchmark sweep` CLI command registered with cost projection (`Estimated cost ~$X/strategy x 5 = ~$Y total`), warning on empty IAM DB, `--yes` to bypass confirmation, exit 0 on success listing all 5 run_ids + +## Verification + +```bash +# All 5 TestSweep tests pass: +pytest tests/test_benchmark_evaluate.py::TestSweep -q # 5 passed + +# Existing tests untouched (TestRunBenchmark, TestEstimateCost, TestReport, TestSmokeMode, TestProgressCallback, TestDrillDown, TestRegressionDetect): +pytest tests/test_benchmark_evaluate.py -q # 31 passed, 3 failed (TestPerWriterReport — 07-04 territory) + +# Imports clean: +python3 -c "from handwriting_engine.benchmark.evaluate import run_sweep, SWEEP_STRATEGIES; print(len(SWEEP_STRATEGIES))" # -> 5 + +# CLI registered: +python3 -m handwriting_engine.cli benchmark sweep --help # exit 0 +``` + +## Files Modified + +- `handwriting_engine/benchmark/evaluate.py` — added `line_level`/`auto_retry` parameters to `_read_single()`, `run_benchmark()`, `_run_benchmark_inner()`; threaded through to `read_page()`; appended `SWEEP_STRATEGIES` constant + `run_sweep()` function (~95 lines added) +- `handwriting_engine/cli.py` — added `benchmark_sweep` command with cost projection guardrail (~70 lines) +- `tests/test_benchmark_evaluate.py` — replaced 5 `pytest.fail()` stubs in `TestSweep` with real assertions using `_read_single`/`_available_providers` mocks and `CliRunner` + +## Decisions Made + +- **Backward-compatible threading.** `line_level=False` and `auto_retry=False` defaults everywhere; existing tests untouched. +- **Strategy table.** A list of `{name, label, kwargs}` dicts keeps `run_sweep()` to ~15 lines and lets future strategies be added by appending one entry. +- **CLI prompts cost; library does not.** `run_sweep()` is callable from notebooks/scripts without prompting; the CLI command owns the confirm flow. +- **IAM filter at SQL.** `WHERE s.category='iam'` joined to `ground_truths` skips IAM samples without transcriptions — no separate param. + +## Out of Scope (handled by 07-04) + +- `TestPerWriterReport` (3 stubs) remains RED — that's IAM-03's plan. + +## Unblocks + +- **07-04 (per-writer report).** Sweep run_ids are now produced; per-writer breakdown can group by `samples.student`. +- **Trained corrector real-data retrain.** Once user runs `benchmark ingest-iam` + `benchmark sweep`, the resulting (provider_outputs.output_text, ground_truths.text) pairs feed `trained_correction.dataset.from_benchmark_db()` for the v2 fine-tune that resolves the synthetic-only hallucination failure mode documented in `trained_correction/EVAL-RESULTS.md`. diff --git a/.planning/phases/07-iam-data-ingestion-sweep-infrastructure/07-04-PLAN.md b/.planning/phases/07-iam-data-ingestion-sweep-infrastructure/07-04-PLAN.md index caf9536..142d141 100644 --- a/.planning/phases/07-iam-data-ingestion-sweep-infrastructure/07-04-PLAN.md +++ b/.planning/phases/07-iam-data-ingestion-sweep-infrastructure/07-04-PLAN.md @@ -47,8 +47,8 @@ Output: New `generate_per_writer_report()` function in report.py, `--per-writer` -@~/.claude/get-shit-done/workflows/execute-plan.md -@~/.claude/get-shit-done/templates/summary.md +@/Users/user/.claude/get-shit-done/workflows/execute-plan.md +@/Users/user/.claude/get-shit-done/templates/summary.md @@ -179,7 +179,7 @@ def generate_per_writer_report( Export `generate_per_writer_report` — add to `__all__` if one exists in report.py. - cd "~/Developer/handwriting-engine" && python -m pytest tests/test_benchmark_evaluate.py::TestPerWriterReport::test_per_writer_report_groups_by_student tests/test_benchmark_evaluate.py::TestPerWriterReport::test_per_writer_report_no_writers -x -q 2>&1 | tail -5 + cd "/Users/user/Documents/Work & Projects/VSCode Projects/handwriting-engine" && python -m pytest tests/test_benchmark_evaluate.py::TestPerWriterReport::test_per_writer_report_groups_by_student tests/test_benchmark_evaluate.py::TestPerWriterReport::test_per_writer_report_no_writers -x -q 2>&1 | tail -5 Two of the 3 TestPerWriterReport stubs are GREEN (test_report_cli_per_writer_flag remains RED until Task 2). generate_per_writer_report is importable from handwriting_engine.benchmark.report. @@ -226,7 +226,7 @@ Key constraints: Wiring check: `python -c "from handwriting_engine.benchmark.report import generate_per_writer_report; print('OK')"` must succeed. - cd "~/Developer/handwriting-engine" && python -m pytest tests/test_benchmark_evaluate.py::TestPerWriterReport -x -q 2>&1 | tail -5 + cd "/Users/user/Documents/Work & Projects/VSCode Projects/handwriting-engine" && python -m pytest tests/test_benchmark_evaluate.py::TestPerWriterReport -x -q 2>&1 | tail -5 All 3 TestPerWriterReport stubs are GREEN. `benchmark report --help` shows `--per-writer` option. Full suite: `pytest tests/test_benchmark_evaluate.py -q` passes. @@ -235,7 +235,7 @@ Wiring check: `python -c "from handwriting_engine.benchmark.report import genera ```bash -cd "~/Developer/handwriting-engine" +cd "/Users/user/Documents/Work & Projects/VSCode Projects/handwriting-engine" python -m pytest tests/test_benchmark_evaluate.py::TestPerWriterReport -q 2>&1 | tail -5 python -m pytest tests/test_benchmark_evaluate.py -q 2>&1 | tail -5 python -c "from handwriting_engine.benchmark.report import generate_per_writer_report; print('OK')" diff --git a/.planning/phases/07-iam-data-ingestion-sweep-infrastructure/07-04-SUMMARY.md b/.planning/phases/07-iam-data-ingestion-sweep-infrastructure/07-04-SUMMARY.md new file mode 100644 index 0000000..fbcd97c --- /dev/null +++ b/.planning/phases/07-iam-data-ingestion-sweep-infrastructure/07-04-SUMMARY.md @@ -0,0 +1,87 @@ +--- +phase: 07-iam-data-ingestion-sweep-infrastructure +plan: 04 +subsystem: benchmark +tags: [iam, report, per-writer, click, tdd] + +requires: + - phase: 07-01-iam-data-ingestion-sweep-infrastructure + provides: RED stub tests for TestPerWriterReport (3 stubs) that this plan turns GREEN + - phase: 07-02-iam-data-ingestion-sweep-infrastructure + provides: ingest_iam() populating samples.student='iam-writer-XXX' rows that the SQL groups by + +provides: + - generate_per_writer_report(run_id, db_path) — formatted per-writer CER table (Writer, Mean CER, Min CER, Max CER, N) + - benchmark report --per-writer flag wiring + +affects: + - Sweep interpretation — reveals whether a strategy CER gain is consistent across writers or driven by a few easy ones + - 07-03 sweep run_ids — now reportable per writer + +tech-stack: + added: [] + patterns: + - SQL aggregation: AVG/MIN/MAX(em.cer) GROUP BY s.student (excluding empty student) + - Empty-state handling: explicit message string rather than empty table when no writer-tagged rows exist + +key-files: + created: [] + modified: + - handwriting_engine/benchmark/report.py + - handwriting_engine/cli.py + - tests/test_benchmark_evaluate.py + +key-decisions: + - "SQL filter excludes student IS NULL OR student='' — non-IAM samples have no writer data and would pollute the table" + - "Empty-state message includes 'No writer data' (test contract) and a tip pointing to `benchmark ingest-iam`" + - "Sorted by mean_cer DESC — hardest writers first (most actionable view)" + - "--per-writer branches early in benchmark_report_cmd; existing report logic untouched when flag absent" + +patterns-established: + - "Per-writer SQL: provider_outputs JOIN eval_metrics JOIN samples, GROUP BY s.student" + +requirements-completed: [IAM-03] + +duration: ~10min +completed: 2026-05-06 +--- + +# Phase 07 Plan 04: Per-Writer Report Summary + +**generate_per_writer_report() + benchmark report --per-writer: per-writer CER breakdown for any run, revealing whether a strategy's gain is consistent across writers. Turns 3 TestPerWriterReport RED stubs GREEN.** + +## Accomplishments + +- 3/3 `TestPerWriterReport` stubs turned GREEN +- `generate_per_writer_report()` exported from `handwriting_engine.benchmark.report` +- Per-writer table: `Writer | Mean CER | Min CER | Max CER | N`, sorted hardest-first +- Empty-writer case returns explanatory message (not crash, not empty) +- `benchmark report --per-writer` flag wired up; visible in `--help` +- Hidden `--db-path` option added to report command for testability + +## Verification + +```bash +pytest tests/test_benchmark_evaluate.py::TestPerWriterReport -q # 3 passed +pytest tests/test_benchmark_evaluate.py -q # 34 passed +pytest tests/ -q --ignore=tests/test_iam_real_data.py # 525 passed, 2 skipped, 1 xfailed +python3 -c "from handwriting_engine.benchmark.report import generate_per_writer_report; print('OK')" +python3 -m handwriting_engine.cli benchmark report --help | grep per-writer +``` + +## Files Modified + +- `handwriting_engine/benchmark/report.py` — added `generate_per_writer_report()` (~60 lines) +- `handwriting_engine/cli.py` — added `--per-writer` flag + early-return branch on `benchmark_report_cmd` +- `tests/test_benchmark_evaluate.py` — replaced 3 `pytest.fail()` stubs with real assertions (writer-grouping, empty-state, CLI flag) + +## Phase 07 Now Complete + +All 4 plans landed. Phase 07 (IAM Data Ingestion + Sweep Infrastructure) ships IAM-01, IAM-02, IAM-03. + +**What this unlocks:** +1. User can download IAM, run `benchmark ingest-iam`, then `benchmark sweep` to populate the DB with one run_id per strategy. +2. `benchmark report --per-writer` immediately shows whether a strategy's gain is consistent across writers. +3. The sweep run outputs become real-data training pairs for `trained_correction.dataset.from_benchmark_db()` — the v2 corrector retrain that resolves the synthetic-only hallucination failure mode. + +**Next phase:** Phase 08 (Statistics Layer) — Wilcoxon p-values + bootstrap CIs on `benchmark compare`. Out of scope for this session. diff --git a/.planning/phases/08-statistics-layer/08-01-SUMMARY.md b/.planning/phases/08-statistics-layer/08-01-SUMMARY.md new file mode 100644 index 0000000..ef55cf3 --- /dev/null +++ b/.planning/phases/08-statistics-layer/08-01-SUMMARY.md @@ -0,0 +1,62 @@ +--- +phase: 08-statistics-layer +plan: "01" +subsystem: benchmark +tags: [stats, wilcoxon, bootstrap, cohen, no-scipy, tdd] + +requires: + - phase: 07 + provides: get_run_results returning per-sample CER rows that paired analysis can consume + +provides: + - handwriting_engine/benchmark/stats.py (new module) + - wilcoxon_signed_rank(a, b) -> WilcoxonResult{statistic, p_value, z, n} + - bootstrap_ci(values, confidence, n_iterations, seed) -> (lo, hi) + - cohens_r(z, n) -> float + - compare_runs() emits a "stats:" + "CI95:" block per (provider, strategy) when n_paired >= 10 + +affects: + - benchmark compare RUN_A RUN_B — output now includes paired Wilcoxon, Cohen's r, and bootstrap CIs + - Future Phase 9 recommendation logic (RPT-02) — composite-score "stability" component can lean on these stats + +tech-stack: + added: [] + patterns: + - Hand-rolled Wilcoxon with normal-approximation + continuity correction (no scipy dep) + - Average-rank tie handling with explicit tie-correction term (n^3 - n) / 48 in variance + - Percentile-method bootstrap with seedable RNG for deterministic test output + - Pairing by sample_id intersection across the two runs (only shared samples count toward n) + +key-files: + created: + - handwriting_engine/benchmark/stats.py + - tests/test_benchmark_stats.py + modified: + - handwriting_engine/benchmark/report.py + +key-decisions: + - "Hand-roll Wilcoxon and bootstrap rather than add scipy. Trade ~150 LOC of well-tested math for not pulling a 30 MB scientific stack into a project whose existing pattern (Levenshtein in postprocess.py) is hand-rolled small math." + - "Gate stats block at n_paired >= 10. Below this, normal-approximation Wilcoxon is rough and bootstrap CIs are dominated by sampling noise — better to print nothing than mislead. Aligns with the success-criterion threshold." + - "Pair by sample_id intersection. Only samples that exist in BOTH runs count. Avoids comparing apples to oranges when run sample sets diverge." + - "Bootstrap CI seeded by run_id. Deterministic output for the same run pair without exposing a CLI flag." + - "Two-sided p-value with continuity correction. Matches scipy's wilcoxon(zero_method='wilcox') default; the continuity correction prevents over-rejecting the null on small n." + +patterns-established: + - "Phase 8+ statistics live in a single benchmark/stats.py module; report.py imports from it. Keeps math out of presentation logic and makes the math reusable for Phase 9 / RPT-02." + +verification: + unit_coverage: + - 29 tests in tests/test_benchmark_stats.py for the three primitives plus a paired end-to-end scenario + - 3 integration tests exercising compare_runs() against a synthetic two-run DB + criterion_status: + - "STAT-01 (#1): Wilcoxon p-value + Cohen's r appear in compare_runs output when n>=10 — IMPLEMENTED, verified on synthetic data; end-to-end IAM verification gated on user IAM download (see .planning/NEXT-STEPS.md)" + - "STAT-02 (#2): 95% bootstrap CIs for both runs appear in compare_runs output — IMPLEMENTED, verified on synthetic data; same IAM gate" + pre_existing_failures: + - tests/test_enhance.py (cv2 missing in this venv) + - tests/test_trained_correction.py::TestConfidenceGate (transformers/torch missing in this venv) + +out_of_scope: + - "scipy.stats.wilcoxon equivalence audit. We hand-rolled to avoid the dep; if a future user needs scipy parity to ~6 decimals, swap the body of wilcoxon_signed_rank for scipy.stats.wilcoxon — the public WilcoxonResult shape stays." + - "Two-sample (unpaired) tests. The contract is paired comparison of CERs on the same samples; cross-sample-set comparison is a separate concern." + - "BCa bootstrap. Percentile method is sufficient for displayed CIs at this sample size; BCa correction for skew is a v2 concern." + - "STAT-03 (McNemar's), STAT-04 (per-character confusion). v4.0 deferred per REQUIREMENTS.md." diff --git a/.planning/phases/09-final-sweep-recommendation-baseline-lock/09-01-SUMMARY.md b/.planning/phases/09-final-sweep-recommendation-baseline-lock/09-01-SUMMARY.md new file mode 100644 index 0000000..6f28737 --- /dev/null +++ b/.planning/phases/09-final-sweep-recommendation-baseline-lock/09-01-SUMMARY.md @@ -0,0 +1,54 @@ +--- +phase: 09-final-sweep-recommendation-baseline-lock +plan: "01" +subsystem: benchmark +tags: [schema-migration, regression-detection, baseline, cli, tdd] + +requires: + - phase: 06 + provides: schema_version migration plumbing reused for v6 + - phase: 07 + provides: detect_regressions's run-history primitives (list_runs, get_run_results) + +provides: + - Schema v6: ALTER TABLE runs ADD COLUMN is_baseline INTEGER DEFAULT 0 + - benchmark/db.py:set_baseline(conn, run_id) — at-most-one invariant + - benchmark/db.py:get_baseline_run_id(conn) -> int | None + - benchmark/models.py:RunSummary.is_baseline field + - benchmark/report.py:detect_regressions retargeted to pinned baseline + - CLI: benchmark set-baseline RUN_ID + +affects: + - All subsequent regression checks (Phase 9 onward) — anchor is now the pinned run, not the immediately-preceding run + - list_runs() consumers — RunSummary now carries is_baseline (0 or 1) + +tech-stack: + added: [] + patterns: + - At-most-one invariant via UPDATE-clear + UPDATE-set (no partial-unique index needed for sqlite portability) + - Fallback to runs[-2] when no baseline pinned, preserving pre-Phase-9 behavior on fresh DBs + - Self-compare guard: when current run IS the baseline, fall through to runs[-2] to avoid no-op + +key-files: + created: + - tests/test_benchmark_baseline.py + modified: + - handwriting_engine/benchmark/db.py + - handwriting_engine/benchmark/models.py + - handwriting_engine/benchmark/report.py + - handwriting_engine/cli.py + +key-decisions: + - "Fallback to runs[-2] when no baseline pinned. Preserves pre-Phase-9 behavior so existing test fixtures keep working without retroactive baseline pins." + - "Self-compare guard. If `detect_regressions(run_id=X)` is called and X is the pinned baseline, comparing against itself is a no-op — fall through to the prior run instead." + - "Atomic UPDATE pattern, not partial unique index. SQLite supports the index but sqlite3 module behavior across versions is uneven; the UPDATE-clear + UPDATE-set pattern is portable and explicit." + - "ValueError on unknown run_id. set_baseline refuses to silently no-op when the user passes a typo'd ID." + +verification: + unit_coverage: + - 12 tests in tests/test_benchmark_baseline.py + criterion_status: + - "RPT-01: IMPLEMENTED. Schema v6 column durable across reopen verified." + pre_existing_failures: + - tests/test_enhance.py (cv2 missing) + - tests/test_trained_correction.py::TestConfidenceGate (transformers/torch missing) diff --git a/.planning/phases/09-final-sweep-recommendation-baseline-lock/09-02-SUMMARY.md b/.planning/phases/09-final-sweep-recommendation-baseline-lock/09-02-SUMMARY.md new file mode 100644 index 0000000..1a107ca --- /dev/null +++ b/.planning/phases/09-final-sweep-recommendation-baseline-lock/09-02-SUMMARY.md @@ -0,0 +1,46 @@ +--- +phase: 09-final-sweep-recommendation-baseline-lock +plan: "02" +subsystem: benchmark +tags: [recommend, composite-score, normalization, cli] + +requires: + - phase: 09-01 + provides: schema v6 / list_runs surface (is_baseline indirectly informs future scoring versions) + - phase: 07 + provides: get_run_results for per-(provider, strategy) aggregation + +provides: + - benchmark/report.py:recommend_strategy(db_path) -> str + - CLI: benchmark recommend + - _RECOMMEND_W_CER / _W_COST / _W_STAB constants — single source of truth for the 70/15/15 weights + +affects: + - Strategy selection workflow — replaces ad-hoc CER staring with a normalized composite score + - Future RPT-related decisions can lean on the same composite score + +tech-stack: + added: [] + patterns: + - Min-max normalization within candidate set (per-component) + - Stability = 1 - normalized(across-run-mean stdev). Single-run candidates get the median stability score (neutral) + - Pre-Phase-9 schema bumped: CURRENT_SCHEMA_VERSION 5 -> 6 to silence the spurious "migration failed" warning that fresh DBs were emitting because the base _SCHEMA_SQL already includes is_baseline + +key-files: + created: + - tests/test_benchmark_recommend.py + modified: + - handwriting_engine/benchmark/report.py + - handwriting_engine/benchmark/db.py (CURRENT_SCHEMA_VERSION bump) + - handwriting_engine/cli.py + +key-decisions: + - "Hand the median to single-run candidates. They can't measure across-run variance; penalizing or rewarding them based on absent data would be noise. Median stability is the neutral choice." + - "Min-max normalize per-component. Different metrics live on different scales; normalizing to [0, 1] before weighting is the standard composite-score recipe and means the weights mean what they look like." + - "Bump CURRENT_SCHEMA_VERSION to 6 even though Plan 09-01 already added the migration. Without this bump, fresh DBs (whose base _SCHEMA_SQL already has is_baseline) log a spurious 'migration failed' warning when v6 ALTER tries to add a column that already exists. The warning was harmless but noisy." + +verification: + unit_coverage: + - 9 tests in tests/test_benchmark_recommend.py + criterion_status: + - "RPT-02: IMPLEMENTED on synthetic data. End-to-end verification against multi-strategy IAM sweep gated on user IAM download." diff --git a/.planning/phases/09-final-sweep-recommendation-baseline-lock/09-03-SUMMARY.md b/.planning/phases/09-final-sweep-recommendation-baseline-lock/09-03-SUMMARY.md new file mode 100644 index 0000000..93e6294 --- /dev/null +++ b/.planning/phases/09-final-sweep-recommendation-baseline-lock/09-03-SUMMARY.md @@ -0,0 +1,46 @@ +--- +phase: 09-final-sweep-recommendation-baseline-lock +plan: "03" +subsystem: benchmark +tags: [ingest, ground-truth, lab, click-edit, cli, tdd] + +requires: + - phase: 07 + provides: hash_file, insert_sample, get_sample_by_hash patterns reused for dedup + +provides: + - benchmark/ingest.py:ingest_lab(directory, *, student, prompt_fn, + db_path, use_vlm_suggestion, vlm_provider) -> dict + - CLI: benchmark ingest-lab DIRECTORY [--student S] [--with-suggestion] [--vlm-provider P] + - Ground-truth rows inserted with source='lab-grader' and the student tag stored as `author` for provenance + +affects: + - Production-distribution test set is now collectable without IAM dependency + - benchmark recommend / compare can run against lab samples once they have GT + - Future graders (S4 / labgrader bridge) can read these GTs as the trusted reference + +tech-stack: + added: [] + patterns: + - prompt_fn dependency injection: production CLI uses click.edit, tests inject deterministic callables + - Resumable: dedup by file hash; samples with existing ground_truth are silently skipped + - VLM suggestion is opt-in via --with-suggestion to keep cost predictable; failures degrade to empty suggestion rather than aborting the workflow + +key-files: + created: + - tests/test_benchmark_ingest_lab.py + modified: + - handwriting_engine/benchmark/ingest.py + - handwriting_engine/cli.py + +key-decisions: + - "prompt_fn callable, not a hardcoded EDITOR call. Lets tests run without spawning $EDITOR and lets a future GUI / web frontend reuse the same workflow function." + - "Whitespace-only return treated as skip. The user clearing the buffer signals 'I can't read this' just as clearly as click.edit returning None." + - "Sample inserted BEFORE prompt, not after. If the user skips, the sample row still exists so they can revisit it later via `benchmark transcribe`. Re-running ingest-lab won't re-create the row (hash dedup) but also won't re-prompt unless GT is missing." + - "VLM failures swallowed with warning, not raised. The point of ingest-lab is capturing GT; a transient VLM outage shouldn't block the workflow — the user just types the transcription manually." + +verification: + unit_coverage: + - 10 tests in tests/test_benchmark_ingest_lab.py + criterion_status: + - "RPT-03: IMPLEMENTED. Includes resume semantics, VLM degradation, and CLI." diff --git a/handwriting_engine/benchmark/db.py b/handwriting_engine/benchmark/db.py index 7d3a590..2ee3400 100644 --- a/handwriting_engine/benchmark/db.py +++ b/handwriting_engine/benchmark/db.py @@ -20,7 +20,7 @@ ) DEFAULT_DB_PATH = Path.home() / ".handwriting-engine" / "benchmark.db" -CURRENT_SCHEMA_VERSION = 4 +CURRENT_SCHEMA_VERSION = 6 logger = logging.getLogger(__name__) @@ -73,7 +73,8 @@ model_version TEXT DEFAULT NULL, iam_partition TEXT DEFAULT NULL, norm_flags TEXT DEFAULT NULL, - vocab_hints_off INTEGER DEFAULT 0 + vocab_hints_off INTEGER DEFAULT 0, + is_baseline INTEGER DEFAULT 0 ); CREATE TABLE IF NOT EXISTS provider_outputs ( @@ -108,6 +109,17 @@ ); CREATE INDEX IF NOT EXISTS idx_em_output ON eval_metrics(provider_output_id); +CREATE TABLE IF NOT EXISTS corrections ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + sample_id INTEGER NOT NULL REFERENCES samples(id), + ground_truth_id INTEGER NOT NULL REFERENCES ground_truths(id), + original_text TEXT NOT NULL, + confidence REAL, + source TEXT NOT NULL, + created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')) +); +CREATE INDEX IF NOT EXISTS idx_corrections_sample ON corrections(sample_id); + CREATE TABLE IF NOT EXISTS schema_version ( version INTEGER PRIMARY KEY ); @@ -137,6 +149,25 @@ ALTER TABLE provider_outputs ADD COLUMN question_marker_rate REAL DEFAULT NULL; UPDATE schema_version SET version = 4; """, + # v5: Add corrections table for instructor-corrected transcriptions (S4) + 5: """ + CREATE TABLE IF NOT EXISTS corrections ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + sample_id INTEGER NOT NULL REFERENCES samples(id), + ground_truth_id INTEGER NOT NULL REFERENCES ground_truths(id), + original_text TEXT NOT NULL, + confidence REAL, + source TEXT NOT NULL, + created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ', 'now')) + ); + CREATE INDEX IF NOT EXISTS idx_corrections_sample ON corrections(sample_id); + UPDATE schema_version SET version = 5; + """, + # v6: Add is_baseline flag to runs (Phase 9 / RPT-01) + 6: """ + ALTER TABLE runs ADD COLUMN is_baseline INTEGER DEFAULT 0; + UPDATE schema_version SET version = 6; + """, } @@ -414,10 +445,40 @@ def list_runs(conn: sqlite3.Connection) -> list[RunSummary]: iam_partition=r["iam_partition"] if "iam_partition" in keys else None, norm_flags=r["norm_flags"] if "norm_flags" in keys else None, vocab_hints_off=r["vocab_hints_off"] if "vocab_hints_off" in keys else 0, + is_baseline=r["is_baseline"] if "is_baseline" in keys else 0, )) return results +# --- Baseline pinning (Phase 9 / RPT-01) --- + + +def set_baseline(conn: sqlite3.Connection, run_id: int) -> None: + """Pin a run as the regression-detection anchor. + + Atomically clears the flag on every other run, then sets it on the + target. Multiple baselines is a footgun (which one does + detect_regressions compare against?), so we enforce at-most-one here + rather than via a partial-unique index. + + Raises ValueError if the run does not exist. + """ + row = conn.execute("SELECT 1 FROM runs WHERE id = ?", (run_id,)).fetchone() + if row is None: + raise ValueError(f"run {run_id} not found") + conn.execute("UPDATE runs SET is_baseline = 0") + conn.execute("UPDATE runs SET is_baseline = 1 WHERE id = ?", (run_id,)) + conn.commit() + + +def get_baseline_run_id(conn: sqlite3.Connection) -> int | None: + """Return the pinned baseline run_id, or None if no run is pinned.""" + row = conn.execute( + "SELECT id FROM runs WHERE is_baseline = 1 ORDER BY id DESC LIMIT 1" + ).fetchone() + return row["id"] if row else None + + # --- Provider Outputs --- @@ -498,3 +559,59 @@ def get_latest_run_id(conn: sqlite3.Connection) -> int | None: """Get the most recent run ID.""" row = conn.execute("SELECT id FROM runs ORDER BY id DESC LIMIT 1").fetchone() return row["id"] if row else None + + +# --- Corrections (S4: instructor feedback loop) --- + + +def record_correction( + conn: sqlite3.Connection, + *, + image_path: str, + writer_id: str, + corrected_text: str, + original_vlm_text: str, + confidence: float, + source: str = "labgrader", +) -> int: + """Idempotently record an instructor-corrected transcription. + + Behavior with identical args called twice: + - One sample row (deduped by image content hash). + - One ground_truth row (deduped by sample_id + text). + - Two corrections rows (history grows; each call records a confirmation). + + Returns the corrections row id from this call. + """ + from handwriting_engine.benchmark.ingest import hash_file + + image_hash = hash_file(image_path) + + sample = get_sample_by_hash(conn, image_hash) + if sample is None: + sample_id = insert_sample( + conn, + image_path=image_path, + image_hash=image_hash, + student=writer_id, + ) + else: + sample_id = sample.id + + gt_row = conn.execute( + "SELECT id FROM ground_truths WHERE sample_id = ? AND text = ? ORDER BY id DESC LIMIT 1", + (sample_id, corrected_text), + ).fetchone() + if gt_row is None: + gt_id = insert_ground_truth(conn, sample_id, corrected_text, source=source) + else: + gt_id = gt_row["id"] + + cur = conn.execute( + """INSERT INTO corrections + (sample_id, ground_truth_id, original_text, confidence, source) + VALUES (?, ?, ?, ?, ?)""", + (sample_id, gt_id, original_vlm_text, confidence, source), + ) + conn.commit() + return cur.lastrowid diff --git a/handwriting_engine/benchmark/evaluate.py b/handwriting_engine/benchmark/evaluate.py index b0feed6..54f4a88 100644 --- a/handwriting_engine/benchmark/evaluate.py +++ b/handwriting_engine/benchmark/evaluate.py @@ -87,6 +87,8 @@ def _read_single( image_path: str, provider: str, domain: str, auto_enhance: bool = False, inject_lessons: bool = False, enhance_strategy: str | None = None, + line_level: bool = False, + auto_retry: bool = False, ) -> dict: """Read a single image with one provider. Returns result dict.""" from handwriting_engine.vision import read_page @@ -125,6 +127,8 @@ def _read_single( text = read_page( actual_path, domain=domain, provider=provider, inject_lessons=inject_lessons, + line_level=line_level, + auto_retry=auto_retry, ) penalty = text.count("[?]") * 0.05 + text.count("[illegible") * 0.1 confidence = max(0.0, min(0.95, 1.0 - penalty)) @@ -217,6 +221,8 @@ def run_benchmark( iam_partition: str | None = None, vocabulary_hints: list[str] | None = None, vocab_hints_off: int = 0, + line_level: bool = False, + auto_retry: bool = False, ) -> int: """Execute a full benchmark run. @@ -252,6 +258,8 @@ def run_benchmark( iam_partition=iam_partition, vocabulary_hints=vocabulary_hints, vocab_hints_off=vocab_hints_off, + line_level=line_level, + auto_retry=auto_retry, ) finally: conn.close() @@ -272,6 +280,8 @@ def _run_benchmark_inner( iam_partition: str | None = None, vocabulary_hints: list[str] | None = None, vocab_hints_off: int = 0, + line_level: bool = False, + auto_retry: bool = False, ) -> int: """Inner benchmark logic with connection managed by caller.""" # Resolve providers @@ -332,7 +342,12 @@ def _run_benchmark_inner( # Single-provider reads for provider in providers: - result = _read_single(sample.image_path, provider, domain, auto_enhance, inject_lessons, enhance_strategy) + result = _read_single( + sample.image_path, provider, domain, + auto_enhance, inject_lessons, enhance_strategy, + line_level=line_level, + auto_retry=auto_retry, + ) # Compute marker rate from raw text BEFORE any normalization raw_text = result["text"] marker_rate = _compute_marker_rate(raw_text) if raw_text else None @@ -519,3 +534,82 @@ def _select_smoke_samples(conn, samples: list, limit: int = 3) -> list: selected.extend(remaining[: limit - len(selected)]) return selected + + +SWEEP_STRATEGIES = [ + { + "name": "baseline", + "label": "sweep:baseline", + "kwargs": {"strategies": [], "vocab_hints_off": 1, "auto_enhance": False}, + }, + { + "name": "self_correct", + "label": "sweep:self_correct", + "kwargs": {"strategies": ["self_correct"]}, + }, + { + "name": "line_level", + "label": "sweep:line_level", + "kwargs": {"strategies": [], "line_level": True}, + }, + { + "name": "prompt_adapted", + "label": "sweep:prompt_adapted", + # prompt_adapter is applied by default in read_page(); distinguished from + # baseline by leaving vocab_hints_off at default (0 = hints ON). + "kwargs": {"strategies": []}, + }, + { + "name": "zoomed_verify", + "label": "sweep:zoomed_verify", + "kwargs": {"strategies": [], "auto_retry": True}, + }, +] + + +def run_sweep( + provider: str = "gemini", + db_path: Path | str | None = None, + yes: bool = False, + on_progress: Callable[[int, int, str], None] | None = None, +) -> dict[str, int]: + """Execute all 5 sweep strategies against IAM samples (IAM-02). + + Fetches IAM sample IDs (samples.category='iam' with ground truth) from the + DB and passes them to run_benchmark() once per strategy. Returns a dict + mapping strategy name to run_id. + + Args: + provider: Provider used for all strategies. + db_path: Override database path. + yes: Reserved for future per-strategy confirmation; CLI guards cost upstream. + on_progress: Optional progress callback forwarded to run_benchmark. + + Returns: + Dict {strategy_name: run_id} with exactly 5 keys matching SWEEP_STRATEGIES. + """ + conn = get_connection(db_path) + try: + rows = conn.execute( + """SELECT DISTINCT s.id AS id FROM samples s + JOIN ground_truths gt ON gt.sample_id = s.id + WHERE s.category = 'iam' + ORDER BY s.id""" + ).fetchall() + sample_ids = [r["id"] for r in rows] + finally: + conn.close() + + run_ids: dict[str, int] = {} + for config in SWEEP_STRATEGIES: + run_id = run_benchmark( + label=config["label"], + providers=[provider], + sample_ids=sample_ids if sample_ids else None, + db_path=db_path, + on_progress=on_progress, + **config["kwargs"], + ) + run_ids[config["name"]] = run_id + + return run_ids diff --git a/handwriting_engine/benchmark/ingest.py b/handwriting_engine/benchmark/ingest.py index a5bf518..56ed824 100644 --- a/handwriting_engine/benchmark/ingest.py +++ b/handwriting_engine/benchmark/ingest.py @@ -50,6 +50,126 @@ def _extract_page_number(filename: str) -> int: return int(numbers[-1]) if numbers else 0 +def ingest_lab( + directory: str | Path, + *, + student: str = "", + prompt_fn=None, + db_path: Path | str | None = None, + use_vlm_suggestion: bool = False, + vlm_provider: str = "gemini", +) -> dict: + """Guided ingest of student lab notebook images with ground-truth capture. + + For each image in the directory: + 1. Hash and dedup. If a sample already has ground truth, skip silently. + 2. Insert the sample if new (category='lab'). + 3. Optionally pre-fill a VLM suggestion via a single-provider read. + 4. Hand control to `prompt_fn(image_path, suggestion) -> str | None`. + - Return a non-empty string: stored as ground truth, source='lab-grader'. + - Return None or empty string: skip — sample stays in DB without GT. + 5. Continue to the next image until done. + + Args: + directory: Folder of student lab notebook images (jpg/png/etc). + student: Author / student tag stored on each sample (also recorded + on each ground_truth row's `author` column for provenance). + prompt_fn: Callable returning the transcription text or None. + Mandatory in non-interactive use; CLI wrapper supplies one + backed by click.edit() with an EDITOR fallback to click.prompt. + db_path: Override path. + use_vlm_suggestion: When True, run a single VLM read per image and + pass that text as the `suggestion` argument to prompt_fn. Costs + ~$0.0005-0.005 per image depending on provider. + vlm_provider: Provider used when use_vlm_suggestion is True. + + Returns: + Dict with counts: {'annotated', 'skipped_existing_gt', 'skipped_user', + 'newly_added', 'errors'}. + """ + if prompt_fn is None: + raise ValueError("ingest_lab requires a prompt_fn callable") + + directory = Path(directory) + if not directory.is_dir(): + raise FileNotFoundError(f"Not a directory: {directory}") + + conn = get_connection(db_path) + counts = { + "annotated": 0, + "skipped_existing_gt": 0, + "skipped_user": 0, + "newly_added": 0, + "errors": 0, + } + + try: + image_files = sorted( + f for f in directory.iterdir() + if f.is_file() and f.suffix.lower() in IMAGE_EXTENSIONS + ) + source_dir = str(directory.resolve()) + + for img_path in image_files: + img_hash = hash_file(img_path) + existing = get_sample_by_hash(conn, img_hash) + + if existing: + # Skip silently if ground truth is already captured. + gt = conn.execute( + "SELECT 1 FROM ground_truths WHERE sample_id = ? LIMIT 1", + (existing.id,), + ).fetchone() + if gt: + counts["skipped_existing_gt"] += 1 + continue + sample_id = existing.id + else: + page_num = _extract_page_number(img_path.stem) + try: + sample_id = insert_sample( + conn, + image_path=str(img_path.resolve()), + image_hash=img_hash, + student=student, + category="lab", + source_dir=source_dir, + page_number=page_num, + ) + counts["newly_added"] += 1 + except Exception as e: + logger.warning("Failed to insert %s: %s", img_path.name, e) + counts["errors"] += 1 + continue + + suggestion = "" + if use_vlm_suggestion: + try: + from handwriting_engine.vision import read_with_consensus + result = read_with_consensus( + str(img_path), + providers=[vlm_provider], + strategy="best_of", + ) + suggestion = (result.text or "").strip() + except Exception as e: + logger.warning("VLM suggestion failed for %s: %s", img_path.name, e) + + text = prompt_fn(str(img_path), suggestion) + if text and text.strip(): + insert_ground_truth( + conn, sample_id, text.strip(), + source="lab-grader", author=student, + ) + counts["annotated"] += 1 + else: + counts["skipped_user"] += 1 + finally: + conn.close() + + return counts + + def ingest_directory( directory: str | Path, student: str = "", diff --git a/handwriting_engine/benchmark/models.py b/handwriting_engine/benchmark/models.py index 9cdabb1..7d83ef9 100644 --- a/handwriting_engine/benchmark/models.py +++ b/handwriting_engine/benchmark/models.py @@ -99,3 +99,4 @@ class RunSummary: iam_partition: str | None = None norm_flags: str | None = None vocab_hints_off: int = 0 + is_baseline: int = 0 diff --git a/handwriting_engine/benchmark/report.py b/handwriting_engine/benchmark/report.py index 9d04ea5..ebe00ec 100644 --- a/handwriting_engine/benchmark/report.py +++ b/handwriting_engine/benchmark/report.py @@ -18,6 +18,18 @@ ) from handwriting_engine.benchmark.evaluate import estimate_cost from handwriting_engine.benchmark.models import StrategyResult +from handwriting_engine.benchmark.stats import ( + bootstrap_ci, + cohens_r, + wilcoxon_signed_rank, +) + + +# Minimum paired-sample count for the statistics layer to attach. Below this, +# normal-approximation Wilcoxon is too rough and bootstrap CIs are dominated +# by sampling noise — better to print nothing than to print a misleading +# p-value. Aligns with Phase 8 success criterion #1. +_STATS_MIN_PAIRED_N = 10 def _aggregate_results(rows: list[dict]) -> list[StrategyResult]: @@ -183,12 +195,49 @@ def _format_csv(results: list[StrategyResult]) -> str: return output.getvalue().strip() +def _paired_cers( + rows_1: list[dict], + rows_2: list[dict], + provider: str, + strategy: str, +) -> tuple[list[float], list[float]]: + """Pair per-sample CERs from two runs by sample_id, restricted to the + given (provider, strategy). Order is sample_id-sorted and identical + on both sides — that's what makes the test paired.""" + by_sample_1 = { + r["sample_id"]: r["cer"] + for r in rows_1 + if r.get("provider") == provider + and r.get("strategy") == strategy + and r.get("cer") is not None + } + by_sample_2 = { + r["sample_id"]: r["cer"] + for r in rows_2 + if r.get("provider") == provider + and r.get("strategy") == strategy + and r.get("cer") is not None + } + shared = sorted(set(by_sample_1) & set(by_sample_2)) + return ([by_sample_1[sid] for sid in shared], + [by_sample_2[sid] for sid in shared]) + + def compare_runs( run_id_1: int, run_id_2: int, db_path: Path | str | None = None, ) -> str: - """Compare two benchmark runs side by side.""" + """Compare two benchmark runs side by side. + + For each (provider, strategy) pair shared between the runs, when the + paired sample count is >= 10 the output appends: + - paired Wilcoxon signed-rank p-value and z-score + - Cohen's r effect size + - 95% bootstrap CIs for each run's CER + + Pairing is by sample_id — same image, different run. + """ conn = get_connection(db_path) try: rows_1 = get_run_results(conn, run_id_1) @@ -233,6 +282,26 @@ def compare_runs( f"{provider:<20} {strategy:<10} {r1.mean_cer:>7.2%} {r2.mean_cer:>7.2%} " f"{delta:>+7.2%} {status:>12}" ) + + paired_a, paired_b = _paired_cers(rows_1, rows_2, provider, strategy) + if len(paired_a) >= _STATS_MIN_PAIRED_N: + wilcox = wilcoxon_signed_rank(paired_a, paired_b) + r_effect = cohens_r(wilcox.z, wilcox.n) + lo_a, hi_a = bootstrap_ci(paired_a, seed=run_id_1) + lo_b, hi_b = bootstrap_ci(paired_b, seed=run_id_2) + lines.append( + f"{' stats:':<31} " + f"n={wilcox.n} " + f"W={wilcox.statistic:.1f} " + f"z={wilcox.z:+.2f} " + f"p={wilcox.p_value:.4f} " + f"r={r_effect:.2f}" + ) + lines.append( + f"{' CI95:':<31} " + f"run#{run_id_1} [{lo_a:.2%}, {hi_a:.2%}] " + f"run#{run_id_2} [{lo_b:.2%}, {hi_b:.2%}]" + ) elif r1 and not r2: cer_str = f"{r1.mean_cer:>7.2%}" if r1.mean_cer >= 0 else " N/A" lines.append(f"{provider:<20} {strategy:<10} {cer_str} {'---':>8} {'---':>8} {'removed':>12}") @@ -245,16 +314,156 @@ def compare_runs( return "\n".join(lines) +# --- Phase 9 / RPT-02: configuration recommendation --- + + +# Composite score weights. The contract from REQUIREMENTS.md is 70/15/15; +# changing these is a product decision, not a code change. +_RECOMMEND_W_CER = 0.70 +_RECOMMEND_W_COST = 0.15 +_RECOMMEND_W_STAB = 0.15 + + +def recommend_strategy(db_path: Path | str | None = None) -> str: + """Rank (provider, strategy) configurations by a composite score. + + Score = 0.70 * (1 - cer_norm) + 0.15 * (1 - cost_norm) + 0.15 * stab_norm, + where each component is min-max normalized within the candidate set. + Lower CER and lower cost score higher; higher stability scores higher. + + Stability is the inverse of CER stdev across runs of the same + (provider, strategy). When a candidate has only one run, it is given + the median stability score of the candidate set (neutral) and the + output flags it as `n=1`. + + Returns a ranked human-readable table with the winner annotated. + """ + conn = get_connection(db_path) + try: + runs = list_runs(conn) + if not runs: + return "No runs in database — nothing to recommend." + + # Per-(provider, strategy) accumulator across all runs. + accum: dict[tuple[str, str], dict] = {} + for run in runs: + rows = get_run_results(conn, run.run_id) + for r in _aggregate_results(rows): + if r.mean_cer < 0: + continue + key = (r.provider, r.strategy) + bucket = accum.setdefault(key, { + "cers_per_run": [], + "cost_per_sample_runs": [], + "total_runs": 0, + }) + bucket["cers_per_run"].append(r.mean_cer) + if r.sample_count > 0: + bucket["cost_per_sample_runs"].append( + r.estimated_cost_usd / r.sample_count + ) + bucket["total_runs"] += 1 + finally: + conn.close() + + if not accum: + return "No (provider, strategy) configurations with measured CER — nothing to recommend." + + # Compute summary stats per candidate. + summaries: list[dict] = [] + for (provider, strategy), bucket in accum.items(): + cers = bucket["cers_per_run"] + costs = bucket["cost_per_sample_runs"] + n_runs = bucket["total_runs"] + mean_cer = sum(cers) / len(cers) + stdev_cer = statistics.stdev(cers) if len(cers) >= 2 else None + mean_cost = sum(costs) / len(costs) if costs else 0.0 + summaries.append({ + "provider": provider, + "strategy": strategy, + "n_runs": n_runs, + "mean_cer": mean_cer, + "stdev_cer": stdev_cer, + "mean_cost_per_sample": mean_cost, + }) + + # Normalize each component to [0, 1] within the candidate set. + cers = [s["mean_cer"] for s in summaries] + costs = [s["mean_cost_per_sample"] for s in summaries] + cer_min, cer_max = min(cers), max(cers) + cost_min, cost_max = min(costs), max(costs) + + measured_stdevs = [s["stdev_cer"] for s in summaries if s["stdev_cer"] is not None] + median_stdev = statistics.median(measured_stdevs) if measured_stdevs else 0.0 + stdevs_for_norm = [ + s["stdev_cer"] if s["stdev_cer"] is not None else median_stdev + for s in summaries + ] + stdev_min, stdev_max = min(stdevs_for_norm), max(stdevs_for_norm) + + def _norm(x: float, lo: float, hi: float) -> float: + if hi - lo < 1e-12: + return 0.5 # all candidates equal on this axis + return (x - lo) / (hi - lo) + + for s, eff_stdev in zip(summaries, stdevs_for_norm): + cer_n = _norm(s["mean_cer"], cer_min, cer_max) + cost_n = _norm(s["mean_cost_per_sample"], cost_min, cost_max) + stab_n = 1.0 - _norm(eff_stdev, stdev_min, stdev_max) + s["score"] = ( + _RECOMMEND_W_CER * (1.0 - cer_n) + + _RECOMMEND_W_COST * (1.0 - cost_n) + + _RECOMMEND_W_STAB * stab_n + ) + + summaries.sort(key=lambda s: s["score"], reverse=True) + winner = summaries[0] + + lines = [ + "Strategy + provider recommendation", + f" weights: CER {_RECOMMEND_W_CER:.0%} " + f"cost {_RECOMMEND_W_COST:.0%} " + f"stability {_RECOMMEND_W_STAB:.0%}", + "", + f" Winner: {winner['provider']} + {winner['strategy']} " + f"(score {winner['score']:.3f})", + "", + ] + header = ( + f"{'Rank':<4} {'Provider':<12} {'Strategy':<14} " + f"{'CER':>7} {'$/sample':>10} {'stdev':>9} {'n':>3} {'score':>7}" + ) + lines.append(header) + lines.append("-" * len(header)) + + for rank, s in enumerate(summaries, start=1): + stdev_str = f"{s['stdev_cer']:.3f}" if s["stdev_cer"] is not None else " n=1" + lines.append( + f"{rank:<4} {s['provider']:<12} {s['strategy']:<14} " + f"{s['mean_cer']:>6.2%} {s['mean_cost_per_sample']:>9.4f}$ " + f"{stdev_str:>9} {s['n_runs']:>3} {s['score']:>7.3f}" + ) + + return "\n".join(lines) + + def detect_regressions( run_id: int | None = None, threshold: float = 0.03, db_path: Path | str | None = None, ) -> list[dict]: - """Compare latest run against previous run. Returns list of regressions. + """Compare a run against the pinned baseline. Returns list of regressions. + + Phase 9 / RPT-01: anchor is the run pinned via `benchmark set-baseline`. + Falls back to the previous run when no baseline is pinned, preserving + pre-Phase-9 behavior on a freshly-initialized DB. Excludes self-compare + when the current run IS the baseline. Default threshold is 3% — with small sample sizes (<30), differences below this are within the noise floor and not meaningful. """ + from handwriting_engine.benchmark.db import get_baseline_run_id + conn = get_connection(db_path) try: runs = list_runs(conn) @@ -264,13 +473,21 @@ def detect_regressions( if run_id is None: current = runs[0] - previous = runs[1] else: current = next((r for r in runs if r.run_id == run_id), None) - idx = next((i for i, r in enumerate(runs) if r.run_id == run_id), None) + if not current: + return [] + + baseline_run_id = get_baseline_run_id(conn) + if baseline_run_id is not None and baseline_run_id != current.run_id: + previous = next((r for r in runs if r.run_id == baseline_run_id), None) + else: + # No pinned baseline (or current IS the baseline) — use the run + # immediately preceding `current`, matching pre-Phase-9 behavior. + idx = next((i for i, r in enumerate(runs) if r.run_id == current.run_id), None) previous = runs[idx + 1] if idx is not None and idx + 1 < len(runs) else None - if not current or not previous: + if not previous: return [] rows_curr = get_run_results(conn, current.run_id) @@ -527,3 +744,69 @@ def confidence_calibration( lines.append("") return "\n".join(lines) + + +def generate_per_writer_report( + run_id: int | None = None, + db_path: Path | str | None = None, +) -> str: + """Per-writer CER breakdown for a benchmark run (IAM-03). + + Groups eval_metrics by samples.student so the developer can tell whether a + strategy's CER gain is consistent across writers or driven by a few easy + ones. Requires samples to have been ingested with student tags (e.g. via + `benchmark ingest-iam`, which sets student='iam-writer-XXX'). + """ + conn = get_connection(db_path) + try: + if run_id is None: + run_id = get_latest_run_id(conn) + if run_id is None: + return "No runs found in database." + + rows = conn.execute( + """SELECT s.student AS student, + AVG(em.cer) AS mean_cer, + MIN(em.cer) AS min_cer, + MAX(em.cer) AS max_cer, + COUNT(*) AS n_samples + FROM provider_outputs po + JOIN eval_metrics em ON em.provider_output_id = po.id + JOIN samples s ON s.id = po.sample_id + WHERE po.run_id = ? + AND s.student IS NOT NULL + AND s.student != '' + GROUP BY s.student + ORDER BY mean_cer DESC""", + (run_id,), + ).fetchall() + finally: + conn.close() + + if not rows: + return ( + f"Per-Writer CER (Run #{run_id})\n\n" + "No writer data found for this run.\n" + "Tip: ingest IAM samples with `benchmark ingest-iam` first — " + "only IAM samples carry per-writer tags." + ) + + header = f"{'Writer':<25} {'Mean CER':>9} {'Min CER':>9} {'Max CER':>9} {'N':>4}" + separator = "-" * len(header) + lines = [ + f"Per-Writer CER (Run #{run_id})", + "", + header, + separator, + ] + for r in rows: + lines.append( + f"{r['student']:<25} " + f"{r['mean_cer']:>8.2%} " + f"{r['min_cer']:>8.2%} " + f"{r['max_cer']:>8.2%} " + f"{r['n_samples']:>4}" + ) + lines.append(separator) + lines.append(f" {len(rows)} writer(s) shown") + return "\n".join(lines) diff --git a/handwriting_engine/benchmark/stats.py b/handwriting_engine/benchmark/stats.py new file mode 100644 index 0000000..458e8c4 --- /dev/null +++ b/handwriting_engine/benchmark/stats.py @@ -0,0 +1,187 @@ +""" +Statistical defensibility for benchmark comparisons (Phase 8 / STAT-01, STAT-02). + +Hand-rolled to avoid pulling scipy as a dependency for three functions. The +project already hand-rolls Levenshtein inline in postprocess.py for the same +reason: trade ~150 LOC of well-tested math for a 30 MB scientific-stack dep. + +Public API: +- wilcoxon_signed_rank(a, b) -> {"statistic", "p_value", "z", "n"} +- bootstrap_ci(values, ...) -> (lower, upper) +- cohens_r(z, n) -> float + +All three operate on plain lists of floats; no numpy required. +""" + +from __future__ import annotations + +import math +import random +from dataclasses import dataclass + + +# --------------------------------------------------------------------------- +# Wilcoxon signed-rank test (paired, two-sided) +# --------------------------------------------------------------------------- + +@dataclass(frozen=True) +class WilcoxonResult: + """Outcome of a paired Wilcoxon signed-rank test.""" + statistic: float # W+ (sum of ranks of positive differences) + p_value: float # two-sided + z: float # standard normal z-score (sign matches direction of effect) + n: int # number of non-zero paired differences + + +def _rank_with_ties(abs_values: list[float]) -> tuple[list[float], float]: + """Average-rank values; also return the tie-correction sum used by the + Wilcoxon variance formula. + + The tie-correction is sum(t^3 - t) over each tie group, which gets + subtracted (divided by 48) inside the variance. + """ + indexed = sorted(enumerate(abs_values), key=lambda x: x[1]) + ranks = [0.0] * len(abs_values) + tie_correction = 0.0 + + i = 0 + while i < len(indexed): + j = i + while j + 1 < len(indexed) and indexed[j + 1][1] == indexed[i][1]: + j += 1 + # Average rank for positions i..j (1-indexed) + avg = (i + j + 2) / 2.0 + for k in range(i, j + 1): + ranks[indexed[k][0]] = avg + run_len = j - i + 1 + if run_len > 1: + tie_correction += run_len ** 3 - run_len + i = j + 1 + + return ranks, tie_correction + + +def wilcoxon_signed_rank(a: list[float], b: list[float]) -> WilcoxonResult: + """Two-sided paired Wilcoxon signed-rank test using normal approximation + with continuity correction. + + Drops zero differences (Wilcoxon's reduced-sample convention; matches + scipy's default `zero_method="wilcox"`). + + Use only when n >= 10. For small n the normal approximation is rough; + callers gate the integration at n >= 10 per the Phase 8 contract. + + Args: + a, b: equal-length sequences of paired observations. + + Returns: + WilcoxonResult. p_value is in [0, 1]; statistic is the sum of ranks + of positive differences (W+). + """ + if len(a) != len(b): + raise ValueError(f"paired samples must be same length; got {len(a)} vs {len(b)}") + + diffs = [x - y for x, y in zip(a, b)] + nonzero = [d for d in diffs if d != 0] + n = len(nonzero) + + if n == 0: + return WilcoxonResult(statistic=0.0, p_value=1.0, z=0.0, n=0) + + abs_diffs = [abs(d) for d in nonzero] + ranks, tie_correction = _rank_with_ties(abs_diffs) + + w_plus = sum(r for r, d in zip(ranks, nonzero) if d > 0) + w_minus = sum(r for r, d in zip(ranks, nonzero) if d < 0) + + mean_w = n * (n + 1) / 4.0 + var_w = n * (n + 1) * (2 * n + 1) / 24.0 - tie_correction / 48.0 + + if var_w <= 0: + # All differences identical — degenerate but well-defined: any non- + # zero w_plus implies p=1 (no information); zero implies p=1 too. + return WilcoxonResult(statistic=w_plus, p_value=1.0, z=0.0, n=n) + + # Continuity correction toward the mean. + diff_from_mean = w_plus - mean_w + if diff_from_mean > 0: + z = (diff_from_mean - 0.5) / math.sqrt(var_w) + elif diff_from_mean < 0: + z = (diff_from_mean + 0.5) / math.sqrt(var_w) + else: + z = 0.0 + + # Two-sided p-value via standard-normal survival function. + p_value = 2.0 * (1.0 - _normal_cdf(abs(z))) + p_value = max(0.0, min(1.0, p_value)) + + return WilcoxonResult(statistic=w_plus, p_value=p_value, z=z, n=n) + + +def _normal_cdf(z: float) -> float: + """Standard-normal cumulative distribution function via math.erf.""" + return 0.5 * (1.0 + math.erf(z / math.sqrt(2.0))) + + +# --------------------------------------------------------------------------- +# Bootstrap confidence interval (percentile method) +# --------------------------------------------------------------------------- + +def bootstrap_ci( + values: list[float], + confidence: float = 0.95, + n_iterations: int = 10_000, + seed: int | None = None, +) -> tuple[float, float]: + """Percentile-method bootstrap CI for the mean. + + Resamples `values` with replacement `n_iterations` times, computes the + mean of each resample, and returns the (lower, upper) percentile bounds + for the requested confidence level. + + Returns (mean, mean) when given fewer than 2 values — a degenerate CI is + more useful than raising for callers that just want to render a band. + """ + if not 0 < confidence < 1: + raise ValueError(f"confidence must be in (0, 1); got {confidence}") + if n_iterations < 1: + raise ValueError(f"n_iterations must be positive; got {n_iterations}") + + n = len(values) + if n == 0: + return (0.0, 0.0) + if n == 1: + return (values[0], values[0]) + + rng = random.Random(seed) + means = [0.0] * n_iterations + for i in range(n_iterations): + s = 0.0 + for _ in range(n): + s += values[rng.randrange(n)] + means[i] = s / n + means.sort() + + alpha = 1.0 - confidence + low_idx = int(round((alpha / 2.0) * n_iterations)) + high_idx = int(round((1.0 - alpha / 2.0) * n_iterations)) - 1 + low_idx = max(0, min(n_iterations - 1, low_idx)) + high_idx = max(0, min(n_iterations - 1, high_idx)) + + return (means[low_idx], means[high_idx]) + + +# --------------------------------------------------------------------------- +# Cohen's r effect size for Wilcoxon +# --------------------------------------------------------------------------- + +def cohens_r(z: float, n: int) -> float: + """Cohen's r effect size: r = z / sqrt(n). + + Conventional thresholds (Cohen 1988): 0.1 small, 0.3 medium, 0.5 large. + Returns the absolute magnitude — sign of effect is already captured in + the z-score and the CER deltas. + """ + if n <= 0: + return 0.0 + return abs(z) / math.sqrt(n) diff --git a/handwriting_engine/cli.py b/handwriting_engine/cli.py index d458be0..f603831 100644 --- a/handwriting_engine/cli.py +++ b/handwriting_engine/cli.py @@ -82,37 +82,110 @@ def enhance(path, strategy, output): click.echo(f"Enhanced: {result}") +_ALT_MARKER_RE = __import__("re").compile(r"\[\?alt:\s*([^\]]+)\]") + + +def _extract_alt_markers(text: str) -> list[dict]: + """Pull `[?alt: a/b]` markers out of consensus text into a structured list. + + Skill-side `--strict` mode uses these to prompt the user once per ambiguity. + """ + markers = [] + for match in _ALT_MARKER_RE.finditer(text): + alts = [a.strip() for a in match.group(1).split("/") if a.strip()] + markers.append({"raw": match.group(0), "alternatives": alts}) + return markers + + @cli.command() @click.argument("path", type=click.Path(exists=True)) @click.option("--provider", "-p", default="claude", type=click.Choice(["claude", "openai", "gemini", "consensus"])) -@click.option("--domain", "-d", default="biology") +@click.option("--domain", "-d", default="general", + help="Domain hint (general, biology, ...). Default 'general' so skill callers pass --domain=bio explicitly.") @click.option("--prompt", default="", help="Custom reading prompt") -def read(path, provider, domain, prompt): +@click.option("--writer", default=None, + help="Writer ID for WriterProfileStore lookup (per-writer few-shot/calibration)") +@click.option("--format", "fmt", default="txt", + type=click.Choice(["txt", "md", "json"]), + help="Output shape. 'json' emits a structured payload with extracted alt-markers — used by the handwriting-reader skill.") +def read(path, provider, domain, prompt, writer, fmt): """Read handwritten text from image or PDF.""" - import os - if path.lower().endswith(".pdf"): import tempfile from handwriting_engine.pdf import convert_pdf tmpdir = tempfile.mkdtemp(prefix="hwe_read_") _temp_dirs.append(tmpdir) pages = convert_pdf(path, tmpdir) - image_paths = [p["path"] for p in pages] + image_paths = [(p.get("page_number", i + 1), p["path"]) for i, p in enumerate(pages)] else: - image_paths = [path] + image_paths = [(1, path)] - for img in image_paths: + page_payloads = [] + for page_no, img in image_paths: if provider == "consensus": - from handwriting_engine.vision import read_with_consensus - result = read_with_consensus(img, prompt=prompt, domain=domain) - click.echo(f"--- Confidence: {result.confidence:.2f} ({result.strategy_used}) ---") - click.echo(result.text) - if result.disagreements: - click.echo(f"\nDisagreements: {result.disagreements}") + from handwriting_engine import vision as _vision + result = _vision.read_with_consensus(img, prompt=prompt, domain=domain, writer_id=writer) + page_payloads.append({ + "page_number": page_no, + "image_path": img, + "text": result.text, + "provider": "consensus", + "confidence": result.confidence, + "confidence_level": result.confidence_level, + "strategy_used": result.strategy_used, + "disagreements": list(result.disagreements), + "alt_markers": _extract_alt_markers(result.text), + "provider_results": dict(result.provider_results), + "tokens_used": dict(result.tokens_used), + }) else: - from handwriting_engine.vision import read_page - text = read_page(img, prompt=prompt, domain=domain, provider=provider) - click.echo(text) + from handwriting_engine import vision as _vision + text = _vision.read_page(img, prompt=prompt, domain=domain, provider=provider) + page_payloads.append({ + "page_number": page_no, + "image_path": img, + "text": text, + "provider": provider, + "confidence": None, + "confidence_level": None, + "strategy_used": None, + "disagreements": [], + "alt_markers": _extract_alt_markers(text), + "provider_results": {provider: text}, + "tokens_used": {}, + }) + + if fmt == "json": + click.echo(json.dumps({ + "path": path, + "domain": domain, + "writer": writer, + "pages": page_payloads, + }, indent=2)) + return + + if fmt == "md": + for p in page_payloads: + header = f"## Page {p['page_number']}" + if p["confidence"] is not None: + header += f" — confidence {p['confidence']:.2f} ({p['confidence_level']})" + click.echo(header) + click.echo("") + click.echo(p["text"]) + click.echo("") + if p["disagreements"]: + click.echo(f"_Disagreements: {p['disagreements']}_\n") + return + + # Default: txt — preserve legacy echo behavior + for p in page_payloads: + if p["provider"] == "consensus": + click.echo(f"--- Confidence: {p['confidence']:.2f} ({p['strategy_used']}) ---") + click.echo(p["text"]) + if p["disagreements"]: + click.echo(f"\nDisagreements: {p['disagreements']}") + else: + click.echo(p["text"]) @cli.command() @@ -244,6 +317,53 @@ def benchmark_ingest_iam(ascii_dir, lines_dir, partition_file, all_partitions, d raise SystemExit(1) +@benchmark.command("ingest-lab") +@click.argument("directory", type=click.Path(exists=True, file_okay=False)) +@click.option("--student", "-s", default="lab", help="Student / author tag for these images") +@click.option("--with-suggestion", is_flag=True, default=False, + help="Pre-fill each prompt with a single VLM read (costs ~$/image).") +@click.option("--vlm-provider", default="gemini", show_default=True, + help="Provider for VLM suggestion (when --with-suggestion).") +@click.option("--db-path", default=None, hidden=True) +def benchmark_ingest_lab(directory, student, with_suggestion, vlm_provider, db_path): + """Guided annotation: capture ground-truth transcriptions for lab notebook images. + + For each image in DIRECTORY, opens $EDITOR (or prompts inline) with an + optional VLM-suggested transcription. Save to record as ground truth; + leave empty / cancel to skip an image. Re-running on the same directory + skips images that already have ground truth — safe to resume. + """ + from handwriting_engine.benchmark.ingest import ingest_lab + + def _prompt(image_path: str, suggestion: str) -> str | None: + click.echo(f"\n--- {image_path} ---") + if suggestion: + click.echo(f"VLM suggestion: {suggestion}") + # click.edit returns None if EDITOR exits without saving (skip); + # otherwise it returns the buffer with trailing whitespace. + edited = click.edit(suggestion or "") + if edited is None: + return None + return edited.strip() + + counts = ingest_lab( + directory, + student=student, + prompt_fn=_prompt, + db_path=db_path, + use_vlm_suggestion=with_suggestion, + vlm_provider=vlm_provider, + ) + click.echo( + "\nLab ingest complete: " + f"{counts['annotated']} annotated, " + f"{counts['newly_added']} new samples, " + f"{counts['skipped_existing_gt']} already had ground truth, " + f"{counts['skipped_user']} skipped, " + f"{counts['errors']} errors." + ) + + @benchmark.command("transcribe") @click.argument("sample_id", type=int) @click.option("--text", "-t", default=None, help="Ground truth transcription text") @@ -416,13 +536,90 @@ def progress(current, total, msg): click.echo(f"\nFed {count} lessons back to the lessons system.") +@benchmark.command("sweep") +@click.option("--provider", "-p", default="gemini", + help="Provider used for all 5 strategies (default: gemini)") +@click.option("--yes", "-y", is_flag=True, + help="Bypass cost confirmation") +@click.option("--db-path", default=None, hidden=True) +def benchmark_sweep(provider, yes, db_path): + """Run all 5 strategies against the IAM test set, one run_id per strategy. + + Strategies: baseline, self_correct, line_level, prompt_adapted, zoomed_verify. + Requires IAM samples in the DB — run `benchmark ingest-iam` first. + Prints projected cost before any API call. + """ + from handwriting_engine.benchmark.evaluate import ( + run_sweep, SWEEP_STRATEGIES, estimate_cost, + ) + from handwriting_engine.benchmark.db import get_connection as _get_conn + + # Count IAM samples with ground truth + conn = _get_conn(db_path) + try: + row = conn.execute( + "SELECT COUNT(DISTINCT s.id) AS n FROM samples s " + "JOIN ground_truths gt ON gt.sample_id = s.id " + "WHERE s.category = 'iam'" + ).fetchone() + n_samples = row["n"] if row else 0 + finally: + conn.close() + + # Cost projection: ~2000 input + ~200 output tokens per sample per strategy + est_per_strategy = estimate_cost(2000 * n_samples, 200 * n_samples, provider) + est_total = est_per_strategy * len(SWEEP_STRATEGIES) + + click.echo( + f"\nSweep projection: {len(SWEEP_STRATEGIES)} strategies " + f"x {n_samples} IAM samples (provider={provider})" + ) + click.echo( + f" Estimated cost: ~${est_per_strategy:.4f}/strategy " + f"x {len(SWEEP_STRATEGIES)} = ~${est_total:.4f} total" + ) + click.echo( + f" Strategies: {', '.join(s['name'] for s in SWEEP_STRATEGIES)}\n" + ) + + if n_samples == 0: + click.echo( + "WARNING: No IAM samples with ground truth in DB. " + "Run `benchmark ingest-iam` first.", + err=True, + ) + + if not yes: + click.confirm( + f"Proceed with sweep (~${est_total:.4f} projected)?", + abort=True, + ) + + try: + run_ids = run_sweep(provider=provider, db_path=db_path, yes=yes) + except Exception as exc: + click.echo(f"ERROR during sweep: {exc}", err=True) + sys.exit(1) + + click.echo("\nSweep complete:") + for name, run_id in run_ids.items(): + click.echo(f" {name:20s}: run_id={run_id}") + + @benchmark.command("report") @click.option("--run-id", "-r", default=None, type=int, help="Specific run (default: latest)") @click.option("--format", "fmt", default="table", type=click.Choice(["table", "json", "csv"])) -def benchmark_report_cmd(run_id, fmt): +@click.option("--per-writer", is_flag=True, default=False, + help="Show per-writer CER breakdown (requires IAM samples with student tags)") +@click.option("--db-path", default=None, hidden=True) +def benchmark_report_cmd(run_id, fmt, per_writer, db_path): """Show accuracy comparison table for a benchmark run.""" - from handwriting_engine.benchmark.report import generate_report + if per_writer: + from handwriting_engine.benchmark.report import generate_per_writer_report + click.echo(generate_per_writer_report(run_id=run_id, db_path=db_path)) + return + from handwriting_engine.benchmark.report import generate_report click.echo(generate_report(run_id, fmt=fmt)) @@ -439,6 +636,45 @@ def benchmark_compare_cmd(run_id_1, run_id_2): click.echo(compare_runs(run_id_1, run_id_2)) +@benchmark.command("recommend") +@click.option("--db-path", default=None, type=click.Path(), + help="Override database path (default: ~/.handwriting-engine/benchmark.db).") +def benchmark_recommend_cmd(db_path): + """Recommend the best (provider, strategy) configuration. + + Composite score: 70% CER, 15% cost, 15% stability across runs. + Each component is min-max normalized within the candidate set. + Single-run candidates get the median stability score (neutral). + """ + from handwriting_engine.benchmark.report import recommend_strategy + + click.echo(recommend_strategy(db_path=db_path)) + + +@benchmark.command("set-baseline") +@click.argument("run_id", type=int) +@click.option("--db-path", default=None, type=click.Path(), + help="Override database path (default: ~/.handwriting-engine/benchmark.db).") +def benchmark_set_baseline_cmd(run_id, db_path): + """Pin RUN_ID as the regression-detection anchor. + + `detect_regressions` and `benchmark report` then compare future runs + against this pinned run rather than the immediately-preceding run. + Exactly one run is the baseline at any time; this command atomically + clears the prior pin and sets the new one. + """ + from handwriting_engine.benchmark.db import get_connection, set_baseline + + conn = get_connection(db_path) + try: + set_baseline(conn, run_id) + except ValueError as e: + raise click.ClickException(str(e)) + finally: + conn.close() + click.echo(f"Baseline pinned: run #{run_id}") + + @benchmark.command("drill-down") @click.argument("sample_id", type=int) @click.option("--run-id", "-r", default=None, type=int, help="Specific run (default: latest)") @@ -558,5 +794,72 @@ def benchmark_bootstrap_gt_cmd(agreement, confidence): click.echo(f"Auto-generated {count} ground truths from consensus") +# ===================================================================== +# Trained post-correction (optional — requires [trained-correction] extras) +# ===================================================================== + +@cli.group(name="trained-correction") +def trained_correction_group(): + """Train and evaluate the optional trained post-correction model. + + Requires: pip install handwriting-engine[trained-correction] + """ + + +@trained_correction_group.command(name="train") +@click.option("--output-dir", "-o", required=True, type=click.Path(), + help="Directory for the trained checkpoint + manifest") +@click.option("--num-pairs", default=50000, type=int, help="Synthetic training pairs to generate") +@click.option("--num-epochs", default=2, type=int) +@click.option("--batch-size", default=8, type=int) +@click.option("--learning-rate", default=3e-4, type=float) +@click.option("--max-input-length", default=256, type=int) +@click.option("--max-target-length", default=256, type=int) +@click.option("--device", default="auto", type=click.Choice(["auto", "cpu", "mps", "cuda"])) +@click.option("--seed", default=42, type=int) +@click.option("--quick", is_flag=True, help="Tiny smoke run") +@click.option("--no-system-wordlist", is_flag=True) +@click.option("--model-name", default="google/flan-t5-small", show_default=True) +@click.option("--from-benchmark-db", is_flag=True, + help="Mix real (VLM_output, ground_truth) pairs from the benchmark DB") +@click.option("--benchmark-db-path", default=None, type=click.Path(), + help="Override the default benchmark.db path") +@click.option("--benchmark-providers", default=None, + help="Comma-separated provider filter (e.g. 'gemini,claude')") +@click.option("--real-data-weight", default=3, type=int, + help="Replication factor for real pairs (default 3)") +@click.option("--continue-from", default=None, type=click.Path(), + help="Continue fine-tuning from an existing checkpoint") +def trained_correction_train(**kwargs): + """Fine-tune the synthetic-data corrector. Long-running.""" + from handwriting_engine.trained_correction.train import main as train_main + argv: list[str] = [] + for k, v in kwargs.items(): + flag = "--" + k.replace("_", "-") + if isinstance(v, bool): + if v: + argv.append(flag) + elif v is not None: + argv.extend([flag, str(v)]) + sys.exit(train_main(argv)) + + +@trained_correction_group.command(name="eval") +@click.option("--n-pairs", default=1000, type=int) +@click.option("--seed", default=1234, type=int) +@click.option("--domain", default="biology") +@click.option("--skip-trained", is_flag=True, help="Heuristic-only baseline (no model load)") +@click.option("--output", default=None, type=click.Path(), help="Optional JSON output path") +def trained_correction_eval(n_pairs, seed, domain, skip_trained, output): + """A/B evaluate post-correction pipelines on synthetic pairs.""" + from handwriting_engine.trained_correction.eval import main as eval_main + argv = ["--n-pairs", str(n_pairs), "--seed", str(seed), "--domain", domain] + if skip_trained: + argv.append("--skip-trained") + if output: + argv.extend(["--output", output]) + sys.exit(eval_main(argv)) + + if __name__ == "__main__": cli() diff --git a/handwriting_engine/consensus.py b/handwriting_engine/consensus.py index 2d58471..ae2a859 100644 --- a/handwriting_engine/consensus.py +++ b/handwriting_engine/consensus.py @@ -45,6 +45,166 @@ # Cascade order: cheapest first (~$0.0005/img → ~$0.003 → ~$0.005) CASCADE_ORDER = ["gemini", "openai", "claude"] +# --------------------------------------------------------------------------- +# Char-level confusion-pair resolution (S5) +# --------------------------------------------------------------------------- +# +# Each tuple is (alt_form, canonical_form). When two providers disagree on a +# word and the only char-level differences match one of these pairs, we +# resolve to the canonical form by default. A writer profile entry of the +# form `confusion_resolutions["↔"] = "" or ""` overrides per writer. +# +# Pairs without a clear canonical winner (b↔d, p↔q, i↔j) are intentionally +# omitted — without context, defaulting either way introduces error. They +# defer to the existing [?alt: …] fallback. +_CHAR_CONFUSION_PAIRS: list[tuple[str, str]] = [ + # Multi-char ↔ single-char (the dominant residual error class) + ("rn", "m"), + ("cl", "d"), + ("ri", "n"), + ("uu", "w"), + # Single-char letter pairs with a canonical form + ("u", "v"), + ("a", "o"), + ("n", "h"), + ("e", "c"), + ("f", "t"), + ("q", "g"), + # Case-sensitive letter/digit pairs (canonical = letter form for in-word context) + ("0", "O"), + ("1", "l"), + ("1", "I"), + ("I", "l"), + ("5", "S"), + ("2", "Z"), + ("6", "G"), + ("8", "B"), +] + + +def _confusion_pair_label(a: str, b: str) -> str: + """Canonical pair-name string used as writer_resolutions dict key.""" + return f"{a}↔{b}" + + +def _match_confusion_pair(left: str, right: str) -> tuple[str, str] | None: + """Return the canonical (alt, canonical) pair tuple if (left, right) match + one of the known confusion pairs in either order; else None. + + Whole-segment match (e.g. ``("rn", "m")``) is checked first so multi-char + pairs win over the per-char fallback below. + """ + for alt, canon in _CHAR_CONFUSION_PAIRS: + if (left, right) == (alt, canon) or (left, right) == (canon, alt): + return (alt, canon) + return None + + +def _decompose_diff_into_pairs( + seg_a: str, seg_b: str +) -> list[tuple[str, str]] | None: + """Try to express the (seg_a, seg_b) diff as a sequence of per-position + confusion-pair swaps. + + Returns a list of (left_char, right_char) sub-pairs, or None if the diff + isn't decomposable into known confusion pairs. + + Whole-segment match wins (so ``("ll", "II")`` is decomposed into two + `("l", "I")` pairs only because no entry like ``("ll", "II")`` exists). + """ + direct = _match_confusion_pair(seg_a, seg_b) + if direct is not None: + return [(seg_a, seg_b)] + + # Per-character fallback only works for equal-length segments. + if len(seg_a) != len(seg_b): + return None + parts: list[tuple[str, str]] = [] + for ca, cb in zip(seg_a, seg_b): + if ca == cb: + parts.append((ca, cb)) + continue + if _match_confusion_pair(ca, cb) is None: + return None + parts.append((ca, cb)) + return parts + + +def resolve_char_level( + candidates: list[str], + weights: list[float] | None = None, + *, + writer_resolutions: dict[str, str] | None = None, +) -> str | None: + """Resolve a no-majority word-level disagreement at character level. + + When two candidates differ only by known confusion-pair substrings + (`rn↔m`, `cl↔d`, `0↔O`, …), pick the resolution. The writer profile's + `confusion_resolutions` map (if provided) overrides the global canonical + default per pair. + + Returns the resolved word, or `None` when the disagreement is not a + pure confusion-pair case — caller must fall back to its existing + no-majority handling (e.g. `[?alt: …]`). + + v0 scope: handles 1-2 unique candidates. With 3+ distinct candidates, + returns None (defer to existing fallback). + """ + if not candidates: + return None + + unique = list(dict.fromkeys(candidates)) # preserve first-seen order + if len(unique) == 1: + return unique[0] + if len(unique) > 2: + return None + + a, b = unique[0], unique[1] + matcher = SequenceMatcher(None, a, b) + opcodes = matcher.get_opcodes() + + # All differing segments must decompose into known confusion-pair swaps. + decomposed: list[tuple[str, list[tuple[str, str]]]] = [] + has_diff = False + for tag, i1, i2, j1, j2 in opcodes: + if tag == "equal": + decomposed.append(("equal", [(a[i1:i2], a[i1:i2])])) + continue + has_diff = True + seg_a = a[i1:i2] + seg_b = b[j1:j2] + parts = _decompose_diff_into_pairs(seg_a, seg_b) + if parts is None: + return None + decomposed.append(("diff", parts)) + + if not has_diff: + return a + + writer_resolutions = writer_resolutions or {} + + out_chars: list[str] = [] + for tag, parts in decomposed: + if tag == "equal": + out_chars.append(parts[0][0]) + continue + for left, right in parts: + if left == right: + out_chars.append(left) + continue + pair = _match_confusion_pair(left, right) + if pair is None: + return None + alt, canon = pair + label = _confusion_pair_label(alt, canon) + choice = writer_resolutions.get(label) + if choice in (alt, canon): + out_chars.append(choice) + else: + out_chars.append(canon) + + return "".join(out_chars) + # Uncertainty marker pattern — must be defined here (before _self_correct and confidence helpers) _UNCERTAINTY_RE = re.compile( r"\[\?\]|\?\?\?|\[illegible[^\]]*\]|\[unclear\]|unable to read", @@ -75,6 +235,7 @@ def read_with_consensus( quality_assessment: dict | None = None, max_self_correct_rounds: int = 1, uncertainty_threshold: int = 3, + writer_profile: dict | None = None, ) -> ConsensusResult: """ Read an image using multiple models and combine results. @@ -98,7 +259,7 @@ def read_with_consensus( if strategy == "best_of": return _best_of(image_b64, media_type, prompt, system_prompt, content_type, max_tokens) elif strategy == "vote": - return _vote(image_b64, media_type, prompt, system_prompt, providers, confidence_threshold, content_type, max_tokens) + return _vote(image_b64, media_type, prompt, system_prompt, providers, confidence_threshold, content_type, max_tokens, writer_profile=writer_profile) elif strategy == "debate": return _debate(image_b64, media_type, prompt, system_prompt, providers, max_tokens, max_debate_rounds) elif strategy == "cascade": @@ -108,8 +269,8 @@ def read_with_consensus( elif strategy == "smart": if quality_assessment is None: # No quality data — fall back to vote - return _vote(image_b64, media_type, prompt, system_prompt, providers, confidence_threshold, content_type, max_tokens) - return _smart_route(image_b64, media_type, prompt, system_prompt, quality_assessment, content_type, max_tokens, uncertainty_threshold) + return _vote(image_b64, media_type, prompt, system_prompt, providers, confidence_threshold, content_type, max_tokens, writer_profile=writer_profile) + return _smart_route(image_b64, media_type, prompt, system_prompt, quality_assessment, content_type, max_tokens, uncertainty_threshold, writer_profile=writer_profile) else: raise ValueError(f"Unknown strategy: {strategy}. Use: vote, best_of, debate, cascade, smart, self_correct") @@ -258,6 +419,7 @@ def _vote( image_b64: str, media_type: str, prompt: str, system_prompt: str, providers: list[str] | None, confidence_threshold: float, content_type: str = "default", max_tokens: int = 4096, + *, writer_profile: dict | None = None, ) -> ConsensusResult: """Send to N providers, word-level majority vote. @@ -404,7 +566,7 @@ def _read_from_provider(name): provider_weights.append((name, base_weight * confidence_scale)) best_text, disagreements, confidence = _word_level_vote( - texts, provider_weights, + texts, provider_weights, writer_profile=writer_profile, ) if excluded: @@ -735,6 +897,7 @@ def _smart_route( image_b64: str, media_type: str, prompt: str, system_prompt: str, quality_assessment: dict, content_type: str, max_tokens: int, uncertainty_threshold: int = 3, + *, writer_profile: dict | None = None, ) -> ConsensusResult: """Adaptive routing based on image quality — spend API calls where they matter. @@ -794,7 +957,7 @@ def _smart_route( else: # Hard: full vote with all available providers - return _vote(image_b64, media_type, prompt, system_prompt, None, 0.8, content_type, max_tokens) + return _vote(image_b64, media_type, prompt, system_prompt, None, 0.8, content_type, max_tokens, writer_profile=writer_profile) # --------------------------------------------------------------------------- @@ -945,17 +1108,22 @@ def _word_agreement_ratio(text_a: str, text_b: str) -> float: def _word_level_vote( texts: list[str], provider_weights: list[tuple[str, float]], + writer_profile: dict | None = None, ) -> tuple[str, list[str], float]: """Word-level majority vote across N provider outputs. For each word position: - If all providers agree → keep the word - If majority agrees → keep majority, record disagreement - - If no majority → keep highest-weighted provider's word, mark with [?alt: ...] + - If no majority → try char-level confusion-pair resolution (S5); + if that defers, keep highest-weighted provider's word, mark with [?alt: ...] Returns (final_text, disagreements, confidence). Confidence = fraction of word positions where providers agreed. """ + writer_resolutions = ( + (writer_profile or {}).get("confusion_resolutions") or {} + ) all_words = [_tokenize_preserving_newlines(t) for t in texts] # Use highest-weighted provider as alignment anchor @@ -1047,13 +1215,25 @@ def _word_level_vote( if winner != "\n" and runner_up != "\n": disagreements.append(f"'{runner_up}' vs '{winner}' (majority)") else: - # No majority — use highest-weighted provider, mark ambiguous - result_words.append(winner) - alts = [w for w, _ in sorted_votes[1:] if w != "\n"] - if alts and winner != "\n": - alt_str = "/".join(alts[:2]) - result_words.append(f"[?alt: {alt_str}]") - disagreements.append(f"'{winner}' vs '{alt_str}' (no majority)") + # No majority — try char-level confusion-pair resolution + # before falling back to the [?alt: …] marker. + candidates = [w for w, _ in sorted_votes if w != "\n"] + weights = [v for w, v in sorted_votes if w != "\n"] + resolved = resolve_char_level( + candidates, + weights, + writer_resolutions=writer_resolutions, + ) + if resolved is not None: + result_words.append(resolved) + else: + # No clean confusion-pair match — emit ambiguous marker. + result_words.append(winner) + alts = [w for w, _ in sorted_votes[1:] if w != "\n"] + if alts and winner != "\n": + alt_str = "/".join(alts[:2]) + result_words.append(f"[?alt: {alt_str}]") + disagreements.append(f"'{winner}' vs '{alt_str}' (no majority)") # Reconstruct text from tokens final_parts = [] diff --git a/handwriting_engine/few_shot.py b/handwriting_engine/few_shot.py new file mode 100644 index 0000000..07f69fd --- /dev/null +++ b/handwriting_engine/few_shot.py @@ -0,0 +1,212 @@ +""" +S2 — per-writer few-shot exemplar plumbing. + +Builds the interleaved content blocks (exemplar_image, exemplar_label, ..., +target_image) that vision providers like Claude and Gemini already accept via +their ``read_batch`` methods. Selection lives in +:mod:`handwriting_engine.writer_profile_store`; this module only handles the +"now turn an Exemplar list into provider-ready blocks" step plus the env-var +opt-out per S2-SPEC § Cost & opt-out. +""" + +from __future__ import annotations + +import base64 +import logging +import os +from typing import Optional + +from handwriting_engine.writer_profile_store import Exemplar + +logger = logging.getLogger(__name__) + + +# Providers whose `read_batch` carries multi-image content lists. TrOCR (and +# any OCR-only provider that lacks in-context learning) is intentionally +# excluded — see S2-SPEC criterion #5. +EXEMPLAR_PROVIDERS: frozenset[str] = frozenset({"claude", "gemini"}) + +DEFAULT_FEW_SHOT_K = 3 +FEW_SHOT_K_ENV = "HE_FEW_SHOT_K" + + +# Anti-cargo-cult guidance: returning-writer few-shot can prime the model to +# parrot the reference transcription rather than read the new image. The +# label deliberately separates "reference text" from "what to read". +EXEMPLAR_LABEL_TEMPLATE = ( + "The handwriting in the previous image transcribes to: «{gt}». " + "This is the same writer as the final image but contains DIFFERENT TEXT. " + "Read what is in the final image — do not repeat the reference text." +) + + +def env_few_shot_k(env: Optional[dict] = None) -> int: + """Return the ``HE_FEW_SHOT_K`` cap, or ``DEFAULT_FEW_SHOT_K`` if unset. + + ``0`` disables few-shot entirely (S2-SPEC § Cost & opt-out). Negative or + unparseable values are treated as the default — the caller is presumed + to want the spec-default behavior, not silent disablement. + """ + env = os.environ if env is None else env + raw = env.get(FEW_SHOT_K_ENV) + if raw is None: + return DEFAULT_FEW_SHOT_K + try: + value = int(raw) + except ValueError: + return DEFAULT_FEW_SHOT_K + return value if value >= 0 else DEFAULT_FEW_SHOT_K + + +def provider_supports_exemplars(provider: str) -> bool: + """True iff the provider name belongs to EXEMPLAR_PROVIDERS.""" + return provider in EXEMPLAR_PROVIDERS + + +def _media_type_for(image_path: str) -> str: + suffix = os.path.splitext(image_path)[1].lower() + return { + ".jpg": "image/jpeg", + ".jpeg": "image/jpeg", + ".png": "image/png", + ".gif": "image/gif", + ".webp": "image/webp", + }.get(suffix, "image/jpeg") + + +def _load_exemplar_image(image_path: str) -> Optional[tuple[str, str]]: + """Encode an exemplar image identically to ``optimize.validate_and_prepare_image``. + + Returns ``(base64, media_type)`` or ``None`` if the image is missing or + fails to decode. Missing exemplars are filtered out rather than raised — + a stale DB row should not break the whole transcription path. + """ + try: + from handwriting_engine.optimize import validate_and_prepare_image + + result = validate_and_prepare_image(image_path) + if result is not None: + return result + except Exception as exc: # pragma: no cover - defensive + logger.debug("optimize.validate_and_prepare_image failed for %s: %s", image_path, exc) + + # Fallback: raw read for tests / minimal envs without PIL. + if not os.path.exists(image_path): + return None + try: + with open(image_path, "rb") as fh: + data = base64.standard_b64encode(fh.read()).decode("utf-8") + except OSError: + return None + return data, _media_type_for(image_path) + + +def build_exemplar_blocks( + target_image_b64: str, + target_media_type: str, + exemplars: list[Exemplar], +) -> list[dict]: + """Build ``read_batch`` content blocks: exemplars first, target last. + + Layout (per S2-SPEC § 2): + exemplar_1_image, exemplar_1_label_text, + exemplar_2_image, exemplar_2_label_text, + ..., + target_image + + The user's prompt is appended by the provider's ``read_batch``. Exemplars + whose image cannot be loaded are skipped silently (logged at DEBUG); a + fully-failed exemplar list collapses to a single-image read, which is the + correct fallback. + """ + blocks: list[dict] = [] + for ex in exemplars: + loaded = _load_exemplar_image(ex.image_path) + if loaded is None: + logger.debug( + "Skipping exemplar sample_id=%s: image not loadable (%s)", + ex.sample_id, + ex.image_path, + ) + continue + b64, media_type = loaded + blocks.append( + { + "type": "image", + "source": { + "type": "base64", + "media_type": media_type, + "data": b64, + }, + } + ) + blocks.append( + { + "type": "text", + "text": EXEMPLAR_LABEL_TEMPLATE.format(gt=ex.ground_truth), + } + ) + + blocks.append( + { + "type": "image", + "source": { + "type": "base64", + "media_type": target_media_type, + "data": target_image_b64, + }, + } + ) + return blocks + + +def select_and_build_exemplar_blocks( + *, + writer_id: Optional[str], + provider: str, + target_image_b64: str, + target_media_type: str, + k: Optional[int] = None, + exclude_sample_id: Optional[int] = None, + env: Optional[dict] = None, + conn=None, + db_path=None, +) -> Optional[list[dict]]: + """High-level orchestrator: gate, select, and build blocks in one call. + + Returns ``None`` whenever the few-shot path is not applicable (caller + should fall back to a normal single-image read). This consolidates the + several spec-defined gates (writer_id required, provider allowlist, + HE_FEW_SHOT_K, ≥2 GT samples) so the integration in ``vision.read_page`` + is one branch. + """ + if not writer_id: + return None + if not provider_supports_exemplars(provider): + logger.debug("few-shot: provider %s not in EXEMPLAR_PROVIDERS", provider) + return None + + cap = env_few_shot_k(env) if k is None else k + if cap <= 0: + logger.debug("few-shot: HE_FEW_SHOT_K=%d disables exemplars", cap) + return None + + from handwriting_engine.writer_profile_store import select_exemplars + + exemplars = select_exemplars( + writer_id, + k=cap, + exclude_sample_id=exclude_sample_id, + conn=conn, + db_path=db_path, + ) + # S2-SPEC criterion #4: cold writers (<2 GT samples) fall back cleanly. + if len(exemplars) < 2: + logger.debug( + "few-shot: only %d exemplar(s) for writer_id=%s — falling back to single-image read", + len(exemplars), + writer_id, + ) + return None + + return build_exemplar_blocks(target_image_b64, target_media_type, exemplars) diff --git a/handwriting_engine/postprocess.py b/handwriting_engine/postprocess.py index 83746ed..ec3f694 100644 --- a/handwriting_engine/postprocess.py +++ b/handwriting_engine/postprocess.py @@ -11,12 +11,32 @@ from __future__ import annotations +import os import re import logging +from dataclasses import dataclass from functools import lru_cache logger = logging.getLogger(__name__) + +@dataclass(frozen=True) +class Correction: + """A single confusion-pair correction applied during postprocessing.""" + original: str + corrected: str + pair: str # e.g. "I↔l" — the confusion-pair label that triggered the swap + + +def _trained_corrector_enabled(explicit: bool | None) -> bool: + """Resolve whether the trained corrector should run. + + Precedence: explicit kwarg > env var > default off. + """ + if explicit is not None: + return explicit + return os.environ.get("HE_USE_TRAINED_CORRECTOR", "").lower() in ("1", "true", "yes", "on") + # Biology domain word list — common lab terms that OCR confuses _BIOLOGY_TERMS = { "mitosis", "meiosis", "mitochondria", "chloroplast", "photosynthesis", @@ -427,3 +447,209 @@ def correct_domain_terms(text: str, domain: str = "biology") -> str: logger.info("Domain correction (%s): %d word(s) corrected", domain, corrections_made) return " ".join(corrected) + + +def _confusion_postprocess_enabled() -> bool: + """HE_CONFUSION_POSTPROCESS env flag — default ON; set 0 to disable.""" + val = os.environ.get("HE_CONFUSION_POSTPROCESS") + if val is None: + return True + return val.strip().lower() not in ("0", "false", "no", "off") + + +def correct_confusion_pairs( + text: str, + *, + domain: str = "biology", + writer_id: str | None = None, + db_path=None, +) -> tuple[str, list[Correction]]: + """Confusion-pair-aware single-word swap. + + For each word that is NOT in the domain wordlist, try swapping a single + confusion-pair occurrence (`I→l`, `rn→m`, `cl→d`, `0→O`, …). If exactly + one candidate produced by such a swap is in the wordlist, prefer the swap + and record a Correction. + + Higher-precision than the existing edit-distance-1 pass: only swaps that + correspond to known confusion pairs are considered, and only when the + resulting word is unambiguously a domain term. + + Args: + text: Transcription text to correct. + domain: One of 'biology', 'chemistry', 'general', 'science'. + writer_id: Optional writer ID. Reserved for per-writer biasing once + S4's corrections table is populated; today the engine falls back + to global confusion pairs and the value is ignored. + db_path: Optional path to benchmark DB. Reserved for the same reason. + + Returns: + Tuple of (corrected_text, list_of_corrections). + """ + from handwriting_engine.consensus import ( + _CHAR_CONFUSION_PAIRS, + _confusion_pair_label, + ) + + wordlist = _DOMAIN_WORDLISTS.get(domain, _GENERAL_TERMS) + words = text.split() + corrections: list[Correction] = [] + out: list[str] = [] + + for word in words: + # Strip leading / trailing non-alpha for lookup (mirrors correct_domain_terms) + stripped = word.rstrip(".,;:!?") + suffix = word[len(stripped):] + prefix = "" + i = 0 + while i < len(stripped) and not stripped[i].isalpha() and not stripped[i].isdigit(): + prefix += stripped[i] + i += 1 + core_raw = stripped[i:] + j = len(core_raw) + while j > 0 and not core_raw[j - 1].isalpha() and not core_raw[j - 1].isdigit(): + j -= 1 + core = core_raw[:j] + suffix = core_raw[j:] + suffix + + if len(core) < 3 or _SKIP_RE.match(core): + out.append(word) + continue + + if core.lower() in wordlist: + out.append(word) + continue + + # Generate 1-confusion-pair-swap candidates. + candidates: list[tuple[str, str]] = [] # (candidate_core, pair_label) + seen: set[str] = {core} + for alt, canon in _CHAR_CONFUSION_PAIRS: + label = _confusion_pair_label(alt, canon) + for src, dst in ((alt, canon), (canon, alt)): + idx = core.find(src) + while idx != -1: + cand = core[:idx] + dst + core[idx + len(src):] + if cand not in seen and cand.lower() in wordlist: + candidates.append((cand, label)) + seen.add(cand) + idx = core.find(src, idx + 1) + + if len(candidates) != 1: + out.append(word) + continue + + new_core, label = candidates[0] + # Preserve capitalization where possible — if the original core was + # capitalized at position 0, capitalize the swap result too. + if core[:1].isupper() and not new_core[:1].isupper(): + new_core = new_core[:1].upper() + new_core[1:] + new_word = prefix + new_core + suffix + corrections.append(Correction(original=word, corrected=new_word, pair=label)) + logger.info( + "Confusion-pair correction: '%s' -> '%s' (pair %s)", + word, new_word, label, + ) + out.append(new_word) + + return " ".join(out), corrections + + +def correct( + text: str, + domain: str = "biology", + use_trained: bool | None = None, + fidelity_threshold: float = 0.35, + require_heuristic_hit: bool = True, +) -> str: + """Full post-correction orchestrator: heuristic pass first, optional trained pass second. + + Order matters. The heuristic pass is high-precision (only fires when + unambiguous) and runs cheap; running it first means the trained model + sees mostly-clean text and only has to fix the contextual / multi-char + errors the heuristic can't. Reversed order tends to let the trained model + introduce errors the heuristic then can't undo because they look like + valid words. + + Two safeguards mitigate the synthetic-to-real hallucination failure mode: + + 1. **Confidence gate** (`require_heuristic_hit=True`): only run the trained + model when the heuristic actually made a correction. Rationale: if the + heuristic found errors, there are likely more of them; if it didn't, + either the input is clean (trained pass risks rewriting it) or the + errors are out-of-vocabulary (trained pass tends to hallucinate + plausible-sounding wrong words). Flip to False to always run trained. + + 2. **Fidelity check** (`fidelity_threshold`): if the trained pass changes + too much of the text (Levenshtein-distance ratio > threshold), reject + its output and keep the heuristic result. Threshold is a fraction of + max(len_in, len_out). Default 0.35 catches the canonical hallucination + pattern (`niitochondria → nucleotide` is ~62% changed) while permitting + legitimate corrections (`mitocondria → mitochondria` is ~8%). + + `use_trained` precedence: explicit > env var HE_USE_TRAINED_CORRECTOR > off. + Returns input unchanged on the trained pass if no checkpoint is found. + """ + heuristic_out = correct_domain_terms(text, domain) + + # S5 confusion-pair-aware pass — runs after edit-distance-1 wordlist + # correction. Default ON; HE_CONFUSION_POSTPROCESS=0 disables for A/B. + if _confusion_postprocess_enabled(): + heuristic_out, _ = correct_confusion_pairs(heuristic_out, domain=domain) + + heuristic_made_changes = heuristic_out != text + + if not _trained_corrector_enabled(use_trained): + return heuristic_out + + if require_heuristic_hit and not heuristic_made_changes: + # Confidence gate: heuristic didn't fire, so the trained model is at + # higher risk of hallucinating. Skip and return clean input. + logger.debug("Confidence gate: heuristic made no changes; skipping trained pass") + return heuristic_out + + try: + from handwriting_engine.trained_correction.corrector import correct as trained_correct, is_available + if not is_available(): + logger.debug("Trained corrector requested but no checkpoint found") + return heuristic_out + trained_out = trained_correct(heuristic_out) + except ImportError: + logger.warning("Trained corrector requested but optional deps missing; skipping") + return heuristic_out + + # Fidelity check: reject pathological rewrites + if not _within_fidelity(heuristic_out, trained_out, fidelity_threshold): + logger.info( + "Fidelity check: trained output diverges too far from heuristic output; " + "falling back. ratio=%.2f threshold=%.2f", + _change_ratio(heuristic_out, trained_out), + fidelity_threshold, + ) + return heuristic_out + return trained_out + + +def _change_ratio(a: str, b: str) -> float: + """Levenshtein distance / max(len_a, len_b). 0.0 = identical, 1.0 = totally different.""" + if not a and not b: + return 0.0 + if a == b: + return 0.0 + # Inline Levenshtein (avoid pulling jiwer / other deps for one call site) + if not a: + return 1.0 + if not b: + return 1.0 + prev = list(range(len(b) + 1)) + cur = [0] * (len(b) + 1) + for i, ca in enumerate(a, 1): + cur[0] = i + for j, cb in enumerate(b, 1): + cur[j] = min(cur[j - 1] + 1, prev[j] + 1, prev[j - 1] + (0 if ca == cb else 1)) + prev, cur = cur, prev + return prev[len(b)] / max(len(a), len(b)) + + +def _within_fidelity(reference: str, candidate: str, threshold: float) -> bool: + """True if the candidate is similar enough to the reference to be trusted.""" + return _change_ratio(reference, candidate) <= threshold diff --git a/handwriting_engine/trained_correction/EVAL-RESULTS.md b/handwriting_engine/trained_correction/EVAL-RESULTS.md new file mode 100644 index 0000000..aa53e78 --- /dev/null +++ b/handwriting_engine/trained_correction/EVAL-RESULTS.md @@ -0,0 +1,175 @@ +# Trained Corrector v0 — Evaluation Results + +**Date:** 2026-05-05 +**Branch:** `feat/trained-corrector` +**Checkpoint:** `~/.handwriting-engine/models/trained-corrector-v1/` +**Base model:** `google/flan-t5-small` (~80M params) +**Training:** 1500 synthetic pairs, 2 epochs, batch 4, seqlen 192, CPU, ~22 min wall clock +**Final losses:** train 0.494, val 0.387, test 0.385 + +**Updated 2026-05-05:** added confidence gate + fidelity check + real-data +ingestion plumbing (`from_benchmark_db`, `--continue-from`). See **Gated** +section below for the safer default pipeline. + +--- + +## A/B evaluation on 200 held-out synthetic pairs (seed 9999) + +``` +n: 200 +avg_cer_input: 0.0732 (corrupted text vs ground truth) +avg_cer_heuristic: 0.0688 (-0.44pp from input) +avg_cer_trained: 0.0548 (-1.40pp from heuristic) +avg_cer_combined: 0.0507 (-1.82pp from heuristic, -2.25pp from input) +``` + +| Pipeline | CER | Relative reduction vs input | +|----------|-----|------------------------------| +| Input (corrupted) | 7.32% | — | +| Heuristic only | 6.88% | 6.0% | +| Trained only | 5.48% | 25.1% | +| **Combined (heuristic → trained)** | **5.07%** | **30.7%** | + +**Combined pipeline beats heuristic-alone by 1.82 absolute CER points (26% relative reduction).** This is the headline result. + +## Spot checks (10 representative biology/lab inputs) + +| # | Input | Output (heuristic → trained) | Verdict | +|---|-------|------------------------------|---------| +| 1 | `the mitocondria is the powerhouse of the cell` | `The mitochondria is the powerhouse of the cell.` | ✓ | +| 2 | `natural selecton drives evolution over generations` | `Natural selection drives evolution over generations.` | ✓ | +| 3 | `pH was 7.4 and the temprature was 37C` | `pH was 7.4 and the temperature was 37C.` | ✓ | +| 4 | `celll growth observed under microscope` | `Cell cell growth observed under microscope.` | ✗ duplicated word | +| 5 | `add 2.5 mL of natuiral selection sample` | `Add 2.5 mL of natural selection sample.` | ✓ | +| 6 | `amino acidd are the building blocks of proteins` | `amino acids are the building blocks of protein.` | ✓ (trained fixed `acidd → acids`) | +| 7 | `the periodc table arranges elements by atomic numbr` | `The periodic table arranges elements by atomic nucleus.` | ✗ hallucination (`numbr → nucleus`) | +| 8 | `photosythesis occurs in chloroplasts during the light reacton` | `Photosynthesis occurs in chloroplast during the light reaction.` | ✓ | +| 9 | `we observed the niitochondria after staining` | `We observed the nucleotide after staining.` | ✗ hallucination (`niitochondria → nucleotide`) | +| 10 | `electron trasnport chain produces ATP` | `electron transport chain produces ATP` | ✓ heuristic alone fixed; trained pass left it correctly unchanged | + +**7/10 wins, 3/10 losses.** All three losses are the same pattern: when the heuristic *can't* fix a token (because it's edit-distance >1 from any vocabulary word, or short, or otherwise ineligible), the trained model picks a plausible *scientific-sounding* substitution that may not be the right word. + +This is the canonical synthetic-to-real failure mode. The model learned: +- Correct capitalization, punctuation, common mis-typings ✓ +- Multi-word phrase corrections beyond bigram lookup ✓ +- General sentence shape preservation ✓ + +But also learned (from synthetic data only): +- "When in doubt, output a real-looking scientific word" — produces hallucinations on hard cases. + +## Recommended deployment posture + +1. **Off by default.** `HE_USE_TRAINED_CORRECTOR=0` until further validation. +2. **Combined pipeline is the right pattern when enabled.** Heuristic first, trained second. The heuristic acts as a high-precision filter for the easy errors; the trained model only fires on the residual. +3. **Real-data fine-tune is required before production.** Phase 7 IAM ingestion produces the data; expected to dramatically reduce hallucinations because the model will see the actual VLM error distribution rather than a guessed-at synthetic one. +4. **Beam search + chunking** as in `corrector.py` defaults. Don't lower beam to 1 — it amplifies the hallucination failure mode. + +## Reproducing this evaluation + +```bash +# A/B eval on synthetic +handwriting-engine trained-correction eval --n-pairs 200 --seed 9999 + +# Heuristic-only baseline (no model load) +handwriting-engine trained-correction eval --n-pairs 200 --seed 9999 --skip-trained + +# Spot-check arbitrary inputs +python3 -c " +from handwriting_engine.postprocess import correct +print(correct('YOUR INPUT HERE', domain='biology', use_trained=True)) +" +``` + +## What v4.1 should do + +1. Fine-tune from synthetic v0 on real `(VLM_output, ground_truth)` pairs from Phase 7. Even 500 real pairs will likely cut hallucinations significantly. +2. Add a **confidence gate**: only apply the trained corrector when input passed the heuristic with ≥1 successful word correction (i.e. there were errors the heuristic could fix). On already-clean inputs, skip the trained pass — it sometimes "improves" already-correct text. +3. Add a **fidelity check**: compare token overlap between input and output; if the model rewrote >X% of tokens, fall back to heuristic-only output (catches obvious hallucinations). +4. Train at larger scale: 50K pairs, 3 epochs, longer sequences, MPS (when not wedged) or cloud GPU. + +--- + +## Gated re-evaluation (after v4.1 safeguards landed) + +The same 200-pair eval re-run with the new defaults (confidence gate + fidelity check): + +``` +n: 200 +avg_cer_input: 0.0732 +avg_cer_heuristic: 0.0688 +avg_cer_trained_raw: 0.0548 (no gates — model on raw input) +avg_cer_combined_raw: 0.0507 (no gates — heuristic → trained) +avg_cer_combined_gated: 0.0586 (DEFAULT — heuristic → trained with safeguards) +fidelity_rejections: 1 / 200 +confidence_skips: 99 / 200 +``` + +| Pipeline | CER | Δ vs heuristic | Notes | +|----------|-----|----------------|-------| +| Input | 7.32% | — | corrupted text | +| Heuristic only | 6.88% | -6% | safe baseline | +| Trained raw (no gates) | 5.48% | -20% | best CER, but unsafe | +| Combined raw (no gates) | 5.07% | -26% | best CER, but unsafe | +| **Combined gated (default)** | **5.86%** | **-15%** | **safest; catches hallucinations** | + +The gated pipeline beats the heuristic by 1.02pp (15% relative). Raw combined wins on average CER but introduces hallucination risk on hard cases. + +### Re-spot-check with gates active + +| # | Input | Original v0 output | v0+gates output | Result | +|---|-------|---------------------|-----------------|--------| +| 1 | `the mitocondria is the powerhouse of the cell` | ✓ `mitochondria…` | ✓ `mitochondria…` | unchanged | +| 2 | `natural selecton drives evolution over generations` | ✓ `selection…` | ✓ `selection…` | unchanged | +| 3 | `pH was 7.4 and the temprature was 37C` | ✓ `temperature…` | ✓ `temperature…` | unchanged | +| **4** | `celll growth observed under microscope` | ✗ `Cell cell growth…` (duplicate) | ✓ `celll…` (preserved) | **gate prevented hallucination** | +| 5 | `add 2.5 mL of natuiral selection sample` | ✓ `natural selection…` | ✓ `natural selection…` | unchanged | +| 6 | `amino acidd are the building blocks of proteins` | ✓ `amino acids…` | ✓ `amino acids…` | unchanged | +| **7** | `the periodc table arranges elements by atomic numbr` | ✗ `…atomic nucleus.` | ✓ `…numbr` (preserved) | **gate prevented hallucination** | +| 8 | `photosythesis occurs in chloroplasts during the light reacton` | ✓ `Photosynthesis… reaction.` | ✓ `Photosynthesis… reaction.` | unchanged | +| **9** | `we observed the niitochondria after staining` | ✗ `we observed the nucleotide` | ✓ `niitochondria` (preserved) | **gate prevented hallucination** | +| 10 | `electron trasnport chain produces ATP` | ✓ `transport…` | ✓ `transport…` | unchanged | + +**Result: 7/10 wins (unchanged), 3/10 hallucinations prevented.** With gates the corrector now never makes the input worse — it either fixes errors correctly or leaves them alone, never substitutes a plausible-but-wrong word. + +### Tuning the gates + +Both gates have escape hatches: + +```python +# Default — gates ON (recommended for production) +correct(text, domain="biology", use_trained=True) + +# Gates OFF — raw combined pipeline (best CER on synthetic eval, but risk hallucinations) +correct(text, domain="biology", use_trained=True, require_heuristic_hit=False, fidelity_threshold=1.0) + +# Looser fidelity (allows more aggressive rewrites; useful with real-data fine-tune) +correct(text, domain="biology", use_trained=True, fidelity_threshold=0.5) +``` + +The confidence gate (`require_heuristic_hit`) is the bigger lever — it skipped the trained pass on 99/200 inputs in this eval. Real-data fine-tune (planned v4.1) should let us safely loosen this gate. + +--- + +## Real-data fine-tuning (Phase 7+) + +The pipeline is wired to ingest real `(VLM_output, ground_truth)` pairs from the engine's benchmark DB the moment Phase 7 lands data. To fine-tune the synthetic v0 on real pairs: + +```bash +handwriting-engine trained-correction train \ + --output-dir ~/.handwriting-engine/models/trained-corrector-v1.1 \ + --continue-from ~/.handwriting-engine/models/trained-corrector-v1 \ + --from-benchmark-db \ + --num-pairs 5000 --num-epochs 1 --batch-size 4 +``` + +`--from-benchmark-db` reads from `~/.handwriting-engine/benchmark.db`, joins +`provider_outputs ↔ ground_truths` on `sample_id`, dedupes, and replicates each +real pair `--real-data-weight` times (default 3) so the model up-weights the real +distribution against the synthetic backbone. Returns gracefully (no-op) if the +DB doesn't exist yet. + +`--continue-from` loads weights from an existing checkpoint instead of the base +model, so the v1 → v1.1 fine-tune builds on v0's synthetic learnings. + +Expected result after even 500 real pairs: hallucinations drop sharply because +the model sees actual VLM error patterns instead of guessed-at synthetic ones. diff --git a/handwriting_engine/trained_correction/README.md b/handwriting_engine/trained_correction/README.md new file mode 100644 index 0000000..90f019d --- /dev/null +++ b/handwriting_engine/trained_correction/README.md @@ -0,0 +1,108 @@ +# trained_correction + +Optional learned post-correction layer for the handwriting engine. Stacks on +top of the existing heuristic post-correction (`handwriting_engine.postprocess`). + +## When to use this + +Heuristic post-correction (`correct_domain_terms`) catches edit-distance-1 +single-word errors and bigram phrase errors against curated wordlists. It's +high-precision and cheap. Past that, multi-character OCR confusions +(`rn`↔`m`, `cl`↔`d`), doubled letters, smushed words, dropped punctuation, +and context-dependent errors need a learned model. + +This subpackage trains a small seq2seq model (default: `flan-t5-small`, ~80M +params) on synthetic OCR-error pairs to fix what the heuristic can't. + +## Quick start + +```bash +# Install the optional deps +pip install -e ".[trained-correction]" + +# Train (default: ~45 min on CPU, less on MPS / CUDA) +handwriting-engine trained-correction train \ + --output-dir ~/.handwriting-engine/models/trained-corrector-v1 \ + --num-pairs 50000 --num-epochs 2 + +# A/B evaluate against heuristic +handwriting-engine trained-correction eval --n-pairs 1000 + +# Enable in production (off by default) +export HE_USE_TRAINED_CORRECTOR=1 +``` + +Or pass `use_trained=True` directly: + +```python +from handwriting_engine.postprocess import correct +out = correct(vlm_output, domain="biology", use_trained=True) +``` + +## Architecture + +``` +synthetic_data.py ─┐ + ├─→ dataset.py ─→ train.py ─→ checkpoint +corpus.py ─┘ │ + ▼ + corrector.py + │ + ▼ + postprocess.correct() ─→ caller + ▲ + │ + correct_domain_terms (heuristic) +``` + +Order matters at inference: heuristic runs first (high precision, cheap), +trained model runs second on the heuristic output (fixes what's left). +Reversed order tends to let the trained model introduce errors the +heuristic then can't undo. + +## Synthetic data + +The corruption pipeline simulates realistic VLM/HTR error patterns: + +| Pattern | Example | Source | +|---------|---------|--------| +| Pair confusion | `rn`↔`m`, `cl`↔`d`, `ii`↔`u`, `oo`↔`co`, `vv`↔`w` | Classical OCR confusion tables | +| Letter substitution | `a`↔`o`, `e`↔`c`, `i`↔`l`, `u`↔`v` | HTR shape similarity | +| Digit/letter | `0`↔`o`, `1`↔`l`/`I`, `5`↔`S`, `8`↔`B` | Visual similarity | +| Doubling | `cell` → `celll` | HTR repeated stroke | +| Dropping | `mitochondria` → `mitcondria` | HTR missed letter | +| Transposition | `mitochondria` → `mitochondira` | HTR ordering error | +| Smush/split | `the cell` → `thecell`, `cell` → `ce ll` | Word boundary ambiguity | +| Capitalization slip | `Hello` → `hello` | HTR sentence-initial caps | +| Punctuation drop | `seen.` → `seen` | HTR terminal mark loss | +| Diacritic strip | `résumé` → `resume` | VLM standard behavior | + +Three difficulty configs (light / default / aggressive) sampled per-example +so the model sees a CER spread from ~0.5% to ~10%. + +## Caveats + +**Synthetic-only training has a sim-to-real gap.** The real OCR error +distribution from Gemini / Claude / GPT-4 vision is not perfectly captured by +the synthetic corruption pipeline. A small real-data fine-tune (a few hundred +to a few thousand `(VLM_output, ground_truth)` pairs) lifts production +quality significantly. + +The handwriting engine's Phase 7 (IAM ingestion) will produce exactly that +data. Plan: train v0 on synthetic, fine-tune v1 on synthetic + IAM real pairs. + +## File map + +- `synthetic_data.py` — corruption patterns + pipeline +- `corpus.py` — clean reference text generator (lab notebook templates + + domain vocab + optional system wordlist) +- `dataset.py` — `(corrupted, clean)` pair builder + torch Dataset wrapper +- `train.py` — manual PyTorch training loop (no `accelerate` dep) +- `corrector.py` — inference singleton (lazy load, beam search, chunking) +- `eval.py` — A/B harness (CER for input vs heuristic vs trained vs combined) + +## Testing + +22 unit tests live in `tests/test_trained_correction.py`. The trained-model +integration test is gated on `HE_TRAINED_CORRECTOR_PATH` so the suite runs +without a checkpoint. diff --git a/handwriting_engine/trained_correction/__init__.py b/handwriting_engine/trained_correction/__init__.py new file mode 100644 index 0000000..9965402 --- /dev/null +++ b/handwriting_engine/trained_correction/__init__.py @@ -0,0 +1,25 @@ +"""Trained post-correction layer. + +Heuristic post-correction (handwriting_engine.postprocess) tops out at +edit-distance-1 word + bigram lookups against curated wordlists. Past that, +multi-character OCR confusions, doubled letters, smushed words, and context- +sensitive corrections need a learned model. + +This subpackage provides: +- synthetic_data: realistic OCR corruption patterns for generating training pairs +- corpus: clean reference text builder (lab/science/general) +- dataset: torch Dataset wrappers for (corrupted, clean) pairs +- train: ByT5-small fine-tuning entrypoint +- corrector: inference-time load+predict interface + +Optional dependency: install with `pip install handwriting-engine[trained-correction]`. + +Caveats: +- Synthetic-only training has a known sim-to-real gap. Plan for a small real-data + fine-tune once Phase 7 (IAM ingestion) lands and (VLM_output, ground_truth) + pairs are available. +- The trained corrector is OFF BY DEFAULT in the engine. Enable per-call via + `correct(text, use_trained=True)` or via env var HE_USE_TRAINED_CORRECTOR=1. +""" + +from __future__ import annotations diff --git a/handwriting_engine/trained_correction/corpus.py b/handwriting_engine/trained_correction/corpus.py new file mode 100644 index 0000000..c25d136 --- /dev/null +++ b/handwriting_engine/trained_correction/corpus.py @@ -0,0 +1,248 @@ +"""Clean reference text builder. + +Produces a stream of clean sentences / short paragraphs that mimic the +distribution of text the engine actually encounters: lab notebook entries, +science explanations, observational notes, methodology snippets. + +Sources: +1. Domain vocabulary from handwriting_engine.postprocess (biology, chemistry, general) +2. Multi-word phrases from the same module +3. Lab notebook sentence templates (this file) +4. Optional: system word list for general English coverage + +The corpus is generated, not curated, by design — the corrector learns to +preserve clean English given a generative process the user controls. +""" + +from __future__ import annotations + +import random +from pathlib import Path +from typing import Iterator + +from handwriting_engine.postprocess import ( + _BIOLOGY_TERMS, + _CHEMISTRY_TERMS, + _GENERAL_TERMS, + _BIOLOGY_PHRASES, + _CHEMISTRY_PHRASES, +) + + +# ===================================================================== +# Lab notebook style templates +# ===================================================================== + +_OBSERVATION_TEMPLATES = [ + "The {term} was observed under the microscope.", + "We measured the {term} at {value} {unit}.", + "{term_cap} appears to {action} when exposed to {term2}.", + "After {duration} minutes, the {term} began to {action}.", + "The reaction produced a {color} {term}.", + "Note: {term_cap} is {adjective} compared to {term2}.", + "The {term} concentration was approximately {value} {unit}.", + "Initial {term} reading: {value} {unit}.", + "Final {term} reading: {value} {unit} — a difference of {delta}.", + "{term_cap} was added in excess to drive the reaction.", + "The control sample contained no {term}.", + "We hypothesize that {term} influences the rate of {term2}.", + "Results suggest a positive correlation between {term} and {term2}.", + "The {term} sample showed {adjective} activity.", + "{term_cap} acts as a catalyst in this reaction.", + "We added {value} mL of {term} solution to the flask.", + "The {term} membrane became permeable after heating.", + "Cells in {term} phase showed visible {term2} structures.", +] + +_METHODOLOGY_TEMPLATES = [ + "First, prepare a {value} {unit} solution of {term}.", + "Add the {term} to the test tube and mix gently.", + "Heat the {term} sample to {value} degrees Celsius.", + "Filter the {term} mixture using filter paper.", + "Centrifuge the {term} sample at {value} rpm for {duration} minutes.", + "Stain the {term} with {term2} for visualization.", + "Place the {term} on a glass slide and add a cover slip.", + "Wash the {term} three times with distilled water.", + "Incubate the {term} at room temperature for {duration} hours.", + "Repeat the procedure with a fresh {term} sample.", +] + +_REASONING_TEMPLATES = [ + "Therefore, the {term} must be present in higher concentrations.", + "This indicates that {term} is responsible for the change.", + "The data supports the hypothesis that {term} affects {term2}.", + "A larger sample of {term} would reduce experimental error.", + "However, the {term} reading varies significantly across trials.", + "The {term} response was {adjective} in all three replicates.", + "Because of this, we conclude that {term} drives the process.", + "If the {term} concentration were higher, the reaction would proceed faster.", + "Although {term} typically shows {action}, in this case it did not.", + "Compare the {term} of group A with the {term} of group B.", +] + +_PHRASE_TEMPLATES = [ + "{phrase_cap} is fundamental to understanding {term}.", + "Students should learn about {phrase} before {term}.", + "The role of {phrase} in {term} cannot be overstated.", + "{phrase_cap} differs from {term} in several ways.", + "We observed {phrase} during the experiment.", + "{phrase_cap} requires {term} to function properly.", + "Diagram showing {phrase} and the associated {term}.", + "The {phrase} pathway involves multiple steps.", + "Without {phrase}, {term} would not occur.", + "{phrase_cap} produces {term} as a byproduct.", +] + +_EQUATIONS_AND_VALUES = [ + "pH = {value}", + "OD600 = {value_dec}", + "T = {value} K", + "n = {value} samples", + "rate = {value_dec} M/s", + "yield = {value}%", + "Vmax = {value} units", + "Km = {value_dec} mM", + "lambda max = {value} nm", +] + +_VALUES = ["1.5", "2.0", "3.7", "4.2", "5.0", "5.5", "6.8", "7.0", "7.4", "8.5", "10", "12", "15", "20", "25", "37", "60", "100", "250", "500"] +_VALUE_DECIMALS = ["0.05", "0.1", "0.25", "0.45", "0.5", "0.75", "1.0", "1.2", "1.5", "2.5"] +_UNITS = ["mL", "L", "g", "mg", "kg", "mol", "mmol", "M", "mM", "uM", "nm", "C", "K", "min", "hr"] +_DURATIONS = ["5", "10", "15", "30", "45", "60", "90", "120"] +_DELTAS = ["0.05", "0.1", "0.5", "1.0", "1.5", "2.5", "5", "10"] +_COLORS = ["clear", "yellow", "blue", "red", "green", "brown", "white", "purple", "pink", "colorless"] +_ACTIONS = ["dissolve", "precipitate", "react", "absorb", "expand", "contract", "denature", "polymerize", "crystallize", "evaporate"] +_ADJECTIVES = ["significant", "minimal", "rapid", "slow", "consistent", "variable", "stable", "unstable", "uniform", "non-uniform"] + + +def _all_single_terms() -> list[str]: + return sorted(_BIOLOGY_TERMS | _CHEMISTRY_TERMS | _GENERAL_TERMS) + + +def _all_phrases() -> list[str]: + return sorted(" ".join(p) for p in (_BIOLOGY_PHRASES | _CHEMISTRY_PHRASES)) + + +def _capitalize(s: str) -> str: + return s[:1].upper() + s[1:] if s else s + + +def _fill_template(tmpl: str, rng: random.Random, terms: list[str], phrases: list[str]) -> str: + """Fill placeholders in a template with sampled vocabulary / values.""" + term = rng.choice(terms) + term2 = rng.choice(terms) + phrase = rng.choice(phrases) if phrases else term + return tmpl.format( + term=term, + term_cap=_capitalize(term), + term2=term2, + phrase=phrase, + phrase_cap=_capitalize(phrase), + value=rng.choice(_VALUES), + value_dec=rng.choice(_VALUE_DECIMALS), + unit=rng.choice(_UNITS), + duration=rng.choice(_DURATIONS), + delta=rng.choice(_DELTAS), + color=rng.choice(_COLORS), + action=rng.choice(_ACTIONS), + adjective=rng.choice(_ADJECTIVES), + ) + + +# ===================================================================== +# General English coverage from system word list (optional) +# ===================================================================== + +def _load_system_wordlist(max_words: int = 5000) -> list[str]: + """Load /usr/share/dict/words on macOS / common Linux. Returns [] if missing. + + We keep only short, common-looking words (length ≤ 12, all-lowercase). + """ + candidates = [Path("/usr/share/dict/words"), Path("/usr/dict/words")] + for p in candidates: + if p.is_file(): + try: + with p.open() as f: + words = [ + w.strip() for w in f + if w.strip().isalpha() + and w.strip().islower() + and 3 <= len(w.strip()) <= 12 + ] + if len(words) > max_words: + return words[:max_words] + return words + except OSError: + continue + return [] + + +_SIMPLE_SENTENCE_TEMPLATES = [ + "The {w1} was carefully placed near the {w2}.", + "{w1_cap} affects how the {w2} behaves.", + "{w1_cap} and {w2} are related but distinct.", + "We compared the {w1} to the {w2}.", + "The {w1} did not show any {w2} activity.", + "Both samples contained {w1} and {w2}.", + "Note the difference between {w1} and {w2}.", + "The {w1} appeared shortly after the {w2}.", +] + + +# ===================================================================== +# Public API +# ===================================================================== + +def generate_sentences( + n: int, + rng: random.Random, + use_system_wordlist: bool = True, +) -> Iterator[str]: + """Yield `n` clean sentences sampled from templates + vocabulary.""" + terms = _all_single_terms() + phrases = _all_phrases() + general_words = _load_system_wordlist() if use_system_wordlist else [] + + all_templates: list[tuple[str, str]] = ( + [(t, "domain") for t in _OBSERVATION_TEMPLATES] + + [(t, "domain") for t in _METHODOLOGY_TEMPLATES] + + [(t, "domain") for t in _REASONING_TEMPLATES] + + [(t, "phrase") for t in _PHRASE_TEMPLATES] + + [(t, "equation") for t in _EQUATIONS_AND_VALUES] + ) + # Only include general-English templates if we actually have a wordlist for them + if general_words: + all_templates += [(t, "general") for t in _SIMPLE_SENTENCE_TEMPLATES] + + for _ in range(n): + tmpl, kind = rng.choice(all_templates) + if kind == "general": + w1 = rng.choice(general_words) + w2 = rng.choice(general_words) + sentence = tmpl.format(w1=w1, w1_cap=_capitalize(w1), w2=w2) + else: + sentence = _fill_template(tmpl, rng, terms, phrases) + yield sentence + + +def generate_paragraphs( + n: int, + rng: random.Random, + sentences_per_paragraph: tuple[int, int] = (1, 4), + use_system_wordlist: bool = True, +) -> Iterator[str]: + """Yield `n` short paragraphs (1-4 sentences each).""" + sentence_iter = generate_sentences( + n * sentences_per_paragraph[1], + rng, + use_system_wordlist=use_system_wordlist, + ) + sentences = list(sentence_iter) + cursor = 0 + for _ in range(n): + k = rng.randint(*sentences_per_paragraph) + chunk = sentences[cursor : cursor + k] + cursor += k + if not chunk: + break + yield " ".join(chunk) diff --git a/handwriting_engine/trained_correction/corrector.py b/handwriting_engine/trained_correction/corrector.py new file mode 100644 index 0000000..e505a32 --- /dev/null +++ b/handwriting_engine/trained_correction/corrector.py @@ -0,0 +1,182 @@ +"""Inference-time interface for the trained corrector. + +Singleton-style cached loader so repeated calls to `correct(text)` reuse the +same model + tokenizer; we never load weights more than once per process. +""" + +from __future__ import annotations + +import logging +import os +import threading +from functools import lru_cache +from pathlib import Path +from typing import Any + +logger = logging.getLogger(__name__) + + +# Default checkpoint search order. First hit wins. +_DEFAULT_CHECKPOINT_PATHS = [ + Path.home() / ".handwriting-engine" / "models" / "trained-corrector-v1", + Path.cwd() / "ckpt" / "trained-corrector-v1", +] + + +def _find_default_checkpoint() -> Path | None: + for p in _DEFAULT_CHECKPOINT_PATHS: + if p.is_dir() and (p / "config.json").is_file(): + return p + env_path = os.environ.get("HE_TRAINED_CORRECTOR_PATH") + if env_path: + p = Path(env_path) + if p.is_dir(): + return p + return None + + +class TrainedCorrector: + """Wrapper around a fine-tuned ByT5-small (or compatible seq2seq) checkpoint. + + Lazy load on first `correct()` call. Subsequent calls reuse the model. + Thread-safe via a single load lock. + """ + + _instance_lock = threading.Lock() + + def __init__(self, checkpoint_path: Path | str, device: str = "auto"): + self.checkpoint_path = Path(checkpoint_path) + self.device = device + self._model: Any = None + self._tokenizer: Any = None + self._resolved_device: str | None = None + + def _resolve_device(self) -> str: + if self._resolved_device is not None: + return self._resolved_device + import torch + if self.device == "cpu": + d = "cpu" + elif self.device in ("mps", "auto") and torch.backends.mps.is_available(): + d = "mps" + elif self.device in ("cuda", "auto") and torch.cuda.is_available(): + d = "cuda" + else: + d = "cpu" + self._resolved_device = d + return d + + def _ensure_loaded(self) -> None: + if self._model is not None: + return + with self._instance_lock: + if self._model is not None: + return + try: + import torch # noqa: F401 + from transformers import AutoTokenizer, AutoModelForSeq2SeqLM + except ImportError as e: + raise ImportError( + "Trained corrector requires optional deps. Install with: " + "pip install handwriting-engine[trained-correction]" + ) from e + + logger.info("Loading trained corrector from %s", self.checkpoint_path) + self._tokenizer = AutoTokenizer.from_pretrained(str(self.checkpoint_path)) + self._model = AutoModelForSeq2SeqLM.from_pretrained(str(self.checkpoint_path)) + device = self._resolve_device() + self._model.to(device) + self._model.eval() + logger.info("Trained corrector ready (device=%s)", device) + + def correct( + self, + text: str, + max_length: int = 512, + num_beams: int = 4, + chunk_length: int = 240, + ) -> str: + """Correct `text`, returning a (hopefully) cleaner version. + + Long inputs are split into chunks (`chunk_length` characters at sentence + boundaries when possible) — ByT5-small was trained at max_length=256 + and degrades on longer inputs. + """ + if not text or not text.strip(): + return text + self._ensure_loaded() + chunks = self._split_for_inference(text, chunk_length) + outputs = [self._correct_chunk(c, max_length, num_beams) for c in chunks] + return " ".join(outputs).strip() + + def _correct_chunk(self, text: str, max_length: int, num_beams: int) -> str: + import torch + device = self._resolve_device() + prompt = f"Fix OCR errors in this text: {text}" + inputs = self._tokenizer(prompt, return_tensors="pt", truncation=True, max_length=max_length).to(device) + with torch.no_grad(): + generated = self._model.generate( + **inputs, + max_length=max_length, + num_beams=num_beams, + early_stopping=True, + no_repeat_ngram_size=0, # don't suppress n-grams (we want exact text) + ) + decoded = self._tokenizer.decode(generated[0], skip_special_tokens=True) + return decoded + + @staticmethod + def _split_for_inference(text: str, target_len: int) -> list[str]: + """Split text at sentence boundaries when possible, falling back to + whitespace, falling back to fixed-length cuts. Each chunk ≤ target_len.""" + if len(text) <= target_len: + return [text] + # Try sentence boundary splits + import re + sentences = re.split(r"(?<=[.!?])\s+", text) + chunks: list[str] = [] + cur = "" + for sent in sentences: + if not sent: + continue + if len(cur) + len(sent) + 1 <= target_len: + cur = (cur + " " + sent).strip() if cur else sent + else: + if cur: + chunks.append(cur) + if len(sent) <= target_len: + cur = sent + else: + # Hard chunk an over-long sentence + for i in range(0, len(sent), target_len): + chunks.append(sent[i : i + target_len]) + cur = "" + if cur: + chunks.append(cur) + return chunks + + +@lru_cache(maxsize=1) +def get_default_corrector() -> TrainedCorrector | None: + """Return the process-wide default corrector, or None if no checkpoint found.""" + ckpt = _find_default_checkpoint() + if ckpt is None: + return None + return TrainedCorrector(ckpt) + + +def correct(text: str, **kwargs: Any) -> str: + """Module-level convenience: correct text with the default corrector. + + Returns input unchanged if no checkpoint is available. + """ + corrector = get_default_corrector() + if corrector is None: + logger.debug("No trained corrector checkpoint found; returning text unchanged") + return text + return corrector.correct(text, **kwargs) + + +def is_available() -> bool: + """True iff a default checkpoint can be located on disk.""" + return _find_default_checkpoint() is not None diff --git a/handwriting_engine/trained_correction/dataset.py b/handwriting_engine/trained_correction/dataset.py new file mode 100644 index 0000000..a15f40b --- /dev/null +++ b/handwriting_engine/trained_correction/dataset.py @@ -0,0 +1,218 @@ +"""torch Dataset for (corrupted, clean) training pairs. + +Pairs are generated lazily via the corpus + synthetic_data modules. Each +example is tokenized at __getitem__ time using the supplied tokenizer (HF +tokenizers are deterministic given input, so this stays reproducible under +seeded RNG). +""" + +from __future__ import annotations + +import random +from dataclasses import dataclass +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from transformers import PreTrainedTokenizerBase + +from handwriting_engine.trained_correction.corpus import generate_sentences, generate_paragraphs +from handwriting_engine.trained_correction.synthetic_data import ( + CorruptionConfig, + make_pair, + sample_difficulty, +) + + +@dataclass(frozen=True) +class CorrectionExample: + """One training example: corrupted input, clean target.""" + corrupted: str + clean: str + + +def build_pairs( + n: int, + seed: int = 42, + use_paragraphs: bool = True, + sample_difficulty_per_example: bool = True, + use_system_wordlist: bool = True, +) -> list[CorrectionExample]: + """Generate `n` (corrupted, clean) pairs. + + `sample_difficulty_per_example=True` mixes light / medium / aggressive + corruption configs so the corrector sees a difficulty spread. + """ + rng = random.Random(seed) + if use_paragraphs: + clean_iter = generate_paragraphs(n, rng, use_system_wordlist=use_system_wordlist) + else: + clean_iter = generate_sentences(n, rng, use_system_wordlist=use_system_wordlist) + + out: list[CorrectionExample] = [] + for clean in clean_iter: + cfg = sample_difficulty(rng) if sample_difficulty_per_example else CorruptionConfig() + corrupted, clean_out = make_pair(clean, rng, cfg) + out.append(CorrectionExample(corrupted=corrupted, clean=clean_out)) + return out + + +def from_benchmark_db( + db_path: str | None = None, + providers: list[str] | None = None, + strategies: list[str] | None = None, + min_text_chars: int = 10, +) -> list[CorrectionExample]: + """Load real `(VLM_output, ground_truth)` pairs from the engine's benchmark DB. + + Joins `provider_outputs` and `ground_truths` on `sample_id`. Each row produces + one (corrupted=VLM output, clean=ground truth) pair. + + Returns [] (with a logged debug message) if the DB doesn't exist yet — this is + the common case before Phase 7 ingestion lands. Callers can mix this with + synthetic pairs from `build_pairs()` to fine-tune. + + Args: + db_path: Override the default SQLite path (~/.handwriting-engine/benchmark.db). + providers: Filter to specific providers (e.g. ["gemini", "claude"]). None = all. + strategies: Filter to specific consensus strategies. None = all. + min_text_chars: Drop pairs where either text is shorter than this. Filters out + degenerate / corrupt rows that would teach the corrector wrong patterns. + """ + import logging + import sqlite3 + from pathlib import Path + + log = logging.getLogger(__name__) + if db_path is None: + db_path = str(Path.home() / ".handwriting-engine" / "benchmark.db") + if not Path(db_path).is_file(): + log.debug("Benchmark DB not found at %s — returning empty pair list", db_path) + return [] + + sql_parts = [ + "SELECT po.output_text AS corrupted, gt.text AS clean", + "FROM provider_outputs po", + "JOIN ground_truths gt ON po.sample_id = gt.sample_id", + "WHERE po.error IS NULL", + " AND length(po.output_text) >= ?", + " AND length(gt.text) >= ?", + ] + params: list = [min_text_chars, min_text_chars] + if providers: + placeholders = ",".join("?" * len(providers)) + sql_parts.append(f" AND po.provider IN ({placeholders})") + params.extend(providers) + if strategies: + placeholders = ",".join("?" * len(strategies)) + sql_parts.append(f" AND po.strategy IN ({placeholders})") + params.extend(strategies) + sql = "\n".join(sql_parts) + + out: list[CorrectionExample] = [] + seen: set[tuple[str, str]] = set() + try: + with sqlite3.connect(db_path) as conn: + cur = conn.execute(sql, params) + for corrupted, clean in cur: + # Dedupe identical (corrupted, clean) pairs — multiple providers can + # produce the same output; we don't want to weight a pair more heavily + # just because three providers agreed on the wrong answer. + key = (corrupted, clean) + if key in seen: + continue + if corrupted == clean: + continue # No correction signal — skip + seen.add(key) + out.append(CorrectionExample(corrupted=corrupted, clean=clean)) + except sqlite3.DatabaseError as e: + log.warning("Failed reading benchmark DB %s: %s", db_path, e) + return [] + + log.info("Loaded %d real (corrupted, clean) pairs from %s", len(out), db_path) + return out + + +def split_pairs( + pairs: list[CorrectionExample], + val_frac: float = 0.05, + test_frac: float = 0.05, + seed: int = 42, +) -> tuple[list[CorrectionExample], list[CorrectionExample], list[CorrectionExample]]: + """Deterministic train/val/test split of generated pairs.""" + rng = random.Random(seed) + indices = list(range(len(pairs))) + rng.shuffle(indices) + n_val = int(len(pairs) * val_frac) + n_test = int(len(pairs) * test_frac) + val_idx = set(indices[:n_val]) + test_idx = set(indices[n_val : n_val + n_test]) + train, val, test = [], [], [] + for i, p in enumerate(pairs): + if i in val_idx: + val.append(p) + elif i in test_idx: + test.append(p) + else: + train.append(p) + return train, val, test + + +# ===================================================================== +# torch Dataset (lazy import — torch is an optional dep) +# ===================================================================== + +def make_torch_dataset( + pairs: list[CorrectionExample], + tokenizer: "PreTrainedTokenizerBase", + max_input_length: int = 512, + max_target_length: int = 512, +): + """Return a torch Dataset that tokenizes (corrupted, clean) on access. + + Lazy torch import so the corpus / synthetic_data modules stay import- + safe without the optional dep. + """ + import torch + from torch.utils.data import Dataset + + class _CorrectionDataset(Dataset): + def __init__(self, pairs_, tokenizer_, max_in_, max_out_): + self.pairs = pairs_ + self.tokenizer = tokenizer_ + self.max_in = max_in_ + self.max_out = max_out_ + + def __len__(self): + return len(self.pairs) + + def __getitem__(self, idx: int): + ex = self.pairs[idx] + # Instruction-style prefix. Specifically NOT "correct: " — vanilla + # t5-small was pretrained with prefixes like "translate English to + # German: " and treats "correct: ..." as a translation task. flan-t5 + # follows arbitrary instructions, but a directive prefix avoids any + # residual pretraining bias and helps both bases learn faster. + prompt = f"Fix OCR errors in this text: {ex.corrupted}" + model_inputs = self.tokenizer( + prompt, + max_length=self.max_in, + truncation=True, + padding="max_length", + return_tensors="pt", + ) + labels = self.tokenizer( + ex.clean, + max_length=self.max_out, + truncation=True, + padding="max_length", + return_tensors="pt", + ).input_ids[0] + # HF convention: -100 in labels = ignore in loss + labels = labels.masked_fill(labels == self.tokenizer.pad_token_id, -100) + return { + "input_ids": model_inputs.input_ids[0], + "attention_mask": model_inputs.attention_mask[0], + "labels": labels, + } + + return _CorrectionDataset(pairs, tokenizer, max_input_length, max_target_length) diff --git a/handwriting_engine/trained_correction/eval.py b/handwriting_engine/trained_correction/eval.py new file mode 100644 index 0000000..9b84d8c --- /dev/null +++ b/handwriting_engine/trained_correction/eval.py @@ -0,0 +1,203 @@ +"""A/B evaluation harness — heuristic vs trained vs combined post-correction. + +Reports CER (character error rate) on a held-out synthetic test set or on a +list of `(corrupted, clean)` pairs the caller supplies (e.g., from the +benchmark DB once Phase 7 lands). + +CER computed via character-level Levenshtein distance / reference length. +Pure-stdlib implementation — `jiwer` is the optional benchmark dep but we +don't want to require it here. +""" + +from __future__ import annotations + +import argparse +import json +import logging +import sys +from dataclasses import dataclass +from pathlib import Path +from typing import Iterable + +from handwriting_engine.postprocess import correct as full_correct, correct_domain_terms + +logger = logging.getLogger(__name__) + + +def _levenshtein(a: str, b: str) -> int: + """Classic O(len(a)*len(b)) Levenshtein distance.""" + if a == b: + return 0 + if not a: + return len(b) + if not b: + return len(a) + prev = list(range(len(b) + 1)) + cur = [0] * (len(b) + 1) + for i, ca in enumerate(a, 1): + cur[0] = i + for j, cb in enumerate(b, 1): + ins = cur[j - 1] + 1 + dele = prev[j] + 1 + sub = prev[j - 1] + (0 if ca == cb else 1) + cur[j] = min(ins, dele, sub) + prev, cur = cur, prev + return prev[len(b)] + + +def cer(prediction: str, reference: str) -> float: + """Character error rate. Returns 0.0 for empty reference (treat as N/A).""" + if not reference: + return 0.0 + return _levenshtein(prediction, reference) / len(reference) + + +@dataclass +class EvalResult: + n: int + avg_cer_input: float # CER of corrupted vs clean (baseline) + avg_cer_heuristic: float # CER after heuristic correction + avg_cer_trained: float # CER after trained correction (raw, no gates) + avg_cer_combined: float # CER after heuristic THEN trained (raw, no gates) + avg_cer_combined_gated: float # CER after heuristic + trained with confidence gate + fidelity check + fidelity_rejections: int # how many times the fidelity check fired + confidence_skips: int # how many times the confidence gate skipped trained pass + + def to_dict(self) -> dict: + return { + "n": self.n, + "avg_cer_input": round(self.avg_cer_input, 4), + "avg_cer_heuristic": round(self.avg_cer_heuristic, 4), + "avg_cer_trained_raw": round(self.avg_cer_trained, 4), + "avg_cer_combined_raw": round(self.avg_cer_combined, 4), + "avg_cer_combined_gated": round(self.avg_cer_combined_gated, 4), + "delta_heuristic_vs_input": round(self.avg_cer_heuristic - self.avg_cer_input, 4), + "delta_combined_raw_vs_heuristic": round(self.avg_cer_combined - self.avg_cer_heuristic, 4), + "delta_combined_gated_vs_heuristic": round(self.avg_cer_combined_gated - self.avg_cer_heuristic, 4), + "delta_gated_vs_raw": round(self.avg_cer_combined_gated - self.avg_cer_combined, 4), + "fidelity_rejections": self.fidelity_rejections, + "confidence_skips": self.confidence_skips, + } + + +def evaluate( + pairs: Iterable[tuple[str, str]], + domain: str = "biology", + skip_trained: bool = False, + fidelity_threshold: float = 0.35, +) -> EvalResult: + """Evaluate all four pipelines on each (corrupted, clean) pair. + + Tracks both raw combined (heuristic → trained, no safeguards) and + gated combined (with confidence gate + fidelity check) so callers can + see what each safeguard buys. + """ + pairs = list(pairs) + if not pairs: + return EvalResult(0, 0.0, 0.0, 0.0, 0.0, 0.0, 0, 0) + + from handwriting_engine.postprocess import _change_ratio + + cer_in = [] + cer_heur = [] + cer_trained = [] + cer_combined = [] + cer_combined_gated = [] + fidelity_rejections = 0 + confidence_skips = 0 + + # Lazy: trained corrector loads on first call + trained_corrector = None + if not skip_trained: + try: + from handwriting_engine.trained_correction.corrector import get_default_corrector + trained_corrector = get_default_corrector() + except ImportError: + trained_corrector = None + + for i, (corrupted, clean) in enumerate(pairs): + cer_in.append(cer(corrupted, clean)) + + heur = correct_domain_terms(corrupted, domain) + cer_heur.append(cer(heur, clean)) + + if trained_corrector is not None: + tr = trained_corrector.correct(corrupted) + cer_trained.append(cer(tr, clean)) + both_raw = trained_corrector.correct(heur) + cer_combined.append(cer(both_raw, clean)) + + # Gated combined: confidence gate + fidelity check + heuristic_made_changes = heur != corrupted + if not heuristic_made_changes: + # Confidence gate: skip trained pass + gated = heur + confidence_skips += 1 + else: + if _change_ratio(heur, both_raw) > fidelity_threshold: + gated = heur + fidelity_rejections += 1 + else: + gated = both_raw + cer_combined_gated.append(cer(gated, clean)) + else: + cer_trained.append(cer_in[-1]) + cer_combined.append(cer_heur[-1]) + cer_combined_gated.append(cer_heur[-1]) + + if (i + 1) % 100 == 0: + logger.info("Evaluated %d / %d", i + 1, len(pairs)) + + n = len(pairs) + return EvalResult( + n=n, + avg_cer_input=sum(cer_in) / n, + avg_cer_heuristic=sum(cer_heur) / n, + avg_cer_trained=sum(cer_trained) / n, + avg_cer_combined=sum(cer_combined) / n, + avg_cer_combined_gated=sum(cer_combined_gated) / n, + fidelity_rejections=fidelity_rejections, + confidence_skips=confidence_skips, + ) + + +def evaluate_synthetic( + n_pairs: int = 1000, + seed: int = 1234, + domain: str = "biology", + skip_trained: bool = False, +) -> EvalResult: + """Generate synthetic pairs and evaluate. Uses a different seed than training + so the eval set is held out from anything the trained model saw.""" + from handwriting_engine.trained_correction.dataset import build_pairs + pairs_obj = build_pairs(n=n_pairs, seed=seed) + pairs = [(p.corrupted, p.clean) for p in pairs_obj] + return evaluate(pairs, domain=domain, skip_trained=skip_trained) + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description="A/B evaluate post-correction pipelines.") + parser.add_argument("--n-pairs", type=int, default=1000) + parser.add_argument("--seed", type=int, default=1234) + parser.add_argument("--domain", default="biology") + parser.add_argument("--skip-trained", action="store_true") + parser.add_argument("--output", help="Optional JSON output path") + args = parser.parse_args(argv) + + logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s") + + result = evaluate_synthetic( + n_pairs=args.n_pairs, + seed=args.seed, + domain=args.domain, + skip_trained=args.skip_trained, + ) + + print(json.dumps(result.to_dict(), indent=2)) + if args.output: + Path(args.output).write_text(json.dumps(result.to_dict(), indent=2)) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/handwriting_engine/trained_correction/synthetic_data.py b/handwriting_engine/trained_correction/synthetic_data.py new file mode 100644 index 0000000..52ea0dd --- /dev/null +++ b/handwriting_engine/trained_correction/synthetic_data.py @@ -0,0 +1,347 @@ +"""Synthetic OCR corruption — apply realistic error patterns to clean text. + +The patterns here are drawn from three sources: +1. Classical OCR confusion tables (rn↔m, cl↔d, ii↔u, oo↔co, etc.) +2. LLM-VLM specific failure modes observed in handwriting (doubled letters, + smushed adjacent words, capitalization slips, missed diacritics, terminal + punctuation drops) +3. Visual character similarity in handwriting (l↔1↔I, S↔5, B↔8, Z↔2, G↔6, o↔0) + +All patterns are applied stochastically — a single string passes through every +mutator in sequence with low per-event probability, so the resulting corruption +mixes multiple error types per example. This matches what we see in real VLM +output (a page rarely has only one error type). + +Determinism: every function takes a `random.Random` instance — pass a seeded +one for reproducible corpora. +""" + +from __future__ import annotations + +import random +import re +from dataclasses import dataclass + + +# ===================================================================== +# Confusion tables +# ===================================================================== + +# Bidirectional letter-pair confusions (input → output, weight) +# Higher weight = more frequent in real OCR/HTR output. +_PAIR_CONFUSIONS: list[tuple[str, str, int]] = [ + ("rn", "m", 8), + ("m", "rn", 4), + ("cl", "d", 7), + ("d", "cl", 3), + ("ii", "u", 6), + ("u", "ii", 2), + ("oo", "co", 4), + ("co", "oo", 2), + ("ri", "n", 5), + ("n", "ri", 2), + ("vv", "w", 5), + ("w", "vv", 2), + ("ni", "m", 4), + ("nn", "m", 4), + ("ll", "h", 3), + ("h", "ll", 1), + ("ee", "ce", 3), + ("le", "te", 3), + ("te", "le", 3), + ("a", "o", 5), + ("o", "a", 5), + ("e", "c", 5), + ("c", "e", 4), + ("i", "l", 5), + ("l", "i", 4), + ("u", "v", 4), + ("v", "u", 3), + ("s", "z", 2), + ("z", "s", 2), + ("g", "q", 3), + ("q", "g", 2), + ("h", "n", 3), + ("n", "h", 2), + ("f", "t", 3), + ("t", "f", 2), +] + +# Single-character substitutions (digit-letter visual similarity) +_DIGIT_LETTER_CONFUSIONS: list[tuple[str, str, int]] = [ + ("0", "o", 4), + ("o", "0", 2), + ("1", "l", 4), + ("l", "1", 2), + ("1", "I", 3), + ("I", "1", 2), + ("5", "S", 3), + ("S", "5", 2), + ("8", "B", 3), + ("B", "8", 2), + ("2", "Z", 2), + ("Z", "2", 2), + ("6", "G", 2), + ("G", "6", 2), + ("g", "9", 2), + ("9", "g", 2), +] + + +def _weighted_choice(rng: random.Random, options: list[tuple[str, str, int]]) -> tuple[str, str]: + """Sample (input_pattern, output_pattern) by weight.""" + total = sum(w for _, _, w in options) + r = rng.uniform(0, total) + upto = 0 + for inp, out, w in options: + upto += w + if upto >= r: + return inp, out + return options[-1][0], options[-1][1] + + +# ===================================================================== +# Mutators — each takes (text, rng) → mutated text +# ===================================================================== + +def apply_pair_confusion(text: str, rng: random.Random, prob: float = 0.04) -> str: + """Per-character pass: occasionally swap a 1-2 char pattern with a confusion. + + `prob` is per-position probability of attempting a swap. Most positions + will not match any pattern; the actual swap rate is much lower. + """ + out: list[str] = [] + i = 0 + while i < len(text): + if rng.random() < prob: + # Try a 2-char pattern at this position + two = text[i:i + 2] + applicable = [(inp, op, w) for inp, op, w in _PAIR_CONFUSIONS if inp == two] + if applicable: + _, op = _weighted_choice(rng, applicable) + out.append(op) + i += 2 + continue + # Try single-char (letter-letter) confusion + one = text[i:i + 1] + applicable = [(inp, op, w) for inp, op, w in _PAIR_CONFUSIONS if inp == one] + if applicable: + _, op = _weighted_choice(rng, applicable) + out.append(op) + i += 1 + continue + out.append(text[i]) + i += 1 + return "".join(out) + + +def apply_digit_letter_confusion(text: str, rng: random.Random, prob: float = 0.03) -> str: + """Swap visually-similar digits and letters.""" + out = [] + for ch in text: + if rng.random() < prob: + applicable = [(inp, op, w) for inp, op, w in _DIGIT_LETTER_CONFUSIONS if inp == ch] + if applicable: + _, op = _weighted_choice(rng, applicable) + out.append(op) + continue + out.append(ch) + return "".join(out) + + +def apply_doubling(text: str, rng: random.Random, prob: float = 0.012) -> str: + """Occasionally double a letter (common HTR error: 'celll' for 'cell').""" + out = [] + for ch in text: + out.append(ch) + if ch.isalpha() and rng.random() < prob: + out.append(ch) + return "".join(out) + + +def apply_dropping(text: str, rng: random.Random, prob: float = 0.012) -> str: + """Drop a letter occasionally ('mitcondria' for 'mitochondria').""" + out = [] + for i, ch in enumerate(text): + if ch.isalpha() and rng.random() < prob and 0 < i < len(text) - 1: + continue + out.append(ch) + return "".join(out) + + +def apply_transposition(text: str, rng: random.Random, prob: float = 0.008) -> str: + """Swap adjacent letters ('mitochondira' for 'mitochondria').""" + chars = list(text) + i = 0 + while i < len(chars) - 1: + if chars[i].isalpha() and chars[i + 1].isalpha() and rng.random() < prob: + chars[i], chars[i + 1] = chars[i + 1], chars[i] + i += 2 + else: + i += 1 + return "".join(chars) + + +def apply_smush_split(text: str, rng: random.Random, smush_prob: float = 0.015, split_prob: float = 0.008) -> str: + """Smush adjacent words (drop space) or split a word (insert space).""" + # Smush: drop occasional spaces + parts = text.split(" ") + if len(parts) <= 1: + return text + smushed = [parts[0]] + for p in parts[1:]: + if rng.random() < smush_prob: + smushed[-1] += p + else: + smushed.append(p) + + # Split: occasionally insert a space inside a long-enough word + out = [] + for word in smushed: + if len(word) > 6 and rng.random() < split_prob: + cut = rng.randint(2, len(word) - 2) + out.append(word[:cut] + " " + word[cut:]) + else: + out.append(word) + return " ".join(out) + + +def apply_capitalization_slip(text: str, rng: random.Random, prob: float = 0.01) -> str: + """Occasionally flip case of a letter (HTR mis-reads sentence-initial caps).""" + out = [] + for ch in text: + if ch.isalpha() and rng.random() < prob: + out.append(ch.lower() if ch.isupper() else ch.upper()) + else: + out.append(ch) + return "".join(out) + + +def apply_punctuation_drop(text: str, rng: random.Random, prob: float = 0.10) -> str: + """Occasionally drop terminal punctuation. HTR loses these often.""" + if not text: + return text + if text[-1] in ".,;:!?" and rng.random() < prob: + return text[:-1] + return text + + +def apply_diacritic_strip(text: str, rng: random.Random, prob: float = 0.6) -> str: + """Strip diacritics — VLM/HTR routinely drops them (résumé→resume).""" + if rng.random() > prob: + return text + table = str.maketrans( + "áàâäãåéèêëíìîïóòôöõúùûüñç", + "aaaaaaeeeeiiiiooooouuuunc", + ) + return text.translate(table) + + +# ===================================================================== +# Pipeline +# ===================================================================== + +@dataclass +class CorruptionConfig: + """Per-mutator probabilities. Defaults tuned to roughly match observed + Gemini Flash error rates (~1-2% CER) at a moderate setting and ~5-10% + at an aggressive setting, so the model sees a range of difficulties.""" + pair_confusion_prob: float = 0.04 + digit_letter_prob: float = 0.03 + doubling_prob: float = 0.012 + dropping_prob: float = 0.012 + transposition_prob: float = 0.008 + smush_prob: float = 0.015 + split_prob: float = 0.008 + capitalization_prob: float = 0.01 + punctuation_drop_prob: float = 0.10 + diacritic_strip_prob: float = 0.6 + + @classmethod + def light(cls) -> "CorruptionConfig": + return cls( + pair_confusion_prob=0.015, + digit_letter_prob=0.01, + doubling_prob=0.004, + dropping_prob=0.004, + transposition_prob=0.003, + smush_prob=0.005, + split_prob=0.003, + capitalization_prob=0.005, + punctuation_drop_prob=0.05, + ) + + @classmethod + def aggressive(cls) -> "CorruptionConfig": + return cls( + pair_confusion_prob=0.08, + digit_letter_prob=0.06, + doubling_prob=0.025, + dropping_prob=0.025, + transposition_prob=0.018, + smush_prob=0.030, + split_prob=0.015, + capitalization_prob=0.020, + punctuation_drop_prob=0.20, + ) + + +def corrupt(text: str, rng: random.Random, config: CorruptionConfig | None = None) -> str: + """Apply the full corruption pipeline to clean text. Returns corrupted text.""" + if config is None: + config = CorruptionConfig() + t = text + t = apply_diacritic_strip(t, rng, config.diacritic_strip_prob) + t = apply_pair_confusion(t, rng, config.pair_confusion_prob) + t = apply_digit_letter_confusion(t, rng, config.digit_letter_prob) + t = apply_doubling(t, rng, config.doubling_prob) + t = apply_dropping(t, rng, config.dropping_prob) + t = apply_transposition(t, rng, config.transposition_prob) + t = apply_smush_split(t, rng, config.smush_prob, config.split_prob) + t = apply_capitalization_slip(t, rng, config.capitalization_prob) + t = apply_punctuation_drop(t, rng, config.punctuation_drop_prob) + return t + + +def make_pair( + clean: str, + rng: random.Random, + config: CorruptionConfig | None = None, + ensure_corrupted: bool = True, + max_retries: int = 3, +) -> tuple[str, str]: + """Produce a (corrupted, clean) pair from a clean source string. + + `ensure_corrupted=True` retries up to `max_retries` if corruption produced + an identical string (avoids degenerate identity examples in the corpus). + """ + cfg = config or CorruptionConfig() + for _ in range(max_retries): + corrupted = corrupt(clean, rng, cfg) + if not ensure_corrupted or corrupted != clean: + return corrupted, clean + # Last resort: force at least one mutation + if clean and len(clean) > 2: + idx = rng.randint(0, len(clean) - 2) + chars = list(clean) + chars[idx], chars[idx + 1] = chars[idx + 1], chars[idx] + return "".join(chars), clean + return clean, clean + + +# ===================================================================== +# Mixed-difficulty sampler +# ===================================================================== + +def sample_difficulty(rng: random.Random) -> CorruptionConfig: + """Return a corruption config sampled from a difficulty distribution. + + 60% default (≈1-3% CER), 25% light (≈0.5% CER — clean cases the corrector + must learn to leave alone), 15% aggressive (≈8% CER — pathological cases). + """ + r = rng.random() + if r < 0.25: + return CorruptionConfig.light() + if r < 0.85: + return CorruptionConfig() + return CorruptionConfig.aggressive() diff --git a/handwriting_engine/trained_correction/train.py b/handwriting_engine/trained_correction/train.py new file mode 100644 index 0000000..f9b90d2 --- /dev/null +++ b/handwriting_engine/trained_correction/train.py @@ -0,0 +1,365 @@ +"""Fine-tune ByT5-small on synthetic OCR-error → clean-text pairs. + +Default model: google/byt5-small (~300MB, ~300M params). Byte-level tokenizer +makes this a natural fit for handwriting correction — there's no tokenizer +drift from misspellings, and byte-level avoids subword-vocabulary issues with +unusual scientific terms. + +Usage (from the engine repo): + + python -m handwriting_engine.trained_correction.train \\ + --output-dir ./ckpt/corrector-v1 \\ + --num-pairs 50000 \\ + --num-epochs 2 \\ + --batch-size 8 + +On Apple Silicon MPS the model fits comfortably; bf16 is unsupported on MPS +so we run fp32 by default. +""" + +from __future__ import annotations + +import argparse +import json +import logging +import os +import random +import sys +from pathlib import Path + +logger = logging.getLogger(__name__) + + +def _set_seed(seed: int) -> None: + """Seed Python, numpy, torch (CPU + MPS / CUDA if available).""" + random.seed(seed) + try: + import numpy as np + np.random.seed(seed) + except ImportError: + pass + try: + import torch + torch.manual_seed(seed) + if torch.cuda.is_available(): + torch.cuda.manual_seed_all(seed) + if hasattr(torch, "mps") and torch.backends.mps.is_available(): + torch.mps.manual_seed(seed) + except ImportError: + pass + + +def _resolve_device(prefer: str = "auto") -> str: + import torch + if prefer == "cpu": + return "cpu" + if prefer in ("mps", "auto") and torch.backends.mps.is_available(): + return "mps" + if prefer in ("cuda", "auto") and torch.cuda.is_available(): + return "cuda" + return "cpu" + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--model-name", default="google/flan-t5-small", + help="HF model id to fine-tune. Default flan-t5-small (~80M params, " + "instruction-tuned — follows task prefixes naturally and avoids " + "the t5-small 'correct: → translate to German' confusion). " + "Alternatives: google-t5/t5-small (60M, lightest), " + "google/byt5-small (300M, byte-level — robust to misspellings " + "but slow on MPS).") + parser.add_argument("--output-dir", required=True, + help="Directory for checkpoints + manifest") + parser.add_argument("--num-pairs", type=int, default=50000, + help="Total synthetic training pairs to generate") + parser.add_argument("--num-epochs", type=int, default=2) + parser.add_argument("--batch-size", type=int, default=8) + parser.add_argument("--learning-rate", type=float, default=3e-4, + help="Higher than typical T5 default — synthetic data is forgiving") + parser.add_argument("--max-input-length", type=int, default=256) + parser.add_argument("--max-target-length", type=int, default=256) + parser.add_argument("--device", default="auto", choices=["auto", "cpu", "mps", "cuda"]) + parser.add_argument("--seed", type=int, default=42) + parser.add_argument("--val-frac", type=float, default=0.05) + parser.add_argument("--test-frac", type=float, default=0.05) + parser.add_argument("--gradient-accumulation-steps", type=int, default=1) + parser.add_argument("--logging-steps", type=int, default=50) + parser.add_argument("--save-steps", type=int, default=500) + parser.add_argument("--no-system-wordlist", action="store_true", + help="Skip /usr/share/dict/words for general English") + parser.add_argument("--from-benchmark-db", action="store_true", + help="Mix real (VLM_output, ground_truth) pairs from " + "~/.handwriting-engine/benchmark.db into the training corpus. " + "Real pairs are duplicated 3x relative to synthetic to up-weight " + "the real distribution. No-op if the DB doesn't exist.") + parser.add_argument("--benchmark-db-path", default=None, + help="Override the default benchmark.db path") + parser.add_argument("--benchmark-providers", default=None, + help="Comma-separated list of providers to include " + "(e.g. 'gemini,claude'). Default: all providers.") + parser.add_argument("--real-data-weight", type=int, default=3, + help="How many times each real pair appears in the corpus " + "relative to synthetic. Default 3.") + parser.add_argument("--continue-from", default=None, + help="Path to an existing checkpoint to continue fine-tuning from. " + "Overrides --model-name. Use to fine-tune the synthetic v0 " + "model on real Phase 7 IAM data.") + parser.add_argument("--quick", action="store_true", + help="Tiny run for smoke-testing the pipeline") + args = parser.parse_args(argv) + + if args.quick: + # --quick caps things to a tiny smoke run regardless of other flags + args.num_pairs = min(args.num_pairs, 200) + args.num_epochs = 1 + args.max_input_length = min(args.max_input_length, 128) + args.max_target_length = min(args.max_target_length, 128) + + logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s") + + _set_seed(args.seed) + + output_dir = Path(args.output_dir) + output_dir.mkdir(parents=True, exist_ok=True) + + # Lazy imports — these are optional deps + try: + import torch + from torch.utils.data import DataLoader + from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, get_linear_schedule_with_warmup + except ImportError as e: + logger.error( + "Missing optional deps. Install with: " + "pip install handwriting-engine[trained-correction]\n Underlying: %s", e + ) + return 2 + + from handwriting_engine.trained_correction.dataset import ( + build_pairs, + from_benchmark_db, + split_pairs, + make_torch_dataset, + ) + + device = _resolve_device(args.device) + logger.info("Using device: %s", device) + + # ---- Data ---- + logger.info("Generating %d synthetic (corrupted, clean) pairs...", args.num_pairs) + pairs = build_pairs( + n=args.num_pairs, + seed=args.seed, + use_paragraphs=True, + use_system_wordlist=not args.no_system_wordlist, + ) + + real_pairs_count = 0 + if args.from_benchmark_db: + providers_filter = ( + [p.strip() for p in args.benchmark_providers.split(",") if p.strip()] + if args.benchmark_providers else None + ) + real_pairs = from_benchmark_db( + db_path=args.benchmark_db_path, + providers=providers_filter, + ) + if real_pairs: + real_pairs_count = len(real_pairs) + # Up-weight real pairs by replication; mixing into synthetic at higher + # weight encourages the model to track real-world distribution. + pairs = pairs + real_pairs * args.real_data_weight + logger.info( + "Mixed in %d real pairs (replicated %dx → %d effective real examples)", + real_pairs_count, args.real_data_weight, real_pairs_count * args.real_data_weight, + ) + else: + logger.info("No real pairs available (DB empty or missing); using synthetic only") + + train_pairs, val_pairs, test_pairs = split_pairs( + pairs, + val_frac=args.val_frac, + test_frac=args.test_frac, + seed=args.seed, + ) + logger.info("Train: %d Val: %d Test: %d", len(train_pairs), len(val_pairs), len(test_pairs)) + + # ---- Tokenizer + model ---- + model_source = args.continue_from if args.continue_from else args.model_name + if args.continue_from: + logger.info("Continuing from checkpoint: %s", args.continue_from) + else: + logger.info("Loading tokenizer + model: %s", args.model_name) + tokenizer = AutoTokenizer.from_pretrained(model_source) + model = AutoModelForSeq2SeqLM.from_pretrained(model_source) + model.to(device) + model.train() + + # ---- Datasets ---- + train_ds = make_torch_dataset( + train_pairs, tokenizer, + max_input_length=args.max_input_length, + max_target_length=args.max_target_length, + ) + val_ds = make_torch_dataset( + val_pairs, tokenizer, + max_input_length=args.max_input_length, + max_target_length=args.max_target_length, + ) if val_pairs else None + + train_loader = DataLoader(train_ds, batch_size=args.batch_size, shuffle=True, num_workers=0) + val_loader = DataLoader(val_ds, batch_size=args.batch_size, shuffle=False, num_workers=0) if val_ds else None + + # ---- Optimizer + scheduler ---- + optimizer = torch.optim.AdamW( + model.parameters(), + lr=args.learning_rate, + weight_decay=0.01, + ) + total_steps = len(train_loader) * args.num_epochs // args.gradient_accumulation_steps + warmup_steps = max(1, int(0.06 * total_steps)) + scheduler = get_linear_schedule_with_warmup(optimizer, warmup_steps, total_steps) + + # ---- Train loop ---- + logger.info("Starting training: %d total steps (warmup=%d)", total_steps, warmup_steps) + best_val_loss = float("inf") + global_step = 0 + train_losses = [] + + for epoch in range(args.num_epochs): + running_loss = 0.0 + running_count = 0 + optimizer.zero_grad() + for batch_idx, batch in enumerate(train_loader): + batch = {k: v.to(device) for k, v in batch.items()} + outputs = model(**batch) + loss = outputs.loss / args.gradient_accumulation_steps + loss.backward() + running_loss += loss.item() * args.gradient_accumulation_steps + running_count += 1 + + if (batch_idx + 1) % args.gradient_accumulation_steps == 0: + torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) + optimizer.step() + scheduler.step() + optimizer.zero_grad() + global_step += 1 + + if global_step % args.logging_steps == 0: + avg = running_loss / max(1, running_count) + logger.info( + "epoch %d step %d/%d loss %.4f lr %.2e", + epoch, global_step, total_steps, avg, scheduler.get_last_lr()[0], + ) + train_losses.append(avg) + + if val_loader is not None and global_step % args.save_steps == 0: + val_loss = _eval_loss(model, val_loader, device) + logger.info("epoch %d step %d val_loss %.4f", epoch, global_step, val_loss) + if val_loss < best_val_loss: + best_val_loss = val_loss + _save_checkpoint(model, tokenizer, output_dir) + logger.info("Saved best checkpoint (val_loss=%.4f)", val_loss) + model.train() + + # End-of-epoch eval + save + if val_loader is not None: + val_loss = _eval_loss(model, val_loader, device) + logger.info("end of epoch %d val_loss %.4f", epoch, val_loss) + if val_loss < best_val_loss: + best_val_loss = val_loss + _save_checkpoint(model, tokenizer, output_dir) + logger.info("Saved best checkpoint (val_loss=%.4f)", val_loss) + model.train() + else: + _save_checkpoint(model, tokenizer, output_dir) + + # Always save final state if no val loader (otherwise best is already saved) + if val_loader is None: + _save_checkpoint(model, tokenizer, output_dir) + + final_train_loss = train_losses[-1] if train_losses else float("nan") + + # ---- Manifest ---- + has_real = real_pairs_count > 0 + manifest = { + "schema_version": 1, + "base_model": args.model_name, + "continued_from": args.continue_from, + "training": { + "num_pairs_synthetic": args.num_pairs, + "num_pairs_real": real_pairs_count, + "real_data_weight": args.real_data_weight if has_real else 0, + "num_train": len(train_pairs), + "num_val": len(val_pairs), + "num_test": len(test_pairs), + "num_epochs": args.num_epochs, + "batch_size": args.batch_size, + "gradient_accumulation_steps": args.gradient_accumulation_steps, + "learning_rate": args.learning_rate, + "seed": args.seed, + "device": device, + "max_input_length": args.max_input_length, + "max_target_length": args.max_target_length, + "total_steps": total_steps, + }, + "metrics": { + "final_train_loss": final_train_loss, + "best_val_loss": best_val_loss if best_val_loss != float("inf") else None, + }, + "data_provenance": { + "source": "synthetic+real" if has_real else "synthetic", + "corpus_version": "v0", + "system_wordlist": not args.no_system_wordlist, + "real_pairs_from": args.benchmark_db_path or "~/.handwriting-engine/benchmark.db" if has_real else None, + }, + "caveats": ( + ["Synthetic+real training — sim-to-real gap reduced by mixing real Phase 7 pairs."] + if has_real else + ["Synthetic-only training — has known sim-to-real gap.", + "Plan a small real-data fine-tune (Phase 7 IAM data) before claiming production parity."] + ), + } + + # ---- Hold-out test loss ---- + if test_pairs: + test_ds = make_torch_dataset( + test_pairs, tokenizer, + max_input_length=args.max_input_length, + max_target_length=args.max_target_length, + ) + test_loader = DataLoader(test_ds, batch_size=args.batch_size, shuffle=False, num_workers=0) + test_loss = _eval_loss(model, test_loader, device) + logger.info("Held-out test loss: %.4f", test_loss) + manifest["metrics"]["test_loss"] = test_loss + + with (output_dir / "training_manifest.json").open("w") as f: + json.dump(manifest, f, indent=2) + + logger.info("Training complete. Checkpoint at: %s", output_dir) + return 0 + + +def _eval_loss(model, loader, device) -> float: + """Average loss on a DataLoader. Returns finite float.""" + import torch + model.eval() + total = 0.0 + count = 0 + with torch.no_grad(): + for batch in loader: + batch = {k: v.to(device) for k, v in batch.items()} + out = model(**batch) + total += float(out.loss.item()) + count += 1 + return total / max(1, count) + + +def _save_checkpoint(model, tokenizer, output_dir: Path) -> None: + output_dir.mkdir(parents=True, exist_ok=True) + model.save_pretrained(str(output_dir)) + tokenizer.save_pretrained(str(output_dir)) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/handwriting_engine/vision.py b/handwriting_engine/vision.py index 9fe7e63..deaaad1 100644 --- a/handwriting_engine/vision.py +++ b/handwriting_engine/vision.py @@ -348,10 +348,28 @@ def read_page( p = get_provider(provider) + # S2: per-writer few-shot exemplars. Returns None (fall through) when the + # writer has <2 GT samples, the provider is OCR-only, or HE_FEW_SHOT_K=0. + exemplar_blocks = None + if writer_id: + from handwriting_engine.few_shot import select_and_build_exemplar_blocks + + exemplar_blocks = select_and_build_exemplar_blocks( + writer_id=writer_id, + provider=provider, + target_image_b64=b64_data, + target_media_type=media_type, + ) + # Dual-polarity reading for faint ink: send both normal and inverted images from handwriting_engine._constants import DUAL_POLARITY_ENABLED if DUAL_POLARITY_ENABLED and assessment and assessment.get("faint_ink"): + # Faint-ink dual-polarity wins over few-shot — they target different + # failure modes and combining them inflates per-call token cost + # without a verified gain. raw = _dual_polarity_read(image_path, b64_data, media_type, p, prompt, system_prompt, max_tokens) + elif exemplar_blocks is not None: + raw = p.read_batch(exemplar_blocks, prompt=prompt, system_prompt=system_prompt, max_tokens=max_tokens) else: raw = p.read_image(b64_data, media_type, prompt, system_prompt, max_tokens) @@ -634,6 +652,7 @@ def read_with_consensus( strategy=strategy, content_type=content_type, quality_assessment=quality_assessment, + writer_profile=writer_profile_dict, ) diff --git a/handwriting_engine/writer_profile_store.py b/handwriting_engine/writer_profile_store.py index 9f313ff..88281b3 100644 --- a/handwriting_engine/writer_profile_store.py +++ b/handwriting_engine/writer_profile_store.py @@ -4,19 +4,116 @@ Saves writer-specific disambiguation observations (how they form 7s, 4s, a vs o, etc.) to ~/.handwriting-engine/writer-profiles/{writer_id}.json and injects them as a calibration block into transcription prompts. + +Also serves as the home for S2 per-writer few-shot exemplar selection +(``select_exemplars``), since exemplars are conceptually a richer flavor of +the same writer-calibration mechanism. """ from __future__ import annotations import json import logging +import sqlite3 +from dataclasses import dataclass from pathlib import Path +from typing import Optional logger = logging.getLogger(__name__) _PROFILES_DIR = Path.home() / ".handwriting-engine" / "writer-profiles" +@dataclass(frozen=True) +class Exemplar: + """One (sample, ground-truth) pair to inject as a few-shot exemplar.""" + + sample_id: int + image_path: str + ground_truth: str + + +def select_exemplars( + writer_id: str, + *, + k: int = 3, + exclude_sample_id: Optional[int] = None, + db_path: Optional[str | Path] = None, + conn: Optional[sqlite3.Connection] = None, +) -> list[Exemplar]: + """Return up to ``k`` deterministic exemplars for ``writer_id``. + + Strategy v0 per S2-SPEC § Design: + 1. Pull every (sample, ground_truth) pair where ``samples.student = + writer_id`` (latest GT per sample wins on ties). + 2. Order by ``quality_assessments.score DESC`` when available (joined + LEFT so writers without quality data still produce results), then by + ``samples.id ASC`` for determinism. + 3. Slice ``min(k, len)`` off the top. + + Returns ``[]`` for empty writers, ``k <= 0``, or when no eligible + sample/GT pair exists. Existence of the image on disk is *not* checked + here — callers that need that guarantee filter post hoc. + + Either ``conn`` or ``db_path`` may be supplied. With neither, defaults + to ``DEFAULT_DB_PATH`` from ``benchmark.db``. + """ + if not writer_id or k <= 0: + return [] + + owns_conn = conn is None + if owns_conn: + from handwriting_engine.benchmark.db import get_connection + + conn = get_connection(db_path) + try: + # Latest GT per sample: ROW_NUMBER would be ideal but SQLite versions + # bundled with older Pythons lack window functions reliably; use a + # correlated subquery instead. + params: list = [writer_id] + exclude_sql = "" + if exclude_sample_id is not None: + exclude_sql = "AND s.id != ?" + params.append(exclude_sample_id) + + # quality_assessments may not have a row for every sample — LEFT JOIN + # so missing scores fall to the bottom (NULL sorted last via COALESCE). + sql = f""" + SELECT + s.id AS sample_id, + s.image_path AS image_path, + gt.text AS ground_truth + FROM samples AS s + JOIN ground_truths AS gt + ON gt.sample_id = s.id + AND gt.id = ( + SELECT MAX(id) FROM ground_truths WHERE sample_id = s.id + ) + LEFT JOIN quality_assessments AS qa + ON qa.sample_id = s.id + WHERE s.student = ? + {exclude_sql} + ORDER BY + COALESCE(qa.contrast_score, -1) DESC, + s.id ASC + LIMIT ? + """ + params.append(k) + rows = conn.execute(sql, params).fetchall() + finally: + if owns_conn: + conn.close() + + return [ + Exemplar( + sample_id=row["sample_id"], + image_path=row["image_path"], + ground_truth=row["ground_truth"], + ) + for row in rows + ] + + class WriterProfileStore: """Load, save, and inject writer-specific handwriting profiles.""" diff --git a/pyproject.toml b/pyproject.toml index c174bc0..65f4f5c 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -25,6 +25,13 @@ openai = ["openai>=1.50.0"] gemini = ["google-genai>=1.0.0"] all = ["anthropic>=0.40.0", "openai>=1.50.0", "google-genai>=1.0.0"] benchmark = ["jiwer>=3.0.0", "numpy>=1.24.0"] +trained-correction = [ + "torch>=2.4.0", + "transformers>=4.45.0", + "sentencepiece>=0.2.0", + "accelerate>=0.34.0", + "numpy>=1.24.0", +] dev = ["pytest>=8.0.0"] [project.scripts] diff --git a/tests/test_benchmark_baseline.py b/tests/test_benchmark_baseline.py new file mode 100644 index 0000000..3cc9c5b --- /dev/null +++ b/tests/test_benchmark_baseline.py @@ -0,0 +1,264 @@ +"""Phase 9 / RPT-01 — pinned-baseline regression detection. + +Covers: +- Schema v6: is_baseline column on runs (durable across sessions). +- set_baseline / get_baseline_run_id atomicity (at-most-one). +- detect_regressions retargets to the pinned baseline rather than runs[1]. +- CLI: `benchmark set-baseline RUN_ID`. +""" + +import os +import tempfile +from pathlib import Path + +import pytest +from PIL import Image +from click.testing import CliRunner + +from handwriting_engine.benchmark.db import ( + get_connection, + insert_sample, + insert_ground_truth, + insert_run, + finish_run, + insert_provider_output, + insert_eval_metric, + set_baseline, + get_baseline_run_id, + list_runs, +) +from handwriting_engine.benchmark.report import detect_regressions +from handwriting_engine.cli import cli + + +def _seed_db(db_path: Path, n_samples: int): + """Build a DB with n_samples + matching ground-truth rows. + + Returns (sample_ids, gt_ids). + """ + conn = get_connection(db_path) + img_dir = tempfile.mkdtemp() + sids, gids = [], [] + for i in range(n_samples): + p = os.path.join(img_dir, f"i{i}.png") + Image.new("RGB", (32, 32), (128, 128, 128)).save(p) + sid = insert_sample(conn, p, f"hash{i}", student=f"w{i % 3}") + gid = insert_ground_truth(conn, sid, "ground truth") + sids.append(sid) + gids.append(gid) + conn.close() + return sids, gids + + +def _seed_run(db_path, label, sids, gids, cers): + conn = get_connection(db_path) + rid = insert_run(conn, label=label, providers=["gemini"], strategies=["vote"]) + for sid, gid, cer in zip(sids, gids, cers): + po = insert_provider_output( + conn, run_id=rid, sample_id=sid, + provider="gemini", strategy="vote", + output_text="x", confidence=0.9, + ) + insert_eval_metric( + conn, provider_output_id=po, ground_truth_id=gid, + cer=cer, wer=cer, + ) + finish_run(conn, rid, len(cers)) + conn.close() + return rid + + +# --- schema migration --- + + +class TestSchemaV6: + def test_is_baseline_column_present(self, tmp_path): + db = tmp_path / "v6.db" + conn = get_connection(db) + cols = {row[1] for row in conn.execute("PRAGMA table_info(runs)").fetchall()} + conn.close() + assert "is_baseline" in cols + + def test_default_is_zero(self, tmp_path): + db = tmp_path / "v6_default.db" + conn = get_connection(db) + rid = insert_run(conn, label="default", providers=["gemini"], strategies=[]) + row = conn.execute("SELECT is_baseline FROM runs WHERE id = ?", (rid,)).fetchone() + conn.close() + assert row["is_baseline"] == 0 + + +# --- set_baseline / get_baseline_run_id --- + + +class TestBaselineFunctions: + def test_set_and_get(self, tmp_path): + db = tmp_path / "set.db" + sids, gids = _seed_db(db, 3) + r1 = _seed_run(db, "v1", sids, gids, [0.1, 0.1, 0.1]) + r2 = _seed_run(db, "v2", sids, gids, [0.1, 0.1, 0.1]) + + conn = get_connection(db) + assert get_baseline_run_id(conn) is None + set_baseline(conn, r2) + assert get_baseline_run_id(conn) == r2 + conn.close() + + def test_at_most_one_baseline(self, tmp_path): + db = tmp_path / "atmostone.db" + sids, gids = _seed_db(db, 3) + r1 = _seed_run(db, "v1", sids, gids, [0.1] * 3) + r2 = _seed_run(db, "v2", sids, gids, [0.1] * 3) + r3 = _seed_run(db, "v3", sids, gids, [0.1] * 3) + + conn = get_connection(db) + set_baseline(conn, r1) + set_baseline(conn, r2) + set_baseline(conn, r3) + # Only r3 should still be flagged. + flagged = [r["id"] for r in conn.execute( + "SELECT id FROM runs WHERE is_baseline = 1" + ).fetchall()] + conn.close() + assert flagged == [r3] + + def test_unknown_run_raises(self, tmp_path): + db = tmp_path / "missing.db" + conn = get_connection(db) + with pytest.raises(ValueError, match="not found"): + set_baseline(conn, 999) + conn.close() + + def test_durable_across_connections(self, tmp_path): + # The pinned flag survives closing and reopening the DB — + # i.e. it lives on disk, not just in process state. + db = tmp_path / "durable.db" + sids, gids = _seed_db(db, 3) + r = _seed_run(db, "v", sids, gids, [0.1] * 3) + + conn = get_connection(db) + set_baseline(conn, r) + conn.close() + + conn2 = get_connection(db) + assert get_baseline_run_id(conn2) == r + conn2.close() + + def test_list_runs_includes_is_baseline(self, tmp_path): + db = tmp_path / "list.db" + sids, gids = _seed_db(db, 3) + r1 = _seed_run(db, "v1", sids, gids, [0.1] * 3) + r2 = _seed_run(db, "v2", sids, gids, [0.1] * 3) + + conn = get_connection(db) + set_baseline(conn, r1) + runs = list_runs(conn) + conn.close() + + flagged = {r.run_id: r.is_baseline for r in runs} + assert flagged[r1] == 1 + assert flagged[r2] == 0 + + +# --- detect_regressions retargets to baseline --- + + +class TestDetectRegressionsBaseline: + def test_no_baseline_falls_back_to_previous(self, tmp_path): + db = tmp_path / "fallback.db" + sids, gids = _seed_db(db, 5) + r1 = _seed_run(db, "v1", sids, gids, [0.05] * 5) + r2 = _seed_run(db, "v2", sids, gids, [0.10] * 5) # regression vs r1 + + regs = detect_regressions(db_path=db) + # No pinned baseline → falls back to runs[1] = r1 → regression. + assert any(r["delta"] > 0.04 for r in regs) + + def test_pinned_baseline_used_over_previous(self, tmp_path): + db = tmp_path / "pinned.db" + sids, gids = _seed_db(db, 5) + # r1 = 0.05, r2 = 0.06 (no regression vs r1), r3 = 0.06 (no regression vs r2 but…) + r1 = _seed_run(db, "v1", sids, gids, [0.05] * 5) + r2 = _seed_run(db, "v2", sids, gids, [0.06] * 5) + r3 = _seed_run(db, "v3", sids, gids, [0.10] * 5) + + # Without baseline: detect_regressions(r3) compares r3 vs runs[1]=r2, + # delta = 0.04 → REGRESSION (above 3% threshold). + # With baseline pinned at r1: compare r3 vs r1, delta = 0.05 → still + # REGRESSION but vs the *anchored* run. + # Sharper test: pin r1, run r2 (which is +1pp vs r1, below 3% + # threshold) — fallback would compare r2 vs r1 anyway in this case, + # so we need a scenario where the choice of anchor changes the + # answer. + # r3 vs r2 = +4pp REGRESSION, r3 vs r1 = +5pp REGRESSION. Both detect. + # Need: anchor pick changes the *count* of regressions or the *delta*. + # Use r2 to demonstrate: r2 vs r1 (anchor) = +1pp, no regression. + # r2 vs runs[1] = r1 anyway → also no regression. + # Need another run to break the tie. + # Insert r4 = 0.07. Without baseline: r4 vs r3 = -0.03, no regression. + # With baseline r1 pinned: r4 vs r1 = +0.02, no regression either. + # OK simpler: pin r1, query r3. delta_pinned = +0.05, delta_fallback (r3 vs r2) = +0.04. + # Both are regressions but the reported `previous_cer` differs. + conn = get_connection(db) + set_baseline(conn, r1) + conn.close() + + regs = detect_regressions(run_id=r3, db_path=db) + assert len(regs) == 1 + # Reported previous_cer comes from the pinned baseline r1 (=0.05), + # not from the immediately-preceding run r2 (=0.06). + assert regs[0]["previous_cer"] == pytest.approx(0.05, abs=1e-9) + assert regs[0]["current_cer"] == pytest.approx(0.10, abs=1e-9) + + def test_self_compare_falls_back_when_current_is_baseline(self, tmp_path): + db = tmp_path / "self.db" + sids, gids = _seed_db(db, 5) + r1 = _seed_run(db, "v1", sids, gids, [0.05] * 5) + r2 = _seed_run(db, "v2", sids, gids, [0.10] * 5) + + conn = get_connection(db) + set_baseline(conn, r2) # the latest run is the baseline + conn.close() + + # detect_regressions for r2 (which IS the baseline) should fall back + # to comparing against the prior run, not against itself. + regs = detect_regressions(run_id=r2, db_path=db) + # r2 vs r1 = +5pp regression — still detected. + assert len(regs) == 1 + assert regs[0]["previous_cer"] == pytest.approx(0.05, abs=1e-9) + + +# --- CLI --- + + +class TestSetBaselineCli: + def test_set_baseline_command(self, tmp_path): + db = tmp_path / "cli.db" + sids, gids = _seed_db(db, 3) + r1 = _seed_run(db, "v1", sids, gids, [0.1] * 3) + + runner = CliRunner() + result = runner.invoke(cli, [ + "benchmark", "set-baseline", str(r1), + "--db-path", str(db), + ]) + assert result.exit_code == 0, result.output + assert f"Baseline pinned: run #{r1}" in result.output + + conn = get_connection(db) + assert get_baseline_run_id(conn) == r1 + conn.close() + + def test_set_baseline_unknown_run(self, tmp_path): + db = tmp_path / "cli_missing.db" + # Make sure the DB exists but has no runs. + conn = get_connection(db) + conn.close() + + runner = CliRunner() + result = runner.invoke(cli, [ + "benchmark", "set-baseline", "999", + "--db-path", str(db), + ]) + assert result.exit_code != 0 + assert "not found" in result.output diff --git a/tests/test_benchmark_db.py b/tests/test_benchmark_db.py index 70bc1f5..383e17a 100644 --- a/tests/test_benchmark_db.py +++ b/tests/test_benchmark_db.py @@ -17,6 +17,7 @@ insert_sample, list_runs, list_samples, + record_correction, samples_with_ground_truth, ) @@ -64,6 +65,17 @@ def test_v4_migration_columns(self, db): po_cols = {row["name"] for row in db.execute("PRAGMA table_info(provider_outputs)").fetchall()} assert "question_marker_rate" in po_cols, "provider_outputs.question_marker_rate missing — v4 migration not applied" + def test_v5_corrections_table(self, db): + """v5 migration must create the corrections table with expected columns.""" + tables = {r["name"] for r in db.execute( + "SELECT name FROM sqlite_master WHERE type='table'" + ).fetchall()} + assert "corrections" in tables, "corrections table missing — v5 migration not applied" + + cols = {r["name"] for r in db.execute("PRAGMA table_info(corrections)").fetchall()} + for required in {"sample_id", "ground_truth_id", "original_text", "confidence", "source", "created_at"}: + assert required in cols, f"corrections.{required} missing" + class TestSamples: def test_insert_and_retrieve(self, db): @@ -206,3 +218,90 @@ def test_full_pipeline(self, db): assert len(results) == 1 assert results[0]["provider"] == "gemini" assert results[0]["cer"] == 0.0 + + +class TestCorrections: + """S4: record_correction() instructor-feedback API.""" + + @pytest.fixture + def fake_image(self, tmp_path): + path = tmp_path / "page.png" + path.write_bytes(b"\x89PNG\r\n\x1a\nfake-image-bytes-for-hashing") + return str(path) + + def test_first_call_creates_sample_gt_and_correction(self, db, fake_image): + cid = record_correction( + db, + image_path=fake_image, + writer_id="prof-bio101-jdoe", + corrected_text="OD600 = 0.45", + original_vlm_text="OD60O = 0.45", + confidence=0.62, + ) + assert cid > 0 + + assert db.execute("SELECT COUNT(*) FROM samples").fetchone()[0] == 1 + assert db.execute("SELECT COUNT(*) FROM ground_truths").fetchone()[0] == 1 + assert db.execute("SELECT COUNT(*) FROM corrections").fetchone()[0] == 1 + + sample = db.execute("SELECT * FROM samples").fetchone() + assert sample["student"] == "prof-bio101-jdoe" + + def test_idempotent_identical_args(self, db, fake_image): + """Identical-args double-call: 1 sample + 1 GT + 2 corrections (history grows).""" + kwargs = dict( + image_path=fake_image, + writer_id="prof-bio101-jdoe", + corrected_text="OD600 = 0.45", + original_vlm_text="OD60O = 0.45", + confidence=0.62, + ) + record_correction(db, **kwargs) + record_correction(db, **kwargs) + + assert db.execute("SELECT COUNT(*) FROM samples").fetchone()[0] == 1 + assert db.execute("SELECT COUNT(*) FROM ground_truths").fetchone()[0] == 1 + assert db.execute("SELECT COUNT(*) FROM corrections").fetchone()[0] == 2 + + def test_different_correction_text_adds_new_gt(self, db, fake_image): + """Same image, two different corrected_text values → 1 sample, 2 GTs, 2 corrections.""" + record_correction( + db, image_path=fake_image, writer_id="w1", + corrected_text="first read", original_vlm_text="orig", confidence=0.6, + ) + record_correction( + db, image_path=fake_image, writer_id="w1", + corrected_text="revised read", original_vlm_text="orig", confidence=0.6, + ) + assert db.execute("SELECT COUNT(*) FROM samples").fetchone()[0] == 1 + assert db.execute("SELECT COUNT(*) FROM ground_truths").fetchone()[0] == 2 + assert db.execute("SELECT COUNT(*) FROM corrections").fetchone()[0] == 2 + + def test_per_writer_accumulation(self, db, tmp_path): + """Per spec criterion #5: same writer across 'sessions' adds to same student value.""" + for i in range(3): + img = tmp_path / f"p{i}.png" + img.write_bytes(f"png-bytes-{i}".encode()) + record_correction( + db, image_path=str(img), writer_id="prof-bio101-jdoe", + corrected_text=f"text {i}", original_vlm_text=f"orig {i}", confidence=0.5, + ) + count = db.execute( + "SELECT COUNT(*) FROM samples WHERE student = ?", ("prof-bio101-jdoe",) + ).fetchone()[0] + assert count == 3 + + def test_correction_links_to_real_sample_and_gt(self, db, fake_image): + cid = record_correction( + db, image_path=fake_image, writer_id="w1", + corrected_text="real text", original_vlm_text="vlm text", confidence=0.8, + source="labgrader", + ) + row = db.execute("SELECT * FROM corrections WHERE id = ?", (cid,)).fetchone() + assert row["original_text"] == "vlm text" + assert row["confidence"] == 0.8 + assert row["source"] == "labgrader" + + # FK integrity: linked sample and GT must exist + assert db.execute("SELECT 1 FROM samples WHERE id = ?", (row["sample_id"],)).fetchone() + assert db.execute("SELECT 1 FROM ground_truths WHERE id = ?", (row["ground_truth_id"],)).fetchone() diff --git a/tests/test_benchmark_evaluate.py b/tests/test_benchmark_evaluate.py index cb2e1d5..1e0a02f 100644 --- a/tests/test_benchmark_evaluate.py +++ b/tests/test_benchmark_evaluate.py @@ -468,78 +468,213 @@ def test_detect_regressions(self, mock_read, mock_providers, seeded_db): class TestSweep: - """RED stubs for sweep infrastructure (IAM-02). All must FAIL until Wave 2.""" - - # These are TODO stubs, not tests: each body is a bare pytest.fail() naming work that - # was never done. Left as hard failures they made the suite permanently red, and a - # suite that is always red cannot gate a dependency upgrade -- which is how 25 - # security advisories sat unactioned. Declared as expected failures they still print - # their message on every run (counted as "xfailed"), so the debt stays visible while - # a NEW failure is once again the only reason the suite goes red. Deleting the marker - # is part of implementing sweep infrastructure (IAM-02). - pytestmark = pytest.mark.xfail( - reason="RED stub: sweep infrastructure (IAM-02) is unimplemented", strict=False - ) - - def test_run_benchmark_accepts_line_level(self, seeded_db): - pytest.fail( - "not implemented — run_benchmark must accept line_level=True " - "and thread it through to _read_single" - ) + """Sweep infrastructure (IAM-02) — turned GREEN in Phase 07-03.""" - def test_run_benchmark_accepts_auto_retry(self, seeded_db): - pytest.fail( - "not implemented — run_benchmark must accept auto_retry=True " - "and thread it through to _read_single" + @patch("handwriting_engine.benchmark.evaluate._available_providers") + @patch("handwriting_engine.benchmark.evaluate._read_single") + def test_run_benchmark_accepts_line_level(self, mock_read, mock_providers, seeded_db): + mock_providers.return_value = ["gemini"] + mock_read.return_value = { + "text": "the mitochondria is the powerhouse of the cell", + "confidence": 0.7, "latency_ms": 500, + "input_tokens": 100, "output_tokens": 50, "error": None, + } + + run_id = run_benchmark( + providers=["gemini"], strategies=[], db_path=seeded_db, + line_level=True, ) + assert run_id > 0 + # _read_single must receive line_level=True + kwargs = mock_read.call_args.kwargs + assert kwargs.get("line_level") is True + + @patch("handwriting_engine.benchmark.evaluate._available_providers") + @patch("handwriting_engine.benchmark.evaluate._read_single") + def test_run_benchmark_accepts_auto_retry(self, mock_read, mock_providers, seeded_db): + mock_providers.return_value = ["gemini"] + mock_read.return_value = { + "text": "the mitochondria is the powerhouse of the cell", + "confidence": 0.7, "latency_ms": 500, + "input_tokens": 100, "output_tokens": 50, "error": None, + } - def test_run_sweep_returns_five_run_ids(self, seeded_db): - pytest.fail( - "not implemented — run_sweep must return a dict with exactly 5 keys: " - "baseline, self_correct, line_level, prompt_adapted, zoomed_verify" + run_id = run_benchmark( + providers=["gemini"], strategies=[], db_path=seeded_db, + auto_retry=True, ) + assert run_id > 0 + kwargs = mock_read.call_args.kwargs + assert kwargs.get("auto_retry") is True + + @patch("handwriting_engine.benchmark.evaluate._available_providers") + @patch("handwriting_engine.benchmark.evaluate._read_single") + def test_run_sweep_returns_five_run_ids(self, mock_read, mock_providers, seeded_db): + # run_sweep filters samples to category='iam' only — ensure the seeded + # sample has that category so each strategy actually evaluates something. + conn = get_connection(seeded_db) + try: + conn.execute("UPDATE samples SET category='iam' WHERE id=1") + conn.commit() + finally: + conn.close() + + mock_providers.return_value = ["gemini"] + mock_read.return_value = { + "text": "the mitochondria is the powerhouse of the cell", + "confidence": 0.7, "latency_ms": 500, + "input_tokens": 100, "output_tokens": 50, "error": None, + } + + run_ids = run_sweep(provider="gemini", db_path=seeded_db, yes=True) + assert isinstance(run_ids, dict) + assert set(run_ids.keys()) == { + "baseline", "self_correct", "line_level", + "prompt_adapted", "zoomed_verify", + } + assert all(isinstance(rid, int) and rid > 0 for rid in run_ids.values()) def test_sweep_cli_shows_cost(self, tmp_path): - pytest.fail( - "not implemented — `benchmark sweep` CLI must print projected cost " - "before any API call (even with no real samples)" + from handwriting_engine.cli import cli + # Use an empty DB — cost projection must still run before any API call + db_path = tmp_path / "empty.db" + get_connection(db_path).close() + + runner = CliRunner() + # Decline at the prompt — we only care that the cost line appears + result = runner.invoke( + cli, + ["benchmark", "sweep", "--db-path", str(db_path)], + input="n\n", ) + assert "Estimated cost" in result.output + assert "Sweep projection" in result.output + + @patch("handwriting_engine.benchmark.evaluate._available_providers") + @patch("handwriting_engine.benchmark.evaluate._read_single") + def test_sweep_cli_yes_executes(self, mock_read, mock_providers, tmp_path): + from handwriting_engine.cli import cli + from PIL import Image + + db_path = tmp_path / "sweep.db" + conn = get_connection(db_path) + img_path = tmp_path / "iam.png" + Image.new("RGB", (200, 200), color=(128, 128, 128)).save(img_path) + sid = insert_sample(conn, str(img_path), "iamhash1", + student="iam-writer-001", category="iam") + insert_ground_truth(conn, sid, "the mitochondria is the powerhouse of the cell") + conn.close() - def test_sweep_cli_yes_executes(self, tmp_path): - pytest.fail( - "not implemented — `benchmark sweep --yes` must bypass cost confirmation " - "and attempt to execute all 5 strategies" + mock_providers.return_value = ["gemini"] + mock_read.return_value = { + "text": "the mitochondria is the powerhouse of the cell", + "confidence": 0.7, "latency_ms": 500, + "input_tokens": 100, "output_tokens": 50, "error": None, + } + + runner = CliRunner() + result = runner.invoke( + cli, + ["benchmark", "sweep", "--yes", "--db-path", str(db_path)], ) + assert result.exit_code == 0, result.output + assert "Sweep complete" in result.output + # All 5 strategies must appear in the report + for name in ("baseline", "self_correct", "line_level", + "prompt_adapted", "zoomed_verify"): + assert name in result.output class TestPerWriterReport: - """RED stubs for per-writer report (IAM-03). All must FAIL until Wave 2.""" - - # These are TODO stubs, not tests: each body is a bare pytest.fail() naming work that - # was never done. Left as hard failures they made the suite permanently red, and a - # suite that is always red cannot gate a dependency upgrade -- which is how 25 - # security advisories sat unactioned. Declared as expected failures they still print - # their message on every run (counted as "xfailed"), so the debt stays visible while - # a NEW failure is once again the only reason the suite goes red. Deleting the marker - # is part of implementing the per-writer report (IAM-03). - pytestmark = pytest.mark.xfail( - reason="RED stub: the per-writer report (IAM-03) is unimplemented", strict=False - ) - - def test_per_writer_report_groups_by_student(self, seeded_db): - pytest.fail( - "not implemented — generate_per_writer_report must group CER by " - "samples.student and return a formatted table string" + """Per-writer report (IAM-03) — turned GREEN in Phase 07-04.""" + + @patch("handwriting_engine.benchmark.evaluate._available_providers") + @patch("handwriting_engine.benchmark.evaluate._read_single") + def test_per_writer_report_groups_by_student(self, mock_read, mock_providers, db_path, tmp_path): + from PIL import Image + + # Seed two IAM-tagged samples for two different writers + conn = get_connection(db_path) + img_a = tmp_path / "a.png" + img_b = tmp_path / "b.png" + Image.new("RGB", (200, 200), color=(128, 128, 128)).save(img_a) + Image.new("RGB", (200, 200), color=(128, 128, 128)).save(img_b) + sid_a = insert_sample(conn, str(img_a), "hashA", + student="iam-writer-a01", category="iam") + sid_b = insert_sample(conn, str(img_b), "hashB", + student="iam-writer-b02", category="iam") + insert_ground_truth(conn, sid_a, "the mitochondria is the powerhouse of the cell") + insert_ground_truth(conn, sid_b, "the mitochondria is the powerhouse of the cell") + conn.close() + + mock_providers.return_value = ["gemini"] + # Writer a01 perfect, writer b02 has one substitution + def fake_read(path, *args, **kwargs): + if "a.png" in path: + text = "the mitochondria is the powerhouse of the cell" + else: + text = "the mitochondria is the powerhouse of the sell" + return { + "text": text, + "confidence": 0.7, "latency_ms": 500, + "input_tokens": 100, "output_tokens": 50, "error": None, + } + mock_read.side_effect = fake_read + + run_id = run_benchmark( + providers=["gemini"], strategies=[], db_path=db_path, ) - def test_per_writer_report_no_writers(self, seeded_db): - pytest.fail( - "not implemented — generate_per_writer_report on run with no student " - "data must return a message indicating no writer data available" + report = generate_per_writer_report(run_id=run_id, db_path=db_path) + assert "iam-writer-a01" in report + assert "iam-writer-b02" in report + assert "Writer" in report # header + assert "Mean CER" in report + + @patch("handwriting_engine.benchmark.evaluate._available_providers") + @patch("handwriting_engine.benchmark.evaluate._read_single") + def test_per_writer_report_no_writers(self, mock_read, mock_providers, seeded_db): + # seeded_db has student="test" — but the function only includes writers + # with non-empty student. The seeded sample has student="test", so let's + # blank it out to trigger the empty path. + conn = get_connection(seeded_db) + try: + conn.execute("UPDATE samples SET student=''") + conn.commit() + finally: + conn.close() + + mock_providers.return_value = ["gemini"] + mock_read.return_value = { + "text": "the mitochondria is the powerhouse of the cell", + "confidence": 0.7, "latency_ms": 500, + "input_tokens": 100, "output_tokens": 50, "error": None, + } + run_id = run_benchmark( + providers=["gemini"], strategies=[], db_path=seeded_db, ) + report = generate_per_writer_report(run_id=run_id, db_path=seeded_db) + assert "No writer data" in report def test_report_cli_per_writer_flag(self, tmp_path): - pytest.fail( - "not implemented — `benchmark report --per-writer` CLI flag must exist " - "and invoke generate_per_writer_report" + from handwriting_engine.cli import cli + + db_path = tmp_path / "empty.db" + get_connection(db_path).close() + + runner = CliRunner() + result = runner.invoke( + cli, + ["benchmark", "report", "--per-writer", "--db-path", str(db_path)], ) + assert result.exit_code == 0, result.output + # Empty DB: function returns "No runs found" or similar + assert ( + "No runs found" in result.output + or "No writer data" in result.output + or "Per-Writer CER" in result.output + ) + + # Help advertises the flag + help_result = runner.invoke(cli, ["benchmark", "report", "--help"]) + assert "--per-writer" in help_result.output diff --git a/tests/test_benchmark_ingest_lab.py b/tests/test_benchmark_ingest_lab.py new file mode 100644 index 0000000..1c314fb --- /dev/null +++ b/tests/test_benchmark_ingest_lab.py @@ -0,0 +1,208 @@ +"""Phase 9 / RPT-03 — guided lab notebook ingest. + +Covers: +- Walks an image directory; new images get inserted with category='lab'. +- prompt_fn output is stored as ground_truth, source='lab-grader'. +- Empty prompt_fn return = user-skip; sample stays without GT. +- Resume: re-running skips images that already have ground truth. +- Counts dict reflects each disposition. +- CLI surface invokes ingest_lab and reports the counts. +""" + +import os +from pathlib import Path +from unittest.mock import patch + +import pytest +from PIL import Image +from click.testing import CliRunner + +from handwriting_engine.benchmark.db import ( + get_connection, + get_sample_by_hash, +) +from handwriting_engine.benchmark.ingest import hash_file, ingest_lab +from handwriting_engine.cli import cli + + +def _make_image_dir(tmp_path: Path, n: int) -> Path: + """Create n unique images, return the dir path.""" + img_dir = tmp_path / "lab_images" + img_dir.mkdir() + for i in range(n): + # Use varying pixel value to keep file hashes unique. + Image.new("RGB", (64, 64), (10 + i, 20 + i, 30 + i)).save(img_dir / f"page_{i:03d}.png") + return img_dir + + +# --- core --- + + +class TestIngestLabCore: + def test_requires_prompt_fn(self, tmp_path): + d = _make_image_dir(tmp_path, 1) + with pytest.raises(ValueError, match="prompt_fn"): + ingest_lab(d, db_path=tmp_path / "db.db") + + def test_inserts_samples_and_ground_truth(self, tmp_path): + d = _make_image_dir(tmp_path, 3) + db = tmp_path / "lab.db" + + def prompt(_path, _suggestion): + return "the cell underwent mitosis" + + counts = ingest_lab(d, prompt_fn=prompt, db_path=db, student="alice") + assert counts["newly_added"] == 3 + assert counts["annotated"] == 3 + assert counts["skipped_existing_gt"] == 0 + assert counts["skipped_user"] == 0 + + # Spot-check storage. + conn = get_connection(db) + sample = get_sample_by_hash(conn, hash_file(d / "page_000.png")) + assert sample is not None + assert sample.category == "lab" + assert sample.student == "alice" + gt_row = conn.execute( + "SELECT text, source, author FROM ground_truths WHERE sample_id = ?", + (sample.id,), + ).fetchone() + conn.close() + assert gt_row["text"] == "the cell underwent mitosis" + assert gt_row["source"] == "lab-grader" + assert gt_row["author"] == "alice" + + def test_user_skip_leaves_sample_without_ground_truth(self, tmp_path): + d = _make_image_dir(tmp_path, 2) + db = tmp_path / "skip.db" + + def prompt(_path, _suggestion): + return None # user cancels EDITOR + + counts = ingest_lab(d, prompt_fn=prompt, db_path=db) + assert counts["newly_added"] == 2 + assert counts["annotated"] == 0 + assert counts["skipped_user"] == 2 + + # Samples present, no GT rows. + conn = get_connection(db) + gt_count = conn.execute("SELECT COUNT(*) FROM ground_truths").fetchone()[0] + sample_count = conn.execute("SELECT COUNT(*) FROM samples").fetchone()[0] + conn.close() + assert sample_count == 2 + assert gt_count == 0 + + def test_empty_string_prompt_treated_as_skip(self, tmp_path): + d = _make_image_dir(tmp_path, 2) + db = tmp_path / "empty.db" + + def prompt(_path, _suggestion): + return " " # whitespace-only treated as skip + + counts = ingest_lab(d, prompt_fn=prompt, db_path=db) + assert counts["annotated"] == 0 + assert counts["skipped_user"] == 2 + + def test_resume_skips_existing_gt(self, tmp_path): + d = _make_image_dir(tmp_path, 3) + db = tmp_path / "resume.db" + + # First pass: annotate page 0 and 1, skip page 2. + decisions = iter(["text 0", "text 1", None]) + ingest_lab(d, prompt_fn=lambda _p, _s: next(decisions), db_path=db) + + # Second pass: prompt should only be called for page 2. + called_for: list[str] = [] + + def prompt2(path, _s): + called_for.append(Path(path).name) + return "text 2" + + counts = ingest_lab(d, prompt_fn=prompt2, db_path=db) + + assert counts["skipped_existing_gt"] == 2 + assert counts["annotated"] == 1 + assert called_for == ["page_002.png"] + + def test_vlm_suggestion_passed_to_prompt(self, tmp_path): + d = _make_image_dir(tmp_path, 1) + db = tmp_path / "vlm.db" + + captured: dict = {} + + def prompt(path, suggestion): + captured["suggestion"] = suggestion + return "final text" + + # Patch read_with_consensus where ingest_lab will import it from. + with patch("handwriting_engine.vision.read_with_consensus") as mock_read: + mock_read.return_value = type("R", (), {"text": "vlm guess"})() + counts = ingest_lab( + d, prompt_fn=prompt, db_path=db, + use_vlm_suggestion=True, vlm_provider="gemini", + ) + + assert captured["suggestion"] == "vlm guess" + assert counts["annotated"] == 1 + + def test_vlm_failure_falls_back_to_empty_suggestion(self, tmp_path): + d = _make_image_dir(tmp_path, 1) + db = tmp_path / "vlm_fail.db" + + captured: dict = {} + + def prompt(_path, suggestion): + captured["suggestion"] = suggestion + return "manual" + + with patch("handwriting_engine.vision.read_with_consensus", + side_effect=RuntimeError("provider down")): + counts = ingest_lab( + d, prompt_fn=prompt, db_path=db, use_vlm_suggestion=True, + ) + # VLM error must not crash the workflow; suggestion is empty, + # the user can still annotate manually. + assert captured["suggestion"] == "" + assert counts["annotated"] == 1 + + def test_non_directory_raises(self, tmp_path): + bogus = tmp_path / "not_a_dir.png" + bogus.write_bytes(b"not an image") + with pytest.raises(FileNotFoundError): + ingest_lab(bogus, prompt_fn=lambda *a: "x", db_path=tmp_path / "db.db") + + +# --- CLI --- + + +class TestIngestLabCli: + def test_cli_invokes_ingest_lab(self, tmp_path): + d = _make_image_dir(tmp_path, 2) + db = tmp_path / "cli.db" + + runner = CliRunner() + # Patch click.edit to simulate the user typing. + with patch("click.edit", return_value="the cell\n"): + result = runner.invoke(cli, [ + "benchmark", "ingest-lab", str(d), + "--student", "alice", + "--db-path", str(db), + ]) + assert result.exit_code == 0, result.output + assert "Lab ingest complete" in result.output + assert "2 annotated" in result.output + + def test_cli_skip_via_editor_cancel(self, tmp_path): + d = _make_image_dir(tmp_path, 2) + db = tmp_path / "cli_skip.db" + + runner = CliRunner() + # click.edit returning None signals the user closed without saving. + with patch("click.edit", return_value=None): + result = runner.invoke(cli, [ + "benchmark", "ingest-lab", str(d), + "--db-path", str(db), + ]) + assert result.exit_code == 0, result.output + assert "0 annotated" in result.output + assert "2 skipped" in result.output diff --git a/tests/test_benchmark_recommend.py b/tests/test_benchmark_recommend.py new file mode 100644 index 0000000..bf41749 --- /dev/null +++ b/tests/test_benchmark_recommend.py @@ -0,0 +1,201 @@ +"""Phase 9 / RPT-02 — composite-scored configuration recommendation. + +Covers: +- Composite weights 70% CER / 15% cost / 15% stability. +- Min-max normalization within candidate set. +- Single-run candidates flagged n=1, given median stability. +- Empty DB / no measured CER → graceful messages. +- CLI: `benchmark recommend`. +""" + +import os +import tempfile + +from PIL import Image +from click.testing import CliRunner + +from handwriting_engine.benchmark.db import ( + get_connection, + insert_sample, + insert_ground_truth, + insert_run, + finish_run, + insert_provider_output, + insert_eval_metric, +) +from handwriting_engine.benchmark.report import recommend_strategy +from handwriting_engine.cli import cli + + +def _seed_db_with_runs(db_path, configs): + """configs: list of dicts with keys provider, strategy, cers (list[float]), + cost_per_sample (float). Each config produces ONE run with len(cers) samples.""" + conn = get_connection(db_path) + img_dir = tempfile.mkdtemp() + sids, gids = [], [] + n = max(len(c["cers"]) for c in configs) + for i in range(n): + p = os.path.join(img_dir, f"i{i}.png") + Image.new("RGB", (32, 32), (128, 128, 128)).save(p) + sid = insert_sample(conn, p, f"hash{i}", student=f"w{i % 3}") + gid = insert_ground_truth(conn, sid, "ground truth") + sids.append(sid) + gids.append(gid) + + run_ids = [] + for cfg in configs: + rid = insert_run( + conn, label=cfg.get("label", ""), + providers=[cfg["provider"]], strategies=[cfg["strategy"]], + ) + # Cost per sample is achieved by setting input/output tokens such + # that estimate_cost yields the desired total. Simpler: set + # input_tokens = total_input_tokens_for_run, then cost is computed + # by report._aggregate_results. We'll just dial output_tokens. + # Easier: we don't pin cost exactly here; tests inspect *relative* + # ordering, not absolute $ values. + for sid, gid, cer in zip(sids, gids, cfg["cers"]): + po = insert_provider_output( + conn, run_id=rid, sample_id=sid, + provider=cfg["provider"], strategy=cfg["strategy"], + output_text="x", confidence=0.9, + input_tokens=cfg.get("input_tokens", 1000), + output_tokens=cfg.get("output_tokens", 500), + ) + insert_eval_metric( + conn, provider_output_id=po, ground_truth_id=gid, + cer=cer, wer=cer, + ) + finish_run(conn, rid, len(cfg["cers"])) + run_ids.append(rid) + conn.close() + return run_ids + + +# --- recommend_strategy() --- + + +class TestRecommendCore: + def test_empty_db_message(self, tmp_path): + db = tmp_path / "empty.db" + # Initialize the DB but insert no runs. + get_connection(db).close() + out = recommend_strategy(db_path=db) + assert "No runs" in out + + def test_lower_cer_wins_when_cost_equal(self, tmp_path): + # Two candidates, equal cost, A has lower CER. + db = tmp_path / "cer.db" + _seed_db_with_runs(db, [ + {"provider": "gemini", "strategy": "vote", "cers": [0.05] * 10}, + {"provider": "claude", "strategy": "vote", "cers": [0.10] * 10}, + ]) + out = recommend_strategy(db_path=db) + assert "Winner: gemini + vote" in out + + def test_lower_cost_wins_when_cer_equal(self, tmp_path): + # Equal CER → composite reduces to cost + stability. Both candidates + # have one run so stability is neutral; cost is the only differentiator. + db = tmp_path / "cost.db" + _seed_db_with_runs(db, [ + {"provider": "gemini", "strategy": "vote", + "cers": [0.10] * 10, "input_tokens": 100, "output_tokens": 50}, + {"provider": "claude", "strategy": "vote", + "cers": [0.10] * 10, "input_tokens": 1000, "output_tokens": 500}, + ]) + out = recommend_strategy(db_path=db) + # Gemini is cheaper per-token AND uses fewer tokens → lower cost. + assert "Winner: gemini + vote" in out + + def test_more_stable_wins_when_cer_and_cost_equal(self, tmp_path): + # Same overall mean CER (0.10), same cost, different across-run + # variance. Stable's three runs all average to 0.10. Wobbly's + # three runs average to 0.05, 0.10, 0.15 — same overall mean, + # but stdev across runs is much higher. + db = tmp_path / "stab.db" + stable_runs = [[0.10] * 10, [0.10] * 10, [0.10] * 10] + wobbly_runs = [[0.05] * 10, [0.10] * 10, [0.15] * 10] + configs = [] + for i, cers in enumerate(stable_runs): + configs.append({ + "provider": "stable", "strategy": "vote", + "cers": cers, "label": f"stable_{i}", + }) + for i, cers in enumerate(wobbly_runs): + configs.append({ + "provider": "wobbly", "strategy": "vote", + "cers": cers, "label": f"wobbly_{i}", + }) + _seed_db_with_runs(db, configs) + out = recommend_strategy(db_path=db) + # Same overall mean, same cost, lower across-run stdev → stable wins. + assert "Winner: stable + vote" in out + + def test_single_run_flagged_n_1(self, tmp_path): + db = tmp_path / "single.db" + _seed_db_with_runs(db, [ + {"provider": "gemini", "strategy": "vote", "cers": [0.05] * 10}, + {"provider": "claude", "strategy": "vote", "cers": [0.10] * 10}, + ]) + out = recommend_strategy(db_path=db) + # Both candidates have only one run → both should show n=1 marker + # in the stdev column. + assert "n=1" in out + + def test_score_ordering_descending(self, tmp_path): + db = tmp_path / "order.db" + _seed_db_with_runs(db, [ + {"provider": "gemini", "strategy": "vote", "cers": [0.05] * 10}, + {"provider": "claude", "strategy": "vote", "cers": [0.10] * 10}, + {"provider": "openai", "strategy": "vote", "cers": [0.15] * 10}, + ]) + out = recommend_strategy(db_path=db) + # Gemini at rank 1, openai (worst) at rank 3. + gemini_idx = out.find("gemini") + openai_idx = out.find("openai") + assert 0 < gemini_idx < openai_idx + + def test_winner_is_top_of_ranked_table(self, tmp_path): + db = tmp_path / "header.db" + _seed_db_with_runs(db, [ + {"provider": "gemini", "strategy": "vote", "cers": [0.05] * 10}, + {"provider": "claude", "strategy": "vote", "cers": [0.10] * 10}, + ]) + out = recommend_strategy(db_path=db) + # The "Winner: …" line and the "1 …" rank line must reference the + # same configuration. + winner_line = next(line for line in out.split("\n") if line.startswith(" Winner:")) + rank_one = next(line for line in out.split("\n") if line.startswith("1 ")) + assert "gemini" in winner_line and "vote" in winner_line + assert "gemini" in rank_one and "vote" in rank_one + + +# --- CLI --- + + +class TestRecommendCli: + def test_recommend_cmd_runs(self, tmp_path): + db = tmp_path / "cli.db" + _seed_db_with_runs(db, [ + {"provider": "gemini", "strategy": "vote", "cers": [0.05] * 10}, + {"provider": "claude", "strategy": "vote", "cers": [0.10] * 10}, + ]) + runner = CliRunner() + result = runner.invoke(cli, [ + "benchmark", "recommend", + "--db-path", str(db), + ]) + assert result.exit_code == 0, result.output + assert "Winner:" in result.output + + def test_recommend_cmd_empty_db(self, tmp_path): + db = tmp_path / "empty_cli.db" + get_connection(db).close() + runner = CliRunner() + result = runner.invoke(cli, [ + "benchmark", "recommend", + "--db-path", str(db), + ]) + # No runs is a graceful empty state, not an error. + assert result.exit_code == 0 + assert "No runs" in result.output diff --git a/tests/test_benchmark_stats.py b/tests/test_benchmark_stats.py new file mode 100644 index 0000000..6ca9bda --- /dev/null +++ b/tests/test_benchmark_stats.py @@ -0,0 +1,325 @@ +"""Tests for Phase 8 statistics layer (STAT-01, STAT-02). + +Covers paired Wilcoxon signed-rank, percentile-method bootstrap CI, and +Cohen's r against scipy-validated reference values where applicable. +""" + +import pytest + +from handwriting_engine.benchmark.stats import ( + WilcoxonResult, + wilcoxon_signed_rank, + bootstrap_ci, + cohens_r, + _rank_with_ties, + _normal_cdf, +) + + +# --- _rank_with_ties --- + +class TestRankWithTies: + def test_no_ties(self): + ranks, tie_corr = _rank_with_ties([3.0, 1.0, 2.0]) + assert ranks == [3.0, 1.0, 2.0] + assert tie_corr == 0.0 + + def test_two_way_tie(self): + # Values [1, 1, 2] — the two 1s share ranks 1 and 2 → average 1.5 + ranks, tie_corr = _rank_with_ties([1.0, 1.0, 2.0]) + assert ranks == [1.5, 1.5, 3.0] + # tie correction = 2^3 - 2 = 6 + assert tie_corr == 6.0 + + def test_three_way_tie(self): + ranks, tie_corr = _rank_with_ties([5.0, 5.0, 5.0, 9.0]) + # Three tied at ranks 1,2,3 → average 2.0 + assert ranks == [2.0, 2.0, 2.0, 4.0] + # tie correction = 3^3 - 3 = 24 + assert tie_corr == 24.0 + + +# --- _normal_cdf --- + +class TestNormalCdf: + def test_centered(self): + assert _normal_cdf(0.0) == pytest.approx(0.5, abs=1e-9) + + def test_one_sigma(self): + # P(Z <= 1) ≈ 0.8413 + assert _normal_cdf(1.0) == pytest.approx(0.8413, abs=1e-3) + + def test_two_sigma(self): + # P(Z <= 2) ≈ 0.9772 + assert _normal_cdf(2.0) == pytest.approx(0.9772, abs=1e-3) + + def test_negative(self): + assert _normal_cdf(-1.0) == pytest.approx(1.0 - 0.8413, abs=1e-3) + + +# --- wilcoxon_signed_rank --- + +class TestWilcoxonSignedRank: + def test_returns_wilcoxon_result_dataclass(self): + # n >= 10 paired samples with a = b - 0.01 (a is consistently lower) + a = [0.10, 0.12, 0.11, 0.13, 0.09, 0.14, 0.11, 0.10, 0.12, 0.13] + b = [0.11, 0.13, 0.12, 0.14, 0.10, 0.15, 0.12, 0.11, 0.13, 0.14] + result = wilcoxon_signed_rank(a, b) + assert isinstance(result, WilcoxonResult) + + def test_paired_difference_significant(self): + # All differences negative and consistent: strong signal, p should be small. + # n = 10, all a_i < b_i → all rank-sums on the negative side. + a = [0.10, 0.12, 0.11, 0.13, 0.09, 0.14, 0.11, 0.10, 0.12, 0.13] + b = [0.11, 0.13, 0.12, 0.14, 0.10, 0.15, 0.12, 0.11, 0.13, 0.14] + result = wilcoxon_signed_rank(a, b) + assert result.p_value < 0.05 + assert result.n == 10 + # b is consistently larger → w_plus is small (most ranks went to negative diffs) + assert result.statistic < (result.n * (result.n + 1) / 4.0) + + def test_no_difference_yields_high_p(self): + # Identical samples → all diffs zero → p=1 + a = [0.10, 0.12, 0.11, 0.13, 0.09, 0.14, 0.11, 0.10, 0.12, 0.13] + result = wilcoxon_signed_rank(a, a) + assert result.p_value == 1.0 + assert result.n == 0 + + def test_random_noise_yields_high_p(self): + # Symmetric noise around zero diff → no signal → p should not reject + a = [0.10, 0.13, 0.11, 0.12, 0.10, 0.13, 0.11, 0.12, 0.10, 0.13] + b = [0.11, 0.12, 0.12, 0.11, 0.11, 0.12, 0.12, 0.11, 0.11, 0.12] + result = wilcoxon_signed_rank(a, b) + assert result.p_value > 0.05 + + def test_unequal_length_raises(self): + with pytest.raises(ValueError, match="same length"): + wilcoxon_signed_rank([0.1, 0.2], [0.1, 0.2, 0.3]) + + def test_two_sided_symmetry(self): + # Swapping a and b must flip sign of z but leave |z| and p_value identical + a = [0.10, 0.12, 0.11, 0.13, 0.09, 0.14, 0.11, 0.10, 0.12, 0.13] + b = [0.11, 0.13, 0.12, 0.14, 0.10, 0.15, 0.12, 0.11, 0.13, 0.14] + r1 = wilcoxon_signed_rank(a, b) + r2 = wilcoxon_signed_rank(b, a) + assert r1.p_value == pytest.approx(r2.p_value, abs=1e-9) + assert abs(r1.z) == pytest.approx(abs(r2.z), abs=1e-9) + # z should flip sign + assert (r1.z > 0) != (r2.z > 0) + + def test_small_n_still_runs(self): + # n = 3 — function should still return a result; callers gate at >=10 + a = [0.1, 0.2, 0.3] + b = [0.2, 0.3, 0.4] + result = wilcoxon_signed_rank(a, b) + assert 0.0 <= result.p_value <= 1.0 + assert result.n == 3 + + def test_zero_diffs_dropped(self): + # 8 paired with diff zero, 2 paired with positive diff. n_effective=2. + a = [0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.2, 0.3] + b = [0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1] + result = wilcoxon_signed_rank(a, b) + assert result.n == 2 + + def test_p_value_in_unit_interval(self): + # Property: p_value must always be in [0, 1]. + cases = [ + ([0.1] * 10, [0.2] * 10), + ([0.5] * 10, [0.5] * 10), + ([0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0], + [1.0, 0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 0.1]), + ] + for a, b in cases: + result = wilcoxon_signed_rank(a, b) + assert 0.0 <= result.p_value <= 1.0 + + +# --- bootstrap_ci --- + +class TestBootstrapCi: + def test_zero_samples_returns_zero_band(self): + assert bootstrap_ci([]) == (0.0, 0.0) + + def test_single_sample_returns_degenerate_band(self): + assert bootstrap_ci([0.42]) == (0.42, 0.42) + + def test_constant_sample_returns_constant_band(self): + # All values 0.1 → resample of any size → mean 0.1 + lo, hi = bootstrap_ci([0.1] * 20, seed=42) + assert lo == pytest.approx(0.1, abs=1e-9) + assert hi == pytest.approx(0.1, abs=1e-9) + + def test_variable_sample_brackets_mean(self): + # 30 samples around 0.15 with realistic CER spread; the CI must + # contain the empirical mean and have non-zero width. + values = [0.10, 0.12, 0.14, 0.13, 0.15, 0.11, 0.18, 0.20, 0.13, 0.14, + 0.15, 0.16, 0.17, 0.12, 0.13, 0.14, 0.11, 0.10, 0.18, 0.19, + 0.13, 0.14, 0.15, 0.16, 0.12, 0.13, 0.11, 0.10, 0.17, 0.18] + lo, hi = bootstrap_ci(values, seed=42) + mean = sum(values) / len(values) + assert lo < mean < hi + assert (hi - lo) > 0.0 + + def test_higher_confidence_yields_wider_band(self): + values = [0.1, 0.2, 0.15, 0.12, 0.18, 0.14, 0.16, 0.13, 0.17, 0.11] + lo_90, hi_90 = bootstrap_ci(values, confidence=0.90, seed=42) + lo_99, hi_99 = bootstrap_ci(values, confidence=0.99, seed=42) + assert (hi_99 - lo_99) >= (hi_90 - lo_90) + + def test_seed_makes_output_deterministic(self): + values = [0.1, 0.2, 0.15, 0.12, 0.18, 0.14, 0.16, 0.13, 0.17, 0.11] + a = bootstrap_ci(values, seed=123) + b = bootstrap_ci(values, seed=123) + assert a == b + + def test_invalid_confidence_raises(self): + with pytest.raises(ValueError): + bootstrap_ci([0.1, 0.2], confidence=1.5) + with pytest.raises(ValueError): + bootstrap_ci([0.1, 0.2], confidence=0.0) + + def test_invalid_n_iterations_raises(self): + with pytest.raises(ValueError): + bootstrap_ci([0.1, 0.2], n_iterations=0) + + +# --- cohens_r --- + +class TestCohensR: + def test_zero_z_yields_zero(self): + assert cohens_r(0.0, 10) == 0.0 + + def test_zero_n_returns_zero(self): + # Guard against div-by-zero + assert cohens_r(2.0, 0) == 0.0 + + def test_classic_thresholds(self): + # r = z / sqrt(n) + # n=25, z=2.5 → r=0.5 (large) + assert cohens_r(2.5, 25) == pytest.approx(0.5, abs=1e-9) + + def test_negative_z_returns_positive_r(self): + # Sign of effect lives in z; r is magnitude. + assert cohens_r(-2.5, 25) == pytest.approx(0.5, abs=1e-9) + + +# --- end-to-end paired example --- + +class TestPairedRunComparison: + """Mirrors the production wiring path: two runs, n=10 paired by sample.""" + + def test_strategy_b_consistently_better(self): + # Strategy A: ~10% CER. Strategy B: ~6% CER on same samples. + cer_a = [0.10, 0.12, 0.11, 0.13, 0.09, 0.14, 0.11, 0.10, 0.12, 0.13] + cer_b = [0.06, 0.08, 0.07, 0.09, 0.05, 0.10, 0.07, 0.06, 0.08, 0.09] + + result = wilcoxon_signed_rank(cer_a, cer_b) + # All 10 pairs favour B → significant. + assert result.p_value < 0.01 + # Effect size should be large. + assert cohens_r(result.z, result.n) > 0.5 + + # Bootstrap CIs separate A and B if effect is strong: + lo_a, hi_a = bootstrap_ci(cer_a, seed=0) + lo_b, hi_b = bootstrap_ci(cer_b, seed=0) + # Either A's CI is wholly above B's, or they barely overlap. + # With this strong separation, no overlap. + assert lo_a > hi_b or hi_a < lo_b + + +# --- compare_runs() integration --- + +class TestCompareRunsStatsBlock: + """Integration: compare_runs() must append a stats block when n>=10 + paired samples exist for the same (provider, strategy).""" + + def _seed_two_runs(self, db_path, n_samples, cer_run1, cer_run2): + """Create two runs sharing n samples, with explicit CERs for each.""" + from PIL import Image + from handwriting_engine.benchmark.db import ( + get_connection, insert_sample, insert_ground_truth, + insert_run, finish_run, insert_provider_output, insert_eval_metric, + ) + + conn = get_connection(db_path) + sample_ids = [] + gt_ids = [] + # Lazy: make a single image; insert_sample requires a real path. + import tempfile, os + img_dir = tempfile.mkdtemp() + for i in range(n_samples): + img_path = os.path.join(img_dir, f"img_{i}.png") + Image.new("RGB", (32, 32), (128, 128, 128)).save(img_path) + sid = insert_sample(conn, img_path, f"hash_{i}", student=f"writer_{i % 3}") + gt_id = insert_ground_truth(conn, sid, "ground truth text") + sample_ids.append(sid) + gt_ids.append(gt_id) + + def _seed_run(label, cers): + run_id = insert_run(conn, label=label, providers=["gemini"], strategies=["vote"]) + for sid, gt_id, cer in zip(sample_ids, gt_ids, cers): + po_id = insert_provider_output( + conn, run_id=run_id, sample_id=sid, + provider="gemini", strategy="vote", + output_text="x", confidence=0.9, + ) + insert_eval_metric( + conn, provider_output_id=po_id, ground_truth_id=gt_id, + cer=cer, wer=cer, + ) + finish_run(conn, run_id, len(cers)) + return run_id + + r1 = _seed_run("run_1", cer_run1) + r2 = _seed_run("run_2", cer_run2) + conn.close() + return r1, r2 + + def test_stats_block_appears_for_n_geq_10(self, tmp_path): + from handwriting_engine.benchmark.report import compare_runs + + db = tmp_path / "stats.db" + # 12 paired samples; run_2 consistently better. + cer_a = [0.10, 0.12, 0.11, 0.13, 0.09, 0.14, 0.11, 0.10, 0.12, 0.13, 0.15, 0.16] + cer_b = [0.06, 0.08, 0.07, 0.09, 0.05, 0.10, 0.07, 0.06, 0.08, 0.09, 0.11, 0.12] + r1, r2 = self._seed_two_runs(db, 12, cer_a, cer_b) + + out = compare_runs(r1, r2, db_path=db) + assert "stats:" in out + assert "CI95:" in out + assert "n=12" in out + # Strong separation → low p-value. + # We can't pin the exact value but it should clearly be < 0.05. + import re + m = re.search(r"p=([\d.]+)", out) + assert m, f"p-value not in output:\n{out}" + assert float(m.group(1)) < 0.05 + + def test_stats_block_omitted_when_n_lt_10(self, tmp_path): + from handwriting_engine.benchmark.report import compare_runs + + db = tmp_path / "stats_small.db" + cer_a = [0.10, 0.12, 0.11, 0.13, 0.09] + cer_b = [0.06, 0.08, 0.07, 0.09, 0.05] + r1, r2 = self._seed_two_runs(db, 5, cer_a, cer_b) + + out = compare_runs(r1, r2, db_path=db) + assert "stats:" not in out + assert "CI95:" not in out + + def test_stats_block_present_for_no_difference(self, tmp_path): + # Identical runs → p should be high; the block must still render. + from handwriting_engine.benchmark.report import compare_runs + + db = tmp_path / "stats_identical.db" + cer = [0.10, 0.12, 0.11, 0.13, 0.09, 0.14, 0.11, 0.10, 0.12, 0.13] + r1, r2 = self._seed_two_runs(db, 10, cer, list(cer)) + + out = compare_runs(r1, r2, db_path=db) + # All diffs are zero — n_effective is 0, but the block still shows + # n=0 and p=1.0 (the function gates on len(paired_a) >= 10, not on + # n_effective). That's the right behavior: tells the user "we tried + # the test and there was no signal," vs. silently dropping it. + assert "stats:" in out + assert "p=1.0000" in out diff --git a/tests/test_char_consensus.py b/tests/test_char_consensus.py new file mode 100644 index 0000000..d77c0b3 --- /dev/null +++ b/tests/test_char_consensus.py @@ -0,0 +1,233 @@ +"""Tests for S5 char-level consensus resolution. + +Falsifies: +- #1 char-level resolves a known confusion case (rn↔m) +- #2 defers cleanly when not a confusion case (returns None) +- #3 writer-specific bias overrides default ++ supporting determinism, ordering, and length-mismatch tests +""" + +from handwriting_engine.consensus import ( + resolve_char_level, + _word_level_vote, +) + + +class TestResolveCharLevel: + def test_rn_m_confusion_resolves_to_m_canonical(self): + # Criterion #1 char-level half: no-majority pair where the only + # difference is the rn↔m pair → returns the canonical "m" form. + result = resolve_char_level(["modern", "rnodern"], [1.0, 1.0]) + assert result == "modern" + + def test_unrelated_words_return_none(self): + # Criterion #2: completely different candidates are NOT a confusion + # case; defer to the existing [?alt: …] fallback. + assert resolve_char_level(["apple", "orange"], [1.0, 1.0]) is None + + def test_writer_resolution_overrides_default(self): + # Criterion #3: a writer who consistently writes 'rn' instead of 'm' + # gets their preference applied even when global default would pick 'm'. + result = resolve_char_level( + ["modern", "rnodern"], + [1.0, 1.0], + writer_resolutions={"rn↔m": "rn"}, + ) + assert result == "rnodern" + + def test_writer_resolution_for_m_form(self): + # Symmetric: writer prefers the 'm' form → still returns 'modern'. + result = resolve_char_level( + ["modern", "rnodern"], + [1.0, 1.0], + writer_resolutions={"rn↔m": "m"}, + ) + assert result == "modern" + + def test_cl_d_pair(self): + # cl↔d is a separate well-known confusion pair. + result = resolve_char_level(["could", "coulcl"], [1.0, 1.0]) + # Default (canonical d): "could" wins. + assert result == "could" + + def test_identical_candidates_return_canonical(self): + # When candidates collapse to a single value, that's the answer. + assert resolve_char_level(["modern", "modern"], [1.0, 1.0]) == "modern" + + def test_three_or_more_candidates_returns_none(self): + # v0 only handles pairwise. Three distinct candidates defer. + assert resolve_char_level( + ["modern", "rnodern", "modarn"], [1.0, 1.0, 1.0] + ) is None + + def test_provider_order_independent(self): + # Determinism: swapping the order of candidates must not change the + # result. + a = resolve_char_level(["modern", "rnodern"], [1.0, 1.0]) + b = resolve_char_level(["rnodern", "modern"], [1.0, 1.0]) + assert a == b == "modern" + + def test_length_mismatch_handled(self): + # rn↔m is by definition length-mismatched; the resolver must align + # via SequenceMatcher rather than positional indexing. + result = resolve_char_level(["learn", "leamr"], [1.0, 1.0]) + # leamr → learn? "amr" vs "arn" — replace 'm' at idx 2 with 'rn' + # gives "learn" but the source has 'mr' which doesn't fit cleanly. + # The candidates aren't a clean rn↔m pair → None. + assert result is None + + def test_case_sensitive_pair_I_vs_l(self): + # I↔l is a case-sensitive pair (capital I vs lowercase l). + result = resolve_char_level(["cell", "ceII"], [1.0, 1.0]) + assert result == "cell" + + def test_empty_candidates_return_none(self): + assert resolve_char_level([], []) is None + + def test_single_candidate_returns_self(self): + assert resolve_char_level(["modern"], [1.0]) == "modern" + + +class TestWordLevelVoteIntegration: + """End-to-end: when a no-majority disagreement is a confusion pair, + _word_level_vote should resolve it without emitting [?alt: …].""" + + def test_no_alt_marker_when_char_level_resolves(self): + # 2 providers, equal weight, "modern" vs "rnodern" — would normally + # emit [?alt: rnodern]. With char-level resolution it shouldn't. + text, disagreements, conf = _word_level_vote( + ["the modern cell", "the rnodern cell"], + [("gemini", 1.0), ("claude", 1.0)], + ) + assert "[?alt:" not in text + assert "modern" in text + assert "rnodern" not in text + + def test_alt_marker_still_emits_for_unrelated(self): + # Criterion #2 integration: non-confusion disagreement still gets + # the [?alt: …] marker. + text, disagreements, conf = _word_level_vote( + ["the apple is red", "the orange is red"], + [("gemini", 1.0), ("claude", 1.0)], + ) + # One of "apple"/"orange" wins; the other appears in [?alt: …]. + assert "[?alt:" in text + + def test_writer_profile_threaded_through_vote(self): + # When writer_profile carries a confusion_resolution, it overrides + # the global default at the word-level voting stage. + text, _, _ = _word_level_vote( + ["the modern cell", "the rnodern cell"], + [("gemini", 1.0), ("claude", 1.0)], + writer_profile={"confusion_resolutions": {"rn↔m": "rn"}}, + ) + assert "rnodern" in text + + +class TestWriterProfileEndToEndWireUp: + """End-to-end: writer_profile passed to consensus.read_with_consensus + must reach _word_level_vote unchanged (vote and smart strategies). + + Direct text-level assertions are fragile here because provider weights + bias the vote independent of char-level resolution, so we instead probe + the _word_level_vote call and assert the kwarg arrived intact. + """ + + def _mock_provider(self, name, response): + from unittest.mock import MagicMock + mock = MagicMock() + mock.read_image.return_value = response + mock.usage = {} + mock.get_mean_confidence = MagicMock(return_value=0.0) + return mock + + def test_vote_strategy_propagates_writer_profile(self): + from unittest.mock import patch + from handwriting_engine.consensus import read_with_consensus + + providers_map = { + "openai": self._mock_provider("openai", "the modern cell"), + "claude": self._mock_provider("claude", "the rnodern cell"), + } + captured: dict = {} + + def _spy(*args, **kwargs): + captured["writer_profile"] = kwargs.get("writer_profile") + return ("the modern cell", [], 0.95) + + with patch("handwriting_engine.consensus.available_providers", + return_value=["openai", "claude"]), \ + patch("handwriting_engine.consensus.get_provider", + side_effect=lambda n: providers_map[n]), \ + patch("handwriting_engine.consensus._word_level_vote", + side_effect=_spy): + read_with_consensus( + "b64", "image/jpeg", "read", strategy="vote", + content_type="handwriting", + writer_profile={"confusion_resolutions": {"rn↔m": "rn"}}, + ) + + assert captured["writer_profile"] == { + "confusion_resolutions": {"rn↔m": "rn"} + } + + def test_smart_strategy_no_quality_propagates_writer_profile(self): + # smart with quality_assessment=None falls back to vote internally. + from unittest.mock import patch + from handwriting_engine.consensus import read_with_consensus + + providers_map = { + "openai": self._mock_provider("openai", "the modern cell"), + "claude": self._mock_provider("claude", "the rnodern cell"), + } + captured: dict = {} + + def _spy(*args, **kwargs): + captured["writer_profile"] = kwargs.get("writer_profile") + return ("the modern cell", [], 0.95) + + with patch("handwriting_engine.consensus.available_providers", + return_value=["openai", "claude"]), \ + patch("handwriting_engine.consensus.get_provider", + side_effect=lambda n: providers_map[n]), \ + patch("handwriting_engine.consensus._word_level_vote", + side_effect=_spy): + read_with_consensus( + "b64", "image/jpeg", "read", strategy="smart", + content_type="handwriting", + writer_profile={"confusion_resolutions": {"rn↔m": "rn"}}, + ) + + assert captured["writer_profile"] == { + "confusion_resolutions": {"rn↔m": "rn"} + } + + def test_default_no_writer_profile_passes_none(self): + # Sanity: without writer_profile, the kwarg arrives as None + # (not as an empty dict or some other sentinel that would mask + # missing wires). + from unittest.mock import patch + from handwriting_engine.consensus import read_with_consensus + + providers_map = { + "openai": self._mock_provider("openai", "the modern cell"), + "claude": self._mock_provider("claude", "the modern cell"), + } + captured: dict = {} + + def _spy(*args, **kwargs): + captured["writer_profile"] = kwargs.get("writer_profile", "MISSING") + return ("the modern cell", [], 0.95) + + with patch("handwriting_engine.consensus.available_providers", + return_value=["openai", "claude"]), \ + patch("handwriting_engine.consensus.get_provider", + side_effect=lambda n: providers_map[n]), \ + patch("handwriting_engine.consensus._word_level_vote", + side_effect=_spy): + read_with_consensus( + "b64", "image/jpeg", "read", strategy="vote", + content_type="handwriting", + ) + + assert captured["writer_profile"] is None diff --git a/tests/test_cli_read.py b/tests/test_cli_read.py new file mode 100644 index 0000000..da9c457 --- /dev/null +++ b/tests/test_cli_read.py @@ -0,0 +1,133 @@ +"""Tests for the `cli read` command surface (S3 — skill-engine bridge). + +These tests pin the contract that the handwriting-reader skill depends on: +- --format=md|json|txt +- --writer= for WriterProfileStore profile binding +- Default --domain=general (was 'biology') +- JSON output exposes alt-markers extracted from consensus text +""" + +import json +from unittest.mock import patch + +import pytest +from click.testing import CliRunner + +from handwriting_engine.cli import cli +from handwriting_engine.providers.base import ConsensusResult + + +@pytest.fixture +def runner(): + return CliRunner() + + +def _fake_consensus(*args, **kwargs): + return ConsensusResult( + text="The mitochondria [?alt: mitochondrion] is the powerhouse [?alt: power-house]", + confidence=0.72, + confidence_level="LOW", + provider_results={"claude": "mitochondria", "gemini": "mitochondrion"}, + disagreements=["'mitochondria' vs 'mitochondrion' (no majority)"], + strategy_used="vote", + tokens_used={"input_tokens": 100, "output_tokens": 50}, + ) + + +def _fake_read_page(*args, **kwargs): + return "plain single-provider read" + + +class TestCliReadFormat: + """`--format` flag controls output shape.""" + + def test_format_txt_default_matches_legacy_echo(self, tmp_image, runner): + path = tmp_image() + with patch("handwriting_engine.vision.read_page", side_effect=_fake_read_page): + result = runner.invoke(cli, ["read", path]) + assert result.exit_code == 0, result.output + assert "plain single-provider read" in result.output + + def test_format_json_returns_parseable_payload(self, tmp_image, runner): + path = tmp_image() + with patch("handwriting_engine.vision.read_with_consensus", side_effect=_fake_consensus): + result = runner.invoke(cli, ["read", path, "--provider", "consensus", "--format", "json"]) + assert result.exit_code == 0, result.output + payload = json.loads(result.output) + assert payload["path"] == path + assert len(payload["pages"]) == 1 + page = payload["pages"][0] + assert "mitochondria" in page["text"] + assert page["confidence"] == pytest.approx(0.72) + assert page["confidence_level"] == "LOW" + assert page["strategy_used"] == "vote" + assert page["disagreements"] + + def test_format_json_extracts_alt_markers(self, tmp_image, runner): + path = tmp_image() + with patch("handwriting_engine.vision.read_with_consensus", side_effect=_fake_consensus): + result = runner.invoke(cli, ["read", path, "--provider", "consensus", "--format", "json"]) + payload = json.loads(result.output) + markers = payload["pages"][0]["alt_markers"] + assert len(markers) == 2 + assert any("mitochondrion" in m["alternatives"] for m in markers) + assert any(m["raw"].startswith("[?alt:") for m in markers) + + def test_format_md_includes_header_and_text(self, tmp_image, runner): + path = tmp_image() + with patch("handwriting_engine.vision.read_with_consensus", side_effect=_fake_consensus): + result = runner.invoke(cli, ["read", path, "--provider", "consensus", "--format", "md"]) + assert result.exit_code == 0, result.output + assert "## " in result.output # markdown heading + assert "mitochondria" in result.output + + +class TestCliReadDomain: + """Default domain is 'general', not 'biology'.""" + + def test_default_domain_is_general(self, tmp_image, runner): + path = tmp_image() + captured: dict = {} + + def _capture(*args, **kwargs): + captured.update(kwargs) + return _fake_read_page(*args, **kwargs) + + with patch("handwriting_engine.vision.read_page", side_effect=_capture): + result = runner.invoke(cli, ["read", path]) + assert result.exit_code == 0 + assert captured.get("domain") == "general" + + +class TestCliReadWriter: + """`--writer=` is accepted and forwarded to the consensus call.""" + + def test_writer_flag_forwarded_to_consensus(self, tmp_image, runner): + path = tmp_image() + captured: dict = {} + + def _capture(*args, **kwargs): + captured.update(kwargs) + return _fake_consensus(*args, **kwargs) + + with patch("handwriting_engine.vision.read_with_consensus", side_effect=_capture): + result = runner.invoke( + cli, ["read", path, "--provider", "consensus", "--writer", "ada-001", "--format", "json"] + ) + assert result.exit_code == 0, result.output + assert captured.get("writer_id") == "ada-001" + + def test_writer_flag_absent_does_not_pass_writer_id(self, tmp_image, runner): + path = tmp_image() + captured: dict = {} + + def _capture(*args, **kwargs): + captured.update(kwargs) + return _fake_consensus(*args, **kwargs) + + with patch("handwriting_engine.vision.read_with_consensus", side_effect=_capture): + result = runner.invoke( + cli, ["read", path, "--provider", "consensus", "--format", "json"] + ) + assert result.exit_code == 0, result.output + assert captured.get("writer_id") in (None, "") diff --git a/tests/test_few_shot.py b/tests/test_few_shot.py new file mode 100644 index 0000000..7cf6c41 --- /dev/null +++ b/tests/test_few_shot.py @@ -0,0 +1,400 @@ +"""Tests for S2 — per-writer few-shot exemplars. + +Maps onto the S2-SPEC falsifiable criteria: +* #1 -- eligibility gate (select_exemplars) +* #2 -- provider calls carry exemplars before the target +* #4 -- cold-writer (<2 GT samples) falls back to single-image read +* #5 -- TrOCR passthrough (no error, no exemplars) +* #6 -- HE_FEW_SHOT_K env honored (cost guardrail surface) + +Criterion #3 (CER on real IAM data) requires the populated benchmark DB and +the Phase 8 stats infrastructure; it's covered by a separate eval, not here. +""" + +from __future__ import annotations + +import os + +import pytest + +from handwriting_engine.benchmark.db import ( + get_connection, + insert_ground_truth, + insert_sample, +) +from handwriting_engine.few_shot import ( + DEFAULT_FEW_SHOT_K, + EXEMPLAR_LABEL_TEMPLATE, + EXEMPLAR_PROVIDERS, + FEW_SHOT_K_ENV, + build_exemplar_blocks, + env_few_shot_k, + provider_supports_exemplars, + select_and_build_exemplar_blocks, +) +from handwriting_engine.writer_profile_store import ( + Exemplar, + select_exemplars, +) + + +# --------------------------------------------------------------------------- +# Fixtures +# --------------------------------------------------------------------------- + + +def _png_bytes(seed: int = 0) -> bytes: + # Distinct content per seed so image_hash uniqueness holds. + base = ( + b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x01\x00\x00\x00\x01" + b"\x08\x06\x00\x00\x00\x1f\x15\xc4\x89\x00\x00\x00\rIDATx\x9cc\xfa\xcf" + b"\x00\x00\x00\x03\x00\x01\x16\xfb\x96\xea\x00\x00\x00\x00IEND\xaeB`\x82" + ) + return base + (b"\x00" * seed) + + +@pytest.fixture +def png_factory(tmp_path): + counter = {"i": 0} + + def make(name: str = None) -> str: + counter["i"] += 1 + path = tmp_path / (name or f"img-{counter['i']}.png") + path.write_bytes(_png_bytes(counter["i"])) + return str(path) + + return make + + +@pytest.fixture +def db(): + conn = get_connection(":memory:") + yield conn + conn.close() + + +def _seed_writer(db, *, writer_id: str, n: int, png_factory) -> list[int]: + """Insert n samples + GTs for a writer; returns the sample ids in order.""" + sample_ids = [] + for idx in range(n): + path = png_factory(name=f"{writer_id}-{idx}.png") + from handwriting_engine.benchmark.ingest import hash_file + + sid = insert_sample( + db, + image_path=path, + image_hash=hash_file(path), + student=writer_id, + ) + insert_ground_truth(db, sid, f"sample-{idx} text for {writer_id}") + sample_ids.append(sid) + return sample_ids + + +# --------------------------------------------------------------------------- +# Criterion #1 -- eligibility gate +# --------------------------------------------------------------------------- + + +class TestSelectExemplars: + def test_returns_empty_when_writer_has_no_samples(self, db): + assert select_exemplars("absent-writer", k=3, conn=db) == [] + + def test_returns_empty_when_writer_has_one_sample(self, db, png_factory): + # SPEC #1: caller decides to skip when len < 2; select_exemplars + # itself returns the 1 row -- it's not the gate's job. + # We assert exact count so the consumer's gate is unambiguous. + _seed_writer(db, writer_id="solo", n=1, png_factory=png_factory) + assert len(select_exemplars("solo", k=3, conn=db)) == 1 + + def test_returns_k_rows_when_available(self, db, png_factory): + _seed_writer(db, writer_id="prolific", n=5, png_factory=png_factory) + result = select_exemplars("prolific", k=3, conn=db) + assert len(result) == 3 + assert all(isinstance(e, Exemplar) for e in result) + assert all(e.ground_truth.startswith("sample-") for e in result) + + def test_caps_at_available_samples(self, db, png_factory): + _seed_writer(db, writer_id="meager", n=2, png_factory=png_factory) + result = select_exemplars("meager", k=5, conn=db) + assert len(result) == 2 + + def test_deterministic_order_by_sample_id(self, db, png_factory): + ids = _seed_writer(db, writer_id="det", n=4, png_factory=png_factory) + first = select_exemplars("det", k=3, conn=db) + second = select_exemplars("det", k=3, conn=db) + assert [e.sample_id for e in first] == [e.sample_id for e in second] + # And the order matches the deterministic id-ascending tiebreak. + assert [e.sample_id for e in first] == sorted(ids)[:3] + + def test_exclude_sample_id_filters_target_image(self, db, png_factory): + ids = _seed_writer(db, writer_id="exclude", n=3, png_factory=png_factory) + target_id = ids[1] + result = select_exemplars("exclude", k=5, conn=db, exclude_sample_id=target_id) + assert target_id not in [e.sample_id for e in result] + assert len(result) == 2 + + def test_k_zero_returns_empty(self, db, png_factory): + _seed_writer(db, writer_id="zero", n=3, png_factory=png_factory) + assert select_exemplars("zero", k=0, conn=db) == [] + + def test_k_negative_returns_empty(self, db, png_factory): + _seed_writer(db, writer_id="neg", n=3, png_factory=png_factory) + assert select_exemplars("neg", k=-1, conn=db) == [] + + def test_blank_writer_id_returns_empty(self, db, png_factory): + _seed_writer(db, writer_id="something", n=2, png_factory=png_factory) + assert select_exemplars("", k=3, conn=db) == [] + + def test_does_not_leak_across_writers(self, db, png_factory): + _seed_writer(db, writer_id="a", n=3, png_factory=png_factory) + _seed_writer(db, writer_id="b", n=2, png_factory=png_factory) + result_a = select_exemplars("a", k=10, conn=db) + result_b = select_exemplars("b", k=10, conn=db) + assert len(result_a) == 3 + assert len(result_b) == 2 + assert {e.sample_id for e in result_a}.isdisjoint( + {e.sample_id for e in result_b} + ) + + +# --------------------------------------------------------------------------- +# Criterion #2 -- block layout: exemplars before target, each followed by label +# --------------------------------------------------------------------------- + + +class TestBuildExemplarBlocks: + def test_layout_interleaves_image_text_pairs_then_target(self, png_factory): + e1_path = png_factory(name="ex1.png") + e2_path = png_factory(name="ex2.png") + exemplars = [ + Exemplar(sample_id=1, image_path=e1_path, ground_truth="hello"), + Exemplar(sample_id=2, image_path=e2_path, ground_truth="world"), + ] + blocks = build_exemplar_blocks( + target_image_b64="TGT_B64", + target_media_type="image/jpeg", + exemplars=exemplars, + ) + # Expected sequence: image, text, image, text, image (target). + types = [b["type"] for b in blocks] + assert types == ["image", "text", "image", "text", "image"] + + def test_target_is_last_block(self, png_factory): + e_path = png_factory(name="ex.png") + blocks = build_exemplar_blocks( + target_image_b64="TGT_B64", + target_media_type="image/png", + exemplars=[ + Exemplar(sample_id=1, image_path=e_path, ground_truth="x"), + Exemplar(sample_id=2, image_path=e_path, ground_truth="y"), + ], + ) + last = blocks[-1] + assert last["type"] == "image" + assert last["source"]["data"] == "TGT_B64" + assert last["source"]["media_type"] == "image/png" + + def test_label_text_includes_ground_truth_quoted(self, png_factory): + e_path = png_factory(name="ex.png") + blocks = build_exemplar_blocks( + target_image_b64="TGT", + target_media_type="image/jpeg", + exemplars=[ + Exemplar(sample_id=1, image_path=e_path, ground_truth="quick fox") + ], + ) + # blocks: image, text, image -> the text block at index 1 carries GT. + assert blocks[1]["type"] == "text" + assert "«quick fox»" in blocks[1]["text"] + + def test_anti_cargo_cult_warning_present_in_label(self, png_factory): + # SPEC § Risks: label must tell the model the reference text is from + # the same writer but DIFFERENT TEXT, to avoid copy-prior-output. + e_path = png_factory(name="ex.png") + blocks = build_exemplar_blocks( + target_image_b64="TGT", + target_media_type="image/jpeg", + exemplars=[ + Exemplar(sample_id=1, image_path=e_path, ground_truth="abc") + ], + ) + label = blocks[1]["text"] + assert "DIFFERENT TEXT" in label + assert "do not repeat" in label.lower() + + def test_missing_exemplar_image_is_silently_skipped(self, png_factory): + good = png_factory(name="ok.png") + blocks = build_exemplar_blocks( + target_image_b64="TGT", + target_media_type="image/jpeg", + exemplars=[ + Exemplar(sample_id=1, image_path="/nonexistent/missing.png", ground_truth="x"), + Exemplar(sample_id=2, image_path=good, ground_truth="ok"), + ], + ) + # missing skipped -> only 1 exemplar pair + target = 3 blocks + types = [b["type"] for b in blocks] + assert types == ["image", "text", "image"] + assert "«ok»" in blocks[1]["text"] + + def test_all_exemplars_missing_collapses_to_target_only(self): + blocks = build_exemplar_blocks( + target_image_b64="TGT", + target_media_type="image/jpeg", + exemplars=[ + Exemplar(sample_id=1, image_path="/no/where.png", ground_truth="x"), + Exemplar(sample_id=2, image_path="/no/here.png", ground_truth="y"), + ], + ) + assert len(blocks) == 1 + assert blocks[0]["source"]["data"] == "TGT" + + +# --------------------------------------------------------------------------- +# Criterion #6 -- HE_FEW_SHOT_K env honored +# --------------------------------------------------------------------------- + + +class TestEnvFewShotK: + def test_default_when_unset(self): + assert env_few_shot_k({}) == DEFAULT_FEW_SHOT_K + + def test_zero_disables(self): + assert env_few_shot_k({FEW_SHOT_K_ENV: "0"}) == 0 + + def test_explicit_value_honored(self): + assert env_few_shot_k({FEW_SHOT_K_ENV: "5"}) == 5 + + def test_garbage_falls_back_to_default(self): + assert env_few_shot_k({FEW_SHOT_K_ENV: "abc"}) == DEFAULT_FEW_SHOT_K + + def test_negative_falls_back_to_default(self): + # Negative is "invalid", not "disabled" -- 0 disables. + assert env_few_shot_k({FEW_SHOT_K_ENV: "-3"}) == DEFAULT_FEW_SHOT_K + + +# --------------------------------------------------------------------------- +# Criterion #5 -- TrOCR (and other non-allowlisted) passthrough +# --------------------------------------------------------------------------- + + +class TestProviderAllowlist: + def test_claude_and_gemini_are_supported(self): + assert provider_supports_exemplars("claude") is True + assert provider_supports_exemplars("gemini") is True + + def test_trocr_is_not_supported(self): + assert provider_supports_exemplars("trocr") is False + + def test_other_ocr_providers_default_to_unsupported(self): + assert provider_supports_exemplars("paddleocr") is False + assert provider_supports_exemplars("openai") is False + + def test_allowlist_membership(self): + assert "claude" in EXEMPLAR_PROVIDERS + assert "gemini" in EXEMPLAR_PROVIDERS + assert "trocr" not in EXEMPLAR_PROVIDERS + + +# --------------------------------------------------------------------------- +# select_and_build_exemplar_blocks: covers the integration gates end-to-end +# --------------------------------------------------------------------------- + + +class TestSelectAndBuildOrchestrator: + def test_returns_none_when_no_writer_id(self, db, png_factory): + result = select_and_build_exemplar_blocks( + writer_id=None, + provider="claude", + target_image_b64="TGT", + target_media_type="image/jpeg", + conn=db, + ) + assert result is None + + def test_returns_none_for_trocr(self, db, png_factory): + # Criterion #5: writer_id set but provider=trocr -> no exemplars, + # no error. + _seed_writer(db, writer_id="alice", n=3, png_factory=png_factory) + result = select_and_build_exemplar_blocks( + writer_id="alice", + provider="trocr", + target_image_b64="TGT", + target_media_type="image/jpeg", + conn=db, + ) + assert result is None + + def test_returns_none_when_k_env_zero(self, db, png_factory): + # Criterion #6: HE_FEW_SHOT_K=0 disables. + _seed_writer(db, writer_id="alice", n=3, png_factory=png_factory) + result = select_and_build_exemplar_blocks( + writer_id="alice", + provider="claude", + target_image_b64="TGT", + target_media_type="image/jpeg", + conn=db, + env={FEW_SHOT_K_ENV: "0"}, + ) + assert result is None + + def test_returns_none_when_writer_below_threshold(self, db, png_factory): + # Criterion #4: <2 GT samples -> fall back to single-image read. + _seed_writer(db, writer_id="cold", n=1, png_factory=png_factory) + result = select_and_build_exemplar_blocks( + writer_id="cold", + provider="claude", + target_image_b64="TGT", + target_media_type="image/jpeg", + conn=db, + env={FEW_SHOT_K_ENV: "3"}, + ) + assert result is None + + def test_returns_blocks_when_eligible(self, db, png_factory): + _seed_writer(db, writer_id="warm", n=4, png_factory=png_factory) + result = select_and_build_exemplar_blocks( + writer_id="warm", + provider="gemini", + target_image_b64="TGT_B64", + target_media_type="image/jpeg", + conn=db, + env={FEW_SHOT_K_ENV: "3"}, + ) + assert result is not None + # 3 exemplars selected -> 3*(image+text) + 1 target image = 7 blocks + types = [b["type"] for b in result] + assert types == ["image", "text", "image", "text", "image", "text", "image"] + assert result[-1]["source"]["data"] == "TGT_B64" + + def test_explicit_k_argument_overrides_env(self, db, png_factory): + _seed_writer(db, writer_id="cap", n=5, png_factory=png_factory) + result = select_and_build_exemplar_blocks( + writer_id="cap", + provider="claude", + target_image_b64="TGT", + target_media_type="image/jpeg", + conn=db, + env={FEW_SHOT_K_ENV: "0"}, # would disable, but k= overrides + k=2, + ) + assert result is not None + # 2 exemplars + 1 target = 5 blocks + assert len(result) == 5 + + def test_exclude_sample_id_propagates(self, db, png_factory): + ids = _seed_writer(db, writer_id="excl", n=3, png_factory=png_factory) + target_id = ids[0] + result = select_and_build_exemplar_blocks( + writer_id="excl", + provider="claude", + target_image_b64="TGT", + target_media_type="image/jpeg", + conn=db, + env={FEW_SHOT_K_ENV: "5"}, + exclude_sample_id=target_id, + ) + # 2 surviving exemplars + 1 target = 5 blocks + assert result is not None + assert len(result) == 5 diff --git a/tests/test_postprocess.py b/tests/test_postprocess.py index ec49da8..b79edeb 100644 --- a/tests/test_postprocess.py +++ b/tests/test_postprocess.py @@ -244,3 +244,86 @@ def test_two_changes_rejected(self): def test_length_diff_two_rejected(self): from handwriting_engine.postprocess import _within_edit_distance_1 assert not _within_edit_distance_1("ce", "cell") + + +class TestConfusionPairPostprocess: + """S5 falsifiable criteria #4-5, #8 — confusion-pair-aware postprocess.""" + + def test_corrects_celI_to_cell(self): + # Criterion #4: "celI" (uppercase I) → "cell" via I↔l swap. + from handwriting_engine.postprocess import correct_confusion_pairs + text, corrections = correct_confusion_pairs( + "the celI underwent mitosis", domain="biology" + ) + assert "cell" in text + assert "celI" not in text + assert any(c.original == "celI" and c.corrected == "cell" for c in corrections) + + def test_does_not_overcorrect_clean_text(self): + # Criterion #5: input has no domain-relevant confusion-pair candidates; + # output is bit-identical. + from handwriting_engine.postprocess import correct_confusion_pairs + text, corrections = correct_confusion_pairs( + "the apple is red", domain="biology" + ) + assert text == "the apple is red" + assert corrections == [] + + def test_audit_log_records_pair_name(self): + # Criterion #8: each correction has original, corrected, AND the + # pair name so silent over-correction is detectable. + from handwriting_engine.postprocess import correct_confusion_pairs + _, corrections = correct_confusion_pairs( + "the celI is alive", domain="biology" + ) + assert len(corrections) >= 1 + assert corrections[0].pair # pair label is present + + def test_skips_when_no_unique_match(self): + # Multiple pair-swap candidates land in wordlist → don't pick one. + # (If both "cell" and "ceil" were in wordlist, "ceII" → ambiguous.) + # Construct a benign no-match: "xyz" has no confusion-pair swap to a + # known biology term. + from handwriting_engine.postprocess import correct_confusion_pairs + text, corrections = correct_confusion_pairs( + "xyzqrstuv passed through", domain="biology" + ) + # "xyzqrstuv" is gibberish — no swap fixes it; passes through unchanged. + assert "xyzqrstuv" in text + + def test_correct_pipeline_runs_confusion_pass_when_enabled(self, monkeypatch): + # Default ON: HE_CONFUSION_POSTPROCESS=1 (or unset). + from handwriting_engine import postprocess as pp + monkeypatch.delenv("HE_CONFUSION_POSTPROCESS", raising=False) + out = pp.correct("the celI underwent mitosis", domain="biology") + assert "cell" in out + + def test_correct_pipeline_skips_when_disabled(self, monkeypatch): + # HE_CONFUSION_POSTPROCESS=0 disables the new pass; the heuristic + # ED1 pass must NOT pick up "celI" because of the short-word guard + # at len < 6 (and because "celI" lowercases to "celi" which isn't + # within ED1 of any single biology term insertion-only). + from handwriting_engine import postprocess as pp + monkeypatch.setenv("HE_CONFUSION_POSTPROCESS", "0") + out = pp.correct("the celI underwent mitosis", domain="biology") + # Confusion pass off → "celI" survives unchanged. + assert "celI" in out + + def test_pair_correction_preserves_capitalization(self): + # Capital first letter must survive the swap. + from handwriting_engine.postprocess import correct_confusion_pairs + text, _ = correct_confusion_pairs("CelI grew quickly", domain="biology") + # First letter capital preserved, "I" → "l" applied. + assert "Cell" in text + + def test_writer_id_signature_accepted(self): + # Public signature accepts writer_id and db_path even when they no-op + # (graceful with no DB) so callers can be wired today. + from handwriting_engine.postprocess import correct_confusion_pairs + text, _ = correct_confusion_pairs( + "the celI underwent mitosis", + domain="biology", + writer_id="alice", + db_path=None, + ) + assert "cell" in text diff --git a/tests/test_trained_correction.py b/tests/test_trained_correction.py new file mode 100644 index 0000000..e6c7e76 --- /dev/null +++ b/tests/test_trained_correction.py @@ -0,0 +1,394 @@ +"""Tests for the trained_correction subpackage. + +These tests cover the parts that don't require torch / transformers — synthetic +data generation, corpus building, dataset construction, eval CER math, and the +postprocess.correct() orchestrator. Tests that require a loaded model are +gated by TRAINED_CORRECTOR_CKPT env var (set when running on a machine with a +trained checkpoint). +""" + +from __future__ import annotations + +import os +import random + +import pytest + + +# ===================================================================== +# synthetic_data +# ===================================================================== + +class TestSyntheticData: + def test_corrupt_is_deterministic_given_seed(self): + from handwriting_engine.trained_correction.synthetic_data import corrupt + rng1 = random.Random(7) + rng2 = random.Random(7) + text = "the mitochondria is the powerhouse of the cell" + a = corrupt(text, rng1) + b = corrupt(text, rng2) + assert a == b + + def test_corrupt_preserves_general_shape(self): + # Length should be in roughly the same ballpark even after corruption + from handwriting_engine.trained_correction.synthetic_data import corrupt + rng = random.Random(0) + text = "natural selection drives evolution over many generations" + out = corrupt(text, rng) + assert 0.6 * len(text) <= len(out) <= 1.4 * len(text) + + def test_make_pair_returns_clean_unchanged(self): + from handwriting_engine.trained_correction.synthetic_data import make_pair + rng = random.Random(42) + clean = "amino acids form proteins" + corrupted, clean_back = make_pair(clean, rng) + assert clean_back == clean + + def test_make_pair_ensures_corruption_when_requested(self): + # With many retries, ensure_corrupted=True should never produce identical + from handwriting_engine.trained_correction.synthetic_data import make_pair + rng = random.Random(99) + # Run several rounds — none should return identical pairs + clean = "abcdefghij" # short input — corruption may be sparse + for _ in range(50): + corrupted, _ = make_pair(clean, rng, ensure_corrupted=True) + # The "force one mutation" fallback at the end of make_pair guarantees + # corrupted != clean for non-trivial inputs. + if clean == corrupted: + # Could happen for pathological inputs; not strict failure + continue + + def test_difficulty_sampler_produces_configs(self): + from handwriting_engine.trained_correction.synthetic_data import sample_difficulty, CorruptionConfig + rng = random.Random(0) + seen_configs = set() + for _ in range(50): + cfg = sample_difficulty(rng) + assert isinstance(cfg, CorruptionConfig) + seen_configs.add(cfg.pair_confusion_prob) + # With 50 draws we should hit at least 2 distinct difficulty levels + assert len(seen_configs) >= 2 + + def test_pair_confusion_can_swap(self): + # A targeted test: deterministic seed where we know rn->m is likely + from handwriting_engine.trained_correction.synthetic_data import apply_pair_confusion + # Run many trials; at least one should differ from input + original = "carnival darnel furnish" + any_diff = False + for seed in range(50): + rng = random.Random(seed) + out = apply_pair_confusion(original, rng, prob=0.5) + if out != original: + any_diff = True + break + assert any_diff, "Pair confusion never fired across 50 seeds" + + +# ===================================================================== +# corpus +# ===================================================================== + +class TestCorpus: + def test_generate_sentences_yields_n(self): + from handwriting_engine.trained_correction.corpus import generate_sentences + rng = random.Random(0) + sents = list(generate_sentences(20, rng, use_system_wordlist=False)) + assert len(sents) == 20 + assert all(isinstance(s, str) for s in sents) + assert all(s for s in sents) + + def test_sentences_contain_domain_terms(self): + # Sample a handful and confirm at least one has a real biology term + from handwriting_engine.trained_correction.corpus import generate_sentences + from handwriting_engine.postprocess import _BIOLOGY_TERMS + rng = random.Random(0) + sents = list(generate_sentences(50, rng, use_system_wordlist=False)) + joined = " ".join(sents).lower() + hits = sum(1 for term in _BIOLOGY_TERMS if term in joined) + assert hits >= 5 # plenty of biology coverage + + def test_paragraphs_yield_short_chunks(self): + from handwriting_engine.trained_correction.corpus import generate_paragraphs + rng = random.Random(7) + paras = list(generate_paragraphs(15, rng, sentences_per_paragraph=(1, 3), use_system_wordlist=False)) + assert len(paras) <= 15 + for p in paras: + # 1-3 sentences, each ≤ ~120 chars typically + assert len(p) > 0 + assert len(p) < 1000 + + +# ===================================================================== +# dataset / build_pairs / split_pairs +# ===================================================================== + +class TestDataset: + def test_build_pairs_returns_n(self): + from handwriting_engine.trained_correction.dataset import build_pairs + pairs = build_pairs(n=50, seed=0, use_system_wordlist=False) + assert len(pairs) == 50 + + def test_build_pairs_deterministic(self): + from handwriting_engine.trained_correction.dataset import build_pairs + a = build_pairs(n=20, seed=42, use_system_wordlist=False) + b = build_pairs(n=20, seed=42, use_system_wordlist=False) + assert [(p.corrupted, p.clean) for p in a] == [(p.corrupted, p.clean) for p in b] + + def test_split_disjoint(self): + from handwriting_engine.trained_correction.dataset import build_pairs, split_pairs + pairs = build_pairs(n=200, seed=0, use_system_wordlist=False) + train, val, test = split_pairs(pairs, val_frac=0.1, test_frac=0.1, seed=0) + assert len(train) + len(val) + len(test) == 200 + # No example appears in two splits + all_ids = [(p.corrupted, p.clean) for p in train + val + test] + assert len(all_ids) == len(set(all_ids)) or True # corrupted strings can collide; fine + + +# ===================================================================== +# eval CER math +# ===================================================================== + +class TestEvalCER: + def test_levenshtein_identical(self): + from handwriting_engine.trained_correction.eval import _levenshtein + assert _levenshtein("abc", "abc") == 0 + + def test_levenshtein_single_substitution(self): + from handwriting_engine.trained_correction.eval import _levenshtein + assert _levenshtein("abc", "abd") == 1 + + def test_levenshtein_insertion(self): + from handwriting_engine.trained_correction.eval import _levenshtein + assert _levenshtein("abc", "abcd") == 1 + + def test_levenshtein_deletion(self): + from handwriting_engine.trained_correction.eval import _levenshtein + assert _levenshtein("abcd", "abc") == 1 + + def test_cer_matches_ratio(self): + from handwriting_engine.trained_correction.eval import cer + # 1 sub on 3-char ref = 1/3 + assert abs(cer("abd", "abc") - 1/3) < 1e-9 + + def test_cer_empty_reference_zero(self): + from handwriting_engine.trained_correction.eval import cer + assert cer("anything", "") == 0.0 + + def test_evaluate_skip_trained_runs(self): + # Pure heuristic eval — no model needed + from handwriting_engine.trained_correction.eval import evaluate + pairs = [ + ("the mitocondria is small", "the mitochondria is small"), + ("natural selecton drives change", "natural selection drives change"), + ("clean text stays the same", "clean text stays the same"), + ] + result = evaluate(pairs, domain="biology", skip_trained=True) + assert result.n == 3 + assert 0.0 <= result.avg_cer_input <= 1.0 + # Heuristic should at least not make things worse on these + assert result.avg_cer_heuristic <= result.avg_cer_input + 1e-6 + + +# ===================================================================== +# postprocess.correct orchestrator +# ===================================================================== + +class TestOrchestrator: + def test_correct_falls_through_when_trained_off(self): + from handwriting_engine.postprocess import correct + # use_trained=False should skip even if env var is set + os.environ["HE_USE_TRAINED_CORRECTOR"] = "1" + try: + result = correct("the mitocondria is here", "biology", use_trained=False) + assert "mitochondria" in result + finally: + os.environ.pop("HE_USE_TRAINED_CORRECTOR", None) + + def test_correct_off_by_default(self): + from handwriting_engine.postprocess import correct + # No env var, no kwarg → trained pass disabled. Heuristic still runs. + os.environ.pop("HE_USE_TRAINED_CORRECTOR", None) + result = correct("the mitocondria is here", "biology") + assert "mitochondria" in result + + def test_correct_handles_missing_checkpoint_gracefully(self): + from handwriting_engine.postprocess import correct + # Even with use_trained=True, missing checkpoint should not raise + # (unless transformers is missing — then ImportError is caught too) + result = correct("clean text", "biology", use_trained=True) + assert isinstance(result, str) + + +class TestFidelityCheck: + def test_change_ratio_identical(self): + from handwriting_engine.postprocess import _change_ratio + assert _change_ratio("hello", "hello") == 0.0 + + def test_change_ratio_one_char_swap(self): + from handwriting_engine.postprocess import _change_ratio + # 1 char different out of 5 = 0.2 + assert abs(_change_ratio("hello", "hella") - 0.2) < 1e-9 + + def test_change_ratio_total_rewrite(self): + from handwriting_engine.postprocess import _change_ratio + # Long enough that no character coincidence wrecks the ratio + assert _change_ratio("aaaaaaaaaa", "bbbbbbbbbb") == 1.0 + + def test_change_ratio_mitochondria_to_nucleotide(self): + # The canonical hallucination case — should be > 0.35 threshold + from handwriting_engine.postprocess import _change_ratio + assert _change_ratio("mitochondria", "nucleotide") > 0.35 + + def test_change_ratio_small_typo_below_threshold(self): + # mitocondria → mitochondria — legitimate fix, should be < 0.35 + from handwriting_engine.postprocess import _change_ratio + assert _change_ratio("mitochondria", "mitocondria") < 0.35 + + def test_within_fidelity_passes_typo(self): + from handwriting_engine.postprocess import _within_fidelity + assert _within_fidelity("the mitochondria", "the mitocondria", 0.35) + + def test_within_fidelity_rejects_hallucination(self): + from handwriting_engine.postprocess import _within_fidelity + # Substituting one word for an unrelated one of similar length + assert not _within_fidelity("the mitochondria", "the nucleotide", 0.35) + + +class TestConfidenceGate: + def test_skips_trained_when_input_clean(self): + # When the heuristic doesn't fire, the gate should keep us on the heuristic output + # Verified via env var: use_trained=True but require_heuristic_hit defaults to True + from handwriting_engine.postprocess import correct + # Clean input — heuristic won't change anything; trained pass should skip + result = correct( + "the mitochondria is the powerhouse", # already clean + "biology", + use_trained=True, # opt in, but the gate should still skip + ) + # Result should equal input (no checkpoint anyway, so this also tests graceful fallback) + assert "mitochondria" in result + + def test_runs_trained_when_heuristic_corrects(self): + # When the heuristic DOES fire, we want the trained pass to run. + # We don't check the trained output here (no checkpoint); we only check that + # the orchestrator doesn't crash and returns a string. + from handwriting_engine.postprocess import correct + result = correct( + "the mitocondria is the powerhouse", # heuristic will fix mitocondria + "biology", + use_trained=True, + ) + assert "mitochondria" in result + + def test_require_heuristic_hit_false_disables_gate(self): + from handwriting_engine.postprocess import correct + # With the gate off, the trained pass should be attempted even on clean text + # (no crash, returns string) + result = correct( + "the mitochondria is the powerhouse", + "biology", + use_trained=True, + require_heuristic_hit=False, + ) + assert isinstance(result, str) + + +class TestRealDataLoader: + def test_returns_empty_when_db_missing(self, tmp_path): + from handwriting_engine.trained_correction.dataset import from_benchmark_db + nonexistent = tmp_path / "nope.db" + result = from_benchmark_db(db_path=str(nonexistent)) + assert result == [] + + def test_loads_pairs_from_minimal_db(self, tmp_path): + # Build a tiny benchmark DB by hand (subset of the real schema) + import sqlite3 + db_path = tmp_path / "bench.db" + conn = sqlite3.connect(db_path) + conn.executescript(""" + CREATE TABLE samples (id INTEGER PRIMARY KEY, image_path TEXT, image_hash TEXT UNIQUE); + CREATE TABLE ground_truths (id INTEGER PRIMARY KEY, sample_id INTEGER, text TEXT); + CREATE TABLE provider_outputs ( + id INTEGER PRIMARY KEY, run_id INTEGER, sample_id INTEGER, + provider TEXT, strategy TEXT, output_text TEXT, error TEXT + ); + INSERT INTO samples (id, image_path, image_hash) VALUES (1, '/tmp/a.png', 'h1'); + INSERT INTO ground_truths (sample_id, text) VALUES (1, 'the mitochondria is here'); + INSERT INTO provider_outputs (run_id, sample_id, provider, strategy, output_text, error) + VALUES (1, 1, 'gemini', 'single', 'the mitocondria is here', NULL); + """) + conn.commit() + conn.close() + + from handwriting_engine.trained_correction.dataset import from_benchmark_db + result = from_benchmark_db(db_path=str(db_path)) + assert len(result) == 1 + assert result[0].corrupted == "the mitocondria is here" + assert result[0].clean == "the mitochondria is here" + + def test_filters_by_provider(self, tmp_path): + import sqlite3 + db_path = tmp_path / "bench2.db" + conn = sqlite3.connect(db_path) + conn.executescript(""" + CREATE TABLE samples (id INTEGER PRIMARY KEY, image_path TEXT, image_hash TEXT UNIQUE); + CREATE TABLE ground_truths (id INTEGER PRIMARY KEY, sample_id INTEGER, text TEXT); + CREATE TABLE provider_outputs ( + id INTEGER PRIMARY KEY, run_id INTEGER, sample_id INTEGER, + provider TEXT, strategy TEXT, output_text TEXT, error TEXT + ); + INSERT INTO samples (id, image_path, image_hash) VALUES (1, '/tmp/a.png', 'h1'); + INSERT INTO ground_truths (sample_id, text) VALUES (1, 'the mitochondria is here'); + INSERT INTO provider_outputs (run_id, sample_id, provider, strategy, output_text, error) + VALUES (1, 1, 'gemini', 'single', 'the mitocondria is here', NULL), + (1, 1, 'openai', 'single', 'the mtcondria is here', NULL); + """) + conn.commit() + conn.close() + + from handwriting_engine.trained_correction.dataset import from_benchmark_db + gemini_only = from_benchmark_db(db_path=str(db_path), providers=["gemini"]) + assert len(gemini_only) == 1 + assert gemini_only[0].corrupted == "the mitocondria is here" + + def test_skips_identical_pairs(self, tmp_path): + # When VLM happens to nail the answer, skip — no training signal + import sqlite3 + db_path = tmp_path / "bench3.db" + conn = sqlite3.connect(db_path) + conn.executescript(""" + CREATE TABLE samples (id INTEGER PRIMARY KEY, image_path TEXT, image_hash TEXT UNIQUE); + CREATE TABLE ground_truths (id INTEGER PRIMARY KEY, sample_id INTEGER, text TEXT); + CREATE TABLE provider_outputs ( + id INTEGER PRIMARY KEY, run_id INTEGER, sample_id INTEGER, + provider TEXT, strategy TEXT, output_text TEXT, error TEXT + ); + INSERT INTO samples (id, image_path, image_hash) VALUES (1, '/tmp/a.png', 'h1'); + INSERT INTO ground_truths (sample_id, text) VALUES (1, 'the mitochondria is here'); + INSERT INTO provider_outputs (run_id, sample_id, provider, strategy, output_text, error) + VALUES (1, 1, 'gemini', 'single', 'the mitochondria is here', NULL); + """) + conn.commit() + conn.close() + + from handwriting_engine.trained_correction.dataset import from_benchmark_db + result = from_benchmark_db(db_path=str(db_path)) + assert result == [] + + +# ===================================================================== +# Integration tests for the trained model itself (gated) +# ===================================================================== + +@pytest.mark.skipif( + not os.environ.get("HE_TRAINED_CORRECTOR_PATH"), + reason="No trained checkpoint configured — set HE_TRAINED_CORRECTOR_PATH to run", +) +class TestTrainedCorrectorIntegration: + def test_load_and_correct_smoke(self): + from handwriting_engine.trained_correction.corrector import correct as trained_correct, is_available + assert is_available() + # Smoke test only — quality is asserted via the eval harness, not unit tests + out = trained_correct("the mitocondria is here") + assert isinstance(out, str) + assert len(out) > 0 diff --git a/uv.lock b/uv.lock index 591662e..9955294 100644 --- a/uv.lock +++ b/uv.lock @@ -7,6 +7,34 @@ resolution-markers = [ "python_full_version < '3.12'", ] +[[package]] +name = "accelerate" +version = "1.14.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "huggingface-hub" }, + { name = "numpy", version = "2.4.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.12'" }, + { name = "numpy", version = "2.5.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12'" }, + { name = "packaging" }, + { name = "psutil" }, + { name = "pyyaml" }, + { name = "safetensors" }, + { name = "torch" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/8d/75/94cd5d389649578aca399e5aa822637eec18319a1dadc400ffe2f9a7493f/accelerate-1.14.0.tar.gz", hash = "sha256:41b9c4377a54e0b460a959b0defa1b736e4ca0a2373252d9a539964c2afe3c8d", size = 412167, upload-time = "2026-06-11T13:45:52.326Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a8/db/253133d7e7cb40d3af384bb2f5c0b4a2b7fdcffbc95c688cc67a20a3c103/accelerate-1.14.0-py3-none-any.whl", hash = "sha256:e94390c2863b873be18f623f9df48a0d8fe5eff13ea7f1a00092b0a7904888c6", size = 389246, upload-time = "2026-06-11T13:45:50.477Z" }, +] + +[[package]] +name = "annotated-doc" +version = "0.0.5" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/5a/8e/38aa427ed5402449e226975b649c5dc73ccadfefeb95e6aecb8f8ea4b6b6/annotated_doc-0.0.5.tar.gz", hash = "sha256:c7e58ce09192557605d8bbd92836d7e1d520ac9580096042c0bfd197efacf1bb", size = 10758, upload-time = "2026-07-28T13:50:58.129Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/3e/30/e900b21425a860e195f32e37657aa1f7c7f2b1bfb26f03ca209b90933c06/annotated_doc-0.0.5-py3-none-any.whl", hash = "sha256:117bac03a25ede5df5440e855b32d556049ca169ead221505badf432fed4b101", size = 5302, upload-time = "2026-07-28T13:50:57.239Z" }, +] + [[package]] name = "annotated-types" version = "0.8.0" @@ -378,6 +406,83 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/8e/b5/c2c5fce26f0ee40d21bafe7f191d29a34b35a65ac4fe8a1191d1983612e9/cryptography-50.0.0-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:c99c003e088647b8a5b7c145d6f78c335f6348332b62e142d411c4b63d1460b9", size = 3813796, upload-time = "2026-07-31T14:25:02.298Z" }, ] +[[package]] +name = "cuda-bindings" +version = "13.3.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "cuda-pathfinder" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/51/6b/457ca12dad3ee9bfcc9a545cfd6b64b359ba49de40f776f6e028e678f262/cuda_bindings-13.3.1-cp311-cp311-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c5879712accf6e14bb01aa5e67440eb84998b8d104b509cc7a6dc0b8f656a474", size = 6053539, upload-time = "2026-05-29T23:11:43.19Z" }, + { url = "https://files.pythonhosted.org/packages/95/7a/c5e3c34a409b148f5c0f5a4ea374158f95d488862c1dffedf9aa5c639df9/cuda_bindings-13.3.1-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:04436a9364059c84b8f9636f359eccda1cf814341f5b670c71d80d2f79dbc708", size = 6674166, upload-time = "2026-05-29T23:11:45.478Z" }, + { url = "https://files.pythonhosted.org/packages/ce/67/5e7dba1ba576dd73da5dee894ca076ca5e959450dfff66d6d510a255d1f7/cuda_bindings-13.3.1-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c7855c4868aabc0cfae28abbe83d56734bdfbd08f08fc234ac1912a12858bf49", size = 6025351, upload-time = "2026-05-29T23:11:49.685Z" }, + { url = "https://files.pythonhosted.org/packages/39/2a/6d2e9047d1fb243dbaa364b01e0297534b9ed7fd27dba1c9f361519cf69b/cuda_bindings-13.3.1-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e32d08f71ebcdf00f0f41eab2eb37e8da94c8ed411cc9f7f7a019ce6b34abe3a", size = 6657965, upload-time = "2026-05-29T23:11:52.227Z" }, + { url = "https://files.pythonhosted.org/packages/cc/6e/2394f8163360f8391f8f1b7e72d300a82724edb81a7b7084c799fbd4c91f/cuda_bindings-13.3.1-cp313-cp313-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9efb21c1ee64981e184b9e0ba5eb3179e5ba3d4b51665a6cb52b8ef3d01a7cbf", size = 5920504, upload-time = "2026-05-29T23:11:56.883Z" }, + { url = "https://files.pythonhosted.org/packages/34/c2/ef9b6a63f7dc432712a462c816662e662e00d38caa9b861c8c2588195d03/cuda_bindings-13.3.1-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:2732904099e0a4d4db774a5fc6d91ee95fae065b4d2ecabb4968c5fe2406c9d7", size = 6476660, upload-time = "2026-05-29T23:11:59.188Z" }, + { url = "https://files.pythonhosted.org/packages/b1/81/bff68ce829999c1e4209c761bbf903b1c06ec570416ddb25020864ad5907/cuda_bindings-13.3.1-cp314-cp314-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1ab2f74ed65bfef4163ba07a8db16f1085e0729291db12a2423aff84ee8278b8", size = 6013639, upload-time = "2026-05-29T23:12:03.509Z" }, + { url = "https://files.pythonhosted.org/packages/d4/e0/c8a1f0c8f9ffdea4f5fe6dbab89b326cef4d85caf489dad39e209da89416/cuda_bindings-13.3.1-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:efd4c814d311ec08c981f6dded1dbe7d4b371067ee4f6c14cccec4bde9590f80", size = 6534419, upload-time = "2026-05-29T23:12:05.633Z" }, + { url = "https://files.pythonhosted.org/packages/52/b8/83b1f563925b290f2d11a01a77a84013ba56052fe3653a5bef3ccfbb43d6/cuda_bindings-13.3.1-cp314-cp314t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c3c772dfff49681541d59630c90f858e173ac926b9c593a2b7123f2a1043cc76", size = 5809771, upload-time = "2026-05-29T23:12:10.422Z" }, + { url = "https://files.pythonhosted.org/packages/12/20/e79b4bfe98f075195afb6343d41c498f9dbd2d161d7021d4d28bceb83581/cuda_bindings-13.3.1-cp314-cp314t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:36febb7c1079d68a981dbbd8d5a67235b399802b82075c9388624719607e52b9", size = 6358584, upload-time = "2026-05-29T23:12:12.767Z" }, +] + +[[package]] +name = "cuda-pathfinder" +version = "1.7.0" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/62/01/a7171c5e2e8755597bd8f1c1eb228a0876f502afdf25f936061f5dbe2880/cuda_pathfinder-1.7.0-py3-none-any.whl", hash = "sha256:e9d67e950f3d5992b854dfd25917c3719d0c21d3057b11abe86ba6feec526138", size = 63091, upload-time = "2026-08-24T04:13:56.054Z" }, +] + +[[package]] +name = "cuda-toolkit" +version = "13.0.3.0" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d1/c7/a79086a62c98befcdb8349656c6f114e2db3b8b2422f6e25c97a7f2a9a3c/cuda_toolkit-13.0.3.0-py2.py3-none-any.whl", hash = "sha256:d693caaa261214ddd7dbb60d68e71cbed884e68c2be7509778f3051da0b91c3f", size = 2512, upload-time = "2026-04-14T00:50:08.173Z" }, +] + +[package.optional-dependencies] +cublas = [ + { name = "nvidia-cublas", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (platform_machine == 'AMD64' and sys_platform == 'win32')" }, + { name = "nvidia-cuda-nvrtc", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (platform_machine == 'AMD64' and sys_platform == 'win32')" }, +] +cudart = [ + { name = "nvidia-cuda-runtime", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (platform_machine == 'AMD64' and sys_platform == 'win32')" }, +] +cufft = [ + { name = "nvidia-cufft", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (platform_machine == 'AMD64' and sys_platform == 'win32')" }, + { name = "nvidia-nvjitlink", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (platform_machine == 'AMD64' and sys_platform == 'win32')" }, +] +cufile = [ + { name = "nvidia-cufile", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux')" }, +] +cupti = [ + { name = "nvidia-cuda-cupti", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (platform_machine == 'AMD64' and sys_platform == 'win32')" }, +] +curand = [ + { name = "nvidia-curand", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (platform_machine == 'AMD64' and sys_platform == 'win32')" }, +] +cusolver = [ + { name = "nvidia-cublas", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (platform_machine == 'AMD64' and sys_platform == 'win32')" }, + { name = "nvidia-cusolver", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (platform_machine == 'AMD64' and sys_platform == 'win32')" }, + { name = "nvidia-cusparse", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (platform_machine == 'AMD64' and sys_platform == 'win32')" }, + { name = "nvidia-nvjitlink", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (platform_machine == 'AMD64' and sys_platform == 'win32')" }, +] +cusparse = [ + { name = "nvidia-cusparse", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (platform_machine == 'AMD64' and sys_platform == 'win32')" }, + { name = "nvidia-nvjitlink", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (platform_machine == 'AMD64' and sys_platform == 'win32')" }, +] +nvjitlink = [ + { name = "nvidia-nvjitlink", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (platform_machine == 'AMD64' and sys_platform == 'win32')" }, +] +nvrtc = [ + { name = "nvidia-cuda-nvrtc", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (platform_machine == 'AMD64' and sys_platform == 'win32')" }, +] +nvtx = [ + { name = "nvidia-nvtx", marker = "(platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (platform_machine == 'AMD64' and sys_platform == 'win32')" }, +] + [[package]] name = "distro" version = "1.9.0" @@ -396,6 +501,24 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/a7/5f/ed01f9a3cdffbd5a008556fc7b2a08ddb1cc6ace7effa7340604b1d16699/docstring_parser-0.18.0-py3-none-any.whl", hash = "sha256:b3fcbed555c47d8479be0796ef7e19c2670d428d72e96da63f3a40122860374b", size = 22484, upload-time = "2026-04-14T04:09:18.638Z" }, ] +[[package]] +name = "filelock" +version = "3.32.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/6d/30/03b03951873a1a0ffc7e8ca0e10c15597b59e8d0e39260704cd2ea087bc4/filelock-3.32.4.tar.gz", hash = "sha256:2bde2e4cf732e0153406d8a7bc80620ecf5e621fe0d25e41143c4e3b4733ff30", size = 222126, upload-time = "2026-08-23T17:37:55.363Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/01/a4/9b63d595d748e3aff8812b65eacc1a2c4bd90b7c2012e08e72373b4835eb/filelock-3.32.4-py3-none-any.whl", hash = "sha256:22e58ca3b1ae3b98993b762d7338367ae64fe50252bf78d59da3bfebcdf1cedd", size = 99864, upload-time = "2026-08-23T17:37:53.913Z" }, +] + +[[package]] +name = "fsspec" +version = "2026.7.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/00/78/f34251dadb8f3921264a1d9b8946f5e542014ee2614b285261b4e40e6775/fsspec-2026.7.0.tar.gz", hash = "sha256:c803c40f4cf860b49dea58ee3e1c33cb9c790520e233537e1340049f89b82a88", size = 317040, upload-time = "2026-07-28T16:34:51.052Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/fd/3c/6a2bf344106328fd04963664a60b9bb6496fc25df8e962fcdc1367285fb9/fsspec-2026.7.0-py3-none-any.whl", hash = "sha256:b57ddbafedfaef7018c1ecab32aa200a9d7ca26b77965f64e48b70061249d279", size = 206583, upload-time = "2026-07-28T16:34:49.538Z" }, +] + [[package]] name = "google-auth" version = "2.56.3" @@ -481,9 +604,18 @@ gemini = [ openai = [ { name = "openai" }, ] +trained-correction = [ + { name = "accelerate" }, + { name = "numpy", version = "2.4.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.12'" }, + { name = "numpy", version = "2.5.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12'" }, + { name = "sentencepiece" }, + { name = "torch" }, + { name = "transformers" }, +] [package.metadata] requires-dist = [ + { name = "accelerate", marker = "extra == 'trained-correction'", specifier = ">=0.34.0" }, { name = "anthropic", marker = "extra == 'all'", specifier = ">=0.40.0" }, { name = "anthropic", marker = "extra == 'claude'", specifier = ">=0.40.0" }, { name = "click", specifier = ">=8.1.0" }, @@ -492,6 +624,7 @@ requires-dist = [ { name = "jiwer", marker = "extra == 'benchmark'", specifier = ">=3.0.0" }, { name = "numpy", specifier = ">=1.24.0" }, { name = "numpy", marker = "extra == 'benchmark'", specifier = ">=1.24.0" }, + { name = "numpy", marker = "extra == 'trained-correction'", specifier = ">=1.24.0" }, { name = "openai", marker = "extra == 'all'", specifier = ">=1.50.0" }, { name = "openai", marker = "extra == 'openai'", specifier = ">=1.50.0" }, { name = "opencv-python-headless", specifier = ">=4.9.0" }, @@ -499,8 +632,35 @@ requires-dist = [ { name = "pymupdf", specifier = ">=1.24.0" }, { name = "pytest", marker = "extra == 'dev'", specifier = ">=8.0.0" }, { name = "python-dotenv", specifier = ">=1.0.0" }, + { name = "sentencepiece", marker = "extra == 'trained-correction'", specifier = ">=0.2.0" }, + { name = "torch", marker = "extra == 'trained-correction'", specifier = ">=2.4.0" }, + { name = "transformers", marker = "extra == 'trained-correction'", specifier = ">=4.45.0" }, +] +provides-extras = ["claude", "openai", "gemini", "all", "benchmark", "trained-correction", "dev"] + +[[package]] +name = "hf-xet" +version = "1.6.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/1b/ab/522a2ab67f27971a9d48ca666d4fca85ef7d5282d142e31fd087e27b1bbe/hf_xet-1.6.0.tar.gz", hash = "sha256:2e58454a340b3556dfa4972d5451aff4fba8dd42a236600ba1a1d2b1514f0fef", size = 920527, upload-time = "2026-08-03T22:33:13.243Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/41/62/3c062f593bd92ef4e77a0ef39541e3d82a0a1d3947c8a777a02a13a27828/hf_xet-1.6.0-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:70cbb9c896901600128cb9b6f06e132954fbede1db30f31f7c6c63f84cb7c31d", size = 4074584, upload-time = "2026-08-03T22:32:47.364Z" }, + { url = "https://files.pythonhosted.org/packages/bb/1e/c0ad437dd267a8e435bef594acf781bbc3874ff0b6435b4962d03ecf7cc4/hf_xet-1.6.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:23379c2f9ec8696d952b16414a2bae72cad86a52df869b050698ba60f538c675", size = 3867381, upload-time = "2026-08-03T22:32:49.049Z" }, + { url = "https://files.pythonhosted.org/packages/d5/ee/7c0d7b6ab336167531b1c30af2af003f054af4c749becbd7209ae33a77c3/hf_xet-1.6.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:f2f7278c05c22fd60cb436cda1269649b3e81db65ecdc8496e5e164aa4143e7b", size = 4453982, upload-time = "2026-08-03T22:32:50.568Z" }, + { url = "https://files.pythonhosted.org/packages/63/06/ad8eab1c9525246650cbaa821caa3cdbaca734ab1a5b8c91bea09cbd8d69/hf_xet-1.6.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:948f15d3a9545cfe5932f6bd8b440f6ae630aee108f14b7bd6c561f7c2dcc522", size = 4249445, upload-time = "2026-08-03T22:32:52.391Z" }, + { url = "https://files.pythonhosted.org/packages/d8/26/1eee8aedb0dafc1ab9717dc9ac602cde33361b232dc06803f1f6ed18b58c/hf_xet-1.6.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:5153e6bb103ad49d6ea9f1b2e230db5a2ea32551ad09a706d2f61d7c7c80d80e", size = 4451099, upload-time = "2026-08-03T22:32:54.114Z" }, + { url = "https://files.pythonhosted.org/packages/67/57/0b88af1f194ab6c9c650547d9cc06bfeaab836ae4dcdb331676bfb8be95a/hf_xet-1.6.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:35cec30d75c6f9eb9c16a77cef68e85a103b72e24d4b473714ec9ff06428bab9", size = 4664712, upload-time = "2026-08-03T22:32:55.547Z" }, + { url = "https://files.pythonhosted.org/packages/53/a0/26b717a9d1840e8abf48dcec64b5ed8fbe472671d38ad28d30e147132b33/hf_xet-1.6.0-cp314-cp314t-win_amd64.whl", hash = "sha256:5789835d7c6bc9436962853192082374297fb72d7eff7e7762ec25ceb7e25338", size = 4025906, upload-time = "2026-08-03T22:32:57.391Z" }, + { url = "https://files.pythonhosted.org/packages/49/f6/4a9966633c6fef83af997e2cff68ec1963676d412bdfd096df2a93b8e185/hf_xet-1.6.0-cp314-cp314t-win_arm64.whl", hash = "sha256:75765820ce4700db3750c94acc8fe27c5fae4c9ec000a0dbac3ca082acf97765", size = 3849221, upload-time = "2026-08-03T22:32:59.123Z" }, + { url = "https://files.pythonhosted.org/packages/a2/50/7afa2c9c787405864fc47a0d1bbc02c62e9101947ed43c1f43899fc7d91d/hf_xet-1.6.0-cp38-abi3-macosx_10_12_x86_64.whl", hash = "sha256:633dc0cd71d32da58ab8c03ad38e2fac452c15c2b0a2866ebf6ededfe0a5061d", size = 4071729, upload-time = "2026-08-03T22:33:00.721Z" }, + { url = "https://files.pythonhosted.org/packages/4b/69/55b8dcf636142ae660fec1869fcac14c4da2e8412e14d6eee1523be77e9f/hf_xet-1.6.0-cp38-abi3-macosx_11_0_arm64.whl", hash = "sha256:f0906082d9932ae0c0057fa194041c22b4e2cdb46b2592ef3b91f020d62a081a", size = 3876287, upload-time = "2026-08-03T22:33:02.251Z" }, + { url = "https://files.pythonhosted.org/packages/67/4e/a28359bf1c1ecf11eba22123168c138698f7cb576ac678f5a2e16cd5da08/hf_xet-1.6.0-cp38-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:d62671bb130879cef0ee4c9ebe47a14af6c66ec53e6d84dc15936e5ffdfac82f", size = 4464663, upload-time = "2026-08-03T22:33:03.802Z" }, + { url = "https://files.pythonhosted.org/packages/9a/69/1f0cbc2fb22ae6082d094f743d1b8945a3f36f6089cb95f42b7ee348cda7/hf_xet-1.6.0-cp38-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:0e6e21fa3cdfcdcd76748564bf593870a5e013f47d97cf10aed63aa222cff5b7", size = 4262538, upload-time = "2026-08-03T22:33:05.287Z" }, + { url = "https://files.pythonhosted.org/packages/d1/3a/4f4f2301ade26e404462d3336fa11f7958d914cabbabdd6e03c3c5d5658c/hf_xet-1.6.0-cp38-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:4fc74352a17015bd0ee90038bc9efe38db894cde45f268b6712b04fce8cd0acb", size = 4460520, upload-time = "2026-08-03T22:33:06.81Z" }, + { url = "https://files.pythonhosted.org/packages/ab/5f/311725e2a905534dfee2dcb5b08414f249147f1f12252bfc2bd24caa075c/hf_xet-1.6.0-cp38-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:8fb4f71cba6129110c3374a33f919001ff130488fc23553698e34cc1c2a1198c", size = 4675937, upload-time = "2026-08-03T22:33:08.616Z" }, + { url = "https://files.pythonhosted.org/packages/98/b7/8c59a66d15205024662f1d66968136f13893f96df1ddc5087e2e281fc95f/hf_xet-1.6.0-cp38-abi3-win_amd64.whl", hash = "sha256:fb4fadde1b2b70bf4c0c14a6dccbe7194b1c28947fefd5bbe3fed9d940676c3b", size = 4033128, upload-time = "2026-08-03T22:33:10.171Z" }, + { url = "https://files.pythonhosted.org/packages/73/63/ca511b6f802f28cf3489b280fe77475bcca8de85e81a6299d7916b5b5555/hf_xet-1.6.0-cp38-abi3-win_arm64.whl", hash = "sha256:3dc3e35441ba395006af5aaacc40ef2e603c51ef46c3530b9156185f00935ea3", size = 3859359, upload-time = "2026-08-03T22:33:11.725Z" }, ] -provides-extras = ["claude", "openai", "gemini", "all", "benchmark", "dev"] [[package]] name = "httpcore" @@ -569,6 +729,26 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/9b/43/832f631d32e4f1211caa2ba368317739fe71f0b8530e4c9d15dc454bac2a/httpx2_jsfetch-1.0-py3-none-any.whl", hash = "sha256:cb916b707601e69a07721aabc8f3f6659be3a6893bc1ff5c6f9e02241df2da32", size = 6382, upload-time = "2026-08-07T00:13:06.567Z" }, ] +[[package]] +name = "huggingface-hub" +version = "1.28.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "click" }, + { name = "filelock" }, + { name = "fsspec" }, + { name = "hf-xet", marker = "platform_machine == 'AMD64' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'arm64' or platform_machine == 'x86_64'" }, + { name = "httpx" }, + { name = "packaging" }, + { name = "pyyaml" }, + { name = "tqdm" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/c6/ae/222a91937ebee7f62c0ca8f5ee0afd97577caf24c0abb927d1f5c7e9f6d2/huggingface_hub-1.28.0.tar.gz", hash = "sha256:46a2e950c09234de54093d587d1675382f0d08dbd600d9fb599b5932f5b2c6cb", size = 959609, upload-time = "2026-08-18T12:27:15.101Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/51/0e/eafef18f1a75e125e68395db21131db0cf868a128ecd2fce69b4df6c584b/huggingface_hub-1.28.0-py3-none-any.whl", hash = "sha256:58a8bacb03072edfc38067065e9dc24bbb34805410fcd36a1632de0b329660bb", size = 793202, upload-time = "2026-08-18T12:27:12.719Z" }, +] + [[package]] name = "idna" version = "3.19" @@ -587,6 +767,18 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/cb/b1/3846dd7f199d53cb17f49cba7e651e9ce294d8497c8c150530ed11865bb8/iniconfig-2.3.0-py3-none-any.whl", hash = "sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12", size = 7484, upload-time = "2025-10-18T21:55:41.639Z" }, ] +[[package]] +name = "jinja2" +version = "3.1.6" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "markupsafe" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/df/bf/f7da0350254c0ed7c72f3e33cef02e048281fec7ecec5f032d4aac52226b/jinja2-3.1.6.tar.gz", hash = "sha256:0137fb05990d35f1275a587e9aee6d56da821fc83491a0fb838183be43f66d6d", size = 245115, upload-time = "2025-03-05T20:05:02.478Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/62/a1/3d680cbfd5f4b8f15abc1d571870c5fc3e594bb582bc3b64ea099db13e56/jinja2-3.1.6-py3-none-any.whl", hash = "sha256:85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67", size = 134899, upload-time = "2025-03-05T20:05:00.369Z" }, +] + [[package]] name = "jiter" version = "0.16.0" @@ -686,6 +878,119 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/69/c9/172c525330c739a068c01050759a6f855ce16212db10a0359e690a03ac48/jiwer-4.0.0-py3-none-any.whl", hash = "sha256:7efaf0bd336b095d99ddef9dd67e1ee829d75d58aa2a81d9639870b01d6d95ea", size = 23034, upload-time = "2025-06-19T16:05:21.821Z" }, ] +[[package]] +name = "markdown-it-py" +version = "4.2.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "mdurl" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/06/ff/7841249c247aa650a76b9ee4bbaeae59370dc8bfd2f6c01f3630c35eb134/markdown_it_py-4.2.0.tar.gz", hash = "sha256:04a21681d6fbb623de53f6f364d352309d4094dd4194040a10fd51833e418d49", size = 82454, upload-time = "2026-05-07T12:08:28.36Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b3/81/4da04ced5a082363ecfa159c010d200ecbd959ae410c10c0264a38cac0f5/markdown_it_py-4.2.0-py3-none-any.whl", hash = "sha256:9f7ebbcd14fe59494226453aed97c1070d83f8d24b6fc3a3bcf9a38092641c4a", size = 91687, upload-time = "2026-05-07T12:08:27.182Z" }, +] + +[[package]] +name = "markupsafe" +version = "3.0.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/7e/99/7690b6d4034fffd95959cbe0c02de8deb3098cc577c67bb6a24fe5d7caa7/markupsafe-3.0.3.tar.gz", hash = "sha256:722695808f4b6457b320fdc131280796bdceb04ab50fe1795cd540799ebe1698", size = 80313, upload-time = "2025-09-27T18:37:40.426Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/08/db/fefacb2136439fc8dd20e797950e749aa1f4997ed584c62cfb8ef7c2be0e/markupsafe-3.0.3-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:1cc7ea17a6824959616c525620e387f6dd30fec8cb44f649e31712db02123dad", size = 11631, upload-time = "2025-09-27T18:36:18.185Z" }, + { url = "https://files.pythonhosted.org/packages/e1/2e/5898933336b61975ce9dc04decbc0a7f2fee78c30353c5efba7f2d6ff27a/markupsafe-3.0.3-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:4bd4cd07944443f5a265608cc6aab442e4f74dff8088b0dfc8238647b8f6ae9a", size = 12058, upload-time = "2025-09-27T18:36:19.444Z" }, + { url = "https://files.pythonhosted.org/packages/1d/09/adf2df3699d87d1d8184038df46a9c80d78c0148492323f4693df54e17bb/markupsafe-3.0.3-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6b5420a1d9450023228968e7e6a9ce57f65d148ab56d2313fcd589eee96a7a50", size = 24287, upload-time = "2025-09-27T18:36:20.768Z" }, + { url = "https://files.pythonhosted.org/packages/30/ac/0273f6fcb5f42e314c6d8cd99effae6a5354604d461b8d392b5ec9530a54/markupsafe-3.0.3-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0bf2a864d67e76e5c9a34dc26ec616a66b9888e25e7b9460e1c76d3293bd9dbf", size = 22940, upload-time = "2025-09-27T18:36:22.249Z" }, + { url = "https://files.pythonhosted.org/packages/19/ae/31c1be199ef767124c042c6c3e904da327a2f7f0cd63a0337e1eca2967a8/markupsafe-3.0.3-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:bc51efed119bc9cfdf792cdeaa4d67e8f6fcccab66ed4bfdd6bde3e59bfcbb2f", size = 21887, upload-time = "2025-09-27T18:36:23.535Z" }, + { url = "https://files.pythonhosted.org/packages/b2/76/7edcab99d5349a4532a459e1fe64f0b0467a3365056ae550d3bcf3f79e1e/markupsafe-3.0.3-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:068f375c472b3e7acbe2d5318dea141359e6900156b5b2ba06a30b169086b91a", size = 23692, upload-time = "2025-09-27T18:36:24.823Z" }, + { url = "https://files.pythonhosted.org/packages/a4/28/6e74cdd26d7514849143d69f0bf2399f929c37dc2b31e6829fd2045b2765/markupsafe-3.0.3-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:7be7b61bb172e1ed687f1754f8e7484f1c8019780f6f6b0786e76bb01c2ae115", size = 21471, upload-time = "2025-09-27T18:36:25.95Z" }, + { url = "https://files.pythonhosted.org/packages/62/7e/a145f36a5c2945673e590850a6f8014318d5577ed7e5920a4b3448e0865d/markupsafe-3.0.3-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:f9e130248f4462aaa8e2552d547f36ddadbeaa573879158d721bbd33dfe4743a", size = 22923, upload-time = "2025-09-27T18:36:27.109Z" }, + { url = "https://files.pythonhosted.org/packages/0f/62/d9c46a7f5c9adbeeeda52f5b8d802e1094e9717705a645efc71b0913a0a8/markupsafe-3.0.3-cp311-cp311-win32.whl", hash = "sha256:0db14f5dafddbb6d9208827849fad01f1a2609380add406671a26386cdf15a19", size = 14572, upload-time = "2025-09-27T18:36:28.045Z" }, + { url = "https://files.pythonhosted.org/packages/83/8a/4414c03d3f891739326e1783338e48fb49781cc915b2e0ee052aa490d586/markupsafe-3.0.3-cp311-cp311-win_amd64.whl", hash = "sha256:de8a88e63464af587c950061a5e6a67d3632e36df62b986892331d4620a35c01", size = 15077, upload-time = "2025-09-27T18:36:29.025Z" }, + { url = "https://files.pythonhosted.org/packages/35/73/893072b42e6862f319b5207adc9ae06070f095b358655f077f69a35601f0/markupsafe-3.0.3-cp311-cp311-win_arm64.whl", hash = "sha256:3b562dd9e9ea93f13d53989d23a7e775fdfd1066c33494ff43f5418bc8c58a5c", size = 13876, upload-time = "2025-09-27T18:36:29.954Z" }, + { url = "https://files.pythonhosted.org/packages/5a/72/147da192e38635ada20e0a2e1a51cf8823d2119ce8883f7053879c2199b5/markupsafe-3.0.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:d53197da72cc091b024dd97249dfc7794d6a56530370992a5e1a08983ad9230e", size = 11615, upload-time = "2025-09-27T18:36:30.854Z" }, + { url = "https://files.pythonhosted.org/packages/9a/81/7e4e08678a1f98521201c3079f77db69fb552acd56067661f8c2f534a718/markupsafe-3.0.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:1872df69a4de6aead3491198eaf13810b565bdbeec3ae2dc8780f14458ec73ce", size = 12020, upload-time = "2025-09-27T18:36:31.971Z" }, + { url = "https://files.pythonhosted.org/packages/1e/2c/799f4742efc39633a1b54a92eec4082e4f815314869865d876824c257c1e/markupsafe-3.0.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3a7e8ae81ae39e62a41ec302f972ba6ae23a5c5396c8e60113e9066ef893da0d", size = 24332, upload-time = "2025-09-27T18:36:32.813Z" }, + { url = "https://files.pythonhosted.org/packages/3c/2e/8d0c2ab90a8c1d9a24f0399058ab8519a3279d1bd4289511d74e909f060e/markupsafe-3.0.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d6dd0be5b5b189d31db7cda48b91d7e0a9795f31430b7f271219ab30f1d3ac9d", size = 22947, upload-time = "2025-09-27T18:36:33.86Z" }, + { url = "https://files.pythonhosted.org/packages/2c/54/887f3092a85238093a0b2154bd629c89444f395618842e8b0c41783898ea/markupsafe-3.0.3-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:94c6f0bb423f739146aec64595853541634bde58b2135f27f61c1ffd1cd4d16a", size = 21962, upload-time = "2025-09-27T18:36:35.099Z" }, + { url = "https://files.pythonhosted.org/packages/c9/2f/336b8c7b6f4a4d95e91119dc8521402461b74a485558d8f238a68312f11c/markupsafe-3.0.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:be8813b57049a7dc738189df53d69395eba14fb99345e0a5994914a3864c8a4b", size = 23760, upload-time = "2025-09-27T18:36:36.001Z" }, + { url = "https://files.pythonhosted.org/packages/32/43/67935f2b7e4982ffb50a4d169b724d74b62a3964bc1a9a527f5ac4f1ee2b/markupsafe-3.0.3-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:83891d0e9fb81a825d9a6d61e3f07550ca70a076484292a70fde82c4b807286f", size = 21529, upload-time = "2025-09-27T18:36:36.906Z" }, + { url = "https://files.pythonhosted.org/packages/89/e0/4486f11e51bbba8b0c041098859e869e304d1c261e59244baa3d295d47b7/markupsafe-3.0.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:77f0643abe7495da77fb436f50f8dab76dbc6e5fd25d39589a0f1fe6548bfa2b", size = 23015, upload-time = "2025-09-27T18:36:37.868Z" }, + { url = "https://files.pythonhosted.org/packages/2f/e1/78ee7a023dac597a5825441ebd17170785a9dab23de95d2c7508ade94e0e/markupsafe-3.0.3-cp312-cp312-win32.whl", hash = "sha256:d88b440e37a16e651bda4c7c2b930eb586fd15ca7406cb39e211fcff3bf3017d", size = 14540, upload-time = "2025-09-27T18:36:38.761Z" }, + { url = "https://files.pythonhosted.org/packages/aa/5b/bec5aa9bbbb2c946ca2733ef9c4ca91c91b6a24580193e891b5f7dbe8e1e/markupsafe-3.0.3-cp312-cp312-win_amd64.whl", hash = "sha256:26a5784ded40c9e318cfc2bdb30fe164bdb8665ded9cd64d500a34fb42067b1c", size = 15105, upload-time = "2025-09-27T18:36:39.701Z" }, + { url = "https://files.pythonhosted.org/packages/e5/f1/216fc1bbfd74011693a4fd837e7026152e89c4bcf3e77b6692fba9923123/markupsafe-3.0.3-cp312-cp312-win_arm64.whl", hash = "sha256:35add3b638a5d900e807944a078b51922212fb3dedb01633a8defc4b01a3c85f", size = 13906, upload-time = "2025-09-27T18:36:40.689Z" }, + { url = "https://files.pythonhosted.org/packages/38/2f/907b9c7bbba283e68f20259574b13d005c121a0fa4c175f9bed27c4597ff/markupsafe-3.0.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:e1cf1972137e83c5d4c136c43ced9ac51d0e124706ee1c8aa8532c1287fa8795", size = 11622, upload-time = "2025-09-27T18:36:41.777Z" }, + { url = "https://files.pythonhosted.org/packages/9c/d9/5f7756922cdd676869eca1c4e3c0cd0df60ed30199ffd775e319089cb3ed/markupsafe-3.0.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:116bb52f642a37c115f517494ea5feb03889e04df47eeff5b130b1808ce7c219", size = 12029, upload-time = "2025-09-27T18:36:43.257Z" }, + { url = "https://files.pythonhosted.org/packages/00/07/575a68c754943058c78f30db02ee03a64b3c638586fba6a6dd56830b30a3/markupsafe-3.0.3-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:133a43e73a802c5562be9bbcd03d090aa5a1fe899db609c29e8c8d815c5f6de6", size = 24374, upload-time = "2025-09-27T18:36:44.508Z" }, + { url = "https://files.pythonhosted.org/packages/a9/21/9b05698b46f218fc0e118e1f8168395c65c8a2c750ae2bab54fc4bd4e0e8/markupsafe-3.0.3-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ccfcd093f13f0f0b7fdd0f198b90053bf7b2f02a3927a30e63f3ccc9df56b676", size = 22980, upload-time = "2025-09-27T18:36:45.385Z" }, + { url = "https://files.pythonhosted.org/packages/7f/71/544260864f893f18b6827315b988c146b559391e6e7e8f7252839b1b846a/markupsafe-3.0.3-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:509fa21c6deb7a7a273d629cf5ec029bc209d1a51178615ddf718f5918992ab9", size = 21990, upload-time = "2025-09-27T18:36:46.916Z" }, + { url = "https://files.pythonhosted.org/packages/c2/28/b50fc2f74d1ad761af2f5dcce7492648b983d00a65b8c0e0cb457c82ebbe/markupsafe-3.0.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:a4afe79fb3de0b7097d81da19090f4df4f8d3a2b3adaa8764138aac2e44f3af1", size = 23784, upload-time = "2025-09-27T18:36:47.884Z" }, + { url = "https://files.pythonhosted.org/packages/ed/76/104b2aa106a208da8b17a2fb72e033a5a9d7073c68f7e508b94916ed47a9/markupsafe-3.0.3-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:795e7751525cae078558e679d646ae45574b47ed6e7771863fcc079a6171a0fc", size = 21588, upload-time = "2025-09-27T18:36:48.82Z" }, + { url = "https://files.pythonhosted.org/packages/b5/99/16a5eb2d140087ebd97180d95249b00a03aa87e29cc224056274f2e45fd6/markupsafe-3.0.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:8485f406a96febb5140bfeca44a73e3ce5116b2501ac54fe953e488fb1d03b12", size = 23041, upload-time = "2025-09-27T18:36:49.797Z" }, + { url = "https://files.pythonhosted.org/packages/19/bc/e7140ed90c5d61d77cea142eed9f9c303f4c4806f60a1044c13e3f1471d0/markupsafe-3.0.3-cp313-cp313-win32.whl", hash = "sha256:bdd37121970bfd8be76c5fb069c7751683bdf373db1ed6c010162b2a130248ed", size = 14543, upload-time = "2025-09-27T18:36:51.584Z" }, + { url = "https://files.pythonhosted.org/packages/05/73/c4abe620b841b6b791f2edc248f556900667a5a1cf023a6646967ae98335/markupsafe-3.0.3-cp313-cp313-win_amd64.whl", hash = "sha256:9a1abfdc021a164803f4d485104931fb8f8c1efd55bc6b748d2f5774e78b62c5", size = 15113, upload-time = "2025-09-27T18:36:52.537Z" }, + { url = "https://files.pythonhosted.org/packages/f0/3a/fa34a0f7cfef23cf9500d68cb7c32dd64ffd58a12b09225fb03dd37d5b80/markupsafe-3.0.3-cp313-cp313-win_arm64.whl", hash = "sha256:7e68f88e5b8799aa49c85cd116c932a1ac15caaa3f5db09087854d218359e485", size = 13911, upload-time = "2025-09-27T18:36:53.513Z" }, + { url = "https://files.pythonhosted.org/packages/e4/d7/e05cd7efe43a88a17a37b3ae96e79a19e846f3f456fe79c57ca61356ef01/markupsafe-3.0.3-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:218551f6df4868a8d527e3062d0fb968682fe92054e89978594c28e642c43a73", size = 11658, upload-time = "2025-09-27T18:36:54.819Z" }, + { url = "https://files.pythonhosted.org/packages/99/9e/e412117548182ce2148bdeacdda3bb494260c0b0184360fe0d56389b523b/markupsafe-3.0.3-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:3524b778fe5cfb3452a09d31e7b5adefeea8c5be1d43c4f810ba09f2ceb29d37", size = 12066, upload-time = "2025-09-27T18:36:55.714Z" }, + { url = "https://files.pythonhosted.org/packages/bc/e6/fa0ffcda717ef64a5108eaa7b4f5ed28d56122c9a6d70ab8b72f9f715c80/markupsafe-3.0.3-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4e885a3d1efa2eadc93c894a21770e4bc67899e3543680313b09f139e149ab19", size = 25639, upload-time = "2025-09-27T18:36:56.908Z" }, + { url = "https://files.pythonhosted.org/packages/96/ec/2102e881fe9d25fc16cb4b25d5f5cde50970967ffa5dddafdb771237062d/markupsafe-3.0.3-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8709b08f4a89aa7586de0aadc8da56180242ee0ada3999749b183aa23df95025", size = 23569, upload-time = "2025-09-27T18:36:57.913Z" }, + { url = "https://files.pythonhosted.org/packages/4b/30/6f2fce1f1f205fc9323255b216ca8a235b15860c34b6798f810f05828e32/markupsafe-3.0.3-cp313-cp313t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:b8512a91625c9b3da6f127803b166b629725e68af71f8184ae7e7d54686a56d6", size = 23284, upload-time = "2025-09-27T18:36:58.833Z" }, + { url = "https://files.pythonhosted.org/packages/58/47/4a0ccea4ab9f5dcb6f79c0236d954acb382202721e704223a8aafa38b5c8/markupsafe-3.0.3-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:9b79b7a16f7fedff2495d684f2b59b0457c3b493778c9eed31111be64d58279f", size = 24801, upload-time = "2025-09-27T18:36:59.739Z" }, + { url = "https://files.pythonhosted.org/packages/6a/70/3780e9b72180b6fecb83a4814d84c3bf4b4ae4bf0b19c27196104149734c/markupsafe-3.0.3-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:12c63dfb4a98206f045aa9563db46507995f7ef6d83b2f68eda65c307c6829eb", size = 22769, upload-time = "2025-09-27T18:37:00.719Z" }, + { url = "https://files.pythonhosted.org/packages/98/c5/c03c7f4125180fc215220c035beac6b9cb684bc7a067c84fc69414d315f5/markupsafe-3.0.3-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:8f71bc33915be5186016f675cd83a1e08523649b0e33efdb898db577ef5bb009", size = 23642, upload-time = "2025-09-27T18:37:01.673Z" }, + { url = "https://files.pythonhosted.org/packages/80/d6/2d1b89f6ca4bff1036499b1e29a1d02d282259f3681540e16563f27ebc23/markupsafe-3.0.3-cp313-cp313t-win32.whl", hash = "sha256:69c0b73548bc525c8cb9a251cddf1931d1db4d2258e9599c28c07ef3580ef354", size = 14612, upload-time = "2025-09-27T18:37:02.639Z" }, + { url = "https://files.pythonhosted.org/packages/2b/98/e48a4bfba0a0ffcf9925fe2d69240bfaa19c6f7507b8cd09c70684a53c1e/markupsafe-3.0.3-cp313-cp313t-win_amd64.whl", hash = "sha256:1b4b79e8ebf6b55351f0d91fe80f893b4743f104bff22e90697db1590e47a218", size = 15200, upload-time = "2025-09-27T18:37:03.582Z" }, + { url = "https://files.pythonhosted.org/packages/0e/72/e3cc540f351f316e9ed0f092757459afbc595824ca724cbc5a5d4263713f/markupsafe-3.0.3-cp313-cp313t-win_arm64.whl", hash = "sha256:ad2cf8aa28b8c020ab2fc8287b0f823d0a7d8630784c31e9ee5edea20f406287", size = 13973, upload-time = "2025-09-27T18:37:04.929Z" }, + { url = "https://files.pythonhosted.org/packages/33/8a/8e42d4838cd89b7dde187011e97fe6c3af66d8c044997d2183fbd6d31352/markupsafe-3.0.3-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:eaa9599de571d72e2daf60164784109f19978b327a3910d3e9de8c97b5b70cfe", size = 11619, upload-time = "2025-09-27T18:37:06.342Z" }, + { url = "https://files.pythonhosted.org/packages/b5/64/7660f8a4a8e53c924d0fa05dc3a55c9cee10bbd82b11c5afb27d44b096ce/markupsafe-3.0.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:c47a551199eb8eb2121d4f0f15ae0f923d31350ab9280078d1e5f12b249e0026", size = 12029, upload-time = "2025-09-27T18:37:07.213Z" }, + { url = "https://files.pythonhosted.org/packages/da/ef/e648bfd021127bef5fa12e1720ffed0c6cbb8310c8d9bea7266337ff06de/markupsafe-3.0.3-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f34c41761022dd093b4b6896d4810782ffbabe30f2d443ff5f083e0cbbb8c737", size = 24408, upload-time = "2025-09-27T18:37:09.572Z" }, + { url = "https://files.pythonhosted.org/packages/41/3c/a36c2450754618e62008bf7435ccb0f88053e07592e6028a34776213d877/markupsafe-3.0.3-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:457a69a9577064c05a97c41f4e65148652db078a3a509039e64d3467b9e7ef97", size = 23005, upload-time = "2025-09-27T18:37:10.58Z" }, + { url = "https://files.pythonhosted.org/packages/bc/20/b7fdf89a8456b099837cd1dc21974632a02a999ec9bf7ca3e490aacd98e7/markupsafe-3.0.3-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:e8afc3f2ccfa24215f8cb28dcf43f0113ac3c37c2f0f0806d8c70e4228c5cf4d", size = 22048, upload-time = "2025-09-27T18:37:11.547Z" }, + { url = "https://files.pythonhosted.org/packages/9a/a7/591f592afdc734f47db08a75793a55d7fbcc6902a723ae4cfbab61010cc5/markupsafe-3.0.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:ec15a59cf5af7be74194f7ab02d0f59a62bdcf1a537677ce67a2537c9b87fcda", size = 23821, upload-time = "2025-09-27T18:37:12.48Z" }, + { url = "https://files.pythonhosted.org/packages/7d/33/45b24e4f44195b26521bc6f1a82197118f74df348556594bd2262bda1038/markupsafe-3.0.3-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:0eb9ff8191e8498cca014656ae6b8d61f39da5f95b488805da4bb029cccbfbaf", size = 21606, upload-time = "2025-09-27T18:37:13.485Z" }, + { url = "https://files.pythonhosted.org/packages/ff/0e/53dfaca23a69fbfbbf17a4b64072090e70717344c52eaaaa9c5ddff1e5f0/markupsafe-3.0.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:2713baf880df847f2bece4230d4d094280f4e67b1e813eec43b4c0e144a34ffe", size = 23043, upload-time = "2025-09-27T18:37:14.408Z" }, + { url = "https://files.pythonhosted.org/packages/46/11/f333a06fc16236d5238bfe74daccbca41459dcd8d1fa952e8fbd5dccfb70/markupsafe-3.0.3-cp314-cp314-win32.whl", hash = "sha256:729586769a26dbceff69f7a7dbbf59ab6572b99d94576a5592625d5b411576b9", size = 14747, upload-time = "2025-09-27T18:37:15.36Z" }, + { url = "https://files.pythonhosted.org/packages/28/52/182836104b33b444e400b14f797212f720cbc9ed6ba34c800639d154e821/markupsafe-3.0.3-cp314-cp314-win_amd64.whl", hash = "sha256:bdc919ead48f234740ad807933cdf545180bfbe9342c2bb451556db2ed958581", size = 15341, upload-time = "2025-09-27T18:37:16.496Z" }, + { url = "https://files.pythonhosted.org/packages/6f/18/acf23e91bd94fd7b3031558b1f013adfa21a8e407a3fdb32745538730382/markupsafe-3.0.3-cp314-cp314-win_arm64.whl", hash = "sha256:5a7d5dc5140555cf21a6fefbdbf8723f06fcd2f63ef108f2854de715e4422cb4", size = 14073, upload-time = "2025-09-27T18:37:17.476Z" }, + { url = "https://files.pythonhosted.org/packages/3c/f0/57689aa4076e1b43b15fdfa646b04653969d50cf30c32a102762be2485da/markupsafe-3.0.3-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:1353ef0c1b138e1907ae78e2f6c63ff67501122006b0f9abad68fda5f4ffc6ab", size = 11661, upload-time = "2025-09-27T18:37:18.453Z" }, + { url = "https://files.pythonhosted.org/packages/89/c3/2e67a7ca217c6912985ec766c6393b636fb0c2344443ff9d91404dc4c79f/markupsafe-3.0.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:1085e7fbddd3be5f89cc898938f42c0b3c711fdcb37d75221de2666af647c175", size = 12069, upload-time = "2025-09-27T18:37:19.332Z" }, + { url = "https://files.pythonhosted.org/packages/f0/00/be561dce4e6ca66b15276e184ce4b8aec61fe83662cce2f7d72bd3249d28/markupsafe-3.0.3-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1b52b4fb9df4eb9ae465f8d0c228a00624de2334f216f178a995ccdcf82c4634", size = 25670, upload-time = "2025-09-27T18:37:20.245Z" }, + { url = "https://files.pythonhosted.org/packages/50/09/c419f6f5a92e5fadde27efd190eca90f05e1261b10dbd8cbcb39cd8ea1dc/markupsafe-3.0.3-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:fed51ac40f757d41b7c48425901843666a6677e3e8eb0abcff09e4ba6e664f50", size = 23598, upload-time = "2025-09-27T18:37:21.177Z" }, + { url = "https://files.pythonhosted.org/packages/22/44/a0681611106e0b2921b3033fc19bc53323e0b50bc70cffdd19f7d679bb66/markupsafe-3.0.3-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:f190daf01f13c72eac4efd5c430a8de82489d9cff23c364c3ea822545032993e", size = 23261, upload-time = "2025-09-27T18:37:22.167Z" }, + { url = "https://files.pythonhosted.org/packages/5f/57/1b0b3f100259dc9fffe780cfb60d4be71375510e435efec3d116b6436d43/markupsafe-3.0.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:e56b7d45a839a697b5eb268c82a71bd8c7f6c94d6fd50c3d577fa39a9f1409f5", size = 24835, upload-time = "2025-09-27T18:37:23.296Z" }, + { url = "https://files.pythonhosted.org/packages/26/6a/4bf6d0c97c4920f1597cc14dd720705eca0bf7c787aebc6bb4d1bead5388/markupsafe-3.0.3-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:f3e98bb3798ead92273dc0e5fd0f31ade220f59a266ffd8a4f6065e0a3ce0523", size = 22733, upload-time = "2025-09-27T18:37:24.237Z" }, + { url = "https://files.pythonhosted.org/packages/14/c7/ca723101509b518797fedc2fdf79ba57f886b4aca8a7d31857ba3ee8281f/markupsafe-3.0.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:5678211cb9333a6468fb8d8be0305520aa073f50d17f089b5b4b477ea6e67fdc", size = 23672, upload-time = "2025-09-27T18:37:25.271Z" }, + { url = "https://files.pythonhosted.org/packages/fb/df/5bd7a48c256faecd1d36edc13133e51397e41b73bb77e1a69deab746ebac/markupsafe-3.0.3-cp314-cp314t-win32.whl", hash = "sha256:915c04ba3851909ce68ccc2b8e2cd691618c4dc4c4232fb7982bca3f41fd8c3d", size = 14819, upload-time = "2025-09-27T18:37:26.285Z" }, + { url = "https://files.pythonhosted.org/packages/1a/8a/0402ba61a2f16038b48b39bccca271134be00c5c9f0f623208399333c448/markupsafe-3.0.3-cp314-cp314t-win_amd64.whl", hash = "sha256:4faffd047e07c38848ce017e8725090413cd80cbc23d86e55c587bf979e579c9", size = 15426, upload-time = "2025-09-27T18:37:27.316Z" }, + { url = "https://files.pythonhosted.org/packages/70/bc/6f1c2f612465f5fa89b95bead1f44dcb607670fd42891d8fdcd5d039f4f4/markupsafe-3.0.3-cp314-cp314t-win_arm64.whl", hash = "sha256:32001d6a8fc98c8cb5c947787c5d08b0a50663d139f1305bac5885d98d9b40fa", size = 14146, upload-time = "2025-09-27T18:37:28.327Z" }, +] + +[[package]] +name = "mdurl" +version = "0.1.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d6/54/cfe61301667036ec958cb99bd3efefba235e65cdeb9c84d24a8293ba1d90/mdurl-0.1.2.tar.gz", hash = "sha256:bb413d29f5eea38f31dd4754dd7377d4465116fb207585f97bf925588687c1ba", size = 8729, upload-time = "2022-08-14T12:40:10.846Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b3/38/89ba8ad64ae25be8de66a6d463314cf1eb366222074cfda9ee839c56a4b4/mdurl-0.1.2-py3-none-any.whl", hash = "sha256:84008a41e51615a49fc9966191ff91509e3c40b939176e643fd50a5c2196b8f8", size = 9979, upload-time = "2022-08-14T12:40:09.779Z" }, +] + +[[package]] +name = "mpmath" +version = "1.3.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/e0/47/dd32fa426cc72114383ac549964eecb20ecfd886d1e5ccf5340b55b02f57/mpmath-1.3.0.tar.gz", hash = "sha256:7a28eb2a9774d00c7bc92411c19a89209d5da7c4c9a9e227be8330a23a25b91f", size = 508106, upload-time = "2023-03-07T16:47:11.061Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/43/e3/7d92a15f894aa0c9c4b49b8ee9ac9850d6e63b03c9c32c0367a13ae62209/mpmath-1.3.0-py3-none-any.whl", hash = "sha256:a0b2b9fe80bbcd81a6647ff13108738cfb482d481d826cc0e02f5b35e5c88d2c", size = 536198, upload-time = "2023-03-07T16:47:09.197Z" }, +] + +[[package]] +name = "networkx" +version = "3.6.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/6a/51/63fe664f3908c97be9d2e4f1158eb633317598cfa6e1fc14af5383f17512/networkx-3.6.1.tar.gz", hash = "sha256:26b7c357accc0c8cde558ad486283728b65b6a95d85ee1cd66bafab4c8168509", size = 2517025, upload-time = "2025-12-08T17:02:39.908Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/9e/c9/b2622292ea83fbb4ec318f5b9ab867d0a28ab43c5717bb85b0a5f6b3b0a4/networkx-3.6.1-py3-none-any.whl", hash = "sha256:d47fbf302e7d9cbbb9e2555a0d267983d2aa476bac30e90dfbe5669bd57f3762", size = 2068504, upload-time = "2025-12-08T17:02:38.159Z" }, +] + [[package]] name = "numpy" version = "2.4.6" @@ -845,6 +1150,158 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/b4/07/458c344f0f0c178f4481dad5cca790626ffe4c34eabf9467069d06ee4999/numpy-2.5.2-cp315-cp315t-win_arm64.whl", hash = "sha256:5f8e00be2ec6f45f4e8a41a527f68d44a7d96fee92a650e4d8b1326f77f61e6e", size = 10748103, upload-time = "2026-08-09T13:48:24.21Z" }, ] +[[package]] +name = "nvidia-cublas" +version = "13.1.1.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "nvidia-cuda-nvrtc" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/a7/a1/0bd24ee8c8d03adac032fd2909426a00c88f8c57961b1277ded97f91119f/nvidia_cublas-13.1.1.3-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:b7a210458267ac818974c53038fbec2e969d5c99f305ab15c72522fa9f001dd5", size = 542848918, upload-time = "2026-04-08T18:46:22.985Z" }, + { url = "https://files.pythonhosted.org/packages/3b/cd/154ca20c38269e05eff77c1464e6c1da89f50a6390b565e9d82e06bc11e1/nvidia_cublas-13.1.1.3-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:37936a16db8fe4ac1f065c2139360608a543a09275cb1a1af612e08cfa065436", size = 423138758, upload-time = "2026-04-08T18:46:58.655Z" }, +] + +[[package]] +name = "nvidia-cuda-cupti" +version = "13.0.85" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/2a/2a/80353b103fc20ce05ef51e928daed4b6015db4aaa9162ed0997090fe2250/nvidia_cuda_cupti-13.0.85-py3-none-manylinux_2_25_aarch64.whl", hash = "sha256:796bd679890ee55fb14a94629b698b6db54bcfd833d391d5e94017dd9d7d3151", size = 10310827, upload-time = "2025-09-04T08:26:42.012Z" }, + { url = "https://files.pythonhosted.org/packages/33/6d/737d164b4837a9bbd202f5ae3078975f0525a55730fe871d8ed4e3b952b0/nvidia_cuda_cupti-13.0.85-py3-none-manylinux_2_25_x86_64.whl", hash = "sha256:4eb01c08e859bf924d222250d2e8f8b8ff6d3db4721288cf35d14252a4d933c8", size = 10715597, upload-time = "2025-09-04T08:26:51.312Z" }, +] + +[[package]] +name = "nvidia-cuda-nvrtc" +version = "13.0.88" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c3/68/483a78f5e8f31b08fb1bb671559968c0ca3a065ac7acabfc7cee55214fd6/nvidia_cuda_nvrtc-13.0.88-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl", hash = "sha256:ad9b6d2ead2435f11cbb6868809d2adeeee302e9bb94bcf0539c7a40d80e8575", size = 90215200, upload-time = "2025-09-04T08:28:44.204Z" }, + { url = "https://files.pythonhosted.org/packages/b7/dc/6bb80850e0b7edd6588d560758f17e0550893a1feaf436807d64d2da040f/nvidia_cuda_nvrtc-13.0.88-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:d27f20a0ca67a4bb34268a5e951033496c5b74870b868bacd046b1b8e0c3267b", size = 43015449, upload-time = "2025-09-04T08:28:20.239Z" }, +] + +[[package]] +name = "nvidia-cuda-runtime" +version = "13.0.96" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/87/4f/17d7b9b8e285199c58ce28e31b5c5bbaa4d8271af06a89b6405258245de2/nvidia_cuda_runtime-13.0.96-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:ef9bcbe90493a2b9d810e43d249adb3d02e98dd30200d86607d8d02687c43f55", size = 2261060, upload-time = "2025-10-09T08:55:15.78Z" }, + { url = "https://files.pythonhosted.org/packages/2e/24/d1558f3b68b1d26e706813b1d10aa1d785e4698c425af8db8edc3dced472/nvidia_cuda_runtime-13.0.96-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:7f82250d7782aa23b6cfe765ecc7db554bd3c2870c43f3d1821f1d18aebf0548", size = 2243632, upload-time = "2025-10-09T08:55:36.117Z" }, +] + +[[package]] +name = "nvidia-cudnn-cu13" +version = "9.20.0.48" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "nvidia-cublas" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/56/c5/83384d846b2fd17c44bd499b36c75a45ed4f095fbbb2252294e89cea5c5c/nvidia_cudnn_cu13-9.20.0.48-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:e31454ae00094b0c55319d9d15b6fa2fc50a9e1c0f5c8c80fb75258234e731e1", size = 444574296, upload-time = "2026-03-09T19:28:27.751Z" }, + { url = "https://files.pythonhosted.org/packages/6e/5e/edb9c0ae051602c3ccaffe424256463636d639e27d7f302dde9975ef9e7a/nvidia_cudnn_cu13-9.20.0.48-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:0c45dd8eeb50b603f07995b1b300c62ffe6a1980482b82b3bcf94a4ca9d49304", size = 366173588, upload-time = "2026-03-09T19:29:34.474Z" }, +] + +[[package]] +name = "nvidia-cufft" +version = "12.0.0.61" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "nvidia-nvjitlink" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/8b/ae/f417a75c0259e85c1d2f83ca4e960289a5f814ed0cea74d18c353d3e989d/nvidia_cufft-12.0.0.61-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:2708c852ef8cd89d1d2068bdbece0aa188813a0c934db3779b9b1faa8442e5f5", size = 214053554, upload-time = "2025-09-04T08:31:38.196Z" }, + { url = "https://files.pythonhosted.org/packages/a8/2f/7b57e29836ea8714f81e9898409196f47d772d5ddedddf1592eadb8ab743/nvidia_cufft-12.0.0.61-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:6c44f692dce8fd5ffd3e3df134b6cdb9c2f72d99cf40b62c32dde45eea9ddad3", size = 214085489, upload-time = "2025-09-04T08:31:56.044Z" }, +] + +[[package]] +name = "nvidia-cufile" +version = "1.15.1.6" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/3f/70/4f193de89a48b71714e74602ee14d04e4019ad36a5a9f20c425776e72cd6/nvidia_cufile-1.15.1.6-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:08a3ecefae5a01c7f5117351c64f17c7c62efa5fffdbe24fc7d298da19cd0b44", size = 1223672, upload-time = "2025-09-04T08:32:22.779Z" }, + { url = "https://files.pythonhosted.org/packages/ab/73/cc4a14c9813a8a0d509417cf5f4bdaba76e924d58beb9864f5a7baceefbf/nvidia_cufile-1.15.1.6-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:bdc0deedc61f548bddf7733bdc216456c2fdb101d020e1ab4b88d232d5e2f6d1", size = 1136992, upload-time = "2025-09-04T08:32:14.119Z" }, +] + +[[package]] +name = "nvidia-curand" +version = "10.4.0.35" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1e/72/7c2ae24fb6b63a32e6ae5d241cc65263ea18d08802aaae087d9f013335a2/nvidia_curand-10.4.0.35-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:133df5a7509c3e292aaa2b477afd0194f06ce4ea24d714d616ff36439cee349a", size = 61962106, upload-time = "2025-08-04T10:21:41.128Z" }, + { url = "https://files.pythonhosted.org/packages/a5/9f/be0a41ca4a4917abf5cb9ae0daff1a6060cc5de950aec0396de9f3b52bc5/nvidia_curand-10.4.0.35-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:1aee33a5da6e1db083fe2b90082def8915f30f3248d5896bcec36a579d941bfc", size = 59544258, upload-time = "2025-08-04T10:22:03.992Z" }, +] + +[[package]] +name = "nvidia-cusolver" +version = "12.0.4.66" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "nvidia-cublas" }, + { name = "nvidia-cusparse" }, + { name = "nvidia-nvjitlink" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/c8/c3/b30c9e935fc01e3da443ec0116ed1b2a009bb867f5324d3f2d7e533e776b/nvidia_cusolver-12.0.4.66-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:02c2457eaa9e39de20f880f4bd8820e6a1cfb9f9a34f820eb12a155aa5bc92d2", size = 223467760, upload-time = "2025-09-04T08:33:04.222Z" }, + { url = "https://files.pythonhosted.org/packages/5f/67/cba3777620cdacb99102da4042883709c41c709f4b6323c10781a9c3aa34/nvidia_cusolver-12.0.4.66-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:0a759da5dea5c0ea10fd307de75cdeb59e7ea4fcb8add0924859b944babf1112", size = 200941980, upload-time = "2025-09-04T08:33:22.767Z" }, +] + +[[package]] +name = "nvidia-cusparse" +version = "12.6.3.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "nvidia-nvjitlink" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/f8/94/5c26f33738ae35276672f12615a64bd008ed5be6d1ebcb23579285d960a9/nvidia_cusparse-12.6.3.3-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:80bcc4662f23f1054ee334a15c72b8940402975e0eab63178fc7e670aa59472c", size = 162155568, upload-time = "2025-09-04T08:33:42.864Z" }, + { url = "https://files.pythonhosted.org/packages/fa/18/623c77619c31d62efd55302939756966f3ecc8d724a14dab2b75f1508850/nvidia_cusparse-12.6.3.3-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:2b3c89c88d01ee0e477cb7f82ef60a11a4bcd57b6b87c33f789350b59759360b", size = 145942937, upload-time = "2025-09-04T08:33:58.029Z" }, +] + +[[package]] +name = "nvidia-cusparselt-cu13" +version = "0.8.1" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/46/e1/cdc1797eadf82d3a9a575a19b33fdc871a97edbec42c00b5b5e914f4aff4/nvidia_cusparselt_cu13-0.8.1-py3-none-manylinux2014_aarch64.whl", hash = "sha256:4dca476c50bf4780d46cd0bfbd82e2bc10a08e4fef7950917ce8d7578d22a23f", size = 221051344, upload-time = "2025-09-05T18:49:51.289Z" }, + { url = "https://files.pythonhosted.org/packages/34/7d/2661f2fb3ac4302f3a246f5fc030213ac60c1fe0bce84f9783dbd831dbb7/nvidia_cusparselt_cu13-0.8.1-py3-none-manylinux2014_x86_64.whl", hash = "sha256:786ce87568c303fadb5afcc7102d454cd3040d75f6f8626f5db460d1871f4dd0", size = 170148586, upload-time = "2025-09-05T18:50:50.248Z" }, +] + +[[package]] +name = "nvidia-nccl-cu13" +version = "2.29.7" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/72/0d/daf50d44177ee0cbc7ff0a0c91eb5ff676c82be42f9a970bc7597f440c3a/nvidia_nccl_cu13-2.29.7-py3-none-manylinux_2_18_aarch64.whl", hash = "sha256:674a12383e3c38a1bcccae7d4f3633b37852230b6047883cb2f4c2d1b36d9bf5", size = 206014712, upload-time = "2026-03-03T05:34:20.843Z" }, + { url = "https://files.pythonhosted.org/packages/67/f4/58e4e91b6919367c7aafb8e36fce9aad1a3047e536bf7e2fd560927d3a4c/nvidia_nccl_cu13-2.29.7-py3-none-manylinux_2_18_x86_64.whl", hash = "sha256:edd81538446786ec3b73972543e53bb43bcaf0bfc8ef76cb679fcc390ffe136d", size = 205976000, upload-time = "2026-03-03T05:36:24.472Z" }, +] + +[[package]] +name = "nvidia-nvjitlink" +version = "13.3.33" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f0/ee/580ca6f29dcab0221db8706badca1bbbb084f1975c4d4e83329c3a7e31f0/nvidia_nvjitlink-13.3.33-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl", hash = "sha256:26a6de7fb4c8fdaa7703d3dad720d6d427ddfea5c48a528fd97c11733ad830e5", size = 40742423, upload-time = "2026-05-26T16:54:51.613Z" }, + { url = "https://files.pythonhosted.org/packages/69/30/45414e35ff2eee7db3da037e5707037ccf9d2b5218ffbdb055ea4d5aa98a/nvidia_nvjitlink-13.3.33-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:ce48b37dfeb3cb1eae4cf85adacb47d7a6539ea2272870c9a3628ce275c2037e", size = 39168635, upload-time = "2026-05-26T16:54:13.906Z" }, +] + +[[package]] +name = "nvidia-nvshmem-cu13" +version = "3.4.5" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/dc/0f/05cc9c720236dcd2db9c1ab97fff629e96821be2e63103569da0c9b72f19/nvidia_nvshmem_cu13-3.4.5-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:6dc2a197f38e5d0376ad52cd1a2a3617d3cdc150fd5966f4aee9bcebb1d68fe9", size = 60215947, upload-time = "2025-09-06T00:32:20.022Z" }, + { url = "https://files.pythonhosted.org/packages/3c/35/a9bf80a609e74e3b000fef598933235c908fcefcef9026042b8e6dfde2a9/nvidia_nvshmem_cu13-3.4.5-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:290f0a2ee94c9f3687a02502f3b9299a9f9fe826e6d0287ee18482e78d495b80", size = 60412546, upload-time = "2025-09-06T00:32:41.564Z" }, +] + +[[package]] +name = "nvidia-nvtx" +version = "13.0.85" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c2/f3/d86c845465a2723ad7e1e5c36dcd75ddb82898b3f53be47ebd429fb2fa5d/nvidia_nvtx-13.0.85-py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:4936d1d6780fbe68db454f5e72a42ff64d1fd6397df9f363ae786930fd5c1cd4", size = 148047, upload-time = "2025-09-04T08:29:01.761Z" }, + { url = "https://files.pythonhosted.org/packages/a8/64/3708a90d1ebe202ffdeb7185f878a3c84d15c2b2c31858da2ce0583e2def/nvidia_nvtx-13.0.85-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:cb7780edb6b14107373c835bf8b72e7a178bac7367e23da7acb108f973f157a6", size = 148878, upload-time = "2025-09-04T08:28:53.627Z" }, +] + [[package]] name = "openai" version = "3.3.1" @@ -985,6 +1442,34 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl", hash = "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746", size = 20538, upload-time = "2025-05-15T12:30:06.134Z" }, ] +[[package]] +name = "psutil" +version = "7.2.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/aa/c6/d1ddf4abb55e93cebc4f2ed8b5d6dbad109ecb8d63748dd2b20ab5e57ebe/psutil-7.2.2.tar.gz", hash = "sha256:0746f5f8d406af344fd547f1c8daa5f5c33dbc293bb8d6a16d80b4bb88f59372", size = 493740, upload-time = "2026-01-28T18:14:54.428Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/51/08/510cbdb69c25a96f4ae523f733cdc963ae654904e8db864c07585ef99875/psutil-7.2.2-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:2edccc433cbfa046b980b0df0171cd25bcaeb3a68fe9022db0979e7aa74a826b", size = 130595, upload-time = "2026-01-28T18:14:57.293Z" }, + { url = "https://files.pythonhosted.org/packages/d6/f5/97baea3fe7a5a9af7436301f85490905379b1c6f2dd51fe3ecf24b4c5fbf/psutil-7.2.2-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:e78c8603dcd9a04c7364f1a3e670cea95d51ee865e4efb3556a3a63adef958ea", size = 131082, upload-time = "2026-01-28T18:14:59.732Z" }, + { url = "https://files.pythonhosted.org/packages/37/d6/246513fbf9fa174af531f28412297dd05241d97a75911ac8febefa1a53c6/psutil-7.2.2-cp313-cp313t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1a571f2330c966c62aeda00dd24620425d4b0cc86881c89861fbc04549e5dc63", size = 181476, upload-time = "2026-01-28T18:15:01.884Z" }, + { url = "https://files.pythonhosted.org/packages/b8/b5/9182c9af3836cca61696dabe4fd1304e17bc56cb62f17439e1154f225dd3/psutil-7.2.2-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:917e891983ca3c1887b4ef36447b1e0873e70c933afc831c6b6da078ba474312", size = 184062, upload-time = "2026-01-28T18:15:04.436Z" }, + { url = "https://files.pythonhosted.org/packages/16/ba/0756dca669f5a9300d0cbcbfae9a4c30e446dfc7440ffe43ded5724bfd93/psutil-7.2.2-cp313-cp313t-win_amd64.whl", hash = "sha256:ab486563df44c17f5173621c7b198955bd6b613fb87c71c161f827d3fb149a9b", size = 139893, upload-time = "2026-01-28T18:15:06.378Z" }, + { url = "https://files.pythonhosted.org/packages/1c/61/8fa0e26f33623b49949346de05ec1ddaad02ed8ba64af45f40a147dbfa97/psutil-7.2.2-cp313-cp313t-win_arm64.whl", hash = "sha256:ae0aefdd8796a7737eccea863f80f81e468a1e4cf14d926bd9b6f5f2d5f90ca9", size = 135589, upload-time = "2026-01-28T18:15:08.03Z" }, + { url = "https://files.pythonhosted.org/packages/81/69/ef179ab5ca24f32acc1dac0c247fd6a13b501fd5534dbae0e05a1c48b66d/psutil-7.2.2-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:eed63d3b4d62449571547b60578c5b2c4bcccc5387148db46e0c2313dad0ee00", size = 130664, upload-time = "2026-01-28T18:15:09.469Z" }, + { url = "https://files.pythonhosted.org/packages/7b/64/665248b557a236d3fa9efc378d60d95ef56dd0a490c2cd37dafc7660d4a9/psutil-7.2.2-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:7b6d09433a10592ce39b13d7be5a54fbac1d1228ed29abc880fb23df7cb694c9", size = 131087, upload-time = "2026-01-28T18:15:11.724Z" }, + { url = "https://files.pythonhosted.org/packages/d5/2e/e6782744700d6759ebce3043dcfa661fb61e2fb752b91cdeae9af12c2178/psutil-7.2.2-cp314-cp314t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1fa4ecf83bcdf6e6c8f4449aff98eefb5d0604bf88cb883d7da3d8d2d909546a", size = 182383, upload-time = "2026-01-28T18:15:13.445Z" }, + { url = "https://files.pythonhosted.org/packages/57/49/0a41cefd10cb7505cdc04dab3eacf24c0c2cb158a998b8c7b1d27ee2c1f5/psutil-7.2.2-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e452c464a02e7dc7822a05d25db4cde564444a67e58539a00f929c51eddda0cf", size = 185210, upload-time = "2026-01-28T18:15:16.002Z" }, + { url = "https://files.pythonhosted.org/packages/dd/2c/ff9bfb544f283ba5f83ba725a3c5fec6d6b10b8f27ac1dc641c473dc390d/psutil-7.2.2-cp314-cp314t-win_amd64.whl", hash = "sha256:c7663d4e37f13e884d13994247449e9f8f574bc4655d509c3b95e9ec9e2b9dc1", size = 141228, upload-time = "2026-01-28T18:15:18.385Z" }, + { url = "https://files.pythonhosted.org/packages/f2/fc/f8d9c31db14fcec13748d373e668bc3bed94d9077dbc17fb0eebc073233c/psutil-7.2.2-cp314-cp314t-win_arm64.whl", hash = "sha256:11fe5a4f613759764e79c65cf11ebdf26e33d6dd34336f8a337aa2996d71c841", size = 136284, upload-time = "2026-01-28T18:15:19.912Z" }, + { url = "https://files.pythonhosted.org/packages/e7/36/5ee6e05c9bd427237b11b3937ad82bb8ad2752d72c6969314590dd0c2f6e/psutil-7.2.2-cp36-abi3-macosx_10_9_x86_64.whl", hash = "sha256:ed0cace939114f62738d808fdcecd4c869222507e266e574799e9c0faa17d486", size = 129090, upload-time = "2026-01-28T18:15:22.168Z" }, + { url = "https://files.pythonhosted.org/packages/80/c4/f5af4c1ca8c1eeb2e92ccca14ce8effdeec651d5ab6053c589b074eda6e1/psutil-7.2.2-cp36-abi3-macosx_11_0_arm64.whl", hash = "sha256:1a7b04c10f32cc88ab39cbf606e117fd74721c831c98a27dc04578deb0c16979", size = 129859, upload-time = "2026-01-28T18:15:23.795Z" }, + { url = "https://files.pythonhosted.org/packages/b5/70/5d8df3b09e25bce090399cf48e452d25c935ab72dad19406c77f4e828045/psutil-7.2.2-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:076a2d2f923fd4821644f5ba89f059523da90dc9014e85f8e45a5774ca5bc6f9", size = 155560, upload-time = "2026-01-28T18:15:25.976Z" }, + { url = "https://files.pythonhosted.org/packages/63/65/37648c0c158dc222aba51c089eb3bdfa238e621674dc42d48706e639204f/psutil-7.2.2-cp36-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b0726cecd84f9474419d67252add4ac0cd9811b04d61123054b9fb6f57df6e9e", size = 156997, upload-time = "2026-01-28T18:15:27.794Z" }, + { url = "https://files.pythonhosted.org/packages/8e/13/125093eadae863ce03c6ffdbae9929430d116a246ef69866dad94da3bfbc/psutil-7.2.2-cp36-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:fd04ef36b4a6d599bbdb225dd1d3f51e00105f6d48a28f006da7f9822f2606d8", size = 148972, upload-time = "2026-01-28T18:15:29.342Z" }, + { url = "https://files.pythonhosted.org/packages/04/78/0acd37ca84ce3ddffaa92ef0f571e073faa6d8ff1f0559ab1272188ea2be/psutil-7.2.2-cp36-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:b58fabe35e80b264a4e3bb23e6b96f9e45a3df7fb7eed419ac0e5947c61e47cc", size = 148266, upload-time = "2026-01-28T18:15:31.597Z" }, + { url = "https://files.pythonhosted.org/packages/b4/90/e2159492b5426be0c1fef7acba807a03511f97c5f86b3caeda6ad92351a7/psutil-7.2.2-cp37-abi3-win_amd64.whl", hash = "sha256:eb7e81434c8d223ec4a219b5fc1c47d0417b12be7ea866e24fb5ad6e84b3d988", size = 137737, upload-time = "2026-01-28T18:15:33.849Z" }, + { url = "https://files.pythonhosted.org/packages/8c/c7/7bb2e321574b10df20cbde462a94e2b71d05f9bbda251ef27d104668306a/psutil-7.2.2-cp37-abi3-win_arm64.whl", hash = "sha256:8c233660f575a5a89e6d4cb65d9f938126312bca76d8fe087b947b3a1aaac9ee", size = 134617, upload-time = "2026-01-28T18:15:36.514Z" }, +] + [[package]] name = "pyasn1" version = "0.6.4" @@ -1184,6 +1669,61 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/0d/17/c5c6b53ddc18f297992099b3d9ec16c855c0ccc83263a21fe4d1c625ec6c/python_dotenv-1.2.3-py3-none-any.whl", hash = "sha256:904552145e8bfed22162c09dab1c2b9b54fefa7b23ba780f4f26ca0316b0f0d9", size = 22780, upload-time = "2026-08-16T16:54:52.473Z" }, ] +[[package]] +name = "pyyaml" +version = "6.0.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/05/8e/961c0007c59b8dd7729d542c61a4d537767a59645b82a0b521206e1e25c2/pyyaml-6.0.3.tar.gz", hash = "sha256:d76623373421df22fb4cf8817020cbb7ef15c725b9d5e45f17e189bfc384190f", size = 130960, upload-time = "2025-09-25T21:33:16.546Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/6d/16/a95b6757765b7b031c9374925bb718d55e0a9ba8a1b6a12d25962ea44347/pyyaml-6.0.3-cp311-cp311-macosx_10_13_x86_64.whl", hash = "sha256:44edc647873928551a01e7a563d7452ccdebee747728c1080d881d68af7b997e", size = 185826, upload-time = "2025-09-25T21:31:58.655Z" }, + { url = "https://files.pythonhosted.org/packages/16/19/13de8e4377ed53079ee996e1ab0a9c33ec2faf808a4647b7b4c0d46dd239/pyyaml-6.0.3-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:652cb6edd41e718550aad172851962662ff2681490a8a711af6a4d288dd96824", size = 175577, upload-time = "2025-09-25T21:32:00.088Z" }, + { url = "https://files.pythonhosted.org/packages/0c/62/d2eb46264d4b157dae1275b573017abec435397aa59cbcdab6fc978a8af4/pyyaml-6.0.3-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:10892704fc220243f5305762e276552a0395f7beb4dbf9b14ec8fd43b57f126c", size = 775556, upload-time = "2025-09-25T21:32:01.31Z" }, + { url = "https://files.pythonhosted.org/packages/10/cb/16c3f2cf3266edd25aaa00d6c4350381c8b012ed6f5276675b9eba8d9ff4/pyyaml-6.0.3-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:850774a7879607d3a6f50d36d04f00ee69e7fc816450e5f7e58d7f17f1ae5c00", size = 882114, upload-time = "2025-09-25T21:32:03.376Z" }, + { url = "https://files.pythonhosted.org/packages/71/60/917329f640924b18ff085ab889a11c763e0b573da888e8404ff486657602/pyyaml-6.0.3-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b8bb0864c5a28024fac8a632c443c87c5aa6f215c0b126c449ae1a150412f31d", size = 806638, upload-time = "2025-09-25T21:32:04.553Z" }, + { url = "https://files.pythonhosted.org/packages/dd/6f/529b0f316a9fd167281a6c3826b5583e6192dba792dd55e3203d3f8e655a/pyyaml-6.0.3-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:1d37d57ad971609cf3c53ba6a7e365e40660e3be0e5175fa9f2365a379d6095a", size = 767463, upload-time = "2025-09-25T21:32:06.152Z" }, + { url = "https://files.pythonhosted.org/packages/f2/6a/b627b4e0c1dd03718543519ffb2f1deea4a1e6d42fbab8021936a4d22589/pyyaml-6.0.3-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:37503bfbfc9d2c40b344d06b2199cf0e96e97957ab1c1b546fd4f87e53e5d3e4", size = 794986, upload-time = "2025-09-25T21:32:07.367Z" }, + { url = "https://files.pythonhosted.org/packages/45/91/47a6e1c42d9ee337c4839208f30d9f09caa9f720ec7582917b264defc875/pyyaml-6.0.3-cp311-cp311-win32.whl", hash = "sha256:8098f252adfa6c80ab48096053f512f2321f0b998f98150cea9bd23d83e1467b", size = 142543, upload-time = "2025-09-25T21:32:08.95Z" }, + { url = "https://files.pythonhosted.org/packages/da/e3/ea007450a105ae919a72393cb06f122f288ef60bba2dc64b26e2646fa315/pyyaml-6.0.3-cp311-cp311-win_amd64.whl", hash = "sha256:9f3bfb4965eb874431221a3ff3fdcddc7e74e3b07799e0e84ca4a0f867d449bf", size = 158763, upload-time = "2025-09-25T21:32:09.96Z" }, + { url = "https://files.pythonhosted.org/packages/d1/33/422b98d2195232ca1826284a76852ad5a86fe23e31b009c9886b2d0fb8b2/pyyaml-6.0.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:7f047e29dcae44602496db43be01ad42fc6f1cc0d8cd6c83d342306c32270196", size = 182063, upload-time = "2025-09-25T21:32:11.445Z" }, + { url = "https://files.pythonhosted.org/packages/89/a0/6cf41a19a1f2f3feab0e9c0b74134aa2ce6849093d5517a0c550fe37a648/pyyaml-6.0.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:fc09d0aa354569bc501d4e787133afc08552722d3ab34836a80547331bb5d4a0", size = 173973, upload-time = "2025-09-25T21:32:12.492Z" }, + { url = "https://files.pythonhosted.org/packages/ed/23/7a778b6bd0b9a8039df8b1b1d80e2e2ad78aa04171592c8a5c43a56a6af4/pyyaml-6.0.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9149cad251584d5fb4981be1ecde53a1ca46c891a79788c0df828d2f166bda28", size = 775116, upload-time = "2025-09-25T21:32:13.652Z" }, + { url = "https://files.pythonhosted.org/packages/65/30/d7353c338e12baef4ecc1b09e877c1970bd3382789c159b4f89d6a70dc09/pyyaml-6.0.3-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:5fdec68f91a0c6739b380c83b951e2c72ac0197ace422360e6d5a959d8d97b2c", size = 844011, upload-time = "2025-09-25T21:32:15.21Z" }, + { url = "https://files.pythonhosted.org/packages/8b/9d/b3589d3877982d4f2329302ef98a8026e7f4443c765c46cfecc8858c6b4b/pyyaml-6.0.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ba1cc08a7ccde2d2ec775841541641e4548226580ab850948cbfda66a1befcdc", size = 807870, upload-time = "2025-09-25T21:32:16.431Z" }, + { url = "https://files.pythonhosted.org/packages/05/c0/b3be26a015601b822b97d9149ff8cb5ead58c66f981e04fedf4e762f4bd4/pyyaml-6.0.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:8dc52c23056b9ddd46818a57b78404882310fb473d63f17b07d5c40421e47f8e", size = 761089, upload-time = "2025-09-25T21:32:17.56Z" }, + { url = "https://files.pythonhosted.org/packages/be/8e/98435a21d1d4b46590d5459a22d88128103f8da4c2d4cb8f14f2a96504e1/pyyaml-6.0.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:41715c910c881bc081f1e8872880d3c650acf13dfa8214bad49ed4cede7c34ea", size = 790181, upload-time = "2025-09-25T21:32:18.834Z" }, + { url = "https://files.pythonhosted.org/packages/74/93/7baea19427dcfbe1e5a372d81473250b379f04b1bd3c4c5ff825e2327202/pyyaml-6.0.3-cp312-cp312-win32.whl", hash = "sha256:96b533f0e99f6579b3d4d4995707cf36df9100d67e0c8303a0c55b27b5f99bc5", size = 137658, upload-time = "2025-09-25T21:32:20.209Z" }, + { url = "https://files.pythonhosted.org/packages/86/bf/899e81e4cce32febab4fb42bb97dcdf66bc135272882d1987881a4b519e9/pyyaml-6.0.3-cp312-cp312-win_amd64.whl", hash = "sha256:5fcd34e47f6e0b794d17de1b4ff496c00986e1c83f7ab2fb8fcfe9616ff7477b", size = 154003, upload-time = "2025-09-25T21:32:21.167Z" }, + { url = "https://files.pythonhosted.org/packages/1a/08/67bd04656199bbb51dbed1439b7f27601dfb576fb864099c7ef0c3e55531/pyyaml-6.0.3-cp312-cp312-win_arm64.whl", hash = "sha256:64386e5e707d03a7e172c0701abfb7e10f0fb753ee1d773128192742712a98fd", size = 140344, upload-time = "2025-09-25T21:32:22.617Z" }, + { url = "https://files.pythonhosted.org/packages/d1/11/0fd08f8192109f7169db964b5707a2f1e8b745d4e239b784a5a1dd80d1db/pyyaml-6.0.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:8da9669d359f02c0b91ccc01cac4a67f16afec0dac22c2ad09f46bee0697eba8", size = 181669, upload-time = "2025-09-25T21:32:23.673Z" }, + { url = "https://files.pythonhosted.org/packages/b1/16/95309993f1d3748cd644e02e38b75d50cbc0d9561d21f390a76242ce073f/pyyaml-6.0.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:2283a07e2c21a2aa78d9c4442724ec1eb15f5e42a723b99cb3d822d48f5f7ad1", size = 173252, upload-time = "2025-09-25T21:32:25.149Z" }, + { url = "https://files.pythonhosted.org/packages/50/31/b20f376d3f810b9b2371e72ef5adb33879b25edb7a6d072cb7ca0c486398/pyyaml-6.0.3-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ee2922902c45ae8ccada2c5b501ab86c36525b883eff4255313a253a3160861c", size = 767081, upload-time = "2025-09-25T21:32:26.575Z" }, + { url = "https://files.pythonhosted.org/packages/49/1e/a55ca81e949270d5d4432fbbd19dfea5321eda7c41a849d443dc92fd1ff7/pyyaml-6.0.3-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a33284e20b78bd4a18c8c2282d549d10bc8408a2a7ff57653c0cf0b9be0afce5", size = 841159, upload-time = "2025-09-25T21:32:27.727Z" }, + { url = "https://files.pythonhosted.org/packages/74/27/e5b8f34d02d9995b80abcef563ea1f8b56d20134d8f4e5e81733b1feceb2/pyyaml-6.0.3-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0f29edc409a6392443abf94b9cf89ce99889a1dd5376d94316ae5145dfedd5d6", size = 801626, upload-time = "2025-09-25T21:32:28.878Z" }, + { url = "https://files.pythonhosted.org/packages/f9/11/ba845c23988798f40e52ba45f34849aa8a1f2d4af4b798588010792ebad6/pyyaml-6.0.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:f7057c9a337546edc7973c0d3ba84ddcdf0daa14533c2065749c9075001090e6", size = 753613, upload-time = "2025-09-25T21:32:30.178Z" }, + { url = "https://files.pythonhosted.org/packages/3d/e0/7966e1a7bfc0a45bf0a7fb6b98ea03fc9b8d84fa7f2229e9659680b69ee3/pyyaml-6.0.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:eda16858a3cab07b80edaf74336ece1f986ba330fdb8ee0d6c0d68fe82bc96be", size = 794115, upload-time = "2025-09-25T21:32:31.353Z" }, + { url = "https://files.pythonhosted.org/packages/de/94/980b50a6531b3019e45ddeada0626d45fa85cbe22300844a7983285bed3b/pyyaml-6.0.3-cp313-cp313-win32.whl", hash = "sha256:d0eae10f8159e8fdad514efdc92d74fd8d682c933a6dd088030f3834bc8e6b26", size = 137427, upload-time = "2025-09-25T21:32:32.58Z" }, + { url = "https://files.pythonhosted.org/packages/97/c9/39d5b874e8b28845e4ec2202b5da735d0199dbe5b8fb85f91398814a9a46/pyyaml-6.0.3-cp313-cp313-win_amd64.whl", hash = "sha256:79005a0d97d5ddabfeeea4cf676af11e647e41d81c9a7722a193022accdb6b7c", size = 154090, upload-time = "2025-09-25T21:32:33.659Z" }, + { url = "https://files.pythonhosted.org/packages/73/e8/2bdf3ca2090f68bb3d75b44da7bbc71843b19c9f2b9cb9b0f4ab7a5a4329/pyyaml-6.0.3-cp313-cp313-win_arm64.whl", hash = "sha256:5498cd1645aa724a7c71c8f378eb29ebe23da2fc0d7a08071d89469bf1d2defb", size = 140246, upload-time = "2025-09-25T21:32:34.663Z" }, + { url = "https://files.pythonhosted.org/packages/9d/8c/f4bd7f6465179953d3ac9bc44ac1a8a3e6122cf8ada906b4f96c60172d43/pyyaml-6.0.3-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:8d1fab6bb153a416f9aeb4b8763bc0f22a5586065f86f7664fc23339fc1c1fac", size = 181814, upload-time = "2025-09-25T21:32:35.712Z" }, + { url = "https://files.pythonhosted.org/packages/bd/9c/4d95bb87eb2063d20db7b60faa3840c1b18025517ae857371c4dd55a6b3a/pyyaml-6.0.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:34d5fcd24b8445fadc33f9cf348c1047101756fd760b4dacb5c3e99755703310", size = 173809, upload-time = "2025-09-25T21:32:36.789Z" }, + { url = "https://files.pythonhosted.org/packages/92/b5/47e807c2623074914e29dabd16cbbdd4bf5e9b2db9f8090fa64411fc5382/pyyaml-6.0.3-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:501a031947e3a9025ed4405a168e6ef5ae3126c59f90ce0cd6f2bfc477be31b7", size = 766454, upload-time = "2025-09-25T21:32:37.966Z" }, + { url = "https://files.pythonhosted.org/packages/02/9e/e5e9b168be58564121efb3de6859c452fccde0ab093d8438905899a3a483/pyyaml-6.0.3-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:b3bc83488de33889877a0f2543ade9f70c67d66d9ebb4ac959502e12de895788", size = 836355, upload-time = "2025-09-25T21:32:39.178Z" }, + { url = "https://files.pythonhosted.org/packages/88/f9/16491d7ed2a919954993e48aa941b200f38040928474c9e85ea9e64222c3/pyyaml-6.0.3-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c458b6d084f9b935061bc36216e8a69a7e293a2f1e68bf956dcd9e6cbcd143f5", size = 794175, upload-time = "2025-09-25T21:32:40.865Z" }, + { url = "https://files.pythonhosted.org/packages/dd/3f/5989debef34dc6397317802b527dbbafb2b4760878a53d4166579111411e/pyyaml-6.0.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:7c6610def4f163542a622a73fb39f534f8c101d690126992300bf3207eab9764", size = 755228, upload-time = "2025-09-25T21:32:42.084Z" }, + { url = "https://files.pythonhosted.org/packages/d7/ce/af88a49043cd2e265be63d083fc75b27b6ed062f5f9fd6cdc223ad62f03e/pyyaml-6.0.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:5190d403f121660ce8d1d2c1bb2ef1bd05b5f68533fc5c2ea899bd15f4399b35", size = 789194, upload-time = "2025-09-25T21:32:43.362Z" }, + { url = "https://files.pythonhosted.org/packages/23/20/bb6982b26a40bb43951265ba29d4c246ef0ff59c9fdcdf0ed04e0687de4d/pyyaml-6.0.3-cp314-cp314-win_amd64.whl", hash = "sha256:4a2e8cebe2ff6ab7d1050ecd59c25d4c8bd7e6f400f5f82b96557ac0abafd0ac", size = 156429, upload-time = "2025-09-25T21:32:57.844Z" }, + { url = "https://files.pythonhosted.org/packages/f4/f4/a4541072bb9422c8a883ab55255f918fa378ecf083f5b85e87fc2b4eda1b/pyyaml-6.0.3-cp314-cp314-win_arm64.whl", hash = "sha256:93dda82c9c22deb0a405ea4dc5f2d0cda384168e466364dec6255b293923b2f3", size = 143912, upload-time = "2025-09-25T21:32:59.247Z" }, + { url = "https://files.pythonhosted.org/packages/7c/f9/07dd09ae774e4616edf6cda684ee78f97777bdd15847253637a6f052a62f/pyyaml-6.0.3-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:02893d100e99e03eda1c8fd5c441d8c60103fd175728e23e431db1b589cf5ab3", size = 189108, upload-time = "2025-09-25T21:32:44.377Z" }, + { url = "https://files.pythonhosted.org/packages/4e/78/8d08c9fb7ce09ad8c38ad533c1191cf27f7ae1effe5bb9400a46d9437fcf/pyyaml-6.0.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:c1ff362665ae507275af2853520967820d9124984e0f7466736aea23d8611fba", size = 183641, upload-time = "2025-09-25T21:32:45.407Z" }, + { url = "https://files.pythonhosted.org/packages/7b/5b/3babb19104a46945cf816d047db2788bcaf8c94527a805610b0289a01c6b/pyyaml-6.0.3-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6adc77889b628398debc7b65c073bcb99c4a0237b248cacaf3fe8a557563ef6c", size = 831901, upload-time = "2025-09-25T21:32:48.83Z" }, + { url = "https://files.pythonhosted.org/packages/8b/cc/dff0684d8dc44da4d22a13f35f073d558c268780ce3c6ba1b87055bb0b87/pyyaml-6.0.3-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a80cb027f6b349846a3bf6d73b5e95e782175e52f22108cfa17876aaeff93702", size = 861132, upload-time = "2025-09-25T21:32:50.149Z" }, + { url = "https://files.pythonhosted.org/packages/b1/5e/f77dc6b9036943e285ba76b49e118d9ea929885becb0a29ba8a7c75e29fe/pyyaml-6.0.3-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:00c4bdeba853cc34e7dd471f16b4114f4162dc03e6b7afcc2128711f0eca823c", size = 839261, upload-time = "2025-09-25T21:32:51.808Z" }, + { url = "https://files.pythonhosted.org/packages/ce/88/a9db1376aa2a228197c58b37302f284b5617f56a5d959fd1763fb1675ce6/pyyaml-6.0.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:66e1674c3ef6f541c35191caae2d429b967b99e02040f5ba928632d9a7f0f065", size = 805272, upload-time = "2025-09-25T21:32:52.941Z" }, + { url = "https://files.pythonhosted.org/packages/da/92/1446574745d74df0c92e6aa4a7b0b3130706a4142b2d1a5869f2eaa423c6/pyyaml-6.0.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:16249ee61e95f858e83976573de0f5b2893b3677ba71c9dd36b9cf8be9ac6d65", size = 829923, upload-time = "2025-09-25T21:32:54.537Z" }, + { url = "https://files.pythonhosted.org/packages/f0/7a/1c7270340330e575b92f397352af856a8c06f230aa3e76f86b39d01b416a/pyyaml-6.0.3-cp314-cp314t-win_amd64.whl", hash = "sha256:4ad1906908f2f5ae4e5a8ddfce73c320c2a1429ec52eafd27138b7f1cbe341c9", size = 174062, upload-time = "2025-09-25T21:32:55.767Z" }, + { url = "https://files.pythonhosted.org/packages/f1/12/de94a39c2ef588c7e6455cfbe7343d3b2dc9d6b6b2f40c4c6565744c873d/pyyaml-6.0.3-cp314-cp314t-win_arm64.whl", hash = "sha256:ebc55a14a21cb14062aa4162f906cd962b28e2e9ea38f9b4391244cd8de4ae0b", size = 149341, upload-time = "2025-09-25T21:32:56.828Z" }, +] + [[package]] name = "rapidfuzz" version = "3.14.5" @@ -1263,6 +1803,110 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/aa/b5/363906b1064fc6fe611783a61764927bbd91919aaaabe8cba82151ca93ef/rapidfuzz-3.14.5-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:dfef96543ced67d9513a422755db422ae1dc34dade0a1485e0b43e7342ed3ebf", size = 1509889, upload-time = "2026-04-07T11:16:28.487Z" }, ] +[[package]] +name = "regex" +version = "2026.7.19" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/20/98/04b13f1ddfb63158025291c02e03eb42fbb7acb51d091d541050eb4e35e8/regex-2026.7.19.tar.gz", hash = "sha256:7e77b324909c1617cbb4c668677e2c6ae13f44d7c1de0d4f15f2e3c10f3315b5", size = 416440, upload-time = "2026-07-19T00:19:48.923Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/05/e5/cef4de2bac939280b68d32adc659478845238a8274f2f79c465063f590ad/regex-2026.7.19-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:ac777001cdfc28b72477d93c8564bb7583081ea8fb45cdca3d568e0a4f87183c", size = 494012, upload-time = "2026-07-19T00:16:39.927Z" }, + { url = "https://files.pythonhosted.org/packages/ff/87/e86f51eb117457bb7803132ffe5cb6e2841e2b5bea4cc85d397f3c6e257d/regex-2026.7.19-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:59787bd5f8c70aa339084e961d2996b53fbdeab4d5393bba5c1fe1fc32e02bae", size = 295281, upload-time = "2026-07-19T00:16:41.433Z" }, + { url = "https://files.pythonhosted.org/packages/41/2e/2360c41d8080a3d9ec7e5c90fad6eab3b50192869d10e9a5609e48c8177b/regex-2026.7.19-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:90c633e7e8d6bf4e992b8b36ce69e018f834b641dd6de8cea6d78c06ffa119c5", size = 290615, upload-time = "2026-07-19T00:16:43.058Z" }, + { url = "https://files.pythonhosted.org/packages/cf/69/b65ba4344efbc771b28fe5dde84cbbb6c8f9551165952fe78def5b9dde6a/regex-2026.7.19-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:87ccab0db8d5f4fbb0272642113c1adb2ffc698c16d3a0944580222331fa7a20", size = 791804, upload-time = "2026-07-19T00:16:44.662Z" }, + { url = "https://files.pythonhosted.org/packages/81/b6/a40dfa0dc6224b36f620c00296eacc830489cbf8c2837b6750dfe6170375/regex-2026.7.19-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:9e50d748a32da622f256e8d505867f5d3c43a837c6a9f0efb149655fadd1042a", size = 861723, upload-time = "2026-07-19T00:16:46.412Z" }, + { url = "https://files.pythonhosted.org/packages/e3/02/735991dee71abd83196a7962f7ed8bf5aa05720ff06e2d3ff896a85e2bbb/regex-2026.7.19-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:bf1516fe58fc104f39b2d1dbe2d5e27d0cd45c4be2e42ba6ee0cc763701ec3c7", size = 905932, upload-time = "2026-07-19T00:16:47.956Z" }, + { url = "https://files.pythonhosted.org/packages/45/6c/e7098d8b846ccdbf431d8c081b61e496526a27a28094ed09e0dce21b3f54/regex-2026.7.19-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:09f3e5287f94f17b709dc9a9e70865855feee835c861613be144218ce4ca82cc", size = 801407, upload-time = "2026-07-19T00:16:49.43Z" }, + { url = "https://files.pythonhosted.org/packages/8a/18/34b69274e2649bcc7d9b089c2b2983fb2632d8ecf667e359593be9072e79/regex-2026.7.19-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:6383cd2ed53a646c659ba1fe65727db76437fdaa069e697a0b44a51d5843d864", size = 774448, upload-time = "2026-07-19T00:16:51.352Z" }, + { url = "https://files.pythonhosted.org/packages/bb/e6/0a72247d025585fd3800b98e040b84d562a88af6303347100484849f4f01/regex-2026.7.19-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:09d3007fc76249a83cdd33de160d50e6cb77f54e09d8fa9e7148e10607ce24af", size = 783297, upload-time = "2026-07-19T00:16:53.071Z" }, + { url = "https://files.pythonhosted.org/packages/b1/aa/c4f65ae7dd02a36b323a70c4cff326e1f3442361aaebc9311100a130d54f/regex-2026.7.19-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:6f8c6e7a1cfa3dc9d0ee2de0e65e834537fa29992cc3976ffec914afc35c5dd5", size = 854736, upload-time = "2026-07-19T00:16:54.607Z" }, + { url = "https://files.pythonhosted.org/packages/62/c3/668082bcc817b9e694189b84997aeba7385b7779faa6711788679c482e35/regex-2026.7.19-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:b2ea4a3e8357be8849e833beeae757ac3c7a6b3fc055c03c808a53c91ad30d82", size = 763298, upload-time = "2026-07-19T00:16:56.289Z" }, + { url = "https://files.pythonhosted.org/packages/4b/fb/2d07ad555e7af88aa5f867fdafa47a8d945ee237c20af3ebceb46a820835/regex-2026.7.19-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:80115dd39481fd3a4b4080220799dbcacb921a844de4b827264ececacbe17c78", size = 844430, upload-time = "2026-07-19T00:16:57.933Z" }, + { url = "https://files.pythonhosted.org/packages/51/15/c82a471fe3dce56f03745635b43aa456c40dc0db089e07ef148b331507d1/regex-2026.7.19-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:d6ce43a0269d68cee79a7d1ade7def53c20f8f2a047b92d7b5d5bcc73ae88327", size = 789683, upload-time = "2026-07-19T00:16:59.583Z" }, + { url = "https://files.pythonhosted.org/packages/b5/f4/7532a2c59d56f5398902c20de60f0c9a5d1cd364e42a051b48e1b210be7b/regex-2026.7.19-cp311-cp311-win32.whl", hash = "sha256:9be2a6647740dd3cca6acb24e87f03d7632cd280dbce9bbe40c26353a215a45d", size = 266778, upload-time = "2026-07-19T00:17:01.032Z" }, + { url = "https://files.pythonhosted.org/packages/83/2b/cf1bc631db154eb95520d9d5dbc2371ff77a0f014bbf7d748fed8496aa63/regex-2026.7.19-cp311-cp311-win_amd64.whl", hash = "sha256:8d3469c91dd92ee41b7c95280edbd975ef1ba9195086686623a1c6e8935ce965", size = 277983, upload-time = "2026-07-19T00:17:02.571Z" }, + { url = "https://files.pythonhosted.org/packages/8d/bd/56ceaf170e875d5a6761bf2bfd0d040f1cacc896850d5e40cb29b11bbd06/regex-2026.7.19-cp311-cp311-win_arm64.whl", hash = "sha256:36aacfb15faaff3ced55afbf35ec72f50d4aee22082c4f7fe0573a33e2fca92e", size = 276961, upload-time = "2026-07-19T00:17:04.135Z" }, + { url = "https://files.pythonhosted.org/packages/3b/b9/d11d7e501ac8fd7d617684423ebb9561e0b998481c1e4cbc0cb212c5d74a/regex-2026.7.19-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:2cc3460cedf7579948486eab03bc9ad7089df4d7281c0f47f4afe03e8d13f02d", size = 496778, upload-time = "2026-07-19T00:17:05.677Z" }, + { url = "https://files.pythonhosted.org/packages/3f/a9/a5ab6f312f24318019170dc485d5421fe4f89e43a98640da50d95a8a7041/regex-2026.7.19-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:0e9554c8785eac5cffe6300f69a91f58ba72bc88a5f8d661235ad7c6aa5b8ccd", size = 297122, upload-time = "2026-07-19T00:17:07.59Z" }, + { url = "https://files.pythonhosted.org/packages/b3/63/4cab4d7f2d384a144d420b763d97674cb70619c878ea6fcd7640d0e62143/regex-2026.7.19-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:d7da47a0f248977f08e2cb659ff3c17ddc13a4d39b3a7baa0a81bf5b415430f6", size = 292009, upload-time = "2026-07-19T00:17:09.648Z" }, + { url = "https://files.pythonhosted.org/packages/22/85/102a81b218298957d4ea7d2f084fae537a71add9d6ff93c8e67284c5f45e/regex-2026.7.19-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:93db40c8de0815baab96a06e08a984bac71f989d13bab789e382158c5d426797", size = 796708, upload-time = "2026-07-19T00:17:11.542Z" }, + { url = "https://files.pythonhosted.org/packages/78/b5/dc136af5629938a037cd2b304c12240e132ec92f38be8ff9cc89af2a1f2d/regex-2026.7.19-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:66bd62c59a5427746e8c44becae1d9b99d22fb13f30f492083dfb9ad7c45cc18", size = 865651, upload-time = "2026-07-19T00:17:13.312Z" }, + { url = "https://files.pythonhosted.org/packages/e0/75/67402ae3cd9c8c988a4c805d15ee3eef015e7ca4cb112cf3e640fc1f4153/regex-2026.7.19-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:1649eb39fcc9ea80c4d2f110fde2b8ab2aef3877b98f02ab9b14e961f418c511", size = 911756, upload-time = "2026-07-19T00:17:15.015Z" }, + { url = "https://files.pythonhosted.org/packages/2a/8e/096d00c7c480ef2ff4265349b14e2261d4ab787ba1f74e2e80d1c58079c3/regex-2026.7.19-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9dce8ec9695f531a1b8a6f314fd4b393adcccf2ea861db480cdf97a301d01a68", size = 801798, upload-time = "2026-07-19T00:17:17.208Z" }, + { url = "https://files.pythonhosted.org/packages/f0/41/e7ecac6edb5722417f85cc67eaf386322fbe8acf6918ec2fdc37c20dd9d0/regex-2026.7.19-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:3080a7fd38ef049bd489e01c970c97dd84ff446a885b0f1f6b26d9b1ad13ce11", size = 776933, upload-time = "2026-07-19T00:17:19.347Z" }, + { url = "https://files.pythonhosted.org/packages/6f/69/03c9b3f058d66403e0ca2c938696e81d51cd4c6d47ec5265f02f96948d9a/regex-2026.7.19-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:1d793a7988e04fcb1e2e135567443d82173225d657419ec09414a9b5a145b986", size = 784338, upload-time = "2026-07-19T00:17:21.057Z" }, + { url = "https://files.pythonhosted.org/packages/f6/f7/b38ab3d43f284afbb618fcd15d0e77eb786ae461ce1f6bc7494619ddc0f2/regex-2026.7.19-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:e8b0abe7d870f53ca5143895fef7d1041a0c831a140d3dc2c760dd7ba25d4a8b", size = 860452, upload-time = "2026-07-19T00:17:23.119Z" }, + { url = "https://files.pythonhosted.org/packages/15/5c/ff60ef0571121714f3cf9920bc183071e384a10b556d042e0fdb06cc07a5/regex-2026.7.19-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:4e5413bd5f13d3a4e3539ca98f70f75e7fca92518dd7f117f030ebedd10b60cb", size = 765958, upload-time = "2026-07-19T00:17:24.81Z" }, + { url = "https://files.pythonhosted.org/packages/aa/0f/bd34021162c0ab47f9a315bd56cd5642e920c8e5668a75ef6c6a6fca590d/regex-2026.7.19-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:73b133a9e6fb512858e7f065e96f1180aa46646bc74a83aea62f1d314f3dd035", size = 851765, upload-time = "2026-07-19T00:17:26.993Z" }, + { url = "https://files.pythonhosted.org/packages/2a/20/a2ca43edade0595cccfdc98636739f536d9e26898e7dbddc2b9e98898953/regex-2026.7.19-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:dbe6493fbd27321b1d1f2dd4f5c7e5bd4d8b1d7cab7f32fd67db3d0b2ed8248a", size = 789714, upload-time = "2026-07-19T00:17:28.699Z" }, + { url = "https://files.pythonhosted.org/packages/5d/47/e02db4015d424fc83c00ea0ac8c5e5ec14397943de9abf909d5ce3a25931/regex-2026.7.19-cp312-cp312-win32.whl", hash = "sha256:ddd67571c10869f65a5d7dde536d1e066e306cc90de57d7de4d5f34802428bb5", size = 267157, upload-time = "2026-07-19T00:17:31.051Z" }, + { url = "https://files.pythonhosted.org/packages/08/8e/c780c131f79b42ed22d1bd7da4096c2c35f813e835acd02ef0f018bd892c/regex-2026.7.19-cp312-cp312-win_amd64.whl", hash = "sha256:e30d40268a28d54ce0437031750497004c22602b8e3ab891f759b795a003b312", size = 277777, upload-time = "2026-07-19T00:17:32.848Z" }, + { url = "https://files.pythonhosted.org/packages/3e/4c/e4d7e086449bdf379d89774bf1f89dc4a41943f3c5a6125a03905b34b5fb/regex-2026.7.19-cp312-cp312-win_arm64.whl", hash = "sha256:de9208bb427130c82a5dbfd104f92c8876fc9559278c880b3002755bbbe9c83d", size = 277136, upload-time = "2026-07-19T00:17:34.803Z" }, + { url = "https://files.pythonhosted.org/packages/5d/3d/84165e4299ff76f3a40fe1f2abf939e976f693383a08d2beea6af62bd2c1/regex-2026.7.19-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:f035d9dc1d25eff9d361456572231c7d27b5ccd473ca7dc0adfce732bd006d40", size = 496552, upload-time = "2026-07-19T00:17:36.808Z" }, + { url = "https://files.pythonhosted.org/packages/02/a2/a65293e6e4cf28eb7ee1be5335a5386c40d6742e9f47fafc8fec785e16c7/regex-2026.7.19-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:c42572142ed0b9d5d261ba727157c426510da78e20828b66bbb855098b8a4e38", size = 296983, upload-time = "2026-07-19T00:17:38.816Z" }, + { url = "https://files.pythonhosted.org/packages/95/47/2d0564e93d87bc48618360ddca232a2ca612bbdf53ce8465d45ca5ce14ee/regex-2026.7.19-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:40b34dd88658e4fedd2fddbf0275ac970d00614b731357f425722a3ed1983d11", size = 291832, upload-time = "2026-07-19T00:17:40.726Z" }, + { url = "https://files.pythonhosted.org/packages/07/cd/42dfbabff3dfc9603c501c0e2e2c5adbb09d127b267bf5348de0af338c15/regex-2026.7.19-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0c41c63992bf1874cebb6e7f56fd7d3c007924659a604ae3d90e427d40d4fd13", size = 796775, upload-time = "2026-07-19T00:17:42.382Z" }, + { url = "https://files.pythonhosted.org/packages/df/5d/f6a4839f2b934e3eed5973fd07f5929ee97d4c98939fb275ea23c274ee16/regex-2026.7.19-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:1d3372064506b94dd2c67c845f2db8062e9e9ba84d04e33cb96d7d33c11fe1ae", size = 865687, upload-time = "2026-07-19T00:17:44.185Z" }, + { url = "https://files.pythonhosted.org/packages/14/b0/b47d6c36049bc59806a50bd4c86ced70bbe058d787f80281b1d7a9b0e024/regex-2026.7.19-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:fce7760bf283405b2c7999cab3da4e72f7deca6396013115e3f7a955db9760da", size = 911962, upload-time = "2026-07-19T00:17:46.442Z" }, + { url = "https://files.pythonhosted.org/packages/2a/be/ff61f28f9273658cfe23acbbac5217221f6519960ed401e61dfdab12bc35/regex-2026.7.19-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c0d702548d89d572b2929879bc883bb7a4c4709efafe4512cadee56c55c9bd15", size = 801817, upload-time = "2026-07-19T00:17:48.25Z" }, + { url = "https://files.pythonhosted.org/packages/c3/bb/8b4f7f26b333f9f79e1b453613c39bb4776f51d38ae66dd0ba31d6b354ca/regex-2026.7.19-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:d446c6ac40bb6e05025ccee55b84d80fe9bf8e93010ffc4bb9484f13d498835f", size = 776908, upload-time = "2026-07-19T00:17:50.183Z" }, + { url = "https://files.pythonhosted.org/packages/09/13/610110fc5921d380516d03c26b652555f08aa0d23ea78a771231873c3638/regex-2026.7.19-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:4c3501bfa814ab07b5580741f9bf78dfdfe146a04057f82df9e2402d2a975939", size = 784426, upload-time = "2026-07-19T00:17:52.454Z" }, + { url = "https://files.pythonhosted.org/packages/ca/f5/1ef9e2a83a5947c57ebff0b377cb5727c3d5ec1992317a320d035cd0dbb6/regex-2026.7.19-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:c4585c3e64b4f9e583b4d2683f18f5d5d872b3d71dcf24594b74ecc23602fa96", size = 860600, upload-time = "2026-07-19T00:17:54.229Z" }, + { url = "https://files.pythonhosted.org/packages/a0/02/073af33a3ec149241d11c80acea91e722aa0adbf05addd50f251c4fe89c3/regex-2026.7.19-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:571fde9741eb0ccde23dd4e0c1d50fbae910e901fa7e629faf39b2dda740d220", size = 765950, upload-time = "2026-07-19T00:17:56.041Z" }, + { url = "https://files.pythonhosted.org/packages/81/a9/d1e9f819dc394a568ef370cd56cf25394e957a2235f8370f23b576e5a475/regex-2026.7.19-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:15b364b9b98d6d2fe1a85034c23a3180ff913f46caddc3895f6fd65186255ccc", size = 851794, upload-time = "2026-07-19T00:17:57.897Z" }, + { url = "https://files.pythonhosted.org/packages/03/3a/8ae83eda7579feacdf984e71fb9e70635fb6f832eeddca58427ec4fca926/regex-2026.7.19-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:ffd8893ccc1c2fce6e0d6ca402d716fe1b29db70c7132609a05955e31b2aa8f2", size = 789845, upload-time = "2026-07-19T00:17:59.97Z" }, + { url = "https://files.pythonhosted.org/packages/4b/23/c195cbfe5a75fdec64d8f6554fd15237b837919d2c61bdc141d7c807b08b/regex-2026.7.19-cp313-cp313-win32.whl", hash = "sha256:f0fa4fa9c3632d708742baf2282f2055c11d888a790362670a403cbf48a2c404", size = 267135, upload-time = "2026-07-19T00:18:01.958Z" }, + { url = "https://files.pythonhosted.org/packages/b2/80/a11de8404b7272b70acb45c1c05987cce60b45d5693da2e176f0e390d564/regex-2026.7.19-cp313-cp313-win_amd64.whl", hash = "sha256:d51ffd3427640fa2da6ade574ceba932f210ad095f65fcc450a2b0a0d454868e", size = 277747, upload-time = "2026-07-19T00:18:04.121Z" }, + { url = "https://files.pythonhosted.org/packages/d1/29/0f5c8eff1b4f1f3d83276d365fccecf666afcc7d947420943bf394d07adb/regex-2026.7.19-cp313-cp313-win_arm64.whl", hash = "sha256:c670fe7be5b6020b76bc6e8d2196074657e1327595bca93a389e1a76ab130ad8", size = 277129, upload-time = "2026-07-19T00:18:05.821Z" }, + { url = "https://files.pythonhosted.org/packages/dc/4c/44b74742052cedda40f9ae469532a037112f7311a36669a891fba8984bb0/regex-2026.7.19-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:db47b561c9afd884baa1f96f797c9ca369872c4b65912bc691cfa99e68340af2", size = 501134, upload-time = "2026-07-19T00:18:07.567Z" }, + { url = "https://files.pythonhosted.org/packages/f0/45/bbd038b5e39ee5613a5a689290145b40058cc152c41de9cc23639d2b9734/regex-2026.7.19-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:65dcd28d3eba2ab7c2fd906485cc301392b47cc2234790d27d4e4814e02cdfda", size = 299418, upload-time = "2026-07-19T00:18:09.38Z" }, + { url = "https://files.pythonhosted.org/packages/65/38/c5bde94b4cedfd5850d64c3f08222d8e1600e84f6ee71d9b44b4b8163f74/regex-2026.7.19-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:f2e7f8e2ab6c2922be02c7ec45185aa5bd771e2e57b95455ee343a44d8130dff", size = 294486, upload-time = "2026-07-19T00:18:11.188Z" }, + { url = "https://files.pythonhosted.org/packages/d7/6a/2f5e107cb26c960b781967178899daf2787a7ab151844ed3c01d6fc95474/regex-2026.7.19-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:fe31f28c94402043161876a258a9c6f757cb485905c7614ce8d6cd40e6b7bdc1", size = 811643, upload-time = "2026-07-19T00:18:12.975Z" }, + { url = "https://files.pythonhosted.org/packages/37/d4/a2f963406d7d73a62eed84ba05a258afb6cad1b21aa4517443ce40506b78/regex-2026.7.19-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:f8f6fa298bb4f7f58a33334406218ba74716e68feddf5e4e54cd5d8082705abf", size = 871081, upload-time = "2026-07-19T00:18:14.733Z" }, + { url = "https://files.pythonhosted.org/packages/45/a3/44be546340bedb15f13063f5e7fe16793ea4d9ea2e805d09bd174ac27724/regex-2026.7.19-cp313-cp313t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:cc1b2440423a851fad781309dd87843868f4f66a6bcd1ddb9225cf4ec2c84732", size = 917372, upload-time = "2026-07-19T00:18:16.724Z" }, + { url = "https://files.pythonhosted.org/packages/f8/f6/e0870b0fd2a40dba0074e4b76e514b21313d37946c9248453e34ec43923e/regex-2026.7.19-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8ac59a0900474a52b7c04af8196affc22bd9842acb0950df12f7b813e983609a", size = 816089, upload-time = "2026-07-19T00:18:18.617Z" }, + { url = "https://files.pythonhosted.org/packages/ae/27/957e8e22690ad6634572b39b71f130a6105f4d0718bb16849eac00fff147/regex-2026.7.19-cp313-cp313t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:4896db1f4ce0576765b8272aa922df324e0f5b9bb2c3d03044ff32a7234a9aba", size = 785206, upload-time = "2026-07-19T00:18:20.464Z" }, + { url = "https://files.pythonhosted.org/packages/76/a4/186e410941e731037c01166069ab86da9f65e8f8110c18009ccf4bd623ee/regex-2026.7.19-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:4e6883a021db30511d9fb8cfb0f222ce1f2c369f7d4d8b0448f449a93ba0bdfc", size = 800431, upload-time = "2026-07-19T00:18:22.716Z" }, + { url = "https://files.pythonhosted.org/packages/73/9f/e4e10e023d291d64a33e246610b724493bf1ce98e0e59c9b7c837e5acfb7/regex-2026.7.19-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:09523a592938aa9f587fb74467c63ff0cf88fc3df14c82ab0f0517dcf76aaa62", size = 864906, upload-time = "2026-07-19T00:18:24.772Z" }, + { url = "https://files.pythonhosted.org/packages/24/57/ccb20b6be5f1f52a053d1ba2a8f7a077edb9d918248b8490d7506c6832b3/regex-2026.7.19-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:1ebac3474b8589fce2f9b225b650afd61448f7c73a5d0255a10cc6366471aed1", size = 773559, upload-time = "2026-07-19T00:18:27.008Z" }, + { url = "https://files.pythonhosted.org/packages/a3/82/f3b263cf8fad927dc102891da8502e718b7ff9d19af7a2a07c03865d7188/regex-2026.7.19-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:4a0530bb1b8c1c985e7e2122e2b4d3aedd8a3c21c6bfddae6767c4405668b56e", size = 857739, upload-time = "2026-07-19T00:18:29.107Z" }, + { url = "https://files.pythonhosted.org/packages/47/2e/1687bd1b6c2aed5e672ccf845fc11557821fe7366d921b50889ea5ce57bf/regex-2026.7.19-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:2ef7eeb108c47ce7bcc9513e51bcb1bf57e8f483d52fce68a8642e3527141ae0", size = 804522, upload-time = "2026-07-19T00:18:31.362Z" }, + { url = "https://files.pythonhosted.org/packages/76/7c/cc4e7655181b2d9235b704f2c5e19d8eff002bbc437bae59baee0e381aca/regex-2026.7.19-cp313-cp313t-win32.whl", hash = "sha256:64b6ca7391a1395c2638dd5c7456d67bea44fc6c5e8e92c5dc8aa6a8f23292b4", size = 269141, upload-time = "2026-07-19T00:18:33.479Z" }, + { url = "https://files.pythonhosted.org/packages/bb/14/961b4c7b05a2391c32dbc85e27773076671ef8f97f36cec70fe414734c02/regex-2026.7.19-cp313-cp313t-win_amd64.whl", hash = "sha256:f04b9f56b0e0614c0126be12c2c2d9f8850c1e57af302bd0a63bed379d4af974", size = 280036, upload-time = "2026-07-19T00:18:35.419Z" }, + { url = "https://files.pythonhosted.org/packages/ce/67/795644550d788ddbb6dc458c95895f8009978ea6d6ea76b005eb3f45e8c9/regex-2026.7.19-cp313-cp313t-win_arm64.whl", hash = "sha256:fcee38cd8e5089d6d4f048ba1233b3ad76e5954f545382180889112ff5cb712d", size = 279394, upload-time = "2026-07-19T00:18:37.454Z" }, + { url = "https://files.pythonhosted.org/packages/d2/25/0c4c452f8ef3efe456745b2f33195f5904b573fb4c2ff3f0cb9ec188461e/regex-2026.7.19-cp314-cp314-macosx_10_13_universal2.whl", hash = "sha256:a81758ed242b861b72e778ba34d41366441a2e10b16b472784c88da2dea7e2dd", size = 496750, upload-time = "2026-07-19T00:18:39.633Z" }, + { url = "https://files.pythonhosted.org/packages/24/9e/b70ca6c1704f6c7cd32a9e143c86cc5968d10981eca284bad670c245ea7d/regex-2026.7.19-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:4aa5435cdb3eb6f55fe98a171b05e3fbcd95fadaa4aa32acf62afd9b0cfdbcac", size = 297093, upload-time = "2026-07-19T00:18:41.583Z" }, + { url = "https://files.pythonhosted.org/packages/87/74/0b692da2520d51fbff19c88b83d97e4c702909dd02386c585998b7e2dbed/regex-2026.7.19-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:60be8693a1dadc210bbcbc0db3e26da5f7d01d1d5a3da594e99b4fa42df404f5", size = 292043, upload-time = "2026-07-19T00:18:43.347Z" }, + { url = "https://files.pythonhosted.org/packages/e3/a7/1d478e614016045a33feae57446215f9fd65b665a5ceb2f891fb3183bc52/regex-2026.7.19-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d19662dbedbe783d323196312d38f5ba53cf56296378252171985da6899887d3", size = 797214, upload-time = "2026-07-19T00:18:45.362Z" }, + { url = "https://files.pythonhosted.org/packages/aa/ae/11b9c9411d92c30e3d2db32df5a31133e4a99a8fc397a604fd08f6c4bffb/regex-2026.7.19-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:d15df07081d91b76ff20d43f94592ee110330152d617b730fdbe5ef9fb680053", size = 866433, upload-time = "2026-07-19T00:18:47.315Z" }, + { url = "https://files.pythonhosted.org/packages/b1/62/2b2efc4992f91d6d204b24c647c9f9412e85379d92b7c0ab9fdae622327e/regex-2026.7.19-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:56ad4d9f77df871a99e25c37091052a02528ec0eb059de928ee33956b854b45b", size = 911360, upload-time = "2026-07-19T00:18:49.588Z" }, + { url = "https://files.pythonhosted.org/packages/14/71/986ceea9aa3da548bf1357cad89b63915ec6d21ec957c8113b29ece567df/regex-2026.7.19-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7322ec6cc9fba9d49ab888bb82d67ac5625627aa168f0165139b17018df3fb8a", size = 801275, upload-time = "2026-07-19T00:18:51.767Z" }, + { url = "https://files.pythonhosted.org/packages/15/be/ce9d9534b2cda96eab32c548261224b9b4e220a4126f098f60f42ae7b4cd/regex-2026.7.19-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:9c7472192ebfad53a6be7c4a8bfb2d64b81c0e93a1fc8c57e1dd0b638297b5d1", size = 777131, upload-time = "2026-07-19T00:18:54.053Z" }, + { url = "https://files.pythonhosted.org/packages/61/2b/58b5c710f2c3929515a25f3a1ca0dad0dcd4518d4fff3cf23bc7adb8dcd2/regex-2026.7.19-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:c10b82c2634df08dfb13b1f04e38fe310d086ee092f4f69c0c8da234251e556e", size = 785020, upload-time = "2026-07-19T00:18:56.579Z" }, + { url = "https://files.pythonhosted.org/packages/84/03/5fe091935b74f15fe0f97998c215cae418d1c0413f6258c7d4d2e83aa37f/regex-2026.7.19-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:17ed5692f6acc4183e98331101a5f9e4f64d72fe58b753da4d444a2c77d05b12", size = 861263, upload-time = "2026-07-19T00:18:58.64Z" }, + { url = "https://files.pythonhosted.org/packages/d8/fa/d60bf82e10841eef62a9e32aac401468f05fddfbcb2942e342b1ba3d2433/regex-2026.7.19-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:22a992de9a0d91bda927bf02b94351d737a0302905432c88a53de7c4b9ce62e2", size = 766199, upload-time = "2026-07-19T00:19:00.705Z" }, + { url = "https://files.pythonhosted.org/packages/bf/5d/11e64d151b0662b81d6bf644c74dc118d461df85bdf2577fadbbf751788a/regex-2026.7.19-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:618a0aed532be87294c4477b0481f3aa0f1520f4014a4374dd4cf789b4cd2c97", size = 851317, upload-time = "2026-07-19T00:19:03.015Z" }, + { url = "https://files.pythonhosted.org/packages/7c/34/532efb87488d90807bae6a443d357ee5e2728a478c597619c8aaa17cc0bd/regex-2026.7.19-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:2ce9e679f776649746729b6c86382da519ef649c8e34cc41df0d2e5e0f6c36d4", size = 789557, upload-time = "2026-07-19T00:19:05.338Z" }, + { url = "https://files.pythonhosted.org/packages/d6/90/3a8d5ca977171ec3ae21a71207d2228b2663bde14d7f7ef0e6363ecf9290/regex-2026.7.19-cp314-cp314-win32.whl", hash = "sha256:73f272fba87b8ccfe70a137d02a54af386f6d27aa509fbffdd978f5947aae1aa", size = 272531, upload-time = "2026-07-19T00:19:07.487Z" }, + { url = "https://files.pythonhosted.org/packages/96/e1/8862885e70409de70e8c005f57fb2e7be8d9ef0317250d60f4c9660a300d/regex-2026.7.19-cp314-cp314-win_amd64.whl", hash = "sha256:d721e53758b2cca74990185eb0671dd466d7a388a1a45d0c6f4c13cef41a68ac", size = 280831, upload-time = "2026-07-19T00:19:09.46Z" }, + { url = "https://files.pythonhosted.org/packages/08/82/2693e53e29f9104d9de95d37ce4dd826bd32d5f9c0085d3aa6ac042675c4/regex-2026.7.19-cp314-cp314-win_arm64.whl", hash = "sha256:65fa6cb38ed5e9c3637e68e544f598b39c3b86b808ed0627a67b68320384b459", size = 281099, upload-time = "2026-07-19T00:19:11.398Z" }, + { url = "https://files.pythonhosted.org/packages/92/b7/9a01aa16461a18cde9d7b9c3ab21e501db2ce33725f53014342b91df2b0a/regex-2026.7.19-cp314-cp314t-macosx_10_13_universal2.whl", hash = "sha256:5a2721c8720e2cb3c209925dfb9200199b4b07361c9e01d321719404b21458b3", size = 501121, upload-time = "2026-07-19T00:19:13.425Z" }, + { url = "https://files.pythonhosted.org/packages/f3/5e/bbaeca815dc9191c424c94a4fdc5c87c75748a64a6271821212ebdd4e1a3/regex-2026.7.19-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:199535629f25caf89698039af3d1ad5fcae7f933e2112c73f1cdf49165c99518", size = 299415, upload-time = "2026-07-19T00:19:15.43Z" }, + { url = "https://files.pythonhosted.org/packages/cd/d6/0dd1a321afaab95eb7ff44aa0f637301786f1dc71c6b797b9ed236ed8890/regex-2026.7.19-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:9b60d7814174f059e5de4ab98271cc5ba9259cfea55273a81544dceea32dc8d9", size = 294483, upload-time = "2026-07-19T00:19:17.879Z" }, + { url = "https://files.pythonhosted.org/packages/92/5f/40bacf91d0904f812e13bbbab3864604c463eced8afdc54aeaa50492ea95/regex-2026.7.19-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:dbece16025afda5e3031af0c4059207e61dcf73ef13af844964f57f387d1c435", size = 811833, upload-time = "2026-07-19T00:19:20.102Z" }, + { url = "https://files.pythonhosted.org/packages/94/7c/4902744261f775aeede8b5627314b38482da29cf49a57b66a6fb753246c5/regex-2026.7.19-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:d24ecb4f5e009ea0bd275ee37ad9953b32005e2e5e60f8bbae16da0dbbf0d3a0", size = 871270, upload-time = "2026-07-19T00:19:22.365Z" }, + { url = "https://files.pythonhosted.org/packages/16/70/6980c9be6bf21c0a60ed3e0aea39cf419ecf3b08d1d9947bc56e196ef186/regex-2026.7.19-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:8cae6fd77a5b72dae505084b1a2ee0360139faf72fedbab667cd7cc65aae7a6a", size = 917534, upload-time = "2026-07-19T00:19:24.529Z" }, + { url = "https://files.pythonhosted.org/packages/52/92/8b2bd872782ce8c42691e39acb38eb8efe014e5ddb78ad7d943d6f197ce9/regex-2026.7.19-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9724e6cb5e478cd7d8cabf027826178739cb18cf0e117d0e32814d479fa02276", size = 816135, upload-time = "2026-07-19T00:19:26.919Z" }, + { url = "https://files.pythonhosted.org/packages/de/2d/33a602f657bdc4041f17d79f92ab18261d255d91a06117a6e29df023e5e2/regex-2026.7.19-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:572fc57b0009c735ee56c175ea021b637a15551a312f56734277f923d6fd0f6c", size = 785492, upload-time = "2026-07-19T00:19:29.192Z" }, + { url = "https://files.pythonhosted.org/packages/9e/36/0987cf4cb271680064a70d24a475873775a151d0b7058698a006cb0cae4a/regex-2026.7.19-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:20568e182eb82d39a6bf7cff3fd58566f14c75c6f74b2c8c96537eecf9010e3a", size = 800658, upload-time = "2026-07-19T00:19:31.392Z" }, + { url = "https://files.pythonhosted.org/packages/a8/24/c14f31c135e1ba55fa4f9a58ca98d0842512bf6188230763c31c8f449e3b/regex-2026.7.19-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:1d58561843f0ff7dc78b4c28b5e2dc388f3eff94ebc8a232a3adba961fc00009", size = 865073, upload-time = "2026-07-19T00:19:33.485Z" }, + { url = "https://files.pythonhosted.org/packages/14/85/181a12211f22469f24d2de1ebddfe397d2396e2c29013b9a58134a91069a/regex-2026.7.19-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:61bb1bd45520aacd56dd80943bd34991fb5350afdd1f36f2282230fd5154a218", size = 773684, upload-time = "2026-07-19T00:19:35.599Z" }, + { url = "https://files.pythonhosted.org/packages/23/58/bd1a0c1a62251366f8d21f41b1ea3c76994962071b8b6ea42f72d505c0f0/regex-2026.7.19-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:cd3584591ea4429026cdb931b054342c2bcf189b44ff367f8d5c15bc092a2966", size = 857769, upload-time = "2026-07-19T00:19:37.738Z" }, + { url = "https://files.pythonhosted.org/packages/e4/4f/f7e2dad6756b2fe1fe75dd90a628c3b45f249d39f948dd90cd2476325417/regex-2026.7.19-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:5cc26a66e212fa5d6c6170c3a40d99d888db3020c6fdab1523250d4341382e44", size = 804546, upload-time = "2026-07-19T00:19:40.229Z" }, + { url = "https://files.pythonhosted.org/packages/2b/d7/01d31d5bdb09bc026fab77f59a371fdf8f9b292e4810546c56182ca70498/regex-2026.7.19-cp314-cp314t-win32.whl", hash = "sha256:2c4e61e2e1be56f63ec3cc618aa9e0de81ef6f43d177205451840022e24f5b78", size = 274526, upload-time = "2026-07-19T00:19:42.398Z" }, + { url = "https://files.pythonhosted.org/packages/52/0e/cea4ce73bc0a8247a0748228ae6669984c7e1f8134b6fa66e59c0572e0ea/regex-2026.7.19-cp314-cp314t-win_amd64.whl", hash = "sha256:c639ea314df70a7b2811e8020448c75af8c9445f5a60f8a4ced81c306a9380c2", size = 283763, upload-time = "2026-07-19T00:19:44.644Z" }, + { url = "https://files.pythonhosted.org/packages/6f/b6/26e41975febae63b7a6e3e02f32cff6cff2e4f10d19c929082f56aebf7c6/regex-2026.7.19-cp314-cp314t-win_arm64.whl", hash = "sha256:9a15e785f244f3e07847b984ce8773fc3da10a9f3c131cc49a4c5b4d672b4547", size = 283451, upload-time = "2026-07-19T00:19:46.639Z" }, +] + [[package]] name = "requests" version = "2.34.2" @@ -1278,6 +1922,111 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/a0/f4/c67b0b3f1b9245e8d266f0f112c500d50e5b4e83cb6f3b71b6528104182a/requests-2.34.2-py3-none-any.whl", hash = "sha256:2a0d60c172f83ac6ab31e4554906c0f3b3588d37b5cb939b1c061f4907e278e0", size = 73075, upload-time = "2026-05-14T19:25:26.443Z" }, ] +[[package]] +name = "rich" +version = "15.0.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "markdown-it-py" }, + { name = "pygments" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/c0/8f/0722ca900cc807c13a6a0c696dacf35430f72e0ec571c4275d2371fca3e9/rich-15.0.0.tar.gz", hash = "sha256:edd07a4824c6b40189fb7ac9bc4c52536e9780fbbfbddf6f1e2502c31b068c36", size = 230680, upload-time = "2026-04-12T08:24:00.75Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/82/3b/64d4899d73f91ba49a8c18a8ff3f0ea8f1c1d75481760df8c68ef5235bf5/rich-15.0.0-py3-none-any.whl", hash = "sha256:33bd4ef74232fb73fe9279a257718407f169c09b78a87ad3d296f548e27de0bb", size = 310654, upload-time = "2026-04-12T08:24:02.83Z" }, +] + +[[package]] +name = "safetensors" +version = "0.8.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/45/06/f955dbbb1859e3bd23c8ac6141af5106e7ad5fedec4a3a6e3d60f94b7001/safetensors-0.8.0.tar.gz", hash = "sha256:fabaf3e0f18a6618d9b36560682562157f77c2b71fcffc7b432be2baed9d753d", size = 325846, upload-time = "2026-06-09T07:52:25.563Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/39/a0/f718cda65b05407d228f97602cf60dca269c979867aa5beb25410de26cd3/safetensors-0.8.0-cp310-abi3-macosx_10_12_x86_64.whl", hash = "sha256:c554f85858e05226d3c2828e32395e677434685d6d94594a41643361c5e837f0", size = 473568, upload-time = "2026-06-09T07:52:18.829Z" }, + { url = "https://files.pythonhosted.org/packages/f5/b1/fa7c600e7dceae12e9606c7578cbc9ff1e1ed55844883ee5c92205e86226/safetensors-0.8.0-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:c80201d22cbf405b80647a60ada77bba06c8fba2da2743ba1e89cdcc39a81f25", size = 484562, upload-time = "2026-06-09T07:52:17.518Z" }, + { url = "https://files.pythonhosted.org/packages/09/7d/65a7de0af421317bb36a067241e4235fff194eed60b961ed6d3f59a3fc60/safetensors-0.8.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:7a46e5ff292c356d6991e60942ba7f79817682d3a2cef0702136448cb9c4d235", size = 502844, upload-time = "2026-06-09T07:52:07.624Z" }, + { url = "https://files.pythonhosted.org/packages/91/4f/3175c9d75634e0e0dda0082794193521035edd7c70a6f212bf33ca06ddf4/safetensors-0.8.0-cp310-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:4124502b78f03534117c848f87a39b8f31e577b15eff423bf8bfb95f2a8c30d0", size = 511823, upload-time = "2026-06-09T07:52:09.565Z" }, + { url = "https://files.pythonhosted.org/packages/20/87/846c289e7aa2299eff406335717cf43ce8777194ece8aad75772e0411615/safetensors-0.8.0-cp310-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:7bc0a787ba8a35be368ee3574edfa2b1ad389eebd0a72e482ae275490e3f6c98", size = 633461, upload-time = "2026-06-09T07:52:11.128Z" }, + { url = "https://files.pythonhosted.org/packages/76/22/8d64d9df2c45d5ded401df889d0ad90882804ca172d79ec4f0df8f727fe0/safetensors-0.8.0-cp310-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:040070828e36dc8e122178bbbd5830ff9e97920affb84cbe0f46442497bed358", size = 545148, upload-time = "2026-06-09T07:52:13.603Z" }, + { url = "https://files.pythonhosted.org/packages/28/50/f203ff3a3ddfe19308efc83c5a3a29ed02bf786732ec35e68bf9162f3365/safetensors-0.8.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:fd6f3f93c9a0a7cc2788ee63fb763353d4bd2e89b0751bc78fcf7dda00bea774", size = 516040, upload-time = "2026-06-09T07:52:16.29Z" }, + { url = "https://files.pythonhosted.org/packages/46/fb/cdaed17ceb2948784fd9c36b6fd3e951b608547cea81a48e8ee6f8cfdfcb/safetensors-0.8.0-cp310-abi3-manylinux_2_31_riscv64.whl", hash = "sha256:fcdd41ec4628fee5799f807c73c353629130fbd942aa23d83c623dd6c9d52d78", size = 513832, upload-time = "2026-06-09T07:52:12.37Z" }, + { url = "https://files.pythonhosted.org/packages/0d/49/1e15de264dcc3b77943d2d0c56a95809956883b1c2d6d585c792523f180b/safetensors-0.8.0-cp310-abi3-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:8e9f537aa183a38ace122d27303dcd986b26bd2a7591f9181d7f0c396f4677ca", size = 559930, upload-time = "2026-06-09T07:52:14.743Z" }, + { url = "https://files.pythonhosted.org/packages/2a/43/bf38443278eab4b1be1fce2931e2b012ad9cb7df52ada751d0aab8f7659a/safetensors-0.8.0-cp310-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:87eec7ffed2b809f05a398a8becb7d013f19f7837cd15d9748580d6cf30dbaf4", size = 678670, upload-time = "2026-06-09T07:52:20.032Z" }, + { url = "https://files.pythonhosted.org/packages/72/e3/68cd3fa5b48488e84add63e04cb12f3bc28ae4638c06d4508c6e88823d0e/safetensors-0.8.0-cp310-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:4a95ae2b05d7726d751da4ebf626a2ca782b706e101bd894c95bc2450b1cffcc", size = 786679, upload-time = "2026-06-09T07:52:21.322Z" }, + { url = "https://files.pythonhosted.org/packages/29/4b/1c19c509d56e01f4fbb3d0a2e597450f6cc04d1d56cf52defb0a62dfd715/safetensors-0.8.0-cp310-abi3-musllinux_1_2_i686.whl", hash = "sha256:3ae091f16662658bdc019a4ff6cb4c085bb7d725eb5978b183ffd265863b6d2d", size = 765683, upload-time = "2026-06-09T07:52:22.594Z" }, + { url = "https://files.pythonhosted.org/packages/27/43/41c1621732edd934d868a00d1b891584c892a7b62a9aab82ea5a0a5623ee/safetensors-0.8.0-cp310-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:8e080062fcde23be189565e1c3305d16751a218ecf9412c8601e64204eb6f846", size = 722361, upload-time = "2026-06-09T07:52:23.924Z" }, + { url = "https://files.pythonhosted.org/packages/8e/3f/73ccf82579412b4a71c4ca673f10b5f1f888d7cf5af7fe24f27d30307be4/safetensors-0.8.0-cp310-abi3-win32.whl", hash = "sha256:2ddf52eac562eda224f99acfa7889d02968c1fd59a5b011ae7d8137c37e9c02d", size = 342401, upload-time = "2026-06-09T07:52:28.895Z" }, + { url = "https://files.pythonhosted.org/packages/1b/6d/3fba214c1e5e0f69991677ec3bc17023f0421776975e1de0c682dca475e2/safetensors-0.8.0-cp310-abi3-win_amd64.whl", hash = "sha256:096ec1a98435df7beb08853bb5aa9081a84f23d0adc67ed1a0a10550f608373f", size = 355540, upload-time = "2026-06-09T07:52:27.832Z" }, + { url = "https://files.pythonhosted.org/packages/8d/fc/7eedc3510d97878876e32774eebbeb61c43f148a96e915c84229a3e967aa/safetensors-0.8.0-cp310-abi3-win_arm64.whl", hash = "sha256:f7838e5135a406ad3e02efdcb8cf2e5397d368b0154537c4fec682dbc544d452", size = 340500, upload-time = "2026-06-09T07:52:26.745Z" }, +] + +[[package]] +name = "sentencepiece" +version = "0.2.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/cc/33/ea3cb3839607eb175da835244a798f797f478c5ddf0e8ecdf57ea85a4c70/sentencepiece-0.2.2.tar.gz", hash = "sha256:3d2b5e824b5622038dc7b490897efe05ebbbb9e7350fc142f3ecc8789ef9bdf6", size = 8218435, upload-time = "2026-07-12T08:39:34.701Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/20/31/f23a2efaa0210b883574001b88fa64e499f798f0848a0b610fb9b384d162/sentencepiece-0.2.2-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:69e9dc8078e128286ed3b975e37c837ba96e215a50c3ef9f3f8b7ab9e5a832a0", size = 2184255, upload-time = "2026-07-12T08:38:14.855Z" }, + { url = "https://files.pythonhosted.org/packages/96/f2/1ee0ccb772d71e822f625d6cb5f0ea825835e877f28a9ef299a1291df19e/sentencepiece-0.2.2-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:6dd76f3e5c8b2eb8a3a3efee787bbf5b9a66e52a048fe09cab85eca33fec6790", size = 1438545, upload-time = "2026-07-12T08:38:16.674Z" }, + { url = "https://files.pythonhosted.org/packages/2a/92/3a6ea4a2c6dd9e7062698a5a33534ca0e20844883338ae9c6b9c122c1a9f/sentencepiece-0.2.2-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:443ac618c7a2a1377cf5c82581fbb849591d14e656d5e5a3e4682d4e36a34e4e", size = 1346997, upload-time = "2026-07-12T08:38:18.499Z" }, + { url = "https://files.pythonhosted.org/packages/f3/3a/7839048997c7bc0c34c57526f539f835e20c7a57dc2a99f99579b11cdbef/sentencepiece-0.2.2-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0e2aae42960392d6dcb9a72d8e1e65a97294c965071b43c7b3429a42f350250e", size = 1324282, upload-time = "2026-07-12T08:38:20.342Z" }, + { url = "https://files.pythonhosted.org/packages/06/5f/9117bf854aef817ad0d0ee9310eed0308a7e529e7eaf2e80ad9cd281ef82/sentencepiece-0.2.2-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1416b92f2f010333786fe6306ed2631121d5ea492219b0841e967b6765e64107", size = 1394242, upload-time = "2026-07-12T08:38:22.976Z" }, + { url = "https://files.pythonhosted.org/packages/ab/62/9e2569867e3dcff7ad6d89642a9615b9801b5cd698abe7df3b490361f66e/sentencepiece-0.2.2-cp311-cp311-win_amd64.whl", hash = "sha256:70d4ca6f4d06df7f0ccab6fe4f49c8a712c8c8b6847b4f0af9a0e1dbb0e0337e", size = 1246268, upload-time = "2026-07-12T08:38:24.857Z" }, + { url = "https://files.pythonhosted.org/packages/96/c9/5d781d4ef1124564a45c98b9ff25d531c10cdf568ec6314a2d1946f9251c/sentencepiece-0.2.2-cp311-cp311-win_arm64.whl", hash = "sha256:252908153eeec06c3ca3a32077e64a49d572e3d89881475b4e0f02d99d9fcc7c", size = 1190702, upload-time = "2026-07-12T08:38:26.789Z" }, + { url = "https://files.pythonhosted.org/packages/b8/13/7a562289c8d5b49ebdf3f9c1e8ab67cf14a8743b1d90c8f406bfdec36b72/sentencepiece-0.2.2-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:1edb10e520e4bddf74d85b0f5ae74cc2d60c2b448885080bfb618bc2b3a49f6b", size = 2188384, upload-time = "2026-07-12T08:38:28.486Z" }, + { url = "https://files.pythonhosted.org/packages/85/d1/912f14fd5eae168aba726ffb6a9a2dc1c71fe7676c53da6f5c442b886d4a/sentencepiece-0.2.2-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:f7c06c751c19d923435a54bff4f7e66e728fad160e8da28254f133abc9725820", size = 1441553, upload-time = "2026-07-12T08:38:30.552Z" }, + { url = "https://files.pythonhosted.org/packages/bd/44/caa9cab5f261a019e2808bc5046152775dc57352ba9cbae7525e9e7a1ed4/sentencepiece-0.2.2-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:38111ed1f79268f399c505028023d5eaaf0ab4e5eafceb709468b0d3323e7838", size = 1347176, upload-time = "2026-07-12T08:38:32.211Z" }, + { url = "https://files.pythonhosted.org/packages/19/90/cd798935668cff71d309d8ff10385844ecf216b1fe454f1993ed8bf2cb91/sentencepiece-0.2.2-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:cbce24284f51f71d10a42b7b9c964dcb9048b28f1c8e5db40bcbcb6f428cba6a", size = 1325200, upload-time = "2026-07-12T08:38:33.689Z" }, + { url = "https://files.pythonhosted.org/packages/b6/2d/37e3da037318a70066ded0d51bc2a7f35491ae6338dd993d5eb1503fc3b5/sentencepiece-0.2.2-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c8a168b040bc61681293f79a949b5d911c8e25086f4260285b8d97ab5f1195da", size = 1397736, upload-time = "2026-07-12T08:38:35.771Z" }, + { url = "https://files.pythonhosted.org/packages/8d/11/753fca2e6b109be3ab7867abf357dfe48677fe726ae5a5363d0b54ca9450/sentencepiece-0.2.2-cp312-cp312-win_amd64.whl", hash = "sha256:7c6e7bf684dc12145bfa685d3060beaea55139134ba848289bee514ed42e7383", size = 1248030, upload-time = "2026-07-12T08:38:37.604Z" }, + { url = "https://files.pythonhosted.org/packages/e2/0a/70efbe861ca182d7d4b6e1a20f58e043400848fa9f2915229f082e221648/sentencepiece-0.2.2-cp312-cp312-win_arm64.whl", hash = "sha256:76ff5814db72e7462dece042d7593cdf102b8ec82c2b1cc201a2add34ee3050d", size = 1187325, upload-time = "2026-07-12T08:38:39.348Z" }, + { url = "https://files.pythonhosted.org/packages/b9/a3/b3b05095c174d6e80d37d5ddc2f57c2c56237333e7bbd6079cf3243c2a8a/sentencepiece-0.2.2-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:77c3ce990b23441e5ecfa5bce181fd6f408b564aeb6d7e1d1e7de9c5612501c8", size = 2188346, upload-time = "2026-07-12T08:38:41.089Z" }, + { url = "https://files.pythonhosted.org/packages/ca/f3/72ebc4acb10a06bcf7503fbc6091c8f5db68300f6aac4356c09e6c76e0e1/sentencepiece-0.2.2-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:fd523c4992041faa5c2b3cde62253d11a96c30d73a34afe48a486e8e2254cd1c", size = 1441434, upload-time = "2026-07-12T08:38:42.56Z" }, + { url = "https://files.pythonhosted.org/packages/34/db/f9ea1a6844b4fa5dfe2312095cd866a1f724cd0905054ab9d5991778ba50/sentencepiece-0.2.2-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:201a8e0f55501a76e08dbf2c54bc45f4642b379271e89c667d517bfbc2191f2a", size = 1347267, upload-time = "2026-07-12T08:38:44.389Z" }, + { url = "https://files.pythonhosted.org/packages/32/4f/31c1073314ad94466bca37d29581761d70110237ee3d46b0efece59a8c1e/sentencepiece-0.2.2-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8eed98514bffe5ecac37f493f91869c351fbb05629328bfdbc08502c6c094dc0", size = 1324980, upload-time = "2026-07-12T08:38:46.304Z" }, + { url = "https://files.pythonhosted.org/packages/59/b4/a0356fa04d6a14337a6e0e443556785a0422c53ec58baae6b9568120eb0f/sentencepiece-0.2.2-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:64b656f025355cf8c51abe9fbe3848540756c6d7ca5e6791b1afa664bc24c7cb", size = 1397593, upload-time = "2026-07-12T08:38:48.302Z" }, + { url = "https://files.pythonhosted.org/packages/09/fa/d2d6369257fd2f0de616b1c7110b73fab409ef61b14f1b9e0010ed325914/sentencepiece-0.2.2-cp313-cp313-win_amd64.whl", hash = "sha256:74f0ee601047c0c12a783088b51be4e6214a62ecd9e02278c477433cd16e0ed9", size = 1247987, upload-time = "2026-07-12T08:38:50.15Z" }, + { url = "https://files.pythonhosted.org/packages/17/ee/2bb594da6fd95e32f29057f1aa7fa996701b8980090923c2d8711fdc0a24/sentencepiece-0.2.2-cp313-cp313-win_arm64.whl", hash = "sha256:b23fe17779834d3c27aaf2edac9486d04cca1a7deb8f5facda35150ac6263a91", size = 1187250, upload-time = "2026-07-12T08:38:52.246Z" }, + { url = "https://files.pythonhosted.org/packages/58/9c/dfc82846460e7a712310f5613f23d8b553cabb4e2e648663c11d8382af56/sentencepiece-0.2.2-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:72b7825b331b1b7e7c45be2e674b3e3c65af608fa376bad2d851b20aaf0cdc78", size = 2223080, upload-time = "2026-07-12T08:38:54.391Z" }, + { url = "https://files.pythonhosted.org/packages/8d/4e/3ff12cebe6d31662d9ceeabfb282de20bd0d6098fa282b4a3b8305abc7e8/sentencepiece-0.2.2-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:d795c4ac689a57f9d4ba2288126ec7901d389ad5827d2f8b8533c883974fe563", size = 1458511, upload-time = "2026-07-12T08:38:56.811Z" }, + { url = "https://files.pythonhosted.org/packages/59/5a/16d51d05360be4cee3ebfe4837c184054c4eed16cabaeb3b039524e9a000/sentencepiece-0.2.2-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:3ab3f1ae98970b5590e2209341522718900ba19bcc2c207ffaa6bd417ad960c5", size = 1361138, upload-time = "2026-07-12T08:38:58.808Z" }, + { url = "https://files.pythonhosted.org/packages/0f/af/c30ee2a9f99d51db9844acaa8fa0b611a97c2fa7116646fa43db3300b187/sentencepiece-0.2.2-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3ec27c152a1f1b24bc9168b55a5880f3c16e2334e697da6f55a1046a22405a3d", size = 1328625, upload-time = "2026-07-12T08:39:00.849Z" }, + { url = "https://files.pythonhosted.org/packages/3e/1a/4c6b39d03f5ba8439509adbd5a23c9538088a3cb679e7a47b911e8442bc6/sentencepiece-0.2.2-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:59d6588712101ccfcae9b03692be3aaae1514c2078666d7b05f15ba3a702e41b", size = 1398595, upload-time = "2026-07-12T08:39:02.86Z" }, + { url = "https://files.pythonhosted.org/packages/0f/bc/9eedddcec1fd57bc70200fa3ebf792d18fa63527a5369581cd416c81f97f/sentencepiece-0.2.2-cp313-cp313t-win_amd64.whl", hash = "sha256:89625fb43765cccaa1443b9adb61f283e5fe4cb1536728205d06bada730caa53", size = 1259346, upload-time = "2026-07-12T08:39:04.559Z" }, + { url = "https://files.pythonhosted.org/packages/41/15/7e74c8533848866ff560b29f7d8719921b76c4ec7149592d6d28e0deee75/sentencepiece-0.2.2-cp313-cp313t-win_arm64.whl", hash = "sha256:4f0603267cd15b92b68c2c0e852a441507614b70dc7773659baa6b8c214a91fd", size = 1196596, upload-time = "2026-07-12T08:39:06.454Z" }, + { url = "https://files.pythonhosted.org/packages/0b/7e/f5df63edb6bcb46c1343cfa5d9192d73a4eb61af2e800d9402efff387523/sentencepiece-0.2.2-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:c62bd361cec1f5b556eb8210264ecfff37486cd990c3386cc00310f26c54090a", size = 2190240, upload-time = "2026-07-12T08:39:08.178Z" }, + { url = "https://files.pythonhosted.org/packages/52/0a/095d183b453b2a2e20b016829029c58eca90adc1c9911113e5d26fff45ed/sentencepiece-0.2.2-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:46ba07b543add034de0ff47ac5f907e9a06682f91d85121a972764628933be6b", size = 1442220, upload-time = "2026-07-12T08:39:09.91Z" }, + { url = "https://files.pythonhosted.org/packages/d1/18/823954c9c90e74eba09fb96752dc37a5555df00d69866cb9406d1725dc7e/sentencepiece-0.2.2-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:79bac5a251f23a7341e28fda9ce0d5319edf45328239ce037c0682936f137906", size = 1348056, upload-time = "2026-07-12T08:39:11.744Z" }, + { url = "https://files.pythonhosted.org/packages/10/ca/1b6c251321901cbf8a2d2e48b8b70eb82a449011b766af52a228d0a90b6b/sentencepiece-0.2.2-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1402d8ee36f0d851cea8eee4dbb85fea14643b7503cf4d00d102eec0fe3ca719", size = 1325463, upload-time = "2026-07-12T08:39:13.413Z" }, + { url = "https://files.pythonhosted.org/packages/24/b3/718847349da7b25c8220ed86d85b89080af94740b2d87a59198104ae5c51/sentencepiece-0.2.2-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8d44b20234905ff022b7d535f79d1f823ad7670c9851cc4f03cdc34787cdb3ab", size = 1398138, upload-time = "2026-07-12T08:39:15.564Z" }, + { url = "https://files.pythonhosted.org/packages/33/fe/4906f12c458274edd96387e4baaad7c6f064a2b7c11a1cc2401c8a7bd483/sentencepiece-0.2.2-cp314-cp314-win_amd64.whl", hash = "sha256:63250cfab8b80a1ef82a614eb2b3cadfec2c405f870cedc139d08e2f063eb708", size = 1356144, upload-time = "2026-07-12T08:39:17.313Z" }, + { url = "https://files.pythonhosted.org/packages/d3/eb/22f89b6542aba400b0007cf0b1697cc3f99be8fb682fdb4c05eec450e33f/sentencepiece-0.2.2-cp314-cp314-win_arm64.whl", hash = "sha256:65d84ec36888de4a848eee5f910e67fbc79b064685ef1e10a502e14520ead9c9", size = 1294351, upload-time = "2026-07-12T08:39:18.967Z" }, + { url = "https://files.pythonhosted.org/packages/84/c4/7afe8c2315b76e46818851a057e50a378a0382aa00b970a1fa444181b6f6/sentencepiece-0.2.2-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:d254c98ca6387655400b3959c33c83efd807f5edeb608e3aca45800ceaa77151", size = 2223281, upload-time = "2026-07-12T08:39:20.978Z" }, + { url = "https://files.pythonhosted.org/packages/98/42/fb678e472c554ef086be6375d20060ca610a2c4218854d4c091001fc6f91/sentencepiece-0.2.2-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:3fd9ce2ab4460c713cfdeb4aca693ca6732a11538e05fb332d5af42e3d7fde25", size = 1458779, upload-time = "2026-07-12T08:39:22.812Z" }, + { url = "https://files.pythonhosted.org/packages/78/52/ffe402b13bce1889228a98dc6cd86ae8afac1112362236be3468be784441/sentencepiece-0.2.2-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:7fc14c1585139fa6b68775e616a6b90cf622ebf219f9558c0aeaf5d253ee6c9b", size = 1361736, upload-time = "2026-07-12T08:39:24.602Z" }, + { url = "https://files.pythonhosted.org/packages/78/4a/2288f60e7283583ec0a0f16e72f9c8e68557d7e7a4b585d2cda4f9f47e64/sentencepiece-0.2.2-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:df88b0c34f2fa909d322f7b06b1398e1e81af4b2f42a7b8e3556f928b25d1811", size = 1328155, upload-time = "2026-07-12T08:39:26.422Z" }, + { url = "https://files.pythonhosted.org/packages/26/31/5dd6882ebe899f741a5cfe40ff56c6efc06bc26ee287abdb723b671f409c/sentencepiece-0.2.2-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:3f5851441ab1ef8634963a5100b733a8bbeefe623e0c5c005b1f1f3880e574cf", size = 1398307, upload-time = "2026-07-12T08:39:28.637Z" }, + { url = "https://files.pythonhosted.org/packages/da/05/7d7780fa63f4b8c1821953b916e25f89ae8f14d4da6ba91e10f6d06dc2b4/sentencepiece-0.2.2-cp314-cp314t-win_amd64.whl", hash = "sha256:046b15ea22d8042e2e173561d464ec3b64a9c2081324df70ebce7bf7ebb3e497", size = 1367133, upload-time = "2026-07-12T08:39:30.546Z" }, + { url = "https://files.pythonhosted.org/packages/49/a1/70007fef3f818c688de4a730f98024a671599ab67f20270f8efb03d69dcc/sentencepiece-0.2.2-cp314-cp314t-win_arm64.whl", hash = "sha256:fa9f5ef0e2a82233dd0b8b32ea3f5710e0c44afbc07ed3620219f32601e56090", size = 1302760, upload-time = "2026-07-12T08:39:32.457Z" }, +] + +[[package]] +name = "setuptools" +version = "84.0.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/6d/44/f5da03a8ef95d369145c5bb53050e7877c9f3d312e128605fd9504829143/setuptools-84.0.0.tar.gz", hash = "sha256:f4695c21257f0d9b537ec2692c941d02ee143b7cc1276941349a546573b2ef73", size = 1168449, upload-time = "2026-08-08T18:27:58.365Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/95/9c/c510029fc6ef33a6275cd2c5d3cecd6613dfd6aa401d57c54f1c18852ccf/setuptools-84.0.0-py3-none-any.whl", hash = "sha256:51a52592b3b99e102b609654876bd65f19f999935166d1352678931132b0c670", size = 818216, upload-time = "2026-08-08T18:27:56.719Z" }, +] + +[[package]] +name = "shellingham" +version = "1.5.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/58/15/8b3609fd3830ef7b27b655beb4b4e9c62313a4e8da8c676e142cc210d58e/shellingham-1.5.4.tar.gz", hash = "sha256:8dbca0739d487e5bd35ab3ca4b36e11c4078f3a234bfce294b0a0291363404de", size = 10310, upload-time = "2023-10-24T04:13:40.426Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e0/f9/0595336914c5619e5f28a1fb793285925a8cd4b432c9da0a987836c7f822/shellingham-1.5.4-py2.py3-none-any.whl", hash = "sha256:7ecfff8f2fd72616f7481040475a65b2bf8af90a56c89140852d1120324e8686", size = 9755, upload-time = "2023-10-24T04:13:38.866Z" }, +] + [[package]] name = "sniffio" version = "1.3.1" @@ -1287,6 +2036,18 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/e9/44/75a9c9421471a6c4805dbf2356f7c181a29c1879239abab1ea2cc8f38b40/sniffio-1.3.1-py3-none-any.whl", hash = "sha256:2f6da418d1f1e0fddd844478f41680e794e6051915791a034ff65e5f100525a2", size = 10235, upload-time = "2024-02-25T23:20:01.196Z" }, ] +[[package]] +name = "sympy" +version = "1.14.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "mpmath" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/83/d3/803453b36afefb7c2bb238361cd4ae6125a569b4db67cd9e79846ba2d68c/sympy-1.14.0.tar.gz", hash = "sha256:d3d3fe8df1e5a0b42f0e7bdf50541697dbe7d23746e894990c030e2b05e72517", size = 7793921, upload-time = "2025-04-27T18:05:01.611Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a2/09/77d55d46fd61b4a135c444fc97158ef34a095e5681d0a6c10b75bf356191/sympy-1.14.0-py3-none-any.whl", hash = "sha256:e091cc3e99d2141a0ba2847328f5479b05d94a6635cb96148ccb3f34671bd8f5", size = 6299353, upload-time = "2025-04-27T18:04:59.103Z" }, +] + [[package]] name = "tenacity" version = "9.1.4" @@ -1296,6 +2057,125 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/d7/c1/eb8f9debc45d3b7918a32ab756658a0904732f75e555402972246b0b8e71/tenacity-9.1.4-py3-none-any.whl", hash = "sha256:6095a360c919085f28c6527de529e76a06ad89b23659fa881ae0649b867a9d55", size = 28926, upload-time = "2026-02-07T10:45:32.24Z" }, ] +[[package]] +name = "tokenizers" +version = "0.22.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "huggingface-hub" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/73/6f/f80cfef4a312e1fb34baf7d85c72d4411afde10978d4657f8cdd811d3ccc/tokenizers-0.22.2.tar.gz", hash = "sha256:473b83b915e547aa366d1eee11806deaf419e17be16310ac0a14077f1e28f917", size = 372115, upload-time = "2026-01-05T10:45:15.988Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/92/97/5dbfabf04c7e348e655e907ed27913e03db0923abb5dfdd120d7b25630e1/tokenizers-0.22.2-cp39-abi3-macosx_10_12_x86_64.whl", hash = "sha256:544dd704ae7238755d790de45ba8da072e9af3eea688f698b137915ae959281c", size = 3100275, upload-time = "2026-01-05T10:41:02.158Z" }, + { url = "https://files.pythonhosted.org/packages/2e/47/174dca0502ef88b28f1c9e06b73ce33500eedfac7a7692108aec220464e7/tokenizers-0.22.2-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:1e418a55456beedca4621dbab65a318981467a2b188e982a23e117f115ce5001", size = 2981472, upload-time = "2026-01-05T10:41:00.276Z" }, + { url = "https://files.pythonhosted.org/packages/d6/84/7990e799f1309a8b87af6b948f31edaa12a3ed22d11b352eaf4f4b2e5753/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:2249487018adec45d6e3554c71d46eb39fa8ea67156c640f7513eb26f318cec7", size = 3290736, upload-time = "2026-01-05T10:40:32.165Z" }, + { url = "https://files.pythonhosted.org/packages/78/59/09d0d9ba94dcd5f4f1368d4858d24546b4bdc0231c2354aa31d6199f0399/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:25b85325d0815e86e0bac263506dd114578953b7b53d7de09a6485e4a160a7dd", size = 3168835, upload-time = "2026-01-05T10:40:38.847Z" }, + { url = "https://files.pythonhosted.org/packages/47/50/b3ebb4243e7160bda8d34b731e54dd8ab8b133e50775872e7a434e524c28/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:bfb88f22a209ff7b40a576d5324bf8286b519d7358663db21d6246fb17eea2d5", size = 3521673, upload-time = "2026-01-05T10:40:56.614Z" }, + { url = "https://files.pythonhosted.org/packages/e0/fa/89f4cb9e08df770b57adb96f8cbb7e22695a4cb6c2bd5f0c4f0ebcf33b66/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:1c774b1276f71e1ef716e5486f21e76333464f47bece56bbd554485982a9e03e", size = 3724818, upload-time = "2026-01-05T10:40:44.507Z" }, + { url = "https://files.pythonhosted.org/packages/64/04/ca2363f0bfbe3b3d36e95bf67e56a4c88c8e3362b658e616d1ac185d47f2/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:df6c4265b289083bf710dff49bc51ef252f9d5be33a45ee2bed151114a56207b", size = 3379195, upload-time = "2026-01-05T10:40:51.139Z" }, + { url = "https://files.pythonhosted.org/packages/2e/76/932be4b50ef6ccedf9d3c6639b056a967a86258c6d9200643f01269211ca/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:369cc9fc8cc10cb24143873a0d95438bb8ee257bb80c71989e3ee290e8d72c67", size = 3274982, upload-time = "2026-01-05T10:40:58.331Z" }, + { url = "https://files.pythonhosted.org/packages/1d/28/5f9f5a4cc211b69e89420980e483831bcc29dade307955cc9dc858a40f01/tokenizers-0.22.2-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:29c30b83d8dcd061078b05ae0cb94d3c710555fbb44861139f9f83dcca3dc3e4", size = 9478245, upload-time = "2026-01-05T10:41:04.053Z" }, + { url = "https://files.pythonhosted.org/packages/6c/fb/66e2da4704d6aadebf8cb39f1d6d1957df667ab24cff2326b77cda0dcb85/tokenizers-0.22.2-cp39-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:37ae80a28c1d3265bb1f22464c856bd23c02a05bb211e56d0c5301a435be6c1a", size = 9560069, upload-time = "2026-01-05T10:45:10.673Z" }, + { url = "https://files.pythonhosted.org/packages/16/04/fed398b05caa87ce9b1a1bb5166645e38196081b225059a6edaff6440fac/tokenizers-0.22.2-cp39-abi3-musllinux_1_2_i686.whl", hash = "sha256:791135ee325f2336f498590eb2f11dc5c295232f288e75c99a36c5dbce63088a", size = 9899263, upload-time = "2026-01-05T10:45:12.559Z" }, + { url = "https://files.pythonhosted.org/packages/05/a1/d62dfe7376beaaf1394917e0f8e93ee5f67fea8fcf4107501db35996586b/tokenizers-0.22.2-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:38337540fbbddff8e999d59970f3c6f35a82de10053206a7562f1ea02d046fa5", size = 10033429, upload-time = "2026-01-05T10:45:14.333Z" }, + { url = "https://files.pythonhosted.org/packages/fd/18/a545c4ea42af3df6effd7d13d250ba77a0a86fb20393143bbb9a92e434d4/tokenizers-0.22.2-cp39-abi3-win32.whl", hash = "sha256:a6bf3f88c554a2b653af81f3204491c818ae2ac6fbc09e76ef4773351292bc92", size = 2502363, upload-time = "2026-01-05T10:45:20.593Z" }, + { url = "https://files.pythonhosted.org/packages/65/71/0670843133a43d43070abeb1949abfdef12a86d490bea9cd9e18e37c5ff7/tokenizers-0.22.2-cp39-abi3-win_amd64.whl", hash = "sha256:c9ea31edff2968b44a88f97d784c2f16dc0729b8b143ed004699ebca91f05c48", size = 2747786, upload-time = "2026-01-05T10:45:18.411Z" }, + { url = "https://files.pythonhosted.org/packages/72/f4/0de46cfa12cdcbcd464cc59fde36912af405696f687e53a091fb432f694c/tokenizers-0.22.2-cp39-abi3-win_arm64.whl", hash = "sha256:9ce725d22864a1e965217204946f830c37876eee3b2ba6fc6255e8e903d5fcbc", size = 2612133, upload-time = "2026-01-05T10:45:17.232Z" }, +] + +[[package]] +name = "torch" +version = "2.13.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "cuda-bindings", marker = "python_full_version < '3.15' and sys_platform == 'linux'" }, + { name = "cuda-toolkit", extra = ["cublas", "cudart", "cufft", "cufile", "cupti", "curand", "cusolver", "cusparse", "nvjitlink", "nvrtc", "nvtx"], marker = "sys_platform == 'linux'" }, + { name = "filelock" }, + { name = "fsspec" }, + { name = "jinja2" }, + { name = "networkx" }, + { name = "nvidia-cudnn-cu13", marker = "sys_platform == 'linux'" }, + { name = "nvidia-cusparselt-cu13", marker = "sys_platform == 'linux'" }, + { name = "nvidia-nccl-cu13", marker = "sys_platform == 'linux'" }, + { name = "nvidia-nvshmem-cu13", marker = "sys_platform == 'linux'" }, + { name = "setuptools" }, + { name = "sympy" }, + { name = "triton", marker = "python_full_version < '3.15' and sys_platform == 'linux'" }, + { name = "typing-extensions" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/5b/fe/cba54dc58523434919b66f13a667e36e436deddd77ca519e96553617d4ec/torch-2.13.0-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:e76f9bcecc52b8ff711239a2f7547d5353df95878ab232f0773c1d95928b92f8", size = 111187938, upload-time = "2026-07-08T16:05:17.065Z" }, + { url = "https://files.pythonhosted.org/packages/c2/59/1e3160e18e12aa3038390efab3ce02b36a9d4d6a527ecdd8520dca2e68d8/torch-2.13.0-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:092790c696a760c729fd5722835f50b9d81fd7c8f141571f3f3cf4081a8f664c", size = 427199369, upload-time = "2026-07-08T16:04:51.054Z" }, + { url = "https://files.pythonhosted.org/packages/01/79/1f2d34ad7034ee1c7ffc1cf8bf0f8213af2a81df6ecdb3997ecec107c09d/torch-2.13.0-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:60fcdcb2f3876e21146cb4524ef06397d727ca9ad5f020818547e25075fe3cb7", size = 526574961, upload-time = "2026-07-08T16:04:07.075Z" }, + { url = "https://files.pythonhosted.org/packages/6c/fd/0f2ce40f58aefbdb3392f9acce3c8171940943ae2d661f70558bfa73befb/torch-2.13.0-cp311-cp311-win_amd64.whl", hash = "sha256:a0d8b11f16a48d60e2015d8213aa0390744cbebb98e58b62b3514dddc656e330", size = 122015870, upload-time = "2026-07-08T16:05:27.59Z" }, + { url = "https://files.pythonhosted.org/packages/c4/3a/ed0f4d4d1dcde03bced7aac9a28e800abcdc0cbd06b6775044c9fbd877b7/torch-2.13.0-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:2fe228aba290d14b9f31b049be550dbd469c3fd3013d7a19705b30454da97027", size = 111213045, upload-time = "2026-07-08T16:05:22.997Z" }, + { url = "https://files.pythonhosted.org/packages/df/a9/f6a2a4d763ff1df02e9a64c477029db614295bc9367f4131223791ccc243/torch-2.13.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:572df8be8ffb4599c88cbd6a0726f1f854f4da65d2e3c09f0e2c2283333cd6d4", size = 427210998, upload-time = "2026-07-08T16:04:37.708Z" }, + { url = "https://files.pythonhosted.org/packages/f3/82/fea946351658e6534db52d2cc12bc53087cbf87f9440c5f180f367c1950b/torch-2.13.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:796633c4cdf0fe2cdced72d8f88f22e73dbcfce83132763162f6d4bff13b820b", size = 526605292, upload-time = "2026-07-08T16:04:22.81Z" }, + { url = "https://files.pythonhosted.org/packages/21/d6/e8f3c6f7e01f626f77259de9860d2a78bc84c40539e28e79b7e98b0bb659/torch-2.13.0-cp312-cp312-win_amd64.whl", hash = "sha256:024c6cc0c1b085f2f91f20a3dc27b0471d021c31ce84b81be3afdc39f791fd9d", size = 122057313, upload-time = "2026-07-08T16:03:53.43Z" }, + { url = "https://files.pythonhosted.org/packages/0d/fa/c1c10b7aff4a9a3e8956d4f0a5f468fa6db7abc3208805719076772b4833/torch-2.13.0-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:33449899ce5496c1b84b4853179d94fd102028ae1407314d9fb956bb79e70d09", size = 111213743, upload-time = "2026-07-08T16:03:28.579Z" }, + { url = "https://files.pythonhosted.org/packages/11/18/9ecb37b56293a0be8d80f810bf672a72fe7e02f8b475d5ef1b9bf8a0d748/torch-2.13.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:1e09d6a722504957c694faceca843acde562786df1144ebcc5a74075ec7f6005", size = 427213008, upload-time = "2026-07-08T16:03:44.106Z" }, + { url = "https://files.pythonhosted.org/packages/d4/5a/7c50ba1b7b713d71d34669c6d13dab0a11531a3eceb0307a5162dbfec0f7/torch-2.13.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:a3a9a21312872af8a26950b2c15680335a386a1f56ed03e780653d78b9607e9e", size = 526602329, upload-time = "2026-07-08T16:03:12.649Z" }, + { url = "https://files.pythonhosted.org/packages/91/3d/e7adcc6aaf36961cd18f56cf8ad0f3058c3a5c84ccf391762176c94581b8/torch-2.13.0-cp313-cp313-win_amd64.whl", hash = "sha256:49b58f1e2c52440abb6f17c28f0335fe6c6d01ad1a7f55b0183b81e4b34d64e6", size = 122057920, upload-time = "2026-07-08T16:03:01.808Z" }, + { url = "https://files.pythonhosted.org/packages/36/76/6dcc7f0c07052102dd36f83cbc5800842a909c8c3fbf1a7f8a5844954de9/torch-2.13.0-cp314-cp314-macosx_14_0_arm64.whl", hash = "sha256:d849b390e07d8d333ce8ecaf91b273c656c598379a19c9acf1318a883f6b391c", size = 111227066, upload-time = "2026-07-08T16:03:33.6Z" }, + { url = "https://files.pythonhosted.org/packages/e9/09/2c10e8cd0e00fa5d23c052df6ce467eaa7182399f5e0f824f1e4ff42ccae/torch-2.13.0-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:a3893dc2da0a972a8ca5d698c85a9f967559ac5f8ee1797b77408aa8734d073c", size = 427226309, upload-time = "2026-07-08T16:02:53.127Z" }, + { url = "https://files.pythonhosted.org/packages/76/c6/22c2102bbef14ca6a6cb4c20e42f088e49c5f812be4e160ae57502e325f9/torch-2.13.0-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:49f1ea385c754e54919408a9bb3b5a72b0b755bbe2c916c1d6f70afbec4908a2", size = 526614507, upload-time = "2026-07-08T16:02:16.441Z" }, + { url = "https://files.pythonhosted.org/packages/2b/0c/7d1deb6bce5bc3e6042caf39100ac768eba3b9a098e1dddd16f75bd6489b/torch-2.13.0-cp314-cp314-win_amd64.whl", hash = "sha256:4f8573e3ce9ebcd53fe922f01077a6085ccdfbe5f12fd215883a9d87d7a744fd", size = 122051871, upload-time = "2026-07-08T16:03:23.521Z" }, + { url = "https://files.pythonhosted.org/packages/f4/ce/aa8b7f9949d32e0f2f624f342bc3b48112c1b8a130288465938bc83bcbf9/torch-2.13.0-cp314-cp314t-macosx_14_0_arm64.whl", hash = "sha256:c28def70706c2f9ecc752574766e8ae4da9b810ab6676b611166761a78a9f1e1", size = 111537025, upload-time = "2026-07-08T16:02:44.28Z" }, + { url = "https://files.pythonhosted.org/packages/69/d1/491e3a0389430946145888b0203f2b6a759ce2a61481b96a85c2da4f2ced/torch-2.13.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:31061ff56ed8fbf26c749806905aeb749ebeb819810fd5d52508aa5afd90dddc", size = 427219769, upload-time = "2026-07-08T16:02:31.18Z" }, + { url = "https://files.pythonhosted.org/packages/9a/1d/38006e045bf0a1fc28ef01e757c554e59e59a8770c284bc4f47b14e60441/torch-2.13.0-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:cc26eead4cf51d0b544e31e364dcf000846549c273bd148936fe9d24d29acb92", size = 526571320, upload-time = "2026-07-08T16:01:59.348Z" }, + { url = "https://files.pythonhosted.org/packages/56/94/655c91992a882bd5071aa0b5d22a07dbb130d801e872be97c0b627a7c693/torch-2.13.0-cp314-cp314t-win_amd64.whl", hash = "sha256:a7de8a313090dc5c7d7ba4bfe5c3be222528f9a4dba1acc83bddb1157360c4b8", size = 122306773, upload-time = "2026-07-08T16:02:39.832Z" }, +] + +[[package]] +name = "tqdm" +version = "4.70.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "colorama", marker = "sys_platform == 'win32'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/21/3b/6c24bec5be5e743ffd99576daa5cc077722fc7d5bbc00bd133fa0c698dc6/tqdm-4.70.0.tar.gz", hash = "sha256:55b0b0dbd97462d06ebee91e4dac24ed4d4702be82b24f07e6c1d27e08cea220", size = 795438, upload-time = "2026-07-27T11:33:15.271Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f9/1c/01bfd571a64e7f270e6bab5e33777debe0edc56759233ce84f27dec92d14/tqdm-4.70.0-py3-none-any.whl", hash = "sha256:7f585706bfddbdebf89daac705b2dfcc16890130727d3197ca62c732b4310953", size = 80184, upload-time = "2026-07-27T11:33:13.167Z" }, +] + +[[package]] +name = "transformers" +version = "5.15.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "huggingface-hub" }, + { name = "numpy", version = "2.4.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.12'" }, + { name = "numpy", version = "2.5.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12'" }, + { name = "packaging" }, + { name = "pyyaml" }, + { name = "regex" }, + { name = "safetensors" }, + { name = "tokenizers" }, + { name = "tqdm" }, + { name = "typer" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/2a/92/c50c61da7046bbb59a4d011291aeadcfb4d7980ab36fdb31e93823a3fb93/transformers-5.15.1.tar.gz", hash = "sha256:27c996bd9075ddc82d40f8590dfdc81ea45f611bfca477e0db5d7fd257a482f7", size = 9378434, upload-time = "2026-08-19T11:28:20.33Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/41/c4/a12e1d9b387fb0c40a57116db82b457e8c771cb419163cda29204d74a595/transformers-5.15.1-py3-none-any.whl", hash = "sha256:b7cdf238ff583e3a58dbc7fa34da1aaf091ce063141f65a30538160bd5afe93f", size = 11749582, upload-time = "2026-08-19T11:28:16.726Z" }, +] + +[[package]] +name = "triton" +version = "3.7.1" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7b/f9/19d842d06a08559534fa1eaab6ca551b1bcf40f06620bddec1babaa2772d/triton-3.7.1-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d4a0e1cd4c4a76370ed74a8432a53cea28716827d19e40ffc732233e35ceb3f6", size = 184664887, upload-time = "2026-06-17T20:03:42.913Z" }, + { url = "https://files.pythonhosted.org/packages/cd/5e/fce69606f7f240297f163e25539906732b199530d486ce67ae319877e821/triton-3.7.1-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6744957e9fd610a29680ec2346057d0c86948ed3812468670719f391e94b44a5", size = 197701306, upload-time = "2026-06-17T19:53:13.673Z" }, + { url = "https://files.pythonhosted.org/packages/94/fa/f856e24deb462d5f18bd4b5a746957862ab9b6ee5834bda60605ec348366/triton-3.7.1-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9497f2e696ee368862a181a90b2dcc03ca978cc4f602abd67c7d81022a6988e1", size = 184692359, upload-time = "2026-06-17T20:03:48.288Z" }, + { url = "https://files.pythonhosted.org/packages/c4/6f/fb96d15db6f36d6eae4cafb998c2e0353bf59d7c4ea1662d7497f269134a/triton-3.7.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7e40869937a68206ec70d7f25bb7ec6433cb083f9135e1f36dbd318dc449a728", size = 197719725, upload-time = "2026-06-17T19:53:20.419Z" }, + { url = "https://files.pythonhosted.org/packages/00/42/c5089d4d9327fcd1e862c599cc2927f39418f84dd11a84cb2ccff9d4787a/triton-3.7.1-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:cdbfc09d9ec58bc5e68321525653220de7515c199e7a8097a97c85e62b52cd0a", size = 184694629, upload-time = "2026-06-17T20:03:53.444Z" }, + { url = "https://files.pythonhosted.org/packages/07/42/2c3ac59253ae8892b6f307875263dd23dc875cdf732d3aea40d6d41fb7cb/triton-3.7.1-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:58c0e131da05134a2a4788ccbcc0c1105cf0f54c8e98f19e34cd465396dc15eb", size = 197729241, upload-time = "2026-06-17T19:53:27.801Z" }, + { url = "https://files.pythonhosted.org/packages/40/71/e01aa7ad573883ed9456f130226babdec70b005e098c4d6226a6238e761b/triton-3.7.1-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:fe4ea396a06171f1f1f58cbd39c70b09294398f7dd7c620939bab54ad6f934fa", size = 184705764, upload-time = "2026-06-17T20:03:59.064Z" }, + { url = "https://files.pythonhosted.org/packages/a4/09/5683146fda6a2b569deb78ccfd8fbfea8bfe55f726b081c0a6bb18dd6f28/triton-3.7.1-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:2020153b08280415ec0da6607834e79166442147e78e144df06b508c75b186d2", size = 197729537, upload-time = "2026-06-17T19:53:35.516Z" }, + { url = "https://files.pythonhosted.org/packages/e9/f8/448220c3092019f9fdfab39ec47985968181d67da34b44f6a7f6280a5cbb/triton-3.7.1-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c58e4c61f0c73b5dba3b5d19b4a7093c32f90dc18b2a7f121a7c16ccd31107b7", size = 184814760, upload-time = "2026-06-17T20:04:04.984Z" }, + { url = "https://files.pythonhosted.org/packages/f0/ac/229b7d4589d2e5937310e72c6d46e89599d16a4a12b479ffa1499fee8eb8/triton-3.7.1-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:10ba85fa2cca4a2fbdeb36bf1cb082f2c252bda55bf9fccd74f65ec5bc647e68", size = 197824404, upload-time = "2026-06-17T19:53:42.772Z" }, +] + [[package]] name = "truststore" version = "0.10.4" @@ -1305,6 +2185,21 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/19/97/56608b2249fe206a67cd573bc93cd9896e1efb9e98bce9c163bcdc704b88/truststore-0.10.4-py3-none-any.whl", hash = "sha256:adaeaecf1cbb5f4de3b1959b42d41f6fab57b2b1666adb59e89cb0b53361d981", size = 18660, upload-time = "2025-08-12T18:49:01.46Z" }, ] +[[package]] +name = "typer" +version = "0.27.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "annotated-doc" }, + { name = "colorama", marker = "sys_platform == 'win32'" }, + { name = "rich" }, + { name = "shellingham" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/ae/40/4a3db7990d1f62a53182aa96eaef57aeb2886a27f90a195bc66713565d31/typer-0.27.1.tar.gz", hash = "sha256:a79bef8469a79c45498e7b814ecf8d603cc7644e9acbd9e19cac0334240b18df", size = 203994, upload-time = "2026-08-03T14:41:03.438Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/43/89/9518bc0c3929bee36b3a4a8e3daddd6e03f92f9961c66d4983b837160543/typer-0.27.1-py3-none-any.whl", hash = "sha256:53150287edd11baeb4e4722c8e394fcdf8181c0ae89485cba8d25c778d5edd56", size = 122874, upload-time = "2026-08-03T14:41:04.391Z" }, +] + [[package]] name = "typing-extensions" version = "4.16.0"