feat: grounded generation and eval harness with 30-doc golden corpus - #4
Merged
Conversation
Generation: - `ollama.py` is an async adapter for the local Ollama `POST /api/chat` endpoint. Tenacity retry on 5xx and transport errors; raises on 4xx. Token counts and latency are surfaced through frozen pydantic models. - `prompt.py` builds the grounded system + user pair. The system prompt explicitly teaches the model to treat `[PERSON_n]` placeholders as opaque tokens. Empty parent contexts short-circuit to a refusal. - `citation.py` extracts `[N]` citations from the answer, deduplicates, and flags any citation outside the retrieved parent set as ungrounded. - `generator.py` ties it all together. The empty-context refusal is enforced in code so Ollama is never called with `(no context available)`. Eval harness: - `models.py` carries the golden corpus shape: `GoldenDocument`, `GoldenEntity`, `GoldenQA`, and `EvalCounts` with separate `recall_hits` / `precision_hits` because a planted entity can be matched by several detector outputs. Counting both directions keeps the metrics honest. - `loader.py` reads the corpus from disk and validates every JSON. - `runner.py` runs the configured detector over the corpus and rolls up per-category and overall counts. - `judge.py` calls Ollama to do binary LLM-as-Judge faithfulness scoring. Informational only; no gate. Scripts: - `scripts/seed_demo_corpus.py` regenerates the 30 synthetic documents (10 HR / 10 contracts / 10 support) with 91 planted PII spans. All values are synthetic (IANA test cards, ISO IBAN examples, example.com emails). - `scripts/eval.py` CLI prints a rich table, writes an HTML report, takes `--with-gliner` for the two-stage path, and gates the recall and precision floors so a regression fails the run. Measured baseline against the 30-doc corpus with GLiNER on: PII Recall 100.0% PII Precision 0.93 (gates >= 85% recall, >= 0.85 precision both pass)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
OllamaClient(tenacity retry on 5xx, raises on 4xx), grounded prompt that teaches the LLM to treat[PERSON_n]placeholders as opaque tokens,[N]citation verifier,Generatorfacade that short-circuits to a refusal when context is emptyrecall_hits/precision_hitscounters, LLM-as-Judge faithfulness scorerscripts/seed_demo_corpus.py, 91 planted PII entities totalscripts/eval.pyCLI prints a rich table, writes an HTML report, and supports--with-gliner/--recall-gate/--precision-gateMeasured baseline (
scripts/eval.py --with-gliner)Test plan
uv run ruff check .cleanuv run pyright0 errors / 0 warningsuv run pytest tests/test_generation_*.py tests/test_eval_*.pyall green