Why
The README roadmap targets a 150-pair x 5-library corpus. More pairs = tighter, more trustworthy accuracy/version/refusal metrics. This is approachable for anyone who can write good documentation Q/A - little core code required.
What to do
Acceptance criteria
- New entries validate against the schema:
uv run pytest -c ../evals/pytest.ini ../evals/tests.
- Balanced in-scope / out-of-scope split noted in the PR.
Pointers
evals/corpus.json, evals/schema.py, evals/README.md.
Why
The README roadmap targets a 150-pair x 5-library corpus. More pairs = tighter, more trustworthy accuracy/version/refusal metrics. This is approachable for anyone who can write good documentation Q/A - little core code required.
What to do
evals/corpus.jsonfollowing the existing schema (evals/schema.py), including out-of-scope entries so the refusal metric stays honest.Acceptance criteria
uv run pytest -c ../evals/pytest.ini ../evals/tests.Pointers
evals/corpus.json,evals/schema.py,evals/README.md.