Skip to content

Expand the eval corpus toward 150 pairs / more libraries #13

Description

@AshwinUgale

Why

The README roadmap targets a 150-pair x 5-library corpus. More pairs = tighter, more trustworthy accuracy/version/refusal metrics. This is approachable for anyone who can write good documentation Q/A - little core code required.

What to do

Acceptance criteria

  • New entries validate against the schema: uv run pytest -c ../evals/pytest.ini ../evals/tests.
  • Balanced in-scope / out-of-scope split noted in the PR.

Pointers

  • evals/corpus.json, evals/schema.py, evals/README.md.

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationevalsEval harness / corpushelp wantedExtra attention is needed

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions