Skip to content

Add evaluation harness M0 and M1#25

Merged
arthurpanhku merged 1 commit into
mainfrom
codex/evaluation-m0-m1
Jul 8, 2026
Merged

Add evaluation harness M0 and M1#25
arthurpanhku merged 1 commit into
mainfrom
codex/evaluation-m0-m1

Conversation

@arthurpanhku

Copy link
Copy Markdown
Owner

Summary

  • Add the evals/ M0 scaffolding with EvalCase, RunConfig, registry, model matrix, and package wiring
  • Implement the M1 OWASP Benchmark SAST triage path: fetch script, adapter, direct AssessmentService runner, hard-key CWE scorer, and JSON/Markdown scorecards
  • Update the root README with evaluation harness usage and project layout references

Data hygiene

  • Raw OWASP Benchmark data is not committed
  • evals/datasets/owasp_benchmark/raw/, downloads, and archives are ignored
  • Registry records the source archive checksum and contamination risk
  • Tests use a tiny synthetic fixture only

Validation

  • .venv/bin/python -m pytest evals/tests passed
  • .venv/bin/ruff check . passed
  • .venv/bin/python -m pytest passed: 108 tests
  • Full OWASP adapter smoke read 2740 cases after fetch; raw data was removed before commit
  • Tiny fixture scorecard smoke was stable across two invocations on the hard-key path

Notes

  • This PR intentionally stops at M1. Matcher, judge, set detection, grounding, calibration, and classification modules are stubs with docstrings for later milestones.

@arthurpanhku arthurpanhku marked this pull request as ready for review July 8, 2026 02:26
@arthurpanhku arthurpanhku merged commit 67973d5 into main Jul 8, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant