test(corpus): pin per-suite task counts so corpus drift fails CI (re #244) - #290
Closed
reacher-z wants to merge 1 commit into
Closed
test(corpus): pin per-suite task counts so corpus drift fails CI (re #244)#290reacher-z wants to merge 1 commit into
reacher-z wants to merge 1 commit into
Conversation
…244) test-cases/v2 currently holds 129 tasks while README, the HF card and the leaderboard denominators say 130 — the divergence went unnoticed because nothing asserts a size. Pin all four built-in suites (v1 152, v2 129, v1-lite 20, claw-eval 19) with a failure message telling the author to update the published counts in the same commit, plus a guard so a newly added suite cannot skip the pin.
Collaborator
|
Task changes on released suites are unusual, thus I believe a dedicated test case/GHA is not needed. I'll keep in mind to update the counts if any changes are made on existing corpuses in the future. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the CI half of #244.
Why
test-cases/v2/holds 129 tasks; the README, the HF dataset card, and the leaderboard denominators all say 130. Nobody noticed because no test asserts a corpus size —test_builtin_case_suites_are_discoverableonly checks that each suite is non-empty. The same blind spot let V1 drift to 152 while the README still advertises 153.A ~1-task divergence silently changes every reported rate (a 3/129 row is not a 3/130 row), and it is exactly the kind of thing a reproducer notices before we do.
What this adds
EXPECTED_SUITE_SIZESintests/test_host_tasks.pypins all four built-in suites — v1 152, v2 129, v1-lite 20, claw-eval 19 — as they exist onmaintoday. Adding or removing tasks is still fine; doing it silently now fails CI with a message that names the surfaces to update:A second test asserts the table covers every entry of
batch.CASE_SUITES, so a new suite cannot slip in unpinned.Not in scope
Whether the truth should be 129 or 130 — i.e. restoring the missing task versus correcting the published number — is the maintainer decision in #244. This PR pins today's reality so the answer can't drift again while that's decided; flipping the pin to 130 is a one-line change once the task is restored.
pytest tests/test_host_tasks.py→ 14 passed; ruff format/check clean.