check-pr-size classify_path has no entry for a per-run bench-evidence directory, so its own suite is red on main

[localai-bot]

Found while adding the conflict-marker gate for #1417, which necessarily edits scripts/check-pr-size.py to register a new checker's creation mutation.

The gap

classify_path fails closed on an unknown path. BENCH_EVIDENCE matches exactly ONE path segment:

BENCH_EVIDENCE = re.compile(r"(?:benchmarks/(?:demo|media)|docs/bench-evidence)/[A-Za-z0-9_.-]+\.(?:json|png|gif|mp4|log)\Z")

docs/bench-evidence/gdn-replayssm-w0-20260818/ landed on 2026-08-18 as a per-run directory of ten files, and none of them matches. Measured on origin/main 9ecaf1bb3, sweeping every tracked path through classify_path:

unclassified tracked paths: 10
   docs/bench-evidence/gdn-replayssm-w0-20260818/RESULT.txt
   docs/bench-evidence/gdn-replayssm-w0-20260818/build_w0.sh
   docs/bench-evidence/gdn-replayssm-w0-20260818/count_sass.sh
   docs/bench-evidence/gdn-replayssm-w0-20260818/job1.log
   docs/bench-evidence/gdn-replayssm-w0-20260818/job2.log
   docs/bench-evidence/gdn-replayssm-w0-20260818/job3_sass.log
   docs/bench-evidence/gdn-replayssm-w0-20260818/probe.cu
   docs/bench-evidence/gdn-replayssm-w0-20260818/run_on_lease.sh
   docs/bench-evidence/gdn-replayssm-w0-20260818/run_sass_on_lease.sh
   docs/bench-evidence/gdn-replayssm-w0-20260818/sass.txt.gz

What it costs

tests/scripts/test_check_pr_size.py::test_every_tracked_and_current_change_path_is_classified sweeps every tracked path, so that suite is RED on origin/main today, verified in a detached worktree at 9ecaf1bb3:

FAIL: test_every_tracked_and_current_change_path_is_classified
Ran 46 tests in 0.546s
FAILED (failures=1)

The suite runs in no preflight SUITES entry, so a plain scripts/agent-preflight.sh is green over it. It surfaces only through the checker-evidence contract: any change that edits scripts/check-pr-size.py must prove its HEAD checker/test pair passes, and that pair cannot pass while the sweep is red. scripts/check-pr-size.py --base origin/main --head HEAD then reports

ERROR: HEAD checker/test pair failed for 'scripts/check-pr-size.py'

which reads as a defect in the change under review rather than as a pre-existing red.

Registering a NEW checker requires an entry in CREATION_MUTATIONS, so every future checker is blocked behind this until it is repaired.

Fourth instance of the same class: #856 (.agents/issue-index.md), #668 (.agents/oracles/), #989 (.agents/reachability.md).

Fix

Name the surface; do not widen a rule. A BENCH_EVIDENCE_RUN pattern for docs/bench-evidence/<run-id>/<file>, restricted to the extensions the directory actually carries (txt, log, gz, sh, cu) and deliberately excluding .md and .json, which already classify as public_document through DOC and would be silently reclassified because the evidence arm is tested first.

Repaired in the same flow as #1417, per AGENTS.md: the person who found the bug has the context. The red-before case already exists and needed no new fixture, and classification is verified to move exactly those 10 paths and nothing else.

[localai-bot]

Fixed in #1450, merged as aeba0de6f.

Verified independently on main at aeba0de6f: python3 tests/scripts/test_check_pr_size.py returns rc 0, Ran 46 tests, OK. Before the fix the same suite on main returned rc 1 with test_every_tracked_and_current_change_path_is_classified failing, "First list contains 10 additional elements" — I confirmed that red myself at 4712dac40 before dispatching the repair.

The cause was that classify_path in scripts/check-pr-size.py had no entry for a per-run docs/bench-evidence/<run-id>/<file> directory, leaving 10 tracked files unclassified since 2026-08-18. The repair adds a BENCH_EVIDENCE_RUN pattern that admits txt|log|gz|sh|cu and deliberately excludes .md and .json, because those already classify as public_document and the evidence arm sits above public_document in classify_path — admitting them would silently reclassify three existing files. A fresh review confirmed that by mutation: adding md|json moves exactly docs/bench-evidence/mxfp4-qwen/{W0-W1-oracle-run.md, W3-e2e-result.md, golden_marlin_w4a16.json} out of public_document. A before/after sweep of all 4811 tracked paths showed exactly the intended 10 change class and nothing else.

The part worth keeping

This suite is in no preflight SUITES entry, which I verified with a grep returning 0. That is why main carried a red for two days with every preflight reporting green: nothing ran it. It surfaced only indirectly, as ERROR: HEAD checker/test pair failed for 'scripts/check-pr-size.py' when an unrelated change touched that checker — which reads as a defect in the change under review rather than in main.

The repair fixes the classification. It does not fix the invisibility, and the fourth instance of this class (after #856, #668 and #989) suggests the invisibility is the more durable defect. A test that exists, fails, and is run by nothing is worse than an absent one, because its presence implies coverage. Whether test_check_pr_size.py should join a preflight suite — and how many other tests/scripts/ suites are in the same position — is a question for the row that owns the preflight surface, and is deliberately not answered here.

A related recurrence risk, named rather than objected to: BENCH_EVIDENCE_RUN admits a fixed extension set, so a future per-run evidence directory carrying a .py, .csv, .png or .nsys-rep reds the same suite again. That is the fail-closed behaviour the protocol asks for, but it is worth knowing it is the fifth instance waiting to happen.