feat(tools): offline ablation-evidence checker and an evidence-bound eval harness

[monotophic]

Authored by Claude Opus in Claude Code, analysis in partnership with @monotophic

The engine's diagnostic modes write artifacts that something else has to read
back: an ablation run writes a JSONL evidence file, and the evaluation harness
reads scoring output off the engine's stdout. Today nothing re-checks either one
independently, and the harness trusts whatever it is handed.

This adds three tools and closes one packaging gap.

tools/engine_evidence.py owns the grammar of the engine's two typed startup
lines (the == GLM C engine banner and the loaded in … record) and the
canonical form of an ablation manifest, so every consumer agrees by construction
instead of each re-deriving the same regular expressions and the same
normalisation.

tools/check_ablate_evidence.py re-checks one manifest and its evidence
artifact against an independently supplied config.json, with no engine and no
test binary: field bounds, every record's key set and types, the header's
binding to the config digest and the manifest digest, and the truncation,
replay and mismatch cases at each record boundary. Three cross-record
invariants are enforced because the producer guarantees them for every row it
can emit; top-k order is deliberately not enforced, because it is unsorted on
the wire by design.

tools/eval_glm.py gains two properties it did not have. Results are appended
per request, so a run that dies mid-task leaves a valid partial file with an
INCOMPLETE marker and still prints the accuracy table over the rows that
landed — before, an interrupted run left nothing. And every line of the scoring
stream is classified against the engine's documented SCORE-mode output, so an
unrecognised line is refused by name instead of silently absorbed.

tools/pack_python.py reached a file two ways — by following imports out from
coli, and by spotting scripts launched as subprocesses — but only followed the
first kind's own imports. A module that only a subprocess-launched script
imports was therefore left out of the release archive while --check still
reported the archive complete. On dev that is tools/iq3_pack.py, which
tools/convert_fp8_to_int4.py imports inside quant_e8(). This is a
pre-existing packaging defect on dev, not one this branch introduces; we
found it while checking that the new helper module would itself be packaged. We
will open a separate issue describing it, once this PR is up so the issue can
point at the fix.

Context. This PR is one of four independent contributions derived from a
single locally-verified working tree (offline evidence tooling, the engine's
ablation scoring mode, two engine-transcript check tools, and a raw wire-format
adapter). This one carries the Python evidence tooling and the packaging fix.
It stands alone — its whole suite is green with the other three absent (169
cases, measured on a checkout of dev carrying only these eight files), and
merging or declining the others does not affect it. The others are proposed
separately, each with its own evidence.

Behavioral contract

• Everything the harness did before, it still does on its own inputs: task
  loading, per-option scoring, exact-vs-greedy accounting, and the output format
  existing consumers read are unchanged, including --dry without a vocabulary
  and the accuracy table on a partial run.
• The offline checker accepts exactly the artifacts the engine really produces,
  including manifests saved with CRLF endings or without a final newline, and
  binds the same digest the engine binds for all three framings of the same
  content.
• Every check the checker performs has a real oracle: literal fixtures, or a
  digest reproducible outside the tool.
• A run whose evidence the engine build cannot supply is reported UNBOUND by
  name; it is never reported as bound.
• The packaged set is closed under imports: a module reached only through a
  subprocess-launched script is packaged, and --check fails when it is not.
• No test here runs a model, and none needs a GPU or a container.

Capstone matrix — one decisive artifact per claim; each would fail if the
claim were false.

claim	decisive evidence
the checker and the engine agree on the canonical manifest digest	the same 64-character known answer is pinned on both sides — tests/test_check_ablate_evidence.py:605 (Python) and tests/test_ablate_mode.c:287 (C, in the engine PR) — and reproduces outside both: printf 'coli-ablate-manifest/2\n0 3 2 0 0 1 2 3\n' | shasum -a 256 → c63a48c375b14ca60f26c7e3c5dd36b5929ffaf669a45511c93deee6e8bbd5ed
the checker accepts real engine output, not just fixtures	on a Linux CPU host, a real ablation run over a real container produced an artifact (coli-ablate/2 header, one item header, three logit records, one terminal record) that the checker validated against the model's own config.json: [ablate-evidence] PASS manifest=4e18e3aba08f4b0a0f5e33619ad910cc6c7d4b90108efde734521bd6fac8b24a items=1 targets=3, exit 0
the harness is honest when the engine cannot supply evidence	the same host ran the harness against the real engine, and it reported its evidence status as UNBOUND — a named status — rather than claiming a binding it did not have; the run exited 0
the harness's dev behaviour is unchanged	a differential against dev's own harness on the same fixtures: byte-identical output, including --dry without a vocabulary and a partial run's table plus its INCOMPLETE line
the packaging fix bites	restore dev's tools/pack_python.py and tests/test_pack_python.py fails at test_engine_evidence_is_needed_by_the_real_tree, naming the file that fell out of the computed set
the packaging test is not self-fulfilling	that assertion runs against the real c/ tree, not a fixture the test built; every other case in the module builds a disposable tree, which is exactly why none of them caught this on dev

Fuller matrix, deferrals, and origin accounting

requirement	instrument	result
helper grammar is source-true	the banner and loaded in regular expressions read against the engine's own format strings and the seven kernel names it can print	match, including the draft upper bound the engine clamps to
checker field bounds and record key sets	literal fixtures per check, plus rejection cases at every record boundary	green
cross-record invariants are producer-guaranteed	each invariant derived from the emitter's own arithmetic and stated in the test module's docstring	three enforced (nll never negative; the correctness flag equals the argmax-equals-gold comparison; the argmax logit never below the target's); top-k order deliberately not enforced
manifest canonicalisation	one implementation, used by the checker before parsing and before hashing	three framings of the same content bind one digest
numeric grammar	the harness and the checker accept both the six-decimal form the engine ships today and the older %.17g form, plus nan/inf/-inf	both accepted; fixtures are non-dyadic
incremental durability	kill the engine stand-in mid-run	valid partial file, INCOMPLETE marker, table printed, child terminated
foreign-line refusal	a line the engine never prints on the scoring stream	refused by name (on dev it was absorbed)
suite runs with nothing else from the set present	python3 -m unittest tests.test_engine_evidence tests.test_check_ablate_evidence tests.test_eval_glm tests.test_pack_python on dev + these eight files	Ran 169 tests … OK

Deferrals, stated plainly: two harness cases that need a test binary this build
does not produce are not present in the module — they are described in its
docstring as deferred, rather than shipped as skips. Nothing here is skipped for
a reason a reviewer cannot see.

Scope note: tools/check_ablate_evidence.py is a repository tool, not an
archive tool — it is not reached from coli, so it is deliberately not in the
release archive. tools/engine_evidence.py is, because the harness imports it,
and that is what the packaging fix restores.

Origin accounting. The helper module, the checker and the adapter-side harness
work are new content. Two changes are fixes to defects that exist on dev
today and were found while reviewing this area: the packaging closure gap
described above, and the harness's silent acceptance of a foreign stdout line
together with its all-or-nothing result writing. The three cross-record
invariants and the byte-level rejection cases were added during review, each
proven against the producer's own arithmetic rather than assumed. No file
outside the eight listed is touched, and no surrounding code was reformatted.

Durable vs current state: the tools, their grammars and the packaging rule
are durable. The counts (169 cases) and the fleet run quoted above are current
state, measured 2026-09-04 against base e1efc68 on macOS (Apple silicon) and,
for the fleet cell, on a Linux CPU host with GCC.

[JustVugg]

Conflict resolved on your branch, pushed as 9e96637. No rebase needed on your side -- pull before your next commit.

What I did, so you can check rather than trust it:

• tools/pack_python.py takes dev's version wholesale. It is a strict superset of this branch's: I diffed them, and the only lines unique here were the three your own commit replaced (the old two-edge docstring, the scripts, script_paths init, the old summary print). Nothing of yours was dropped.
• tests/test_pack_python.py takes dev's version and keeps your engine_evidence assertion as an additional case, not a replacement. On dev it could not run, since engine_evidence.py is introduced here. On this branch it pins the edge this branch adds, sitting next to the one that pins the historical bug. Both are real-tree assertions and they defend different things.

12 tests, green locally. CI is running.

One thing I noticed while verifying, worth a decision from you

tools/check_ablate_evidence.py is not in the computed set:

engine_evidence.py           True
eval_glm.py                  True
check_ablate_evidence.py     False

Not a defect in your change: nothing in the code names it, and it takes argparse arguments, so its caller is a person. Static analysis cannot reach a human. tools/k3_tokenizer.py is in exactly the same position and release.yml copies it explicitly for that reason.

So it is a decision, not a bug: if the checker is meant to be usable from a release archive, it wants the same explicit copy plus a test -f gate. If it is a repository-side tool for people working on the engine, it is already right as it is. You know which one you intended.

Thanks again for #1359. Cherry-picking your commit rather than rewriting it was the least I could do after duplicating another contributor's work last week by not checking the PR list first.

[monotophic]

@JustVugg: Thanks for patching the bugs, resolving the conflict on this branch, and the write-up. I checked the merge: pack_python.py at 9e96637 is dev's file with nothing of ours dropped, and test_pack_python.py keeps the engine_evidence case as an added assertion. I have pulled 9e96637 and have not rewritten the branch.

On check_ablate_evidence.py: it should ship in the release archive. It verifies the evidence that eval_glm.py produces, and that harness is in the archive, so anyone running a released engine and producing ablation evidence wants the checker beside it. Added in ba2802a: the same copy and test -f gate that release.yml gives k3_tokenizer.py, plus a parse check, and a real-tree test that pins both halves (the walk does not reach either human-only tool; the workflow copies and gates each).

#1356
and
#1357
stack on this branch; I merged 9e96637 into each the same way rather than rebasing, so their histories stay put.

[Authored by Claude Fable 5.1 in Claude Code, analysis in partnership with @monotophic.]