test(tools): gate the logprob gap check on the engine preamble; add the native-MTP witness

[monotophic]

Authored by Claude Opus in Claude Code, analysis in partnership with @monotophic

Two command-line checks that read an engine transcript and say whether it is
internally consistent.

tests/check_data_logprob_gaps.py already existed and had two weaknesses. It
read a transcript without first establishing that the transcript came from an
engine it understands, so an unrecognised header was treated as no header; and
it reported problems without saying where in the byte stream they were, which
makes a disagreement on a long capture tedious to localise. It now parses the
engine's preamble through a shared helper and fails loudly when that preamble
is unparsable, the protocol grammar is exact, the numeric tail and the top-k
cap are enforced, and every reported problem cites the byte offset of the frame
it concerns. The numeric grammar accepts both the fixed six-decimal form the
engine ships today and an earlier %.17g form (plus nan/inf/-inf); a
token can satisfy both spellings exactly, so form classification stays
informational — a tally in the summary line — rather than a pass/fail rule.

tests/check_native_mtp_witness.py is new. It captures and validates a witness
that the native multi-token-prediction path really accepted the tokens it
claims, binding the capture to the binary, the container and the request bytes
by SHA-256 so a witness cannot be replayed against different inputs. It reuses
the gap check's frame grammar by importing it rather than restating it, so the
two cannot drift apart. Its exit statuses are distinct on purpose: 0 is a
live-capture pass; 1 is every ordinary malformed, incomplete or inconsistent
witness, including an offline replay, which is always incomplete on provenance
even when clean; and 3 is the separate case where the engine build cannot
produce this witness at all — that is a fact about the build, not a malformed
witness, so it prints its own stderr prefix and does not share an exit status
with real failures.

That last case is the common one today: the per-emission record this proof
depends on is not printed by the engine as it stands, so on a current build the
tool reports UNSUPPORTED and exits 3 rather than inventing a verdict. The
docstring says so.

Context. This PR is one of four independent contributions derived from a
single locally-verified working tree (offline evidence tooling, the engine's
ablation scoring mode, these two transcript checks, and a raw wire-format
adapter). This one carries the two transcript checks and their suites. It uses
one file from the evidence-tooling PR (#1355, #1355),
tools/engine_evidence.py, which both tools import at module load for the
preamble grammar — measured: with that PR absent both modules fail to import, and
with it present the two suites here run 105 cases green. The others are proposed
separately, each with its own evidence.

Behavioral contract

• A transcript whose preamble cannot be parsed is a named failure quoting the
  offending line, never a silent pass.
• Every reported problem names the byte offset of the frame it concerns.
• Both numeric spellings the engine has used are accepted; neither is treated as
  evidence of anything by itself.
• A witness that cannot be produced by this engine build is reported as
  unsupported, with its own exit status, and never as a pass or an ordinary
  failure.
• No model is run by anything here, and no fixture is downloaded: every case is
  built from literals inside the suite.

Capstone matrix — one decisive artifact per claim.

claim	decisive evidence
an unparsable preamble cannot pass	test_unparsed_banner_is_a_named_failure_quoting_the_line, test_unparsed_loaded_is_a_named_failure_quoting_the_line, and test_full_capture_with_unparsed_preamble_fails_the_whole_check — the last one drives a complete, otherwise-valid capture and requires the whole check to fail
the preamble gate cannot fail open	test_preamble_returning_none_is_defensively_named_not_silently_passed — the case where the parser returns nothing at all is asserted to be a named failure, which is the shape a "no news is good news" gate would pass
the numeric grammar is source-true, both forms	test_fixed6_form_is_accepted_and_reported, test_c17g_form_is_accepted_and_reported, and test_a_token_matching_both_grammars_is_ambiguous_and_still_accepted; the fixtures use the engine's own printf formats
the grammar checks bite independently	test_done_domains_and_fixed_grammar_bite_independently — each rule is shown to fail on its own input rather than being masked by a neighbour
an unsupported engine build is distinguishable at the command line	test_engine_witness_unsupported_when_mtpemit_absent and test_engine_witness_unsupported_is_distinguishable_at_the_cli — the second asserts the exit status and the stderr prefix, not just the internal status
the witness bindings each bite	test_every_artifact_binding_bites_independently, plus test_capture_owned_paths_cannot_be_replaced_before_binding: a capture whose file is swapped between hashing and binding is refused, and each binding is shown to fail on its own
the two tools cannot drift	the witness tool loads the gap check's own module and calls its check/parse_frames; there is no second copy of the grammar to keep in step

Fuller matrix and origin accounting

requirement	instrument	result
preamble gate	four named cases, including a full-capture case	loud failure in every shape
byte offsets	offsets asserted in the problem strings	present on every reported problem
Windows	the witness's payload-identity check needs POSIX stat semantics (fstat of the descriptor agreeing with lstat of the path on device and inode); CI's Windows job saw the two disagree on some fixture files	the witness refuses on Windows with an error naming the requirement, the fixture suite skips there, and two tests that run on every platform pin the refusal and the working case; the gap check is unaffected
exact protocol grammar	malformed and misordered global records, malformed framing	refused by name
numeric tail and top-k cap	literal fixtures in both spellings, non-dyadic values	accepted; classification informational
replay and provenance	offline replay of a clean capture	stays incomplete on provenance by design
unsupported build	absent per-emission record	exit 3 with its own prefix
suites run with the tooling PR present and nothing else from the set	python3 -m unittest tests.test_check_data_logprob_gaps tests.test_check_native_mtp_witness on dev plus these four files and that PR's	Ran 105 tests … OK

A limitation we would rather state than have you find: the gap this check hunts
is not producible by the engine's current multiplexing loop, so on today's builds
it is regression coverage for an invariant rather than a live detector, and the
preamble check binds no binary or container digest by itself. Both facts are in
the tool's own docstring.

Origin accounting. The witness tool is new content. The gap check's changes are
re-expressed from work done against this engine earlier and brought onto current
dev; the byte offsets and the explicit classification of the capture's shape
were added during review, as was the separate exit status for the unsupported
case. No file outside the four listed is touched.

Durable vs current state: the grammars, the gate and the exit-status
contract are durable. The case count (105) is current state, measured 2026-09-04
against base e1efc68 on macOS (Apple silicon) with Python 3.12.