feat(BACKEND-BENCH-CUDA-SGLANG-PREFLIGHT): a third benchmark subject can be named, and the key that names it NAMES its checkpoint (#1594)

[localai-bot]

tools/bench/run_serve_low.py declared --model-key as choices=("27", "35"), and that value is the key of the evidence tree: it is a directory name in BenchRun.corpus_path, in BenchRun.output_path, and again in the two in-container paths build_bench_command hands the pinned sglang.bench_serving. #1574's subject, r0b0tlab/Qwen3.8-27B-NVFP4-MTP-sm121 @ 36f717a2, could therefore only be measured by filing its evidence under an existing subject's key.

The third key is q38mtp, deliberately not digit-prefixed: 27 already names a DIFFERENT 27B checkpoint, unsloth/Qwen3.6-27B-NVFP4, and online_gate.py's neighbouring 27/27n pair records that two 27B checkpoints share no goldens and no comparable ratios.

A key that names no checkpoint is the same defect as a key nothing admits

The failure this guards is not a crash. It is a run that completes, writes a plausible raw result, and files it under another subject's key. A choices edit alone does not stop that, and neither does a choices edit plus a fail-closed guard on the same list, which is what the first head of this branch shipped. Measured on that head: --model-key 27 accepted BOTH 890bdef7 (the 27 subject) and 36f717a2 (the q38mtp subject), each run exited 0, and summarize_serve_low.py aggregated them as one subject. This branch materially widened that hole rather than opening it, because 27 and q38mtp are both 27B NVFP4 checkpoints, where 27 and 35 at least differed in size.

The mitigation already existed one file away. tools/bench/online_gate.py:244-255 derives its choices from key-to-repository and key-to-revision maps, so a key cannot be named there without naming its checkpoint. That record is now serve_low_common.MODEL_CHECKPOINTS, read by every tool that keys on a subject:

• admitted_model_keys refuses to admit a key the registry does not carry, so admitting a key and pinning the weights it means are one edit and cannot drift apart.
• require_model_checkpoint binds --model-revision to the key.
• build_bench_command additionally requires the host to hold <model-repo>/snapshots/<revision>. That is the tokenizer path the command hands the container, so a repository directory without it is the wrong checkpoint under the right name, and refusing on the host turns a confusing in-container failure into one line.
• online_gate.py reads the same record instead of keeping a second copy of the same four pins. Its admitted SET stays its own: its goldens, MAX_NUM_BATCHED_TOKENS, MAX_MODEL_LEN and model-gate contracts are keyed on exactly 27, 27n, 35 and q3mxfp4.

Two more paths keyed on the subject

build_dry_run_manifest hardcoded a two-entry "models" block, so a q38mtp run through scripts/dgx-sglang-low-concurrency.sh --dry-run wrote a manifest.json naming neither the subject nor its checkpoint. The manifest is now derived from the admitted keys, which is the second reason the registry had to come first.

make_serve_low_corpus.py declared --model-key with no choices at all and wrote the value verbatim into the corpus manifest, while the output directory came from an independent --out. Both its parser and generate_corpus now refuse, and --tokenizer-revision is bound to the key as well, because a corpus is that subject's corpus only if its tokenizer is. Its admitted set is the whole registry rather than this campaign's three, since online_gate.py's planned corpus command and scripts/mxfp4-online-serving-grid.sh build their corpora with the same tool and legitimately pass q3mxfp4.

Evidence

The tests enter through main() on a real argv with a stubbed subprocess.run that resolves the container --output-file back through the bind mount the same command declared, so the host derivation and the container derivation must agree or the run cannot read its own result back. For every admitted key they assert that all four derived paths carry that key and that no other admitted key's tree was named or created. The manifest case runs the campaign shell itself, because a manifest built by calling build_dry_run_manifest directly would prove the function and not that the driver reaches it. Each expectation is written out in the test rather than read back from the harness.

Eleven negative mutations across the branch, each printed as an applied diff and restored byte-identically against a pre-taken sha256. Three from the first head: the corpus path hardcoded to 27, refused as missing corpus partition; the container raw path alone hardcoded to 27, which the corpus precondition cannot see and which surfaces as a host FileNotFoundError on the run's own result; and the key guard deleted. Eight from the repair: a fourth admitted key with no checkpoint, which now makes the tool refuse to import at all with unknown model key: 38; 35 deleted from the admitted set, and separately from the registry; the revision binding, the repository-snapshot binding, the corpus key binding and the corpus choices each deleted; and the manifest re-listed as the two-entry literal. Every one is red. tests/tools 340/340.

Records

The spec .agents/specs/cuda-sglang-low-concurrency.md is reconciled with the tree, which the first head left untouched. Its Shared checkpoint candidates table carries the third subject with its repository and exact revision and says plainly that it is not the 27 subject; its ## Now no longer reports any Qwen3.8-27B point as unsupported for want of an instrument, and separates what the harness can now name from the measurement and the equivalence classification that stay open; and the two ## Tests to port and add rows describe what those files now assert.

#1605 is filed and indexed, not fixed here. tests/tools/test_serve_low_client.py:135 asserts _CompletionHandler.peak <= 2 and is genuinely flaky: 1 failure in 30 standalone runs here, AssertionError: 3 not less than or equal to 2, and tests/CMakeLists.txt:12 registers all of tests/tools as one CTest target, so it reds CI at random. The number is the fixture's, not the client's: peak counts handler threads inside do_POST, a window strictly wider than the client's in-flight window. Repairing it changes what the case measures, so under ## Every change starts from an issue it takes its own row, spec and fresh review rather than riding inside a change about model keys.

No public document is owed. tools/bench, run_serve_low and --model-key appear zero times in docs/USAGE.md, docs/FEATURES.md and README.md: this harness is documented nowhere public, so there is no projection to update, and adding the first entry for one flag would create an obligation rather than project a changed one. No workload shape, concurrency ladder or timed semantic changed, and nothing starts a server or a container.

Closes #1594.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]