diff --git a/.agents/specs/nemotron-h-model.md b/.agents/specs/nemotron-h-model.md index b5723fa23..ec080b1b5 100644 --- a/.agents/specs/nemotron-h-model.md +++ b/.agents/specs/nemotron-h-model.md @@ -485,6 +485,22 @@ with the revision pinned as `parity::kNemotron35LightningNvfP4Revision` to. Output is coherent — Fibonacci is `0, 1, 1, 2, 3, 5, 8, 13, 21, 34` — so this is a healthy denominator, not a degraded one. +**★ The golden records no ENGINE configuration, and that is now stated in the +file itself (#926).** This section called it a pin. It is not one: `oracle.json` +recorded the model, the revision, `temperature`, `max_tokens` and the three +library versions, and not one engine knob, and this capture ran from an +uncommitted driver in `$HOME` on a host that was reimaged two days later. The +configuration is **unrecoverable** — six independent searches for it are recorded +in [`nemotron-oracle-golden-provenance.md`](nemotron-oracle-golden-provenance.md) +§2 — and two later runs under fully recorded configurations reproduce prompts 0 +and 1 exactly and neither reproduces prompt 2. The golden is kept and its tokens +are unchanged, but it now carries a `capture` block saying it is unattributed and +naming the issue that owes the re-derivation, +`scripts/nemotron-h-oracle-capture.py` can regenerate one that is attributed, and +the gates refuse a golden that says nothing either way. Read every token score +taken against this reference as a difference from an unattributable reference +until that re-derivation lands. + **Two traps recorded here because they cost time.** (1) `$HOME/venvs/vllm-oracle` on dgx symlinks to `vllm-oracle-v0.25.0-stage`, which predates `NemotronHMoEDecoderLayer`; a run through it fails and reads as "the model is @@ -2041,6 +2057,14 @@ forward is still the HOST reference and nothing runs on the paged runner (W6). ## Owed +- [#926](https://github.com/mudler/vllm.cpp/issues/926) — the golden's capture + configuration is unrecoverable, so it can be approximated but not re-derived. + The provenance half has landed (a committed generator, the `capture` block, and + gates that refuse a silent golden); the **re-derivation under a named + configuration** is still owed and is blocked on + [#1431](https://github.com/mudler/vllm.cpp/issues/1431). Spec + [`nemotron-oracle-golden-provenance.md`](nemotron-oracle-golden-provenance.md). + - [#847](https://github.com/mudler/vllm.cpp/issues/847) — **SWEPT, no longer owed here.** Row `FIX-REGISTRY-DOWNCAST-SWEEP` claimed the registry type-confusion class this row's §6d fix named but did not sweep, and closed it: diff --git a/.agents/specs/nemotron-oracle-golden-provenance.md b/.agents/specs/nemotron-oracle-golden-provenance.md new file mode 100644 index 000000000..406b1fd04 --- /dev/null +++ b/.agents/specs/nemotron-oracle-golden-provenance.md @@ -0,0 +1,636 @@ +# The Nemotron oracle golden cannot be regenerated, and now it says so + +**Issue:** [#926](https://github.com/mudler/vllm.cpp/issues/926) +**Row:** `MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm` ([model matrix](../model-matrix.md)) +**Owed out of:** [#517](https://github.com/mudler/vllm.cpp/issues/517), which +committed the golden +**Blocks:** [#1289](https://github.com/mudler/vllm.cpp/pull/1289) is being scored +against this golden at 95/96 + +## 0. Scope (headline verdict) + +`tests/parity/goldens/nemotron_35_lightning_greedy/oracle.json` is the reference +for every Nemotron-3.5-Lightning token claim in this tree. **Its capture +configuration is unrecoverable.** The recovery was attempted and failed on +evidence, not on effort, and §2 records the six independent places it was looked +for. + +What this row does instead: it commits a generator, extends the golden's format +to carry the engine configuration, and makes the artifact **state which of two +things it is** — attributed, or unattributed-and-saying-so. The third state, +silence, was what #926 filed, and it is what this removes. The lost +configuration is not invented, and no gate is weakened to accommodate its +absence. + +**In scope** + +- `scripts/nemotron-h-oracle-capture.py` — capture, verify and validate. +- The `capture` provenance block, written truthfully into the shipped golden. +- The contract, gated in three places that cannot drift apart: the generator's + `--check`, `tests/scripts/test_nemotron_h_oracle_capture.py`, and the C++ + consumer in `tests/vllm/models/test_nemotron_h_loader.cpp`. +- The A3 driver printing what it is being held to. + +**Out of scope** + +- Running the oracle. [#1431](https://github.com/mudler/vllm.cpp/issues/1431) + owns that and five attempts have already been killed by the host-memory + watchdog. This unit needs no GPU, which is why it could be done now. +- The index-29 top-2 margin ([#1388](https://github.com/mudler/vllm.cpp/issues/1388)). +- Deciding whether #1289's moved token is a defect. + +## 1. What `oracle.json` recorded, and what it omitted + +Seven top-level keys, as `af8170154` wrote them: + +| Key | Value | +|---|---| +| `vllm` | `0.23.1rc1.dev1511+g555967922` | +| `transformers` | `5.14.1` | +| `flashinfer` | `0.6.15.post1` | +| `model` | `/mnt/nas_share/checkpoints/nemotron-3.5-lightning-30b-nvfp4` | +| `revision` | `29f2d1746d8f41e316523194b19018707749b1b1` | +| `sampling` | `{temperature: 0.0, max_tokens: 32}` | +| `golden` | 3 entries: `prompt`, `prompt_token_ids`, `token_ids`, `text` | + +**Not one engine knob.** `enforce_eager`, `max_model_len`, `max_num_seqs`, +`max_num_batched_tokens`, `gpu_memory_utilization`, `num_gpu_blocks_override`, +`block_size`, the batch shape, `seed`, `ignore_eos`, the compile mode, the +cudagraph capture sizes, the attention backend and the MoE backend are all +absent. Every one of them can move a greedy argmax at a near-tie, because each +one changes batching, prefill chunking, paging or the kernel that runs — that +is, the reduction order. + +For contrast, this tree already knows how to do this: the sibling golden +`tests/parity/goldens/gdn_ba_projection_bf16_sm121/oracle.json` records +`device`, `compute_capability`, `cuda_runtime`, `torch_version`, `dispatch`, +`implementation`, `repetitions` and `vllm_target_commit`. The Nemotron golden is +the outlier, not the norm. + +## 2. The generator was never committed, and the configuration is gone + +Six checks, each of which could have found it: + +1. `git show --stat af8170154` adds exactly **three** files — + `.agents/specs/nemotron-h-model.md`, this golden, and + `tests/parity/hf_snapshot.h`. None is a generator. +2. `git log --all --oneline -- tests/parity/goldens/nemotron_35_lightning_greedy/` + returns **exactly one commit** in the whole history. The golden has never + been revised, re-attributed or regenerated. +3. `grep -rn nemotron_35_lightning_greedy scripts/` returns nothing, and a + per-branch `git ls-tree` sweep for a Nemotron capture script finds none. The + Nemotron scripts that do exist (`nemotron-h-a2q1-neartie-gap.py`, + `nemotron-h-a2q1-dgx-gate.sh`, `nemotron-h-a2q2b-gpu-gate.sh`) are all later + and none captures a golden. `git log --diff-filter=D -- scripts/` shows none + was deleted either. +4. The capture ran from `$HOME/venvs/vllm-oracle-next` on `dgx.casa` + (`af8170154`'s message and `nemotron-h-model.md` §5a). **`dgx.casa` was + reimaged on 2026-08-14**, two days later. +5. Nothing of the run reached shared storage: a recursive grep for + `MODEL_LOADED_OK` and `ORACLE_IDENTITY_OK` across `/mnt/nas_share/{rc, + experiments,staging,...}` returns nothing, and the oldest `rc/` job directory + is **2026-08-17** — the share post-dates the capture. +6. `af8170154`'s own message is the fullest surviving description of the run, + and it names the venv, the identity assertions, the arch, the layer count and + the block pattern. It names **no engine knob**. + +The only timestamp that exists is the commit's author date, +`2026-08-12T22:37:57Z`, and the golden now records it as that rather than as a +run time. + +**Verdict: unrecoverable.** Not "not found yet". + +## 3. The lead that looked implicit and is a COMMON TERM + +The logs of the rebuilt oracle show `kv_cache_dtype=fp8_e4m3` selected without +anyone asking for it, with `Checkpoint does not provide a q scaling factor. +Setting it to k_scale` (`kv_cache.py:134`) beside it. That is exactly the shape +of an unrecorded implicit choice that moves tokens — so it was checked rather +than assumed, and it is **not** a candidate difference. + +The **checkpoint** carries it: + +- `config.json` → `quantization_config.kv_cache_scheme` = + `{"dynamic": false, "num_bits": 8, "type": "float"}` +- `hf_quant_config.json` → `quantization.kv_cache_quant_algo` = `"FP8"` + +and at the pin, the default `kv_cache_dtype="auto"` is resolved from that +before `CacheConfig` is built: + +- `vllm/engine/arg_utils.py:1916` — `resolved_cache_dtype = + resolve_kv_cache_dtype_string(self.kv_cache_dtype, model_config)`, whose + result is passed as `cache_dtype=` at `:1928`. `:1916` is the **unique call + site of `resolve_kv_cache_dtype_string`** — the symbol has exactly four + occurrences at the pin: this call, the import (`:115`), a defensive comment + (`attention.py:282`) and the definition (`torch_utils.py:374`). It is *not* + the unique place the string `cache_dtype` appears, which is 29 occurrences on + 21 lines of `arg_utils.py`; the resolver is what is unique, and `:1928` is + where its result reaches `CacheConfig`. +- `vllm/utils/torch_utils.py:374-392` — returns early unless `kv_cache_dtype == + "auto"`, then reads `hf_config.quantization_config`. +- `vllm/utils/torch_utils.py:310-362` — for `quant_method` starting `modelopt`, + maps a `kv_cache_scheme` dict of exactly that shape to `"fp8"`. + +So **every** unoverridden run of this checkpoint at this pin gets fp8_e4m3, +including the 2026-08-12 capture. Same on both sides is a measurement, and the +golden now records it under `capture.forced_by_checkpoint_or_device` together +with the MoE backend (MARLIN — a device without native FP4 takes the first +supported backend and at this pin no environment knob selects another), the +dtype and the quantization method. + +**This narrows the unrecoverable set to the knobs a driver passes**, which is +also the set §1 lists. It does not recover any of them. + +## 4. Two configurations, two answers, both repeatable + +| Run | Configuration | Prompt 0 | Prompt 1 | Prompt 2 | +|---|---|---|---|---| +| 2026-08-18 `oracle_only.sh` attempt `a` | `max_model_len=512`, `max_num_seqs=8`, `gpu_memory_utilization=0.30`, `max_num_batched_tokens=512`, `enforce_eager=False`, `TokensPrompt`, one prompt per `generate()` | 32/32 | 32/32 | **26/32** | +| the #926 rebuild | `enforce_eager=True`, `gpu_memory_utilization=0.25`, `max_model_len=4096` | 32/32 | 32/32 | **29/32** | + +The first ran its configuration **twice in one process** (`ORACLE_LEG 1`, +`ORACLE_LEG 2`) with identical results, `ORACLE TOKEN MATCH: 180/192`, log at +`/mnt/nas_share/rc/nhspeed/oracle.a.out` (the worker's `/workspace/nhspeed`). + +That is **configuration sensitivity, not non-determinism**. The distinction +decides the gate form, and it decides it against weakening: AGENTS.md admits a +ratified distributional gate **only** where the oracle's own greedy decode is +non-deterministic, and here it is not. **A distributional gate is inadmissible +on this evidence.** What is licensed is re-deriving the golden under a named +configuration. + +Note what the table also says: **prompt 2 has never been reproduced by anything +this repository can name.** Prompts 0 and 1 have been, twice. + +## 5. Design + +### 5.1 The contract + +A Nemotron oracle golden is in exactly one of two states, and the file says +which: + +- `capture.engine_config_recorded = true` — then `capture.engine.resolved` + carries **every** key in `REQUIRED_ENGINE_KEYS`, `capture.batch.shape` says how + the prompts were submitted, `capture.legs >= 2` and `capture.legs_agree` is + true. +- `capture.engine_config_recorded = false` — then + `capture.unrecoverable_reason` says why and `capture.issue` names the issue + that owes the re-derivation, and `capture.engine` is null. "Unrecorded" and + "here is the record" cannot both be true. + +A null inside `resolved` is refused for every key but +`num_gpu_blocks_override`: **a value that could not be read is not a value that +was default.** That is the same rule AGENTS.md states for `.env` — a missing +value never becomes an assumption. + +The unattributed arm needs a **second** half, and the first round of this row +shipped without it. Everything above is satisfied by a file that says +"unrecorded", names an issue and argues **nothing**. The attributed arm is gated +by structure — twenty keys are in `resolved` or they are not — but the +unattributed arm's whole record is prose, and a contract that asks only whether +the prose is *truthy* gates the shape and not the substance. + +The fresh review demonstrated it rather than predicting it: gut `evidence`, +`forced_by_checkpoint_or_device` and `captured_utc_is`, put the single word +`"dunno"` in `unrecoverable_reason`, and **all four gates stayed green** while +`--check` kept printing `engine_config_recorded=False`. Every argument this +artifact rests on — that nothing has ever reproduced prompt 2, that +`kv_cache_dtype` is a common term rather than a candidate difference, that a +distributional gate is inadmissible — lived in ungated prose inside a data file. +That is silence wearing the shape of a record, which is the state #926 filed, +reached from the other side. + +So an unattributed golden also carries, and the contract checks: + +| Requirement | Why it is named | +|---|---| +| `captured_utc_is`, non-empty | a commit's author date read as a capture time is a **fabricated** provenance | +| `forced_by_checkpoint_or_device`, an object with `kv_cache_dtype`, `moe_backend`, `dtype`, `quantization`, each non-empty | these are the terms COMMON to every unoverridden run of this checkpoint. They are what **narrows** "unrecoverable" to the knobs a driver passes (§3), so deleting one widens the unrecoverable set without saying so | +| `evidence`, an object with `never_reproduced` and `gate_form`, each non-empty | whether anything ever reproduced this golden, and which gate form its behaviour licenses. `gate_form` is the field that refuses a distributional gate; losing it loses the reason the refusal was on evidence rather than on taste | + +and four of those fields carry a **length floor of 80 characters**, because their +content is an *argument* rather than a value: `unrecoverable_reason`, +`forced_by_checkpoint_or_device.kv_cache_dtype`, `evidence.never_reproduced` and +`evidence.gate_form`. + +**What the floor is, and what it is not.** It cannot prove the prose is *true* — +nothing in a checker can. A keyword grep would be worse: it proves only the +checker's own vocabulary and it reds an honest rewording. What a floor detects is +**removal**, which is the threat that was actually demonstrated, one word in +place of a paragraph. 80 is set from measurement rather than taste. In the +shipped golden those four fields are 814, 448, 702 and 293 characters, so the +tightest margin is **3.7x** and no honest rewrite of an argument approaches it, +while `dunno`, `unknown`, `TBD` and `see the spec` are all under it. AGENTS.md's +rule that a gate firing on ordinary work is the defect is why the floor is taken +from the *shortest* real field and not from the longest. + +Scoping the floor to the four *argument* fields is load-bearing rather than +tidy: `forced.dtype` and `forced.quantization` are **35 characters each**, so a +floor applied to every forced term would make the shipped golden red itself. + +`captured_utc_is` is required but deliberately **not** floored. Its job is to say +what the timestamp is, and that can honestly be said in a clause — +"af8170154's author date, not a run time" is 44 characters and is not a +hand-wave. Flooring it would be a gate that fires on ordinary work. + +The requirement is scoped to the **unattributed arm only**. An attributed golden +records `kv_cache_dtype`, `dtype`, `quantization` and `moe_backend` in +`engine.resolved` as *values*, so it owes no prose about them, and a requirement +that fired on both arms would red every future capture this generator writes. +`test_the_attributed_arm_is_not_burdened_by_these_keys` holds that scoping. + +The shipped golden is in the second state. It is **kept**, because deleting +evidence to make a gate green is never the repair. + +### 5.2 The generator + +`scripts/nemotron-h-oracle-capture.py`, three modes: + +| Mode | Needs | Does | +|---|---|---| +| `--check ` | nothing | validates the contract. This is the CI gate. | +| `--verify ` | the oracle | runs and compares, reporting a **configuration** difference before a token difference | +| `--capture --out

` | the oracle | runs and writes a golden that records its own configuration | + +Four properties are deliberate: + +1. **Identity is asserted, never assumed** — the pin substring `555967922` must + be in `vllm.__version__` and the run aborts otherwise. `$HOME/venvs/ + vllm-oracle` on dgx has resolved to a 0.25.0 rollback that predates + `NemotronHMoEDecoderLayer`, and a run through it fails in a way that reads as + "the model is unsupported". +2. **The configuration is read BACK OUT of the built engine**, not echoed from + the kwargs. `kv_cache_dtype`, the block size, the block count and the + backends are chosen by vLLM, so what a driver passed is not what it ran. +3. **`--capture` refuses to write** a golden that fails its own contract, or + whose legs disagree. A golden written from disagreeing legs records a coin + flip. +4. **There is no builder for the unattributed block, deliberately.** One + existed and had exactly one caller: a fixture in the test suite. `--capture` + cannot reach it — a capture that runs records its configuration, which is the + mode's whole point — so no production path produced the shape it described, + and the one unattributed golden this repository has was hand-written and + carried three keys the builder could not emit. Under "Nothing lands dead" + that is a helper documenting a production shape nothing in production + produces, and it is gone. Deleting it also repaired the fixture: the suite's + own rule is that a fixture must never be derived from the module under test, + because setup and expectation then move together and a key dropped from the + checker drops from the fixture too. That fixture was the one place the suite + broke its own rule. It is a test-owned literal now. +5. **The body is under `if __name__ == "__main__":`** — vLLM v1 spawns + EngineCore, the module re-imports, and an unguarded driver fails as a + `multiprocessing` traceback naming neither vLLM nor the caller. The tell is + the banner printing twice. + +### 5.3 The named profile + +`--profile nhspeed-a` is the 2026-08-18 configuration of §4: the only oracle +configuration on this checkpoint for which this repository has determinism +evidence, with its full resolved config readable at +`/mnt/nas_share/rc/nhspeed/oracle.a.out`, and with CUDA graphs ON because +`--enforce-eager` is never the denominator. **It is a name for a run that +happened. It is not a reconstruction of the lost one**, and nothing in this row +claims it is. + +It is a token-golden configuration and **not** a speed denominator: +`max_num_batched_tokens=512` against the denominator's 8192 is a regime you +cannot tune down and keep a ratio through. + +## 6. Gates + +```sh +python3 scripts/nemotron-h-oracle-capture.py --check \ + tests/parity/goldens/nemotron_35_lightning_greedy/oracle.json +python3 tests/scripts/test_nemotron_h_oracle_capture.py +ctest --test-dir build -R test_nemotron_h_loader --output-on-failure +``` + +All three run with no vLLM, no GPU and no checkpoint. That is the point: the +provenance defect is a records defect, and a gate for it must not need the +hardware whose absence caused it. + +The contract exists in three copies — the generator's `REQUIRED_ENGINE_KEYS`, +the C++ consumer's `required` list, and this suite's `EXPECTED_ENGINE_KEYS`. The +suite asserts all three agree and **owns the expectation itself**, so a key +deleted from both production copies is still red. + +## 7. Evidence + +**Measured on** `mudler-ubuntu-box` (x86_64, 20 cores), in the worktree +`.claude/worktrees/row-926-golden-provenance` at base `5d548d003`, CPU-only +Release build (`-DVLLM_CPP_CUDA=OFF`), nothing overlaid. `libvllm 0.0.3 (ABI 23, +header 23)`. vLLM anchors read at `/home/mudler/_git/vllm` HEAD +`5559679229bc961848b121ccdeaa8fa5d79bec98`, remote `vllm-project/vllm`, +verified before citing. + +### RED first + +With the golden as `af8170154` left it, the suite collected 25 cases and +**3 were red**: `test_the_shipped_golden_satisfies_the_contract`, +`test_check_reads_the_shipped_golden` and +`test_the_shipped_golden_is_not_silently_attributed`, each reporting +`oracle.json: missing 'capture'`. After the provenance block: **26/26 OK**. + +### Mutation proof — the golden (data) + +Each mutation applied alone, `git diff --stat` printed to prove it APPLIED, the +target rebuilt to prove `compile_rc=0`, the case rerun, then the tree restored +and its **sha256** re-asserted against the pre-mutation baseline +`659c26bd2301317d4a6999df0b7afc3243dcff129de89abcb66b46817dd6f9e9`. A mutation +that never applied and a mutation that failed to build both read as a passing +test, so neither is inferred here. + +| # | Mutation | diff | Result | +|---|---|---|---| +| — | baseline | — | 33 assertions, 0 failed, SUCCESS | +| M1 | delete the `capture` block (the `af8170154` shape) | -24 | 20 assertions, **1 failed** | +| M2 | claim `engine_config_recorded: true` while `engine` is null | 1/1 | 28, **1 failed** | +| M3 | blank `unrecoverable_reason` | 1/1 | 33, **1 failed** | +| M4 | point `issue` at a non-vllm.cpp string | 1/1 | 33, **1 failed** | +| M5 | truncate one golden row below `max_tokens` | 1/-28 | 33, **1 failed** | +| M6 | empty the `golden` array | 1/-148 | 7, **1 failed** | +| M7 | drop `engine_config_recorded` so the file says NOTHING | -1 | 21, **1 failed** | + +M5 and M6 are the anti-vacuity arms: a comparison over zero elements reports a +perfect score, so the width is asserted rather than trusted. M7 is the third +state — not "unrecorded", but silent — which is the state #926 filed. + +### Mutation proof — the case is the SOLE holder (source) + +`compile_rc` is load-bearing here, and it is printed. **M8**: delete the +`capture` requirement from the C++ case itself (`compile_rc=0`, 0 compile +errors, `git diff --stat` 1 insertion), then re-apply M1. The **whole binary** +then reads `3 passed | 0 failed`, **26 assertions** against the restored 40, and +`Status: SUCCESS!` over a golden with no provenance at all. + +Nothing else in this tree holds the guarantee. Restored, the binary is 40 +assertions and both sha256s match. + +### Mutation proof — the checker (source) + +Each applied alone, parsed (`parse_rc=0` — a mutation that does not parse reads +as a passing test), the suite rerun, restored by sha256 +`a477bcefbcc90da30d8e0cae016fc474e281b74889ed5107553a6f851ce54fce`. + +| # | Mutation | Result | +|---|---|---| +| M9 | stop requiring a reason on an unattributed golden | **1 failure** | +| M10 | stop requiring every engine key (`for key in ()`) | **2 failures** | +| M11 | drop the anti-vacuity width check | **1 failure** | + +`test_each_required_engine_key_is_load_bearing` additionally drops each of the +20 engine keys in turn and asserts the contract names the one it dropped, so +M10 is a floor and not the whole proof. + +### The driver states all three cases + +`nemotron-h-gen --golden-info` on the shipped golden, on a copy with +`engine_config_recorded: true`, and on a copy with the block removed: + +``` +capture: engine configuration UNRECORDED — a token difference below is + UNATTRIBUTABLE, not yet a defect; owed by .../issues/926 +capture: engine configuration RECORDED +capture: the golden does not SAY whether its engine configuration was recorded + (no capture block) +``` + +The same line prints beside `DIVERGENCE`, because a reader who sees `DIVERGENCE` +and stops reading is the reader it is for. + +### A trap this hit + +`REQUIRE_MESSAGE(..., "capture is missing '" << key << "'")` over a +`const char*` printed `capture is missing '1'`: doctest stringifies a bare +`char*` as a **bool**. The loop iterates `std::string` now, and the messages read +`capture is missing 'schema'`. + +### The repair round — closing the SHAPE-not-SUBSTANCE gap + +The fresh review refuted nothing and returned findings. The substantive one is +above: the contract gated the shape of the unattributed block and not its +content. The re-run below is the proof that it no longer does. + +**Mutation E, re-applied verbatim** to the shipped golden: `unrecoverable_reason` +→ `"dunno"`, `captured_utc_is` → `""`, `forced_by_checkpoint_or_device` → `{}`, +`evidence` → `{}`. Applied alone, `json_parse_rc=0` (a mutation that does not +parse reads as a passing test), `git diff --stat` **5 insertions, 13 deletions** +so it demonstrably APPLIED, then restored and its sha256 re-asserted against the +pre-mutation baseline +`659c26bd2301317d4a6999df0b7afc3243dcff129de89abcb66b46817dd6f9e9`. + +| Gate | Before the repair | After | +|---|---|---| +| `--check` | 0 problems, **rc=0** | **8 problems, rc=1**, naming all four gutted fields | +| the Python suite | 26/26 OK | **2 failures**, rc=1 | +| the C++ consumer | `SUCCESS!` | **2 failed, `Status: FAILURE!`** | + +The C++ arm is the one that matters for the cross-gate: the guarantee is not +held by one file. `Status:` is read as well as `assertions:`, because an +`assertions:` line can say `0 failed` while cases threw. + +**The assertion COUNT under this mutation is shape-dependent, so it is not the +evidence.** A `REQUIRE` aborts its case, so the total depends on which one trips +first, which depends on the mutation's exact diff shape and on whether the count +is read for the single case or for the whole binary. This round measured **38** +for the provenance case alone at diff shape `+5/-13`; the fresh reviewer +measured **45** at `+4/-12`. Both are correct measurements of different shapes, +and an evidence table carrying a number that does not reproduce is worse than +one that omits it. What DOES reproduce, and what the gate rests on, is +**`2 failed`, `Status: FAILURE!`, rc=1**. + +Green again on the restored tree: `--check` 0 problems, the suite **41 cases OK** +(26 before this round, 13 new substance cases, 2 new cross-gate cases), and the +C++ case **63 assertions** against the previous 40, `Status: SUCCESS!`. + +The new contract holds the **shipped golden byte-for-byte unchanged**. Its +sha256 is the same value before this round and after it, and its diff against +`af8170154` is still 24 insertions and **zero** deletions. + +### The three copies still cannot drift, including the new ones + +`test_the_cpp_consumer_names_every_unattributed_key` parses `kForcedTermKeys` and +`kEvidenceKeys` out of the C++ source as initializer lists and compares them as +sets, and `test_the_cpp_consumer_carries_the_same_argument_floor` reads +`kMinArgumentChars` out of it. Parsed, not grepped, and the distinction is +load-bearing: `"dtype"` and `"quantization"` already appear in the C++ +*engine-key* list, so `assertIn('"dtype"', source)` would have passed without the +unattributed arm naming either of them. The expectation is the suite's own +literal, so a key deleted from both production copies is still red. + +`test_each_forced_term_is_load_bearing` and `test_each_evidence_key_is_load_bearing` +drop each key in turn and assert the contract names the one dropped, and +`test_each_argument_field_refuses_a_one_word_answer` puts `"dunno"` in each +floored field in turn. `test_an_argument_at_the_floor_is_accepted` is the other +side of the floor, so the cases cannot pass by refusing everything. + +### Mutation proof — the NEW gate, and the new cross-gate + +Four more, each applied alone, `parse_rc=0` printed, `git diff --stat` printed to +prove it applied, restored by sha256 (`5b81281f7fa76824253d4b119459c1dad0b54ed2e35c6d4265beb53db9967411` +for the checker, `812808faf1e149378fb13c84be6ccb0a661d2b45c62f5d652361d14d9a60b93e` +for the C++ consumer). + +| # | Mutation | Result | +|---|---|---| +| M12 | lower the checker's floor, `MIN_ARGUMENT_CHARS = 80 → 5`, to hide a shrinking record | **3 failures** | +| M13 | stop requiring the evidence keys, `REQUIRED_EVIDENCE_KEYS = ()` | **4 failures** | +| M14 | drop `quantization` from the **C++** `kForcedTermKeys` alone, leaving both other copies intact | **1 failure**, `test_the_cpp_consumer_names_every_unattributed_key` | +| M15 | change the **C++** `kMinArgumentChars` to 5 alone | **1 failure**, `test_the_cpp_consumer_carries_the_same_argument_floor` | + +M14 and M15 are the ones worth reading. Nothing in Python imports the C++ file, +so a mutation confined to it could only be caught by a case that reads that +source and holds it to a suite-owned expectation. Both fire, and they fire on the +key that was actually removed. The floor and the key lists cannot be quietly +relaxed in either direction: raising the floor to hide a shrinking record is red, +and lowering it to admit one is red too. + +### One more disagreement between the copies, found while proving the fix + +The checker tested `unrecoverable_reason` for **truthiness** while the C++ copy +had always spelled it `is_string()`. So `"unrecoverable_reason": 123` returned +**zero problems** from `--check` and was refused by the C++ gate: two copies of +one contract disagreeing about what satisfies it, which is exactly the drift +this three-copy design promises cannot happen. The weaker copy moved to +`_is_prose`, and `test_a_non_string_reason_is_refused` holds it there. Suite +41 → 42 cases. + +### The copies disagreed a SECOND time, on whitespace + +Found by the fresh review, and it is the `123` divergence again with the weaker +copy on the other side. Python has always spelled these `value.strip()`; the C++ +consumer spelled them `.empty()` and `.size()` on the raw string. So a golden +carrying **200 spaces** in `unrecoverable_reason`, `evidence.never_reproduced` +and `evidence.gate_form` read: + +| Copy | Before | After | +|---|---|---| +| `--check` | **3 problems, rc=1** | 3 problems, rc=1 | +| the C++ consumer | **70 assertions, 0 failed, `SUCCESS!`, rc=0** | **79 assertions, 6 failed, `Status: FAILURE!`, rc=1** | + +A blank paragraph is the record going missing exactly as surely as a deleted +one. `TrimmedProse()` now matches `_is_prose`/`len(value.strip())` exactly. + +**The scope is deliberate and is not "trim everything".** Python trims in +`_is_prose` and in the argument floor, and it does **not** trim +`capture.batch.shape`, which it tests for truthiness. Trimming that one in C++ +would have repaired this divergence by opening its mirror image, so it is left +alone. + +**The C++ arm can now hold its half alone.** The provenance case reads the one +committed golden, so it can only ever exercise the shape that golden happens to +have — which is why this arm could not notice. `NemotronH golden: a blank +paragraph is not prose` tests `TrimmedProse` directly and is the counterpart of +`test_the_floor_is_not_met_by_whitespace` on the Python side. Proven: reverting +`TrimmedProse` to the untrimmed behaviour reds it **5 of 9 assertions with the +golden untouched** (`compile_rc=0`, 0 compile errors, `git diff --stat` +printed), restored by sha256 +`00cd7e05a32fa4be3cc852156633dbdc4b3069bc183bbeb4889183c7613f80cb`. It asserts +both sides of the rule, so it cannot pass by refusing everything. + +### The driver's third state named a cause it could not know + +`nemotron-h-gen` printed `(no capture block)` for `capture_recorded == -1`. The +tri-state was right and the parenthetical was not: two different files reach that +arm — a golden with no `capture` block at all, and a golden that HAS one whose +`engine_config_recorded` flag is missing or unreadable, which is the file the +reviewer actually fed it. The message now names both and asserts neither. +Verified on all four inputs: `recorded=false` → UNRECORDED, `recorded=true` → +RECORDED, no block → the tri-state line, block-with-flag-deleted → the same +tri-state line. + +### One red that is not this row's + +`scripts/agent-preflight.sh` reports `test_cpu_x86_llamacpp_floor` failing on +`test_a_contended_leg_is_discarded_and_never_summarised`. That is +[#618](https://github.com/mudler/vllm.cpp/issues/618) by its exact case name — +the harness is load-dependent and this box had just finished a build. The suite +and its script are **byte-identical to `origin/main`** in this diff +(`git diff --stat origin/main...HEAD -- tests/scripts/test_cpu_x86_llamacpp_floor.py +scripts/cpu-x86-llamacpp-floor.sh` is empty, against a positive control on this +row's own files that is not). Every other preflight gate is green. + +## 8. Risks + +- **A named profile could be mistaken for the recovered one.** Mitigated by + saying so in the generator, in the golden and in §5.3, and by the fact that + `nhspeed-a` reproduces prompt 2 at 26/32 — it is visibly not the lost + configuration. +- **Three copies of the key list can drift.** The suite asserts they agree and + owns the expectation. +- **The re-derivation needs the oracle**, which #1431 blocks. Nothing here + depends on it; the contract admits the unattributed state precisely so this + work did not have to wait. + +## 9. Stop conditions + +- Do **not** relax the contract to admit a silent golden. +- Do **not** ratify a distributional gate for this row. §4 shows the oracle is + deterministic at a fixed configuration, which is the condition AGENTS.md + requires be **absent**. +- Do **not** write a reconstructed configuration into the golden. An invented + provenance is worse than a stated absence. + +## 10. Now + +Landed: the generator, the contract in **both** its halves — the attributed +arm's twenty keys and the unattributed arm's required, floored record — its +three gates, the truthful provenance block, and the driver line. The golden's +tokens are **byte-for-byte unchanged** — the diff against `af8170154` is 24 +inserted lines and zero deletions, and its sha256 is the same before and after +the repair round. + +## 11. Owed + +- **The re-derivation, and it needs a decision.** Re-deriving the golden under + `--profile nhspeed-a` would replace an unattributable reference with an + attributable one and would change the reference #1289 is scored against — from + 32/32, 32/32, 32/32 to 32/32, 32/32, 26/32 on the oracle side. That is a + change to a gate's reference and is the developer's call, not an implementer's. + It is blocked on #1431 either way. +- **The index-29 top-2 margin** (#1388), also blocked on #1431. Note the + ordering this row establishes: the margin measures how close the two + candidates are; it does not tell you which configuration produced the + reference. Both are needed and this one was cheaper. +- **The other goldens.** This contract is Nemotron-only by design; whether the + rest of `tests/parity/goldens/` can name their capture configurations is + unmeasured and is not claimed either way here. + +## 12. Outcome + +Recorded because the code does not say it: + +- **The recovery was attempted and failed.** §2 is the negative result, and it + is worth more than a plausible reconstruction would have been. +- **The `kv_cache_dtype` lead was closed by reading the checkpoint and the + pinned source**, not by running anything. It looked like the best candidate + and it is a common term. +- **The contract does not demand the configuration back.** Demanding it would + have made this row wait for #1431 and would have made `main` red on an + artifact nobody can currently fix. Demanding that the file *say which state it + is in* costs nothing, is checkable today, and is the property that was + actually missing. +- **A distributional gate was available and was rejected**, on the evidence in + §4 rather than on preference. +- **The first round gated the SHAPE and not the SUBSTANCE, and only a mutation + found it.** Both gate arms read as symmetric — one requires twenty keys, the + other requires a reason and an issue — and both were green on the shipped + artifact, so nothing in the diff looked wrong. What made the asymmetry visible + was gutting the block and watching four gates stay green. The rule this leaves + behind: **where a record is prose rather than structure, a truthiness check is + not a gate.** It admits the exact artifact the row exists to refuse, and it + admits it silently. +- **A length floor was chosen over a keyword grep, and the choice is a + limitation, not a feature.** The floor cannot see whether the prose is true; it + sees only that somebody removed it. That is a genuinely weaker guarantee than + the attributed arm's, and it is recorded as weaker rather than described as + equivalent. The alternative — asserting the text contains "distributional" or + "reimaged" — would gate the checker's own vocabulary and would red an honest + rewrite, which is the failure mode AGENTS.md names when it says a gate that + fires on ordinary work is the defect. +- **A helper with one caller, and that caller a test, is dead code even in a + script.** `unattributed_capture()` looked like production structure. It could + not emit three of the keys the artifact it described actually carries, and + `--capture` could never reach it. Deleting it removed a fourth copy of the + contract that could drift and repaired a fixture that was deriving its setup + from the module under test. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 733c8aa37..1ade07515 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -320,6 +320,20 @@ jobs: python3 scripts/check-oracle-pins.py python3 scripts/check-oracle-pins.py --self-test python3 tests/scripts/test_check_oracle_pins.py + - name: The Nemotron oracle golden says whether it can be regenerated (#926) + # A golden whose provenance is one person's home directory on one box is + # not a pin. `af8170154` committed this reference with no generator and + # no engine configuration, and the host was reimaged two days later, so + # nobody can re-derive it or say what produced it -- while a 95/96 is + # being scored against it. The contract does not demand the lost + # configuration back. It demands that the file state which of the two it + # is, so silence stops being an option. Runs with no vLLM, no GPU and no + # checkpoint, deliberately: the defect is a records defect and its gate + # must not need the hardware whose absence caused it. + run: | + python3 scripts/nemotron-h-oracle-capture.py --check \ + tests/parity/goldens/nemotron_35_lightning_greedy/oracle.json + python3 tests/scripts/test_nemotron_h_oracle_capture.py - name: cuBLASLt GEMM invocation stays dtype-faithful (template parity) run: | python3 scripts/check-gemv-invocation-consistency.py diff --git a/examples/nemotron_h_gen/main.cpp b/examples/nemotron_h_gen/main.cpp index 448c972f9..680a77267 100644 --- a/examples/nemotron_h_gen/main.cpp +++ b/examples/nemotron_h_gen/main.cpp @@ -118,6 +118,25 @@ std::string ReadStringAfter(const std::string& s, const std::string& key, return s.substr(open + 1, close - open - 1); } +// Read the boolean value that follows `"":`, as a tri-state: 1 true, +// 0 false, -1 the key is absent. Absent is its own answer here and must not +// collapse into false -- "the golden says its configuration is unrecorded" and +// "the golden says nothing at all" are exactly the two states #926 separates. +int ReadBoolAfter(const std::string& s, const std::string& key, size_t from) { + const size_t k = FindKey(s, key, from); + if (k == std::string::npos) return -1; + const size_t colon = s.find(':', k); + if (colon == std::string::npos) return -1; + const size_t t = s.find("true", colon); + const size_t f = s.find("false", colon); + const size_t stop = s.find(',', colon); + const bool t_ok = t != std::string::npos && (stop == std::string::npos || t < stop); + const bool f_ok = f != std::string::npos && (stop == std::string::npos || f < stop); + if (t_ok && (!f_ok || t < f)) return 1; + if (f_ok) return 0; + return -1; +} + struct GoldenEntry { std::vector prompt_token_ids; std::vector token_ids; @@ -127,6 +146,10 @@ struct Golden { std::string vllm_version; std::string model; std::string revision; + // #926: whether the golden records the ENGINE CONFIGURATION it was captured + // under. 1 recorded, 0 recorded-as-unrecorded, -1 the file does not say. + int capture_recorded = -1; + std::string capture_issue; std::vector entries; }; @@ -136,6 +159,11 @@ Golden ReadGolden(const std::string& path) { g.vllm_version = ReadStringAfter(s, "vllm", 0); g.model = ReadStringAfter(s, "model", 0); g.revision = ReadStringAfter(s, "revision", 0); + const size_t cap = FindKey(s, "capture", 0); + if (cap != std::string::npos) { + g.capture_recorded = ReadBoolAfter(s, "engine_config_recorded", cap); + g.capture_issue = ReadStringAfter(s, "issue", cap); + } const size_t arr = FindKey(s, "golden", 0); if (arr == std::string::npos) throw std::runtime_error("golden: no \"golden\" array in " + path); @@ -254,6 +282,32 @@ int main(int argc, char** argv) { " entries=%zu\n", golden_path.c_str(), gold.vllm_version.c_str(), gold.model.c_str(), gold.revision.c_str(), gold.entries.size()); + // ── What this run is being held to (#926) ────────────────────────────── + // The tokens below are a difference from a REFERENCE, and a difference is + // only a defect once the reference can be regenerated. This golden's + // capture configuration was never recorded and its capture host has since + // been reimaged, so this line prints beside every score taken against it. + if (gold.capture_recorded == 1) { + std::fprintf(stderr, + " capture: engine configuration RECORDED\n"); + } else if (gold.capture_recorded == 0) { + std::fprintf(stderr, + " capture: engine configuration UNRECORDED — a " + "token difference below is UNATTRIBUTABLE, not yet a defect%s%s\n", + gold.capture_issue.empty() ? "" : "; owed by ", + gold.capture_issue.c_str()); + } else { + // Two different files reach this arm and the message must not name + // either: a golden with NO `capture` block at all (the af8170154 shape), + // and a golden that HAS one whose `engine_config_recorded` flag is + // missing or unreadable. The tri-state is what separates them from + // "unrecorded"; it does not separate them from each other, and a + // parenthetical that picked one was wrong for the other. + std::fprintf(stderr, + " capture: the golden does not SAY whether its " + "engine configuration was recorded (no `capture` block, or " + "one with no readable `engine_config_recorded` flag)\n"); + } if (gold.entries.empty()) { std::fprintf(stderr, "[nemotron-h] REFUSING: the golden carries ZERO entries, so " @@ -414,6 +468,17 @@ int main(int argc, char** argv) { } if (total_matched != total_compared) { std::fprintf(stderr, "[nemotron-h] DIVERGENCE\n"); + // #926: say it HERE too, not only in the header 400 lines up. A reader who + // sees "DIVERGENCE" and stops reading is the reader this line is for. + if (gold.capture_recorded != 1) { + const std::string owed = + gold.capture_issue.empty() ? std::string() : (" (" + gold.capture_issue + ")"); + std::fprintf(stderr, + "[nemotron-h] ...against a golden whose engine configuration " + "is NOT recorded, so this is a difference from an " + "unattributable reference%s\n", + owed.c_str()); + } return 1; } std::fprintf(stderr, "[nemotron-h] STRICT PASS\n"); diff --git a/scripts/nemotron-h-oracle-capture.py b/scripts/nemotron-h-oracle-capture.py new file mode 100755 index 000000000..cdcf98230 --- /dev/null +++ b/scripts/nemotron-h-oracle-capture.py @@ -0,0 +1,760 @@ +#!/usr/bin/env python3 +# Nemotron-3.5-Lightning-30B-A3B-NVFP4 (`NemotronHForCausalLM`) — the generator +# for tests/parity/goldens/nemotron_35_lightning_greedy/oracle.json, and the +# validator for its provenance contract (#926). +# +# ── WHY THIS FILE EXISTS ──────────────────────────────────────────────────── +# The golden was committed by `af8170154` (#517) with NO generator. That capture +# ran from `$HOME/venvs/vllm-oracle-next` on `dgx.casa`, and the host was +# reimaged on 2026-08-14, so the driver, its log and its engine configuration are +# gone. `oracle.json` recorded the model, the revision, the sampling parameters +# and the vllm/transformers/flashinfer versions — and NOT ONE engine knob. A +# reference nobody can regenerate and nobody can attribute is not a pin, and the +# A2-Q1 device mamba arm's 95/96 (#1289, #1388) is currently being scored against +# exactly that. `.agents/specs/nemotron-oracle-golden-provenance.md` carries the +# recovery attempt and its negative result. +# +# Two runs have since failed to reproduce the golden's third prompt, each under a +# DIFFERENT and FULLY RECORDED configuration, which is what makes the missing +# record the defect rather than a curiosity: +# +# * 2026-08-18, `/workspace/nhspeed/oracle_only.sh` attempt `a` on `dgx:gpu0` +# (`/mnt/nas_share/rc/nhspeed/oracle.a.out`): 32/32, 32/32, **26/32**, twice +# in one process, `ORACLE TOKEN MATCH: 180/192`. +# * the #926 rebuild (`enforce_eager=True`, `gpu_memory_utilization=0.25`, +# `max_model_len=4096`): 32/32, 32/32, **29/32**, diverging at index 29. +# +# Same box, same checkpoint, same greedy sampling, two different configurations, +# two different answers, each internally repeatable. That is CONFIGURATION +# SENSITIVITY, not non-determinism, and AGENTS.md admits a ratified +# distributional gate only for a non-deterministic greedy decode. So the licensed +# repair is to re-derive the golden under a NAMED configuration — which is what +# `--capture` below does, and why it refuses to write a golden it cannot +# attribute. +# +# ── THE ONE LEAD THAT IS ALREADY CLOSED ───────────────────────────────────── +# `kv_cache_dtype=fp8_e4m3` is auto-selected on this checkpoint and vLLM imputes +# the missing q scale ("Checkpoint does not provide a q scaling factor. Setting +# it to k_scale"). It reads like an unrecorded implicit choice. It is not a +# candidate difference: the CHECKPOINT carries it. `config.json`'s +# `quantization_config.kv_cache_scheme` is `{dynamic: false, num_bits: 8, type: +# "float"}` and `hf_quant_config.json`'s `quantization.kv_cache_quant_algo` is +# `"FP8"`, and at the pin `vllm/engine/arg_utils.py:1916` resolves the DEFAULT +# `kv_cache_dtype="auto"` through `vllm/utils/torch_utils.py:374-392` -> +# `:310-362`, which maps that dict to `"fp8"`. Every run of this checkpoint that +# does not explicitly override the knob gets fp8_e4m3, including the capture. It +# is a COMMON TERM. Recorded anyway, because "the same on both sides" is a +# measurement and not an assumption. +# +# ── MODES ─────────────────────────────────────────────────────────────────── +# --check validate the provenance contract. Needs NO vLLM, no GPU +# and no checkpoint; this is what CI runs. +# --verify run the oracle and compare against a committed golden. +# Reports a CONFIGURATION difference before a token +# difference, because an unattributable token count is +# not evidence. +# --capture --out

run the oracle and WRITE a golden that records the +# configuration it ran under. +# +# `--verify` and `--capture` need the pinned oracle. Run them under an `rc` lease +# on `dgx:gpu0` (never `ssh` to a fleet device), from the pinned venv: +# +# $VENV/bin/python scripts/nemotron-h-oracle-capture.py --capture \ +# --model $CHECKPOINT_ROOT/nemotron-3.5-lightning-30b-nvfp4 \ +# --profile nhspeed-a --legs 2 \ +# --out tests/parity/goldens/nemotron_35_lightning_greedy/oracle.json +# +# NOTE the body is guarded by `if __name__ == "__main__":`. That is not style: +# vLLM v1 spawns EngineCore, the module re-imports, and an unguarded driver fails +# as a `multiprocessing` traceback naming neither vLLM nor the caller. The tell +# is the banner printing twice (`.agents/specs/nemotron-h-model.md` §5a). +"""Capture, verify and validate the Nemotron-3.5-Lightning greedy oracle golden.""" + +from __future__ import annotations + +import argparse +import json +import os +import sys +import time + +# The three prompts the committed golden carries, in order. Kept here so a +# capture that drifts from the committed battery is caught by the writer rather +# than by a reader months later. +PROMPTS = [ + "The capital of France is", + "Write the first five Fibonacci numbers:", + "Explain what a state space model is, in one sentence:", +] + +# The parity pin. `vllm.__version__` at this commit spells it +# `0.23.1rc1.dev1511+g555967922` from a source tree and `0.1.dev1+g555967922` +# from the FlashInfer-only wheel, so the ASSERTION is on the commit substring, +# never on the whole version string. +ORACLE_COMMIT = "555967922" + +# The checkpoint revision the golden belongs to, mirrored from +# `parity::kNemotron35LightningNvfP4Revision` (tests/parity/hf_snapshot.h). +CHECKPOINT_REVISION = "29f2d1746d8f41e316523194b19018707749b1b1" + +SCHEMA = 2 + +# ── Named engine configurations ───────────────────────────────────────────── +# A profile is a NAME for a configuration that this repository has a recorded +# run of. It is not a guess at the lost one. `nhspeed-a` is the 2026-08-18 run +# whose full resolved config is readable at +# /mnt/nas_share/rc/nhspeed/oracle.a.out (the worker's /workspace/nhspeed), and +# it is the only oracle configuration on this checkpoint for which this +# repository has determinism evidence: two legs in one process, byte-identical. +# CUDA graphs stay ON in it, because `--enforce-eager` is never the denominator. +PROFILES = { + "nhspeed-a": { + "max_model_len": 512, + "max_num_seqs": 8, + "gpu_memory_utilization": 0.30, + "max_num_batched_tokens": 512, + "enforce_eager": False, + "num_gpu_blocks_override": None, + }, +} + +# Every engine key a re-derived golden must carry. The list is the set of knobs +# that can move a greedy argmax on this engine — batch shape, prefill chunking, +# paging, graph capture, the resolved kernel backends, and the dtypes — plus the +# two that identify the run at all. A key that is absent is not "default"; it is +# unrecorded, which is the whole defect this file exists to close. +REQUIRED_ENGINE_KEYS = ( + "attention_backend", + "block_size", + "cudagraph_capture_sizes", + "cudagraph_mode", + "compilation_mode", + "dtype", + "enable_chunked_prefill", + "enable_prefix_caching", + "enforce_eager", + "gpu_memory_utilization", + "kv_cache_dtype", + "max_model_len", + "max_num_batched_tokens", + "max_num_seqs", + "moe_backend", + "num_gpu_blocks", + "num_gpu_blocks_override", + "quantization", + "seed", + "tensor_parallel_size", +) + +# `num_gpu_blocks_override` and `cudagraph_capture_sizes` are legitimately null +# or empty (no override; eager). Everything else must carry a value: a null +# there means the reader failed, and a failed read must not be written down as +# a fact. +NULLABLE_ENGINE_KEYS = frozenset({"num_gpu_blocks_override"}) + +ISSUE_PREFIX = "https://github.com/mudler/vllm.cpp/issues/" + +# ── What an UNATTRIBUTED golden has to carry (#926) ───────────────────────── +# The attributed arm is gated by STRUCTURE: `engine.resolved` either carries all +# twenty keys or it does not, and a checker can answer that exactly. The +# unattributed arm has no structure to gate, because its whole record IS prose — +# and a contract that asks only whether the prose is TRUTHY gates the shape and +# not the substance. +# +# That gap was demonstrated, not imagined. A reviewer gutted +# `forced_by_checkpoint_or_device`, `evidence` and `captured_utc_is` and replaced +# an 814-character `unrecoverable_reason` with the single word "dunno", and all +# four gates stayed green while `--check` kept printing +# `engine_config_recorded=False`. Every argument this artifact rests on — that +# nothing has ever reproduced prompt 2, that `kv_cache_dtype` is a common term +# rather than a candidate difference, that a distributional gate is inadmissible +# — lived in ungated prose inside a data file. Silence wearing the shape of a +# record is exactly what #926 filed, so the gap is closed here rather than noted. +# +# It is closed in the two ways a checker can actually answer, and in no way it +# cannot. + +# 1. STRUCTURE. Present, an object, and every sub-key a non-empty string. +# +# `forced_by_checkpoint_or_device` names the terms that are COMMON to every +# unoverridden run of this checkpoint on this class of device. It is what +# NARROWS "unrecoverable" to the knobs a driver passes, so deleting an entry +# widens the unrecoverable set without saying so. +REQUIRED_FORCED_TERM_KEYS = ("kv_cache_dtype", "moe_backend", "dtype", "quantization") + +# `evidence` carries the two findings that decide how this golden may be USED: +# whether anything has ever reproduced it, and which gate form its behaviour +# licenses. `gate_form` is the one that refuses a distributional gate; a golden +# that loses it loses the reason the refusal was on evidence rather than taste. +REQUIRED_EVIDENCE_KEYS = ("never_reproduced", "gate_form") + +# 2. SUBSTANCE. A length floor, on the four fields whose content is an ARGUMENT +# rather than a value. +# +# A floor cannot prove the prose is TRUE. Nothing in a checker can. A keyword +# grep would be worse than useless here: it proves only the checker's own +# vocabulary, and it reds an honest rewording. What a floor detects is REMOVAL, +# which is the threat that was actually demonstrated — one word in place of a +# paragraph. +# +# 80 is set from measurement rather than taste. In the shipped golden the four +# floored fields are 814, 448, 702 and 293 characters, so the tightest margin is +# 3.7x and no honest rewrite of an argument comes near it, while "dunno", +# "unknown", "TBD" and "see the spec" are all under it. AGENTS.md's rule that a +# gate firing on ordinary work is the defect is why the floor is set from the +# SHORTEST real field and not from the longest. +MIN_ARGUMENT_CHARS = 80 + +# The floored fields, as paths under `capture`. `captured_utc_is` is required +# and must be a non-empty string, but it is deliberately NOT floored: its job is +# to say what the timestamp is, and that can honestly be said in a clause — +# "af8170154's author date, not a run time" is 44 characters and is not a +# hand-wave. Flooring it would be a gate that fires on ordinary work. +ARGUMENT_FIELDS = ( + ("unrecoverable_reason",), + ("forced_by_checkpoint_or_device", "kv_cache_dtype"), + ("evidence", "never_reproduced"), + ("evidence", "gate_form"), +) + + +def _is_prose(value): + """True when `value` is a string carrying something other than whitespace.""" + return isinstance(value, str) and bool(value.strip()) + + +# ── The provenance contract ───────────────────────────────────────────────── + + +class ContractError(Exception): + """A golden that cannot be attributed, with the reason it cannot.""" + + +def check_golden(doc): + """Return the list of contract violations in `doc`. Empty means it holds. + + Pure: no vLLM, no GPU, no checkpoint. This is the gate. + """ + problems = [] + + def need(container, key, where): + if not isinstance(container, dict) or key not in container: + problems.append(f"{where}: missing '{key}'") + return False + return True + + for key in ("vllm", "transformers", "flashinfer", "model", "revision", + "sampling", "golden", "capture"): + need(doc, key, "oracle.json") + + sampling = doc.get("sampling") + if isinstance(sampling, dict): + for key in ("temperature", "max_tokens"): + need(sampling, key, "sampling") + elif "sampling" in doc: + problems.append("sampling: not an object") + + # A comparison over zero elements reports a perfect score, so the golden's + # own width is asserted rather than trusted. + golden = doc.get("golden") + if not isinstance(golden, list) or not golden: + problems.append("golden: must be a non-empty array") + else: + want = sampling.get("max_tokens") if isinstance(sampling, dict) else None + for i, entry in enumerate(golden): + where = f"golden[{i}]" + for key in ("prompt", "prompt_token_ids", "token_ids", "text"): + need(entry, key, where) + if not isinstance(entry, dict): + continue + if not entry.get("prompt_token_ids"): + problems.append(f"{where}: empty prompt_token_ids") + ids = entry.get("token_ids") + if not ids: + problems.append(f"{where}: empty token_ids") + elif isinstance(want, int) and len(ids) != want: + problems.append( + f"{where}: {len(ids)} token_ids against sampling.max_tokens={want}") + + capture = doc.get("capture") + if not isinstance(capture, dict): + if "capture" in doc: + problems.append("capture: not an object") + return problems + + if capture.get("schema") != SCHEMA: + problems.append(f"capture.schema: expected {SCHEMA}, got {capture.get('schema')!r}") + for key in ("captured_utc", "host", "generator"): + if not capture.get(key): + problems.append(f"capture: '{key}' is empty") + + recorded = capture.get("engine_config_recorded") + if not isinstance(recorded, bool): + problems.append("capture.engine_config_recorded: must be true or false") + return problems + + if not recorded: + # An UNATTRIBUTED golden is allowed to exist — deleting it would delete + # evidence — but it has to say so out loud and name the issue that owes + # the re-derivation. Silence is what #926 is. + # `_is_prose`, not truthiness. The C++ copy of this contract already + # spells it `is_string()`, and a bare truthiness test let a NUMBER + # through here -- `"unrecoverable_reason": 123` returned zero problems + # from this checker while the C++ gate refused it. Two copies of one + # contract disagreeing about what satisfies it is the drift this design + # promises cannot happen, so the weaker copy is the one that moves. + if not _is_prose(capture.get("unrecoverable_reason")): + problems.append( + "capture.engine_config_recorded is false and " + "'unrecoverable_reason' is not prose: an unattributed golden " + "must state IN WORDS why it cannot be attributed") + issue = capture.get("issue") or "" + if not str(issue).startswith(ISSUE_PREFIX): + problems.append( + "capture.engine_config_recorded is false and 'issue' does not " + f"name a vllm.cpp issue (got {issue!r})") + if capture.get("engine") is not None: + problems.append( + "capture.engine_config_recorded is false but 'engine' is not " + "null: a configuration that is recorded is not unrecorded") + + # ── The substance, not only the shape ─────────────────────────────── + # Everything above this point is satisfied by a file that says + # "unrecorded", names an issue and argues NOTHING. That file passes as + # a record while being one, and it is the state #926 filed. What + # follows requires the record to still be there. + if not _is_prose(capture.get("captured_utc_is")): + problems.append( + "capture.captured_utc_is is empty: an unattributed golden's " + "timestamp is not a run time unless the file says what it is, " + "and a commit's author date read as a capture time is a " + "fabricated provenance") + for parent, keys in (("forced_by_checkpoint_or_device", + REQUIRED_FORCED_TERM_KEYS), + ("evidence", REQUIRED_EVIDENCE_KEYS)): + block = capture.get(parent) + if not isinstance(block, dict): + problems.append( + f"capture.{parent}: must be an object on an unattributed " + f"golden, got {type(block).__name__} -- an unrecoverable " + "configuration is a claim, and a claim without its " + "supporting record is the silence this contract refuses") + continue + for key in keys: + if not _is_prose(block.get(key)): + problems.append( + f"capture.{parent}['{key}'] is empty: it is named by " + "this contract because deleting it removes an argument " + "the golden's admissibility rests on") + for path in ARGUMENT_FIELDS: + value = capture + for step in path: + value = value.get(step) if isinstance(value, dict) else None + # Absence and emptiness are already reported above, so this arm + # only judges LENGTH and cannot report the same defect twice. + if _is_prose(value) and len(value.strip()) < MIN_ARGUMENT_CHARS: + problems.append( + "capture.%s is %d characters, under the %d this contract " + "requires of a field whose content is an ARGUMENT: a " + "one-word answer here is the record going missing while " + "the file keeps its shape" % ( + ".".join(path), len(value.strip()), MIN_ARGUMENT_CHARS)) + return problems + + engine = capture.get("engine") + if not isinstance(engine, dict): + problems.append("capture.engine: must be an object when the config is recorded") + return problems + resolved = engine.get("resolved") + if not isinstance(resolved, dict): + problems.append("capture.engine.resolved: must be an object") + return problems + for key in REQUIRED_ENGINE_KEYS: + if key not in resolved: + problems.append(f"capture.engine.resolved: missing '{key}'") + elif resolved[key] is None and key not in NULLABLE_ENGINE_KEYS: + problems.append( + f"capture.engine.resolved['{key}'] is null: a value that could " + "not be read is not a value that was default") + for key in ("torch", "device"): + if not engine.get(key): + problems.append(f"capture.engine: '{key}' is empty") + + batch = capture.get("batch") + if not isinstance(batch, dict) or not batch.get("shape"): + problems.append( + "capture.batch.shape is empty: batching re-orders reductions, so a " + "golden that does not say how the prompts were submitted is not " + "reproducible") + legs = capture.get("legs") + if not isinstance(legs, int) or legs < 2: + problems.append( + f"capture.legs: expected at least 2 legs, got {legs!r} -- one leg " + "cannot show the configuration is deterministic") + if capture.get("legs_agree") is not True: + problems.append( + "capture.legs_agree is not true: a golden whose own legs disagree " + "records a coin flip") + return problems + + +# There is deliberately NO builder for the unattributed provenance block here. +# +# One lived at this line and had exactly one caller: a fixture in +# tests/scripts/test_nemotron_h_oracle_capture.py. `--capture` cannot reach it — +# a capture that runs records its configuration, which is the whole point of the +# mode — so no production path produced the shape it described, and the one +# unattributed golden this repository has was written by hand and carried three +# keys the builder could not emit. Under AGENTS.md's "Nothing lands dead" that +# is a helper documenting a production shape nothing in production produces. +# +# Deleting it also repairs the fixture. This suite's own rule is that a fixture +# must never be derived from the module under test, because setup and +# expectation then move together and a key deleted from the checker deletes its +# own test. The unattributed fixture was the one place that broke that rule. It +# is a test-owned literal now, and the contract's requirements are proven key by +# key by dropping each one from it. +# +# `--capture` builds the ATTRIBUTED block inline in main(), where the values it +# needs are in scope; that block is reached, and check_golden() refuses it if it +# is wrong. + + +# ── The oracle side ───────────────────────────────────────────────────────── + + +def assert_oracle_identity(): + """Abort unless the importable vLLM IS the pin. Identity is asserted, never + assumed: `$HOME/venvs/vllm-oracle` on dgx has resolved to a 0.25.0 rollback + that predates `NemotronHMoEDecoderLayer` entirely, and a run through it fails + in a way that reads as 'the model is unsupported'.""" + import vllm + + version = vllm.__version__ + where = vllm.__file__ + print(f"vllm {version} {where}", flush=True) + if ORACLE_COMMIT not in version: + raise SystemExit( + f"WRONG ORACLE: vllm {version} does not carry the pin {ORACLE_COMMIT}") + identity = {"vllm": version, "vllm_file": where} + for name in ("transformers", "flashinfer"): + try: + module = __import__(name) + identity[name] = getattr(module, "__version__", "unknown") + except Exception as exc: # noqa: BLE001 - reported, never swallowed + identity[name] = f"IMPORT_FAILED: {exc}" + print("ORACLE_IDENTITY_OK", json.dumps(identity), flush=True) + return identity + + +def read_resolved_config(llm): + """Read the engine configuration BACK OUT of the built engine. + + Reading back is the point. `kv_cache_dtype`, the attention backend, the MoE + backend, the block size and the block count are all chosen by vLLM from the + checkpoint and the device, so the kwargs a driver passed are not the + configuration it ran. A key this cannot read stays ABSENT and the writer + refuses; it is never filled in with a plausible default. + """ + config = llm.llm_engine.vllm_config + cache = config.cache_config + scheduler = config.scheduler_config + model = config.model_config + parallel = config.parallel_config + compilation = config.compilation_config + resolved = {} + + def put(key, value): + if value is not None: + resolved[key] = value + + put("block_size", getattr(cache, "block_size", None)) + put("num_gpu_blocks", getattr(cache, "num_gpu_blocks", None)) + resolved["num_gpu_blocks_override"] = getattr(cache, "num_gpu_blocks_override", None) + put("gpu_memory_utilization", getattr(cache, "gpu_memory_utilization", None)) + put("kv_cache_dtype", str(getattr(cache, "cache_dtype", "") or "") or None) + put("enable_prefix_caching", getattr(cache, "enable_prefix_caching", None)) + put("max_model_len", getattr(model, "max_model_len", None)) + put("dtype", str(getattr(model, "dtype", "") or "") or None) + put("quantization", getattr(model, "quantization", None)) + put("seed", getattr(model, "seed", None)) + put("enforce_eager", getattr(model, "enforce_eager", None)) + put("max_num_seqs", getattr(scheduler, "max_num_seqs", None)) + put("max_num_batched_tokens", getattr(scheduler, "max_num_batched_tokens", None)) + put("enable_chunked_prefill", getattr(scheduler, "enable_chunked_prefill", None)) + put("tensor_parallel_size", getattr(parallel, "tensor_parallel_size", None)) + put("compilation_mode", str(getattr(compilation, "mode", "") or "") or None) + put("cudagraph_mode", str(getattr(compilation, "cudagraph_mode", "") or "") or None) + sizes = getattr(compilation, "cudagraph_capture_sizes", None) + if sizes is not None: + resolved["cudagraph_capture_sizes"] = list(sizes) + for key, env in (("attention_backend", "VLLM_ATTENTION_BACKEND"), + ("moe_backend", "VLLM_FUSED_MOE_BACKEND")): + # vLLM PRINTS the backend it selected and does not always expose it on + # the config, so the environment override is recorded when it is set and + # the log line is the authority otherwise. `--attention-backend` and + # `--moe-backend` below let the caller state what the log said, and the + # writer refuses if neither is available. + value = os.environ.get(env) + if value: + resolved[key] = value + return resolved + + +def generate(llm, sampling_params, prompt_inputs): + """One prompt per `generate()` call. Batching re-orders reductions, and this + tree has already misread that as non-determinism once (see + scripts/deepseek-v2-oracle-capture.py). The shape is RECORDED either way.""" + results = [] + for prompt in prompt_inputs: + out = llm.generate([prompt], sampling_params) + results.append(list(out[0].outputs[0].token_ids)) + return results + + +def run_oracle(args): + """Build the engine, run `args.legs` legs, and return (identity, resolved, + per-leg token ids, per-prompt token ids, texts).""" + from vllm import LLM, SamplingParams + from vllm.inputs import TokensPrompt + + identity = assert_oracle_identity() + profile = dict(PROFILES[args.profile]) + for key in ("max_model_len", "max_num_seqs", "gpu_memory_utilization", + "max_num_batched_tokens", "enforce_eager", "num_gpu_blocks_override"): + override = getattr(args, key, None) + if override is not None: + profile[key] = override + kwargs = {k: v for k, v in profile.items() if v is not None} + kwargs["enforce_eager"] = profile["enforce_eager"] + kwargs["model"] = args.model + print("ORACLE_KW", json.dumps({k: v for k, v in kwargs.items() if k != "model"}), + flush=True) + + started = time.time() + llm = LLM(**kwargs) + print("ORACLE_LOAD_S=%.1f" % (time.time() - started), flush=True) + + resolved = read_resolved_config(llm) + for key, value in ((("attention_backend"), args.attention_backend), + (("moe_backend"), args.moe_backend)): + if value: + resolved[key] = value + print("ORACLE_RESOLVED", json.dumps(resolved, default=str), flush=True) + + golden_prompts = None + if args.golden_prompt_ids: + golden_prompts = args.golden_prompt_ids + if args.tokens_prompt: + if golden_prompts is None: + raise SystemExit( + "--tokens-prompt needs prompt_token_ids, which only a golden " + "carries; pass --verify or drop the flag") + inputs = [TokensPrompt(prompt_token_ids=ids) for ids in golden_prompts] + shape = "one TokensPrompt per generate() call (pre-tokenized)" + else: + inputs = list(PROMPTS) + shape = "one text prompt per generate() call" + + sampling_params = SamplingParams(temperature=0.0, max_tokens=args.max_tokens, + ignore_eos=args.ignore_eos) + legs = [] + texts = None + for leg in range(args.legs): + print("ORACLE_LEG %d" % (leg + 1), flush=True) + out = [] + leg_texts = [] + for prompt in inputs: + result = llm.generate([prompt], sampling_params) + out.append(list(result[0].outputs[0].token_ids)) + leg_texts.append(result[0].outputs[0].text) + if texts is None: + pass + legs.append(out) + if leg == 0: + texts = leg_texts + prompt_ids = [list(r) for r in _prompt_ids(llm, inputs, sampling_params)] + return identity, resolved, legs, prompt_ids, texts, shape + + +def _prompt_ids(llm, inputs, sampling_params): + """vLLM's OWN tokenization of each prompt, taken from the RequestOutput. + + A silently unapplied BOS has scored 0/6 in this tree while emitting fluent + English, and the committed tokenization is what caught it. + """ + ids = [] + for prompt in inputs: + result = llm.generate([prompt], sampling_params) + ids.append(list(result[0].prompt_token_ids)) + return ids + + +def compare(expected, got): + n = min(len(expected), len(got)) + return sum(1 for i in range(n) if expected[i] == got[i]), n + + +def main(argv=None): + parser = argparse.ArgumentParser(description=__doc__) + mode = parser.add_mutually_exclusive_group(required=True) + mode.add_argument("--check", metavar="GOLDEN", + help="validate a committed golden's provenance contract " + "(no vLLM, no GPU, no checkpoint)") + mode.add_argument("--verify", metavar="GOLDEN", + help="run the oracle and compare against a committed golden") + mode.add_argument("--capture", action="store_true", + help="run the oracle and write a golden") + parser.add_argument("--out", help="output path for --capture") + parser.add_argument("--model", help="checkpoint directory") + parser.add_argument("--profile", default="nhspeed-a", choices=sorted(PROFILES), + help="named engine configuration (default: nhspeed-a)") + parser.add_argument("--legs", type=int, default=2, + help="greedy legs in one process; >=2 or the golden " + "cannot claim determinism at its own configuration") + parser.add_argument("--max-tokens", type=int, default=32) + parser.add_argument("--ignore-eos", action="store_true", default=True) + parser.add_argument("--no-ignore-eos", dest="ignore_eos", action="store_false") + parser.add_argument("--tokens-prompt", action="store_true", + help="feed the golden's prompt_token_ids instead of the " + "prompt text, skipping the tokenizer") + parser.add_argument("--attention-backend", + help="the backend vLLM's startup log named, e.g. FLASHINFER") + parser.add_argument("--moe-backend", + help="the NvFp4 MoE backend vLLM's startup log named, e.g. MARLIN") + parser.add_argument("--host", help="the box this ran on, recorded into the golden") + for key in ("max_model_len", "max_num_seqs", "max_num_batched_tokens", + "num_gpu_blocks_override"): + parser.add_argument("--" + key.replace("_", "-"), type=int, default=None) + parser.add_argument("--gpu-memory-utilization", type=float, default=None) + parser.add_argument("--enforce-eager", dest="enforce_eager", action="store_true", + default=None, + help="NOT the denominator; only for a diagnostic leg") + args = parser.parse_args(argv) + + if args.check: + with open(args.check, encoding="utf-8") as handle: + doc = json.load(handle) + problems = check_golden(doc) + capture = doc.get("capture") or {} + for problem in problems: + print("CONTRACT: " + problem, file=sys.stderr) + print("checked %s: %d golden entr%s, engine_config_recorded=%r, %d problem(s)" + % (args.check, len(doc.get("golden") or []), + "y" if len(doc.get("golden") or []) == 1 else "ies", + capture.get("engine_config_recorded"), len(problems))) + return 1 if problems else 0 + + if args.capture and not args.out: + parser.error("--capture needs --out") + if not args.model: + parser.error("--verify and --capture need --model") + + existing = None + if args.verify: + with open(args.verify, encoding="utf-8") as handle: + existing = json.load(handle) + args.golden_prompt_ids = [e["prompt_token_ids"] for e in existing["golden"]] + args.max_tokens = int(existing["sampling"]["max_tokens"]) + else: + args.golden_prompt_ids = None + + if args.legs < 2: + raise SystemExit("--legs must be at least 2") + + identity, resolved, legs, prompt_ids, texts, shape = run_oracle(args) + + agree = all(leg == legs[0] for leg in legs[1:]) + print("ORACLE_LEGS_AGREE=%s" % agree, flush=True) + + if existing is not None: + recorded = (existing.get("capture") or {}).get("engine_config_recorded") + if not recorded: + print("CONFIGURATION: the committed golden records NO engine " + "configuration, so this run cannot be held to it -- a token " + "difference below is UNATTRIBUTABLE, not a defect (#926)", + flush=True) + total_matched = total_compared = 0 + for i, entry in enumerate(existing["golden"]): + matched, compared = compare(entry["token_ids"], legs[0][i]) + total_matched += matched + total_compared += compared + print("ORACLE prompt %d: compared=%d matched=%d" % (i, compared, matched), + flush=True) + print("ORACLE TOKEN MATCH: %d/%d" % (total_matched, total_compared), flush=True) + return 0 if total_matched == total_compared else 2 + + if not agree: + raise SystemExit( + "the legs of this capture DISAGREE with each other; a golden written " + "from them would record a coin flip. Nothing was written.") + + document = { + "vllm": identity["vllm"], + "transformers": identity.get("transformers"), + "flashinfer": identity.get("flashinfer"), + "model": args.model, + "revision": CHECKPOINT_REVISION, + "sampling": {"temperature": 0.0, "max_tokens": args.max_tokens, + "ignore_eos": args.ignore_eos}, + "capture": { + "schema": SCHEMA, + "generator": "scripts/nemotron-h-oracle-capture.py", + "captured_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), + "host": args.host or os.uname().nodename, + "engine_config_recorded": True, + "unrecoverable_reason": None, + "issue": ISSUE_PREFIX + "926", + "engine": { + "profile": args.profile, + "resolved": resolved, + "torch": _torch_version(), + "device": _device_name(), + }, + "batch": {"shape": shape, "prompts": len(PROMPTS)}, + "legs": args.legs, + "legs_agree": agree, + }, + "golden": [ + {"prompt": PROMPTS[i], "prompt_token_ids": prompt_ids[i], + "token_ids": legs[0][i], "text": texts[i]} + for i in range(len(PROMPTS)) + ], + } + problems = check_golden(document) + if problems: + for problem in problems: + print("CONTRACT: " + problem, file=sys.stderr) + raise SystemExit( + "this capture does not satisfy the provenance contract, so it was " + "NOT written: a golden that cannot be attributed is what #926 is") + with open(args.out, "w", encoding="utf-8") as handle: + json.dump(document, handle, indent=2) + handle.write("\n") + print("WROTE %s" % args.out) + return 0 + + +def _torch_version(): + try: + import torch + + return torch.__version__ + except Exception: # noqa: BLE001 + return None + + +def _device_name(): + try: + import torch + + if torch.cuda.is_available(): + major, minor = torch.cuda.get_device_capability(0) + return "%s (sm_%d%d)" % (torch.cuda.get_device_name(0), major, minor) + except Exception: # noqa: BLE001 + pass + return None + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/parity/goldens/nemotron_35_lightning_greedy/oracle.json b/tests/parity/goldens/nemotron_35_lightning_greedy/oracle.json index 1ebe98a29..22d96833d 100644 --- a/tests/parity/goldens/nemotron_35_lightning_greedy/oracle.json +++ b/tests/parity/goldens/nemotron_35_lightning_greedy/oracle.json @@ -8,6 +8,30 @@ "temperature": 0.0, "max_tokens": 32 }, + "capture": { + "schema": 2, + "generator": "none -- af8170154 committed no generator (#926)", + "captured_utc": "2026-08-12T22:37:57Z", + "captured_utc_is": "af8170154's author date, the only timestamp that exists for this capture; the run itself was not timestamped", + "host": "dgx.casa (GB10, sm_121), from $HOME/venvs/vllm-oracle-next", + "engine_config_recorded": false, + "unrecoverable_reason": "The capture ran from an uncommitted driver in $HOME on dgx.casa and wrote no log to shared storage. dgx.casa was reimaged on 2026-08-14, so the driver and its engine configuration are gone. af8170154 added exactly three files -- this golden, .agents/specs/nemotron-h-model.md and tests/parity/hf_snapshot.h -- and none of them is a generator; this directory has exactly one commit in the whole history; no script in scripts/ on any branch writes this path; and no artifact of the run survives on the NAS, whose oldest job directory is 2026-08-17. So enforce_eager, max_model_len, max_num_seqs, max_num_batched_tokens, gpu_memory_utilization and the batch shape are UNRECORDED and cannot be recovered. Every one of them can move a greedy argmax at a near-tie, so these tokens can be approximated but not re-derived.", + "issue": "https://github.com/mudler/vllm.cpp/issues/926", + "engine": null, + "batch": null, + "legs": null, + "legs_agree": null, + "forced_by_checkpoint_or_device": { + "kv_cache_dtype": "fp8_e4m3, and NOT a candidate difference: the checkpoint carries it. config.json quantization_config.kv_cache_scheme is {dynamic:false, num_bits:8, type:float} and hf_quant_config.json quantization.kv_cache_quant_algo is FP8, and at the pin vllm/engine/arg_utils.py:1916 resolves the default kv_cache_dtype='auto' through vllm/utils/torch_utils.py:374-392 -> :310-362 to 'fp8'. Every unoverridden run of this checkpoint gets it, including this one.", + "moe_backend": "MARLIN on GB10 -- a device without native FP4 support takes the first backend fused_moe/oracle/nvfp4.py supports, and at this pin there is no environment knob to select another", + "dtype": "torch.bfloat16, from the checkpoint", + "quantization": "modelopt_mixed, from the checkpoint" + }, + "evidence": { + "never_reproduced": "Two later runs, each under a fully recorded configuration, reproduce prompts 0 and 1 exactly and neither reproduces prompt 2. 2026-08-18 /workspace/nhspeed/oracle_only.sh attempt a (max_model_len=512, max_num_seqs=8, gpu_memory_utilization=0.30, max_num_batched_tokens=512, enforce_eager=False, TokensPrompt, one prompt per generate call): 32/32, 32/32, 26/32, twice in one process, ORACLE TOKEN MATCH 180/192, log at /mnt/nas_share/rc/nhspeed/oracle.a.out. The #926 rebuild (enforce_eager=True, gpu_memory_utilization=0.25, max_model_len=4096): 32/32, 32/32, 29/32, diverging at index 29. Repeatable within a configuration and different between configurations: config sensitivity, not non-determinism.", + "gate_form": "A ratified distributional gate is INADMISSIBLE here. AGENTS.md permits one only where the oracle's own greedy decode is non-deterministic, and it is not. What is licensed is re-deriving the golden under a named configuration: scripts/nemotron-h-oracle-capture.py --capture --profile nhspeed-a." + } + }, "golden": [ { "prompt": "The capital of France is", diff --git a/tests/scripts/test_nemotron_h_oracle_capture.py b/tests/scripts/test_nemotron_h_oracle_capture.py new file mode 100644 index 000000000..f01bc70bb --- /dev/null +++ b/tests/scripts/test_nemotron_h_oracle_capture.py @@ -0,0 +1,514 @@ +#!/usr/bin/env python3 +"""Unit and mutation checks for scripts/nemotron-h-oracle-capture.py (#926). + +The rule under test is the golden's PROVENANCE CONTRACT: a Nemotron oracle +golden either records the engine configuration it was captured under, or says in +the file itself that it does not and names the issue that owes the +re-derivation. Silence is the third state, and silence is what #926 is. + +Every case is red before the rule exists. The file the contract was written +against -- `oracle.json` exactly as `af8170154` committed it -- is reproduced +here as `AF8170154_SHAPE` and is asserted to FAIL, so the suite fails if the +contract is ever widened enough to admit the artifact that motivated it. + +The suite runs with no vLLM, no GPU and no checkpoint, which is the point: the +provenance defect is a records defect and its gate must not need the hardware +that the defect blocks. +""" + +from __future__ import annotations + +import contextlib +import importlib.util +import io +import json +import re +import sys +import tempfile +import unittest +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[2] +GENERATOR = ROOT / "scripts/nemotron-h-oracle-capture.py" +SHIPPED_GOLDEN = ROOT / "tests/parity/goldens/nemotron_35_lightning_greedy/oracle.json" +CPP_CONSUMER = ROOT / "tests/vllm/models/test_nemotron_h_loader.cpp" +SPEC = importlib.util.spec_from_file_location("nemotron_h_oracle_capture", GENERATOR) +assert SPEC is not None and SPEC.loader is not None +capture = importlib.util.module_from_spec(SPEC) +sys.modules[SPEC.name] = capture +SPEC.loader.exec_module(capture) + + +# ── Test-owned literals ───────────────────────────────────────────────────── +# Never derived from the module under test. A fixture built from the production +# constant makes setup and expectation move together, and then a key deleted +# from REQUIRED_ENGINE_KEYS deletes its own test. +EXPECTED_ENGINE_KEYS = { + "attention_backend", + "block_size", + "compilation_mode", + "cudagraph_capture_sizes", + "cudagraph_mode", + "dtype", + "enable_chunked_prefill", + "enable_prefix_caching", + "enforce_eager", + "gpu_memory_utilization", + "kv_cache_dtype", + "max_model_len", + "max_num_batched_tokens", + "max_num_seqs", + "moe_backend", + "num_gpu_blocks", + "num_gpu_blocks_override", + "quantization", + "seed", + "tensor_parallel_size", +} + +# The knobs #926 named by name when it said the goldens can be approximated but +# not re-derived. Asserted separately from the set above, because these three +# are the ones the rebuild differed on and are the reason the contract exists. +KNOBS_926_NAMED = ("enforce_eager", "gpu_memory_utilization", "max_model_len") + +# The unattributed arm's own expectation, owned here for the same reason. The +# attributed arm is gated by structure; this arm's whole record is prose, so +# "present and truthy" gates the shape and not the substance -- a reviewer +# replaced an 814-character `unrecoverable_reason` with the word "dunno", +# emptied `evidence`, `forced_by_checkpoint_or_device` and `captured_utc_is`, +# and every gate stayed green. +EXPECTED_FORCED_TERM_KEYS = {"kv_cache_dtype", "moe_backend", "dtype", "quantization"} +EXPECTED_EVIDENCE_KEYS = {"never_reproduced", "gate_form"} + +# The fields whose content is an ARGUMENT rather than a value, and the floor +# they carry. The floor detects REMOVAL; it does not and cannot claim the prose +# is true. 80 is owned here as a literal so that raising it in the checker to +# hide a shrinking record is red, and lowering it to admit one is red too. +EXPECTED_MIN_ARGUMENT_CHARS = 80 +EXPECTED_ARGUMENT_FIELDS = { + ("unrecoverable_reason",), + ("forced_by_checkpoint_or_device", "kv_cache_dtype"), + ("evidence", "never_reproduced"), + ("evidence", "gate_form"), +} + +# Long enough to clear EXPECTED_MIN_ARGUMENT_CHARS without being a paraphrase of +# the shipped golden's text. The fixture must not borrow the artifact's prose, +# or a case that deletes the artifact's prose still passes. +_ARGUMENT = ( + "A sentence long enough to be an argument rather than a hand-wave, which is " + "the only property a length floor can honestly test for." +) + +RESOLVED = { + "attention_backend": "FLASHINFER", + "block_size": 512, + "compilation_mode": "CompilationMode.VLLM_COMPILE", + "cudagraph_capture_sizes": [1, 2, 4, 8, 16], + "cudagraph_mode": "CUDAGraphMode.FULL_AND_PIECEWISE", + "dtype": "torch.bfloat16", + "enable_chunked_prefill": True, + "enable_prefix_caching": False, + "enforce_eager": False, + "gpu_memory_utilization": 0.30, + "kv_cache_dtype": "fp8_e4m3", + "max_model_len": 512, + "max_num_batched_tokens": 512, + "max_num_seqs": 8, + "moe_backend": "MARLIN", + "num_gpu_blocks": 1258, + "num_gpu_blocks_override": None, + "quantization": "modelopt_mixed", + "seed": 0, + "tensor_parallel_size": 1, +} + + +def _cpp_string_list(source, name): + """The set of string literals in the C++ initializer list called `name`. + + Parsed rather than grepped: two of the keys under test also appear in the + C++ engine-key list, so `assertIn('"dtype"', source)` would pass without the + unattributed arm naming it at all. + """ + match = re.search(re.escape(name) + r"\s*=\s*\{(.*?)\};", source, re.S) + if match is None: + raise AssertionError(f"{CPP_CONSUMER.name} has no list called {name}") + return set(re.findall(r'"([^"]*)"', match.group(1))) + + +def _entry(prompt, prompt_ids, tokens): + return {"prompt": prompt, "prompt_token_ids": list(prompt_ids), + "token_ids": list(tokens), "text": " x"} + + +def attributed(): + """A golden that DOES record its configuration.""" + return { + "vllm": "0.23.1rc1.dev1511+g555967922", + "transformers": "5.14.1", + "flashinfer": "0.6.15.post1", + "model": "/checkpoints/nemotron-3.5-lightning-30b-nvfp4", + "revision": "29f2d1746d8f41e316523194b19018707749b1b1", + "sampling": {"temperature": 0.0, "max_tokens": 4, "ignore_eos": True}, + "capture": { + "schema": 2, + "generator": "scripts/nemotron-h-oracle-capture.py", + "captured_utc": "2026-08-21T00:00:00Z", + "host": "dgx:gpu0 (GB10)", + "engine_config_recorded": True, + "unrecoverable_reason": None, + "issue": "https://github.com/mudler/vllm.cpp/issues/926", + "engine": { + "profile": "nhspeed-a", + "resolved": dict(RESOLVED), + "torch": "2.13.0+cu130", + "device": "NVIDIA GB10 (sm_121)", + }, + "batch": {"shape": "one text prompt per generate() call", "prompts": 1}, + "legs": 2, + "legs_agree": True, + }, + "golden": [_entry("The capital of France is", [1, 2, 3], [9, 8, 7, 6])], + } + + +def unattributed(): + """A golden that does NOT record its configuration and SAYS so. + + A TEST-OWNED literal, deliberately. This fixture used to be built by + `capture.unattributed_capture()`, which made setup and expectation move + together: a key dropped from the checker's requirements was also dropped + from the fixture, so the case could not go red. That helper had no other + caller and is gone (#926); the shape lives here, where the suite owns it. + """ + doc = attributed() + doc["capture"] = { + "schema": 2, + "generator": "none -- af8170154 committed no generator", + "captured_utc": "2026-08-12T22:37:57Z", + "captured_utc_is": "the committing author date, not a recorded run time", + "host": "dgx.casa (GB10)", + "engine_config_recorded": False, + "unrecoverable_reason": _ARGUMENT, + "issue": "https://github.com/mudler/vllm.cpp/issues/926", + "engine": None, + "batch": None, + "legs": None, + "legs_agree": None, + "forced_by_checkpoint_or_device": { + "kv_cache_dtype": _ARGUMENT, + "moe_backend": "a backend the device forced", + "dtype": "a dtype the checkpoint forced", + "quantization": "a quantization the checkpoint forced", + }, + "evidence": { + "never_reproduced": _ARGUMENT, + "gate_form": _ARGUMENT, + }, + } + return doc + + +# `oracle.json` exactly as af8170154 shaped it: seven top-level keys and no +# provenance at all. Reproduced here so the contract is pinned against the +# artifact that motivated it rather than against a paraphrase of it. +AF8170154_SHAPE = { + "vllm": "0.23.1rc1.dev1511+g555967922", + "transformers": "5.14.1", + "flashinfer": "0.6.15.post1", + "model": "/mnt/nas_share/checkpoints/nemotron-3.5-lightning-30b-nvfp4", + "revision": "29f2d1746d8f41e316523194b19018707749b1b1", + "sampling": {"temperature": 0.0, "max_tokens": 4}, + "golden": [_entry("The capital of France is", [1, 2, 3], [9, 8, 7, 6])], +} + + +class RequiredKeysTests(unittest.TestCase): + def test_the_required_engine_keys_are_the_ones_this_suite_owns(self) -> None: + self.assertEqual(set(capture.REQUIRED_ENGINE_KEYS), EXPECTED_ENGINE_KEYS) + + def test_every_knob_926_named_is_required(self) -> None: + for knob in KNOBS_926_NAMED: + self.assertIn(knob, capture.REQUIRED_ENGINE_KEYS, knob) + + def test_the_cpp_consumer_names_every_required_key(self) -> None: + # The C++ gate that reads this golden carries its own copy of the key + # list, because a test cannot import a Python constant. Two copies can + # drift; this is the case that refuses to let them. The expectation is + # owned by this suite, so a key deleted from BOTH copies is still red. + source = CPP_CONSUMER.read_text(encoding="utf-8") + for key in sorted(EXPECTED_ENGINE_KEYS): + self.assertIn(f'"{key}"', source, + f"{CPP_CONSUMER.name} does not name '{key}'") + + def test_the_cpp_consumer_names_every_unattributed_key(self) -> None: + # The same cross-gate as above, for the unattributed arm. A SUBSTRING + # grep would pass vacuously here -- "dtype" and "quantization" already + # appear in the C++ engine-key list -- so the two initializer lists are + # PARSED out of the source and compared as sets. The expectation is this + # suite's, so a key deleted from both production copies is still red. + source = CPP_CONSUMER.read_text(encoding="utf-8") + self.assertEqual(_cpp_string_list(source, "kForcedTermKeys"), + EXPECTED_FORCED_TERM_KEYS) + self.assertEqual(_cpp_string_list(source, "kEvidenceKeys"), + EXPECTED_EVIDENCE_KEYS) + + def test_the_cpp_consumer_carries_the_same_argument_floor(self) -> None: + source = CPP_CONSUMER.read_text(encoding="utf-8") + match = re.search(r"kMinArgumentChars\s*=\s*(\d+)", source) + self.assertIsNotNone(match, "the C++ consumer names no argument floor") + self.assertEqual(int(match.group(1)), EXPECTED_MIN_ARGUMENT_CHARS) + + def test_the_prompt_battery_is_the_committed_one(self) -> None: + shipped = json.loads(SHIPPED_GOLDEN.read_text(encoding="utf-8")) + self.assertEqual([e["prompt"] for e in shipped["golden"]], list(capture.PROMPTS)) + + +class ContractTests(unittest.TestCase): + def test_the_shipped_golden_satisfies_the_contract(self) -> None: + shipped = json.loads(SHIPPED_GOLDEN.read_text(encoding="utf-8")) + self.assertEqual(capture.check_golden(shipped), []) + + def test_the_shipped_golden_is_not_silently_attributed(self) -> None: + # It records no engine configuration, and the file has to say so. If a + # later capture attributes it, this case is the one that must be + # updated deliberately rather than a green that drifted. + shipped = json.loads(SHIPPED_GOLDEN.read_text(encoding="utf-8")) + recorded = shipped["capture"]["engine_config_recorded"] + self.assertIsInstance(recorded, bool) + if not recorded: + self.assertTrue(shipped["capture"]["unrecoverable_reason"]) + self.assertTrue(shipped["capture"]["issue"].endswith("/926")) + + def test_the_af8170154_shape_is_refused(self) -> None: + problems = capture.check_golden(AF8170154_SHAPE) + self.assertTrue(any("capture" in p for p in problems), problems) + + def test_an_attributed_golden_holds(self) -> None: + self.assertEqual(capture.check_golden(attributed()), []) + + def test_an_unattributed_golden_that_says_so_holds(self) -> None: + self.assertEqual(capture.check_golden(unattributed()), []) + + def test_an_unattributed_golden_with_no_reason_is_refused(self) -> None: + doc = unattributed() + doc["capture"]["unrecoverable_reason"] = "" + self.assertTrue(capture.check_golden(doc)) + + def test_an_unattributed_golden_with_no_issue_is_refused(self) -> None: + doc = unattributed() + doc["capture"]["issue"] = "ask mudler" + self.assertTrue(capture.check_golden(doc)) + + def test_unrecorded_and_an_engine_block_cannot_both_be_true(self) -> None: + doc = unattributed() + doc["capture"]["engine"] = {"resolved": dict(RESOLVED)} + self.assertTrue(capture.check_golden(doc)) + + def test_a_missing_recorded_flag_is_refused(self) -> None: + doc = attributed() + del doc["capture"]["engine_config_recorded"] + self.assertTrue(capture.check_golden(doc)) + + def test_each_required_engine_key_is_load_bearing(self) -> None: + for key in sorted(EXPECTED_ENGINE_KEYS): + doc = attributed() + del doc["capture"]["engine"]["resolved"][key] + problems = capture.check_golden(doc) + self.assertTrue(any(key in p for p in problems), + f"dropping '{key}' left the contract green: {problems}") + + def test_a_null_value_is_not_a_recorded_value(self) -> None: + doc = attributed() + doc["capture"]["engine"]["resolved"]["kv_cache_dtype"] = None + self.assertTrue(capture.check_golden(doc)) + + def test_a_null_block_override_is_a_legitimate_value(self) -> None: + doc = attributed() + doc["capture"]["engine"]["resolved"]["num_gpu_blocks_override"] = None + self.assertEqual(capture.check_golden(doc), []) + + def test_one_leg_cannot_claim_determinism(self) -> None: + doc = attributed() + doc["capture"]["legs"] = 1 + self.assertTrue(capture.check_golden(doc)) + + def test_disagreeing_legs_are_refused(self) -> None: + doc = attributed() + doc["capture"]["legs_agree"] = False + self.assertTrue(capture.check_golden(doc)) + + def test_the_batch_shape_is_required(self) -> None: + doc = attributed() + doc["capture"]["batch"] = {"prompts": 3} + self.assertTrue(capture.check_golden(doc)) + + def test_an_empty_golden_is_refused(self) -> None: + doc = attributed() + doc["golden"] = [] + self.assertTrue(capture.check_golden(doc)) + + def test_a_short_token_row_is_refused(self) -> None: + doc = attributed() + doc["golden"][0]["token_ids"] = [9, 8] + problems = capture.check_golden(doc) + self.assertTrue(any("max_tokens" in p for p in problems), problems) + + def test_an_empty_prompt_tokenization_is_refused(self) -> None: + doc = attributed() + doc["golden"][0]["prompt_token_ids"] = [] + self.assertTrue(capture.check_golden(doc)) + + def test_a_wrong_schema_is_refused(self) -> None: + doc = attributed() + doc["capture"]["schema"] = 1 + self.assertTrue(capture.check_golden(doc)) + + +class UnattributedSubstanceTests(unittest.TestCase): + """The unattributed arm's record has to still BE there, not merely fit. + + The reviewer's mutation that motivated every case below: gut `evidence`, + `forced_by_checkpoint_or_device` and `captured_utc_is`, and set + `unrecoverable_reason` to "dunno". The file keeps its shape, `--check` keeps + printing `engine_config_recorded=False`, and every argument the artifact + rests on is gone. That is silence wearing the shape of a record, which is + the state #926 filed. + """ + + def test_the_forced_term_keys_are_the_ones_this_suite_owns(self) -> None: + self.assertEqual(set(capture.REQUIRED_FORCED_TERM_KEYS), + EXPECTED_FORCED_TERM_KEYS) + + def test_the_evidence_keys_are_the_ones_this_suite_owns(self) -> None: + self.assertEqual(set(capture.REQUIRED_EVIDENCE_KEYS), EXPECTED_EVIDENCE_KEYS) + + def test_the_argument_fields_and_their_floor_are_the_ones_this_suite_owns(self) -> None: + self.assertEqual({tuple(f) for f in capture.ARGUMENT_FIELDS}, + EXPECTED_ARGUMENT_FIELDS) + self.assertEqual(capture.MIN_ARGUMENT_CHARS, EXPECTED_MIN_ARGUMENT_CHARS) + + def test_the_reviewers_gutting_mutation_is_refused(self) -> None: + # Mutation E, reproduced verbatim as a case so it can never go green + # again silently. + doc = unattributed() + doc["capture"]["unrecoverable_reason"] = "dunno" + doc["capture"]["captured_utc_is"] = "" + doc["capture"]["forced_by_checkpoint_or_device"] = {} + doc["capture"]["evidence"] = {} + problems = capture.check_golden(doc) + for named in ("unrecoverable_reason", "captured_utc_is", + "forced_by_checkpoint_or_device", "evidence"): + self.assertTrue(any(named in p for p in problems), + f"gutting '{named}' left the contract silent: {problems}") + + def test_each_forced_term_is_load_bearing(self) -> None: + for key in sorted(EXPECTED_FORCED_TERM_KEYS): + doc = unattributed() + del doc["capture"]["forced_by_checkpoint_or_device"][key] + problems = capture.check_golden(doc) + self.assertTrue(any(key in p for p in problems), + f"dropping '{key}' left the contract green: {problems}") + + def test_each_evidence_key_is_load_bearing(self) -> None: + for key in sorted(EXPECTED_EVIDENCE_KEYS): + doc = unattributed() + del doc["capture"]["evidence"][key] + problems = capture.check_golden(doc) + self.assertTrue(any(key in p for p in problems), + f"dropping '{key}' left the contract green: {problems}") + + def test_a_missing_forced_block_is_refused(self) -> None: + doc = unattributed() + del doc["capture"]["forced_by_checkpoint_or_device"] + self.assertTrue(capture.check_golden(doc)) + + def test_a_missing_evidence_block_is_refused(self) -> None: + doc = unattributed() + del doc["capture"]["evidence"] + self.assertTrue(capture.check_golden(doc)) + + def test_a_missing_timestamp_gloss_is_refused(self) -> None: + # A commit's author date read as a capture time is a fabricated + # provenance, so the file has to say which one it is. + doc = unattributed() + del doc["capture"]["captured_utc_is"] + self.assertTrue(capture.check_golden(doc)) + + def test_each_argument_field_refuses_a_one_word_answer(self) -> None: + for path in sorted(EXPECTED_ARGUMENT_FIELDS): + doc = unattributed() + block = doc["capture"] + for step in path[:-1]: + block = block[step] + block[path[-1]] = "dunno" + problems = capture.check_golden(doc) + self.assertTrue(any(".".join(path) in p for p in problems), + f"'{'.'.join(path)}' accepted one word: {problems}") + + def test_a_non_string_reason_is_refused(self) -> None: + # The C++ copy has always spelled this `is_string()`. A bare truthiness + # test in the checker let a number through, so the two copies of one + # contract disagreed about what satisfied it. + doc = unattributed() + doc["capture"]["unrecoverable_reason"] = 123 + problems = capture.check_golden(doc) + self.assertTrue(any("unrecoverable_reason" in p for p in problems), problems) + + def test_the_floor_is_not_met_by_whitespace(self) -> None: + doc = unattributed() + doc["capture"]["evidence"]["gate_form"] = " " * 200 + self.assertTrue(capture.check_golden(doc)) + + def test_an_argument_at_the_floor_is_accepted(self) -> None: + # The other side of the floor, so the case cannot pass by refusing + # everything. A gate that fires on ordinary work is the defect. + doc = unattributed() + doc["capture"]["evidence"]["gate_form"] = "x" * EXPECTED_MIN_ARGUMENT_CHARS + self.assertEqual(capture.check_golden(doc), []) + + def test_the_attributed_arm_is_not_burdened_by_these_keys(self) -> None: + # `engine.resolved` already records kv_cache_dtype, dtype, quantization + # and moe_backend as VALUES, so an attributed golden owes no prose about + # them. Scoping matters: a requirement that fired on both arms would red + # every future capture this generator writes. + doc = attributed() + for key in ("captured_utc_is", "forced_by_checkpoint_or_device", "evidence"): + self.assertNotIn(key, doc["capture"]) + self.assertEqual(capture.check_golden(doc), []) + + +class CliTests(unittest.TestCase): + def _check(self, doc): + with tempfile.TemporaryDirectory() as tmp: + path = Path(tmp) / "oracle.json" + path.write_text(json.dumps(doc), encoding="utf-8") + out, err = io.StringIO(), io.StringIO() + with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err): + code = capture.main(["--check", str(path)]) + return code, out.getvalue() + err.getvalue() + + def test_check_exits_zero_on_an_attributed_golden(self) -> None: + code, text = self._check(attributed()) + self.assertEqual(code, 0, text) + self.assertIn("0 problem(s)", text) + + def test_check_exits_one_on_the_af8170154_shape(self) -> None: + code, text = self._check(AF8170154_SHAPE) + self.assertEqual(code, 1, text) + self.assertIn("CONTRACT:", text) + + def test_check_reads_the_shipped_golden(self) -> None: + out, err = io.StringIO(), io.StringIO() + with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err): + code = capture.main(["--check", str(SHIPPED_GOLDEN)]) + self.assertEqual(code, 0, out.getvalue() + err.getvalue()) + # The count is stated, because a validator that examined nothing and a + # validator that examined everything print the same exit code. + self.assertIn("3 golden entries", out.getvalue()) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/vllm/models/test_nemotron_h_loader.cpp b/tests/vllm/models/test_nemotron_h_loader.cpp index 96b6ba76c..3268b5bb0 100644 --- a/tests/vllm/models/test_nemotron_h_loader.cpp +++ b/tests/vllm/models/test_nemotron_h_loader.cpp @@ -145,8 +145,288 @@ void NoteRan(const std::string& case_name) { Verdicts()[case_name] = "RAN"; } +// #926: the provenance contract's notion of "prose", matching +// `_is_prose`/`len(value.strip())` in scripts/nemotron-h-oracle-capture.py +// EXACTLY. +// +// This exists because the two copies disagreed on whitespace. Python has always +// spelled these `value.strip()`; this file spelled them `.empty()` and `.size()` +// on the raw string. So a golden carrying 200 SPACES in `unrecoverable_reason`, +// `evidence.never_reproduced` and `evidence.gate_form` was refused by `--check` +// with 3 problems and accepted here at 70 assertions, 0 failed, `SUCCESS!`. A +// blank paragraph is the record going missing exactly as surely as a deleted +// one, and two copies of one contract disagreeing about what satisfies it is +// the drift this three-copy design exists to prevent. +// +// Scope is deliberate and is NOT "trim everything". Python trims in `_is_prose` +// and in the argument floor, and it does NOT trim `capture.batch.shape`, which +// it tests for truthiness. Trimming that one here would repair this divergence +// by opening the mirror image of it. +std::string TrimmedProse(const nlohmann::json& node) { + if (!node.is_string()) return std::string(); + const std::string raw = node.get(); + const char* kSpace = " \t\n\r\f\v"; + const size_t first = raw.find_first_not_of(kSpace); + if (first == std::string::npos) return std::string(); + return raw.substr(first, raw.find_last_not_of(kSpace) - first + 1); +} + } // namespace +// ── The golden's PROVENANCE, asserted before any token is compared (#926) ─── +// Everything below this line that touches `oracle.json` compares tokens against +// it. This case asks the prior question: can that reference be regenerated, and +// does it say what configuration produced it? +// +// As `af8170154` committed it, the answer was no and the file was silent about +// it. `oracle.json` recorded the model, the revision, `temperature` and +// `max_tokens` and the vllm/transformers/flashinfer versions, and NOT ONE engine +// knob; the capture ran from an uncommitted driver in `$HOME` on `dgx.casa`, +// which was reimaged two days later. `enforce_eager`, `max_model_len`, +// `max_num_seqs`, `max_num_batched_tokens`, `gpu_memory_utilization` and the +// batch shape each move a greedy argmax at a near-tie, and two later runs under +// fully recorded configurations reproduce prompts 0 and 1 exactly and neither +// reproduces prompt 2 (32/32, 32/32, 26/32 and 32/32, 32/32, 29/32). +// +// So the contract this case holds is not "the configuration is recorded" -- it +// cannot be, and inventing one would be worse than the silence. It is that the +// golden STATES which of the two it is. A golden that records its configuration +// must record ALL of it; a golden that cannot must say so in the file and name +// the issue that owes the re-derivation. The third state, silence, is the defect, +// and it is what this case removes. +// +// It needs no checkpoint and no GPU: the golden is committed, so this runs on +// every runner. That is deliberate. The provenance defect is a records defect, +// and a gate for it must not need the hardware whose absence caused it. +// #926: the whitespace half of the contract, pinned HERE rather than only in +// the Python suite. +// +// The provenance case below reads the one committed golden, so it can only ever +// exercise the shape that golden happens to have. That made this arm unable to +// hold its own half: a blank-paragraph golden was refused by `--check` and +// accepted here. `test_the_floor_is_not_met_by_whitespace` in +// tests/scripts/test_nemotron_h_oracle_capture.py pins the Python side; this +// case is its counterpart, so neither copy depends on the other to notice. +// +// The values are the reviewer's: 200 spaces, which is over the 80-character +// floor by every untrimmed measure and is not one word of a record. +TEST_CASE("NemotronH golden: a blank paragraph is not prose") { + CHECK(TrimmedProse(nlohmann::json(std::string(200, ' '))).empty()); + CHECK(TrimmedProse(nlohmann::json(std::string("\t\n\r\f\v "))).empty()); + CHECK(TrimmedProse(nlohmann::json(std::string())).empty()); + + // 200 spaces measures 200 raw and 0 trimmed. The floor must read the second. + const nlohmann::json blank = std::string(200, ' '); + CHECK(blank.get().size() == 200); + CHECK(TrimmedProse(blank).size() == 0); + + // A non-string is not prose either, matching `_is_prose`'s isinstance check — + // this is the `"unrecoverable_reason": 123` divergence, held on both sides. + CHECK(TrimmedProse(nlohmann::json(123)).empty()); + CHECK(TrimmedProse(nlohmann::json()).empty()); + + // And the other side of the rule, so the case cannot pass by refusing + // everything: real prose survives, and surrounding whitespace is not counted + // toward the floor. + CHECK(TrimmedProse(nlohmann::json(std::string(" a real reason "))) == + "a real reason"); + const std::string padded = " " + std::string(80, 'x') + " "; + CHECK(TrimmedProse(nlohmann::json(padded)).size() == 80); +} + +TEST_CASE("NemotronH golden: the reference says whether it can be regenerated") { + const std::string path = std::string(NEMOTRON_H_GOLDENS_DIR) + "/oracle.json"; + std::ifstream in(path); + REQUIRE_MESSAGE(in.good(), "cannot read the committed golden at " << path); + nlohmann::json doc = nlohmann::json::parse(in); + + // Anti-vacuity first. A comparison over zero elements reports a perfect + // score, so the width every later case will compare over is asserted here + // rather than trusted. + REQUIRE(doc.contains("sampling")); + REQUIRE(doc["sampling"].contains("max_tokens")); + const int width = doc["sampling"]["max_tokens"].get(); + REQUIRE(width > 0); + REQUIRE(doc.contains("golden")); + REQUIRE(doc["golden"].is_array()); + REQUIRE(!doc["golden"].empty()); + for (const auto& entry : doc["golden"]) { + REQUIRE(entry.contains("prompt_token_ids")); + REQUIRE(!entry["prompt_token_ids"].empty()); + REQUIRE(entry.contains("token_ids")); + CHECK(static_cast(entry["token_ids"].size()) == width); + } + MESSAGE("golden: " << doc["golden"].size() << " prompts x " << width + << " tokens = " << doc["golden"].size() * width + << " token comparisons available"); + + REQUIRE_MESSAGE(doc.contains("capture"), + "the golden carries no `capture` block, so nothing in this " + "tree can say what configuration produced it -- see #926 and " + "scripts/nemotron-h-oracle-capture.py"); + const nlohmann::json& capture = doc["capture"]; + REQUIRE(capture.contains("engine_config_recorded")); + REQUIRE(capture["engine_config_recorded"].is_boolean()); + for (const std::string& key : {std::string("schema"), std::string("generator"), + std::string("captured_utc"), std::string("host")}) { + // std::string, not const char*: doctest stringifies a bare char* as a + // BOOL, and this message read "capture is missing '1'" until it did not. + REQUIRE_MESSAGE(capture.contains(key), "capture is missing '" << key << "'"); + } + + if (!capture["engine_config_recorded"].get()) { + // UNATTRIBUTED, and saying so is the whole contract. The golden is kept -- + // deleting evidence to make a gate green is never the repair -- but every + // token score taken against it is a difference from an unattributable + // reference, and this run says that out loud. + REQUIRE(capture.contains("unrecoverable_reason")); + CHECK(capture["unrecoverable_reason"].is_string()); + CHECK(!TrimmedProse(capture["unrecoverable_reason"]).empty()); + REQUIRE(capture.contains("issue")); + const std::string issue = capture["issue"].get(); + CHECK(issue.rfind("https://github.com/mudler/vllm.cpp/issues/", 0) == 0); + // "unrecorded" and "here is the record" cannot both be true. + REQUIRE(capture.contains("engine")); + CHECK(capture["engine"].is_null()); + + // ── The substance, not only the shape ────────────────────────────────── + // Everything above is satisfied by a file that says "unrecorded", names an + // issue and argues NOTHING. The attributed arm below is gated by STRUCTURE + // -- twenty keys are there or they are not -- but this arm's whole record + // is prose, and a check that asks only whether the prose is non-empty gates + // the shape and not the substance. Gut `evidence`, + // `forced_by_checkpoint_or_device` and `captured_utc_is`, put the word + // "dunno" in `unrecoverable_reason`, and the file still passes as a record + // while being one. That is the state #926 filed, reached from the other + // side. + // + // `forced_by_checkpoint_or_device` names the terms COMMON to every + // unoverridden run of this checkpoint, which is what narrows + // "unrecoverable" to the knobs a driver passes. `evidence` carries whether + // anything ever reproduced this golden and which gate form its behaviour + // licenses. The floor below detects REMOVAL of an argument; it does not and + // cannot claim the prose is true, and it is set from the shortest real + // field so that an honest rewording does not red it. + // + // Mirrored from REQUIRED_FORCED_TERM_KEYS, REQUIRED_EVIDENCE_KEYS and + // MIN_ARGUMENT_CHARS in scripts/nemotron-h-oracle-capture.py; + // tests/scripts/test_nemotron_h_oracle_capture.py parses these very lists + // out of this file and refuses to let the copies drift. + const size_t kMinArgumentChars = 80; + const std::vector kForcedTermKeys = { + "kv_cache_dtype", "moe_backend", "dtype", "quantization"}; + const std::vector kEvidenceKeys = {"never_reproduced", + "gate_form"}; + + REQUIRE_MESSAGE(capture.contains("captured_utc_is"), + "capture is missing 'captured_utc_is': a commit's author " + "date read as a capture time is a fabricated provenance, so " + "the file has to say which one this is"); + CHECK(!TrimmedProse(capture["captured_utc_is"]).empty()); + + for (const auto& block : {std::make_pair(std::string("forced_by_checkpoint_or_device"), + kForcedTermKeys), + std::make_pair(std::string("evidence"), kEvidenceKeys)}) { + REQUIRE_MESSAGE(capture.contains(block.first), + "capture is missing '" << block.first + << "': an unrecoverable " + "configuration is a claim, and " + "a claim without its supporting " + "record is silence"); + REQUIRE(capture[block.first].is_object()); + for (const std::string& key : block.second) { + REQUIRE_MESSAGE(capture[block.first].contains(key), + "capture." << block.first << " is missing '" << key + << "', so an argument this golden's " + "admissibility rests on is gone"); + CHECK_MESSAGE(!TrimmedProse(capture[block.first][key]).empty(), + "capture." << block.first << "['" << key + << "'] is empty or blank"); + } + } + + // The four fields whose content is an ARGUMENT rather than a value. + for (const std::vector& path : + std::vector>{ + {"unrecoverable_reason"}, + {"forced_by_checkpoint_or_device", "kv_cache_dtype"}, + {"evidence", "never_reproduced"}, + {"evidence", "gate_form"}}) { + const nlohmann::json* node = &capture; + bool reachable = true; + std::string dotted; + for (const std::string& step : path) { + dotted += (dotted.empty() ? "" : ".") + step; + if (!node->is_object() || !node->contains(step)) { + reachable = false; + break; + } + node = &(*node)[step]; + } + REQUIRE_MESSAGE(reachable, "capture." << dotted << " is absent"); + REQUIRE_MESSAGE(node->is_string(), "capture." << dotted << " is not prose"); + CHECK_MESSAGE(TrimmedProse(*node).size() >= kMinArgumentChars, + "capture." << dotted << " is " + << TrimmedProse(*node).size() + << " characters, under the " << kMinArgumentChars + << " an ARGUMENT needs: a one-word answer here is " + "the record going missing while the file keeps " + "its shape"); + } + + MESSAGE( + "UNATTRIBUTED GOLDEN: this reference records no engine configuration, " + "so a token difference against it is not yet a defect. Owed by " + << issue + << ". Re-derive with scripts/nemotron-h-oracle-capture.py --capture " + "--profile nhspeed-a."); + return; + } + + // ATTRIBUTED: then it is attributed COMPLETELY. Each key below can move a + // greedy argmax at a near-tie, and a key that is absent is not "the default", + // it is unrecorded -- which is the state this whole case exists to refuse. + // The list is duplicated in scripts/nemotron-h-oracle-capture.py + // (REQUIRED_ENGINE_KEYS) and tests/scripts/test_nemotron_h_oracle_capture.py + // asserts the three agree, so neither copy can drift alone. + REQUIRE(capture.contains("engine")); + REQUIRE(capture["engine"].is_object()); + REQUIRE(capture["engine"].contains("resolved")); + const nlohmann::json& resolved = capture["engine"]["resolved"]; + const std::vector required = { + "attention_backend", "block_size", "compilation_mode", + "cudagraph_capture_sizes", "cudagraph_mode", "dtype", + "enable_chunked_prefill", "enable_prefix_caching", "enforce_eager", + "gpu_memory_utilization", "kv_cache_dtype", "max_model_len", + "max_num_batched_tokens", "max_num_seqs", "moe_backend", + "num_gpu_blocks", "num_gpu_blocks_override", "quantization", + "seed", "tensor_parallel_size"}; + for (const std::string& key : required) { + REQUIRE_MESSAGE(resolved.contains(key), + "capture.engine.resolved is missing '" + << key << "', so this golden is only partly attributed"); + if (key != "num_gpu_blocks_override") { + CHECK_MESSAGE(!resolved[key].is_null(), + "capture.engine.resolved['" + << key + << "'] is null: a value that could not be read is not a " + "value that was default"); + } + } + REQUIRE(capture.contains("batch")); + REQUIRE(capture["batch"].contains("shape")); + CHECK(!capture["batch"]["shape"].get().empty()); + REQUIRE(capture.contains("legs")); + CHECK_MESSAGE(capture["legs"].get() >= 2, + "one leg cannot show the configuration is deterministic"); + REQUIRE(capture.contains("legs_agree")); + CHECK(capture["legs_agree"].get()); + MESSAGE("ATTRIBUTED GOLDEN: profile " + << capture["engine"].value("profile", "") << ", " + << required.size() << " engine keys recorded, " + << capture["legs"].get() << " agreeing legs"); +} + TEST_CASE("NemotronH: the REAL checkpoint loads and the forward produces logits") { const std::string kCase = "real_checkpoint_loads_and_forwards"; std::string why;