The Nemotron goldens' capture configuration is unrecorded, so they can be approximated but not re-derived: a rebuilt oracle reproduces 2 of 3 token-exactly

[localai-bot]

What happened

The pinned vLLM oracle was destroyed when dgx.casa was reimaged on 2026-08-14 and has now been rebuilt from source at the recorded pin, with identity asserted in code:

vllm 0.23.1rc1.dev1511+g555967922   transformers 5.14.1   flashinfer 0.6.15.post1
torch 2.13.0+cu130   GB10 sm_121
vllm_file /home/mudler/venvs/vllm-oracle-pin-555967922/vllm-src/vllm/__init__.py
ORACLE_IDENTITY_OK

Run against tests/parity/goldens/nemotron_35_lightning_greedy/oracle.json — same prompts, temperature=0, max_tokens=32:

Prompt	prompt_token_ids	token_ids
The capital of France is	MATCH	MATCH
Write the first five Fibonacci numbers:	MATCH	MATCH
Explain what a state space model is, in one sentence:	MATCH	DIFFERS at index 29

93 of 96 generated tokens exact. The third sequence is token-identical for 29 tokens of a long technical sentence, then takes 11286 ("a transition equation for") where the golden has 9943 ("a combination of state").

The actual defect

The goldens do not record the configuration they were captured under. oracle.json and the capture commit (af8170154, #517) record only temperature: 0.0 and max_tokens: 32. They do not record:

• enforce_eager (the rebuild used True)
• gpu_memory_utilization (the rebuild used 0.25)
• max_model_len (the rebuild used 4096)
• whether the three prompts were submitted in one generate() call or separately — i.e. the batch shape

Every one of those can change the reduction order of a greedy decode, and therefore the argmax at a near-tie. So the goldens can be approximated but not re-derived: there is no configuration a future run can be held to.

That is a records defect, not a code defect. AGENTS.md requires a gate's recipe to be recorded precisely because a number that cannot be reproduced is not evidence.

What is NOT yet established

Whether this divergence is a near-tie is a hypothesis, not a result. A top-2 margin probe at index 29 has been queued on the GPU host but has not produced a number yet. Nothing was adjusted in response to the mismatch, and nothing should be until that number exists.

Two readings remain open and the margin distinguishes them:

1. Near-tie — the two candidates are within noise, and the divergence is a reduction-order artifact of the unrecorded config difference. This is the likely reading: a genuinely wrong stack diverges early and across all prompts, not in the last three tokens of one.
2. A real difference in the rebuilt stack, which would matter for every number measured against it.

Note the precedent: a Qwen 35B "divergence" investigated earlier in this project turned out to be a bit-exact oracle tie, and the lesson recorded then was to ask the oracle's top-2 margin first.

What done looks like

1. Extend the golden capture format to record the full run configuration — at minimum enforce_eager, gpu_memory_utilization, max_model_len, batch composition, and the device — and have the consumer assert them, so a mismatch is reported as a configuration difference rather than a token failure.
2. Land the index-29 top-2 margin measurement and record which of the two readings it supports.
3. If near-tie: the row's gate needs the ratified distributional treatment rather than a bare token-exact comparison, which AGENTS.md already provides for where the oracle's own greedy decode is non-deterministic.
4. Re-capture the goldens under a recorded configuration once the format carries it.

Found while rebuilding the oracle for the speed-parity work owed by #517.

[localai-bot]

This is still open, it now blocks a second row, and there is a named candidate mechanism

Found again while repairing the A2-Q1 divergence record (#1388, PR #1432). Recording three things this issue did not have.

1. It was never in .agents/issue-index.md

Open since 2026-08-15 and no index row referenced it, so a session hitting the same wall re-derived it instead of finding it. PR #1432 appends a row that names MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm as the owner and lists it under ## Owed in .agents/specs/nemotron-h-a2q1-fp8-mamba.md.

2. It BLOCKS #1388, and it precedes #1431

A2-Q1's device mamba arm moves one token in 96 on GB10 (prompt 2, generation position 32, 11286 where the golden holds 3468). The measurement that decides whether that is a near-tie or a defect is the oracle's own top-2 margin at that position. But a margin measured against an unreproducible golden is a margin against an unknown, so this issue is prior to that measurement. #1431 (the oracle cannot complete engine start-up on dgx:gpu0) blocks the margin too, and it needs a GPU lease. This one does not. It should be done first.

3. A candidate mechanism, visible in the lease logs and UNTESTED

kv_cache_dtype=fp8_e4m3 is auto-selected on this checkpoint, and vLLM says outright it can move tokens:

INFO [cache.py:296] Using fp8_e4m3 data type to store kv cache. It reduces the GPU
memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop
without a proper scaling factor
WARNING [kv_cache.py:134] Checkpoint does not provide a q scaling factor. Setting it
to k_scale. This only matters for FP8 Attention backends (flash-attn or flashinfer).

The checkpoint carries no q scale, so vLLM imputes one from k_scale, and the run is on flashinfer — the backend the warning names. Add kv_cache_dtype to the list of terms the capture format must record, alongside enforce_eager, gpu_memory_utilization, max_model_len and the batch shape.

Observed in /workspace/nhspeed/oracle.a.out (2026-08-18) and /workspace/a2q1-neartie/20260820T030818Z/neartie.ref.out (2026-08-20), both dgx:gpu0. Untested as a cause: nobody has re-run with kv_cache_dtype="auto".

4. The oracle is CONFIG-SENSITIVE, not non-deterministic — which decides the gate FORM

/workspace/nhspeed/oracle.a.out ran one configuration twice inside one process. ORACLE_LEG 1 and ORACLE_LEG 2 each returned matched=32, matched=32, matched=26, for ORACLE TOKEN MATCH: 180/192. At a fixed configuration the pinned oracle is deterministic; the committed 32/32 golden came from a different, unrecorded configuration.

That rules out one of this issue's own options. AGENTS.md permits a ratified distributional gate only where the oracle's greedy decode is non-deterministic, and this measures the opposite — so step 3 of "What done looks like" is not available on the current evidence. Steps 1 and 4, recording the configuration and re-capturing under it, are what the evidence licenses.

5. The 11286 fork in this issue is the same fork A2-Q1 sees

This issue's rebuilt oracle diverged on prompt 2 at index 29 to 11286 transition. That is, position for position, the continuation our host mamba arm produces on Thor sm_110: transition, equation, for. So a pinned vLLM oracle under its own configuration takes that fork. It is evidence about the position, and it says nothing yet about the device arm's position 32.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]

[localai-bot]

Cross-reference: the measured consequence of this issue is now recorded on PR #1289.

Short form: prompt 2 has never been reproduced by any nameable configuration — prompts 0 and 1 reproduce exactly, twice, under two different recorded configs; prompt 2 reproduces under neither (#926 rebuild 29/32 diverging at index 29; nhspeed-a 26/32, twice in one process). #1289's single divergence sits at index 31, inside the tail both other runs also fail.

So this issue is not a records-hygiene item. It is the reason a 7.28x speed win is held on evidence that cannot currently adjudicate it. PR #1645 commits a generator and makes the golden state what it does and does not record; the re-derivation this issue owes remains blocked on #1431.