record(A2-Q1): the moved token is not yet shown to be a defect, and the measurement that would decide it is blocked (#1388, #1431) - #1432
Draft
localai-bot wants to merge 33 commits into
Draft
Conversation
…e FP8 W8A8 seam (#517) The 23 Mamba2 layers were the last host bounce of a decode step, and they were the expensive one. The host arm reaches its two projections through `Linear(..., const NemotronHOwned&)`, whose `DenseFor` calls `NemotronHOwned::DenseBf16()` -- a FULL dequant of the fp8 tower into a fresh bf16 buffer ON EVERY CALL. That is 23 x (10304x2688 + 2688x4096) = 890e6 elements re-expanded per token, plus a download of the normed hidden and an upload of the mixer output per layer. The block moves whole or not at all: `mixer.in_proj` produces the fused `zxbcdt` that the causal conv and the SSD scan both consume (mamba_mixer2.py:550, split :692-696), so there is no intermediate landing in which the conv is on the device and `in_proj` is not. That is why the shared FP8 W8A8 linear seam had to be extracted first (#940, `dense_fp8_gemm.h`), and this change routes through it rather than re-typing its entry points. `NemotronHMamba2MixerDevice` mirrors the host arm statement for statement -- same vt:: ops, same order, same dtypes, different backend, which is the property A2-R established and the numeric gate reads: in_proj (FP8 W8A8) -> QkvSplit(z | xBC | dt) -> CausalConv1dFwd(silu) -> QkvSplit(x | B | C) -> Mamba2ChunkScan -> RmsNormGatedGroup(n_groups) -> out_proj (FP8 W8A8) The one substitution is the split: the host arm copies columns with `SliceCols` because `vt::Mamba2ChunkScan` validates every operand contiguous, and `vt::QkvSplit` is exactly that copy on the device. Both forwards select it at runtime, never by preprocessor: `MambaIsFp8` names only `NemotronHWeightForm` and `vt::OpRegistered` is the op table's own answer, so a dense NemotronH or a device without the fp8 pair keeps the host bounce. The paged arm hands the mixer the recurrent rows `vt::GdnStateGather` already gathered and zeroed and it advances them IN PLACE, so the two downloads, the host mixer and the two uploads per layer are gone; A2-P's gather/scatter pair and its `has_initial = true` reasoning are unchanged. The `ssm_dtype == f32` term in the paged selection is not decoration. The gather widens the page into an f32 working buffer by op contract and the host arm then narrows it back to `ssm_dtype`, so on a checkpoint whose `mamba_ssm_cache_dtype` is not f32 the two arms would round differently and the per-block numeric gate would be comparing two different computations. The device residency is built on first use and keyed on a `ResidentSlot` the weights own, never on an address (#237): across two engine builds in one process an address-keyed cache returns the previous engine's device pointers, which is plausible wrong values rather than a crash. The e4m3 staging copy is released as soon as `ResidentFp8` has uploaded it, so the conversion peaks at one projection rather than at the whole 890 MB tower. `dense_fp8::ResidentFp8` still does not account its own upload -- that is #974, and this change does not touch the shared header -- so A2-Q1 accounts what IT uploads at the site that causes it, through the same `load_stats::AddDeviceUpload` `ResidentWeight` and `ResidentNvfp4` call. That counter is also the instrument the new residency case reads: an arm that re-uploaded the tower every step returns identical numbers to one that uploads it once. G-SAFE is untouched. All three clauses of the interlock stand, this change creates no paging, no carried state and no batching that A2-P did not already create, and `num_reqs <= 1` remains A2-B's to remove. `tests/vllm/models/test_nemotron_h_mamba_device.cpp` is the cheap arm in front of the real-checkpoint gate, for the same reason A2-Q2a's device MoE file exists: `BuildTiny` is all-`kDense`, so nothing in the existing device suite could reach the FP8 W8A8 arm at all. The two arms are NOT bit-comparable by construction -- the host reference is W8A16 and the device arm is W8A8 as vLLM is -- so every band is measured in the run against a defect the fixture separates, and the element count is asserted against the geometry so a maximum over zero elements cannot read as agreement. `scripts/nemotron-h-a2q1-dgx-gate.sh` records the GB10 recipe rather than leaving it to be retyped, including the three environment facts that have each voided a run before: the CUDA lane is `sbsa`, `nvcc --version` is not a sufficient postcondition, and `/workspace` is CIFS and refuses symlinks. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…ms it divided, or refuses `examples/nemotron_h_gen` prints neither a rate nor a duration. It prints the engine load time and a `TOKEN MATCH: m/n` line, so the per-output-token number this row is measured against has to be derived from the wall clock, and a derived number that hides its terms is exactly how a rate over an unknown denominator comes to be quoted as if it had been measured. `scripts/nemotron-h-a2q1-per-token.py` prints the wall time, the load it subtracts and the token count it divides by on separate lines, and it refuses in the two cases where a plausible zero would otherwise be printed: when either term is absent from the log, and when the wall clock is not greater than the load it contains. The second case is not hypothetical -- the terms coming from different runs yields a NEGATIVE rate, which still formats like a measurement. The gate script calls it once per arm, so the device-on and device-off runs report the same shape. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…e it had to accept, so it gates the STATE Thor (sm_110) ran the focused suite and one case failed. The device arm itself passed everything else -- 49 assertions where a GPU-less box runs 4, the fresh block matched the host reference at T=1/8/12 (0.164, 0.282, 0.309 against a band of 0.5), both refusals threw, and the fp8 tower uploaded 61760 B on the first call and 0 B on the second. The failure is the instrument. The carry case banded the SECOND LEG'S OUTPUT against the separation of a dropped carry. On Thor the second leg agreed to 0.705 while a dropped carry separated by only 0.205, so the derived band (0.102) sat BELOW the deviation a FRESH leg already shows on this fixture. The two arms are W8A8 against W8A16, so a fresh leg already disagrees by the e4m3 activation quantization and a second leg compounds that with the same disagreement propagated through the carried state. A defect whose separation is smaller than the noise the comparison must accept is not resolvable from that comparison, and widening the band until it passes is what the spec's §8.1 says to stop for. So the assertion moves to what the carry actually IS. A dropped carry hands the next leg zeros, so the separation between the advanced state and a zeroed one is 1.0 by construction -- about six times the noise floor, which this fixture can genuinely resolve. The conv window and the SSM state are now banded separately, each against its own zeroed twin, each with its element count asserted against the geometry. The noise floor is MEASURED in the run, at the same width, from a fresh device leg against a fresh host leg, and printed beside the separation. The second leg's output is still measured, and it carries an assertion only when the separation exceeds twice that floor; the condition is printed either way, so "no assertion here" is a stated measurement rather than a silent hole. What this cannot see -- a carry advanced but wrong by less than the band -- is written down, and the real-checkpoint per-block gate is named as the instrument for it. Two things the failure exposed beside the band. The fixture left `mamba_ssm_cache_dtype` unset and so resolved bf16, which is NOT the configuration the paged forward selects the device arm for (`ssm_dtype == f32`); it now says float32 as the released checkpoint does, and the case asserts it, because a state silently halved is invisible to every comparison. And the whole file was a skip on a GPU-less box, so a new CPU-runnable case pins the op contract the split depends on -- three outputs of widths (I, conv_dim, num_heads) taking the head, middle and tail of an in_proj row. That case states plainly that it does not pin the production call site, which only the device numeric case covers. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
left open The spec's §3 measured Thor (sm_110) as having HALF an fp8 arm: the GEMM present through the unconditionally registered `kMatmulFp8CublasLt`, the activation quant trapped in a CUTLASS-gated translation unit that no sm_110 build compiles. #991 moved the registration out. Whether that actually reaches this model's arm was never run, and the spec said the conclusion had inverted twice during scoping, so it is worth writing down that it now has an answer. It does. On a build whose feature table reads `ENABLED for [110]: 1 ; DISABLED cells: 7` -- only `marlin-nvfp4`, with `cutlass-fp8` and both `scaledmm-c3x` cells off -- the FP8 W8A8 mamba arm executes: 49 assertions where a GPU-less box reports 4, the fresh block matching the host reference at three widths, and the fp8 tower uploading exactly once. The section also records the carry gate's failure and its repair, because the failure is the more instructive half: the case banded a defect whose separation (0.205) was SMALLER than the noise the comparison had to accept (0.164 on a fresh leg), which is §5.2's lesson arriving from the direction the spec did not anticipate. And it states which host can answer the occupancy question at all -- the 6.31% baseline is a GB10 number, so a Thor busy fraction would measure different silicon and answer something else. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…side a Thor number The A3 gate ran on Thor and printed `GPU busy in 358 of 2336 samples = 15.33% busy (baseline 6.31%)`. The denominator is right and the sample count is there, but the parenthesis is wrong: 6.31% was measured on GB10, and printing it beside an sm_110 number invites a comparison across two different pieces of silicon, two different clocks and two different memory systems. It answers a different question than the one it appears to answer. That is the failure this repository already knows by name -- a number quoted often enough comes to be treated as measured -- and a gate script is exactly where it starts, because the script's output is what gets pasted into a report. The baseline is now quoted only when `ARCH` is `121a`. On any other arch the line says so and names the comparison that IS valid there: the same-binary `VT_NEMOTRON_H_DEVICE_MAMBA` ON/OFF A/B on that same box. All three arms are self-tested, including the empty-sample arm, which reports UNMEASURED rather than 0% -- a busy fraction over zero samples formats identically to a genuinely idle GPU. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…line it cannot be read against The A3 gate ran on Thor (sm_110) with the FP8 W8A8 mamba arm on the device and returned 96/96, mode=decode, STRICT PASS, exit 0, with zero reference-tier lines. That last number is the one worth recording beside the verdict: the portable reference tier is numerically CORRECT, so a pass obtained on it is invisible in every other figure, and its absence is the only thing that separates the two. The busy fraction was 358 of 2336 samples, 15.33%. Both surfaces say explicitly that this is NOT comparable to the 6.31% baseline the unit is accepted against, because that baseline was taken on GB10 and this is different silicon. Recording the number without that sentence would have been the more useful-looking and less true entry, and the GB10 leg is still queued. BENCHMARKS also records what the run did NOT produce: no per-token time, because the Thor build predates the timing helper and its script line needed `bc`, which the container lacks. STATUS moves from one passing gate to two and names which arm the second one exercised. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…ered the wrong question (#1290) The Thor A/B ran both arms of the same binary against the same checkpoint and golden, differing only by VT_NEMOTRON_H_DEVICE_MAMBA. Two things came out of it and only one of them is a number this row may keep. THE ARM IS TOKEN-EXACT WHERE THE HOST ARM IS NOT. With the device arm on, the A3 gate read 96/96 mode=decode STRICT PASS, exit 0. With it off -- which is what main does on a GPU today -- it read 93/96 DIVERGENCE, exit 1. Zero reference-tier lines on both, so neither run took the portable tier. The mechanism is named rather than guessed: the golden comes from an oracle that computes these projections W8A8, and `DenseBf16` states outright at nemotron_h.cpp:419-422 that the host arm carries `input_scale` and does not apply it. That is #1290, filed while landing this row and fixed by it. It is n=1 per arm, and the row says so. THE BUSY FRACTION IS VOID, AND IT IS THIS UNIT'S ACCEPTANCE TEST. It read 15.33% on and 14.73% off, and neither figure means what it appears to: the sampler started with the process, so the multi-minute GPU-IDLE 20.1 GiB engine load sat inside the same window as the decode and diluted both arms toward each other. That is the same defect as summing prefill and decode into one profile, and a fraction over the wrong window is worse than no fraction, because it still formats like a measurement. No occupancy claim is made from that run in either direction. `run_gate` now starts the driver first, waits for `engine loaded in Ns`, and samples only after it; when that line never appears it reports the fraction as NOT REPORTED rather than sampling a window whose meaning is unknown. The per-token helper takes the decode window directly and subtracts nothing, since the load is already outside its brackets, and it refuses on a non-positive window -- the brackets failing to span a decode would otherwise print a negative or infinite rate that still looks like a measurement. What survives from the run without an instrument caveat is the wall-time ratio from the sample counts, one sampler at one nominal interval: 2.44x shorter with the arm on. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
`origin/main` moved while this row's Thor A/B was running, so the branch is merged rather than rebased and the gate is re-run over the merged tree. AGENTS.md R4: merge an immutable SHA and re-run the full gate after merging rather than reading the diff. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…a counter-measurement The issue as filed said the host mamba arm is what main does "on a GPU today" and therefore that main is token-wrong on a GPU. That is established for sm_110 ONLY. On GB10 (sm_121a) the SAME host arm reads 96/96 STRICT PASS -- the A3 run that closed #1157, /usr/local/nas_share/rc/nh1157/gate_fixed.out, same golden and same oracle revision. Left as written, the issue would get dismissed the first time somebody checked on GB10 and got a pass, and a silicon-specific defect would go with it. So the row, the spec and the benchmark entry now all say what was measured WHERE: host arm 93/96 on sm_110 with fa2 and cutlass-fp8 both DISABLED, host arm 96/96 on sm_121a with both ENABLED, n=1 per arm. What is established is narrower and still worth having: on ONE box, holding the entire rest of the tower constant, flipping only the mamba arm moves 93/96 to 96/96. The `DenseBf16` reading stays the leading mechanism -- the golden's oracle is W8A8 and the host arm is W8A16 because `input_scale` is carried and not applied (nemotron_h.cpp:419-422) -- but it now carries the obligation to explain why GB10 is clean, and the honest reading is that the perturbation is marginal rather than grossly wrong. One proposed candidate is excluded rather than chased. "The two arms resolve different fp8 GEMMs" cannot be the differentiator, because that configuration runs NO fp8 GEMM on either box: `DenseFor` dequantizes and hands the result to vt::MatmulBT on the CPU queue (nemotron_h_device.cpp:2027). The device-side difference that IS checkable is attention -- CudaArchFeatures.cmake:349 provides fa2 for 12.1a and not for 11.0, so GB10 runs FlashAttention-2 over the 6 GQA layers where Thor runs the portable fallback. Next step named rather than the question closed: the oracle's top-2 margin at the three moved tokens, because this family has already produced a "divergence" that turned out to be a bit-exact near-tie; then a NemotronHTrace layer bisect; then a repeat to lift n=1. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…of its four behaviours ARE refusals The helper exists because `examples/nemotron_h_gen` reports neither a rate nor a duration, so the per-output-token number this row is measured on has to be derived. A derived number that prints 0, or a negative, or a rate over an unknown denominator is indistinguishable from a measurement once it reaches a report, so most of the helper is guards -- and nothing in the tree pinned them. Six cases now do: the rate is printed with the window, the excluded load and the token count beside it; the load is excluded rather than subtracted a second time; a non-positive window refuses; a missing TOKEN MATCH line refuses; zero compared tokens refuses; and the vLLM denominator is the pinned 0.014369 s rather than a constant that could drift and silently restate the gap. Armed, not merely present. Deleting the non-positive-window guard reds the suite; making the missing-token path print `per output token 0.000000 s` reds it. The tree was restored byte-for-byte between mutations and after, sha256 be32e5f5c786ccb283c88b88e256f9c0e5d79af87eeef5615a3c10a615d3ce8c both times, and the suite is green at that sha. The whole `run_gate` window change was also driven end to end against a fake driver and a fake nvidia-smi before it is allowed near the scarce GB10 lease: a 3 s load plus a 4 s decode yields 39 samples over a 3.994 s window rather than ~70 over ~7 s, so the load really is outside the brackets; a driver that dies during load reports `NO busy fraction is sampled` and still surfaces its exit code 3; and a 93/96 divergence surfaces exit 1 with `matched 93` beside the rate. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…se doctest stringifies char* as bool
The Thor re-run confirmed the repaired carry gate -- 5 cases, 63 assertions, 0
failed -- and printed the numbers that justify it: a noise floor of 0.2465 at
T=1, a carried conv window agreeing to 0.1746 against a band of 0.5, and a
carried SSM state agreeing to 0.0614 against the same band, at 2.9x and 8.1x
margins.
It also proved the diagnosis that drove the repair. A dropped carry separates the
second leg's output by 0.2045 while the noise the comparison must accept is
0.2465, so the defect really is smaller than the noise, `separation > 2 *
noise_floor` is false, and the case correctly asserts nothing there.
Correctly, but not legibly. The line that reports WHY no assertion was made
printed `1`. doctest stringifies a `const char*` as a bool, and the message
streamed a `char*` ternary, so it printed the truthiness of a non-null pointer
whichever branch was live -- an unconditional `1` that says nothing about the
condition it names.
That line exists to make "no assertion was made here" a STATED result rather than
a silent hole, so a version of it that cannot say what it means is the same class
of defect as the band it reports on: an instrument that formats like information.
It now builds a std::string. Reproduced against doctest 2.5.2 in both forms
first: `MESSAGE("x " << (false ? "yes" : "prose"))` prints `x 1`, and the
std::string form prints `x prose`.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
…ide a Thor decode The decode-window sampler works. With the arm on, Thor read 42.55% busy over 564 DECODE samples where the old conflated window read 15.33%, and the reason is now measured rather than argued: the engine load is 500.9 s and the decode is 75.4 s, so the old window was 87% load. The same run also printed `per output token 0.785606 s (vLLM 0.014369 s; ratio 54.7x)`. That ratio is wrong in the way the busy fraction was wrong an hour ago: 0.014369 s is a GB10 figure and this decode happened on Thor, so the quotient compares two different pieces of silicon. Fixing the busy-fraction surface and leaving its twin is how a bad comparison survives a correction, and 54.7x is exactly the kind of number that gets quoted once it exists. The rate still prints, because it is a real measurement of that box. Only the comparison is withheld, and it is withheld BY NAME rather than silently dropped, so the reader knows a ratio was declined rather than forgotten. The gate script now passes ARCH through. Two cases pin both arms, and the guard is armed: quoting the ratio unconditionally reds the suite. Tree restored byte-for-byte after the mutation, sha256 3d94497009c384bcb36709a48b3e1c5da203fea23db020ecb097b79af035cae1, green at that sha. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…n Thor, and the divergence reproduced The acceptance test of this unit was never a ratio: the GPU busy fraction had to rise, measured with its denominator. On the corrected instrument it does. Second Thor lease, fresh build and clone, same binary, same checkpoint, same golden, differing only by VT_NEMOTRON_H_DEVICE_MAMBA: arm ON 96/96 STRICT PASS rc=0 240/564 = 42.55% busy 0.785606 s/token arm OFF 93/96 DIVERGENCE rc=1 700/3808 = 18.38% busy 5.633442 s/token +24.17 points, a 2.31x rise in occupancy, and 7.17x less time per decode token. Peak host 44070 MiB. Zero reference-tier lines on either arm, so neither ran on the portable tier. READ ON THE BOX IT WAS TAKEN ON. These are sm_110 figures, and BOTH references this row is normally quoted against -- the 6.31% baseline and the 0.014369 s per token -- are GB10's. Neither supports a ratio against these numbers, the instrument withholds both by name off 121a, and the stale `ratio 54.7x` and `ratio 392.1x` strings in that run's log predate the per-token arch gate and must not be quoted. The per-token VALUES are sound; the ratios beside them are not. The divergence also reproduced, which lifts the n=1 caveat for Thor: two independent leases with separate builds agree exactly, 96/96 on and 93/96 off both times. GB10 stays n=1 in the other direction, and no GB10 run of the DEVICE arm exists at all yet -- so the hypothesis is SUPPORTED on sm_110 and UNMEASURED on sm_121a, and the owed list now says so rather than implying one covers the other. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
`origin/main` moved again while the second Thor lease was running. Merged rather than rebased, and the gate is re-run over the merged tree: AGENTS.md R4 says to merge an immutable SHA and re-run the full gate rather than reading the diff. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
`main` advanced to f06b9e9 carrying #1297, which routes `test_dspark_block_size_guard.cpp` through `tests/support/process_id.h`. This branch's `build-newest-gcc` failure was that inherited red, so it is merged forward to pick the fix up rather than re-run against a stale base. `docs/BENCHMARKS.md` was a GENUINE content conflict, not a driver artifact: this branch inserts the A2-Q1 row and main inserted the `ENG-RECORD-ANCHOR-RATCHET` row at the same point. They are different keys, so BOTH are kept -- main's first, then this row's. Neither was dropped, and no ratchet was raised to fit them. The union auto-merge of `.agents/issue-index.md` was DISCARDED: main's file taken wholesale, only rows whose KEY main lacks re-appended, main's file asserted a strict PREFIX. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
GitHub does not run the `merge=union` driver `.gitattributes:7` sets, so a branch touching `.agents/issue-index.md` re-conflicts whenever main appends a row. Resolved locally where the driver runs: the union auto-merge was DISCARDED, main's file taken wholesale, only rows whose KEY main lacks re-appended, and main's file asserted a strict PREFIX with zero duplicate keys. Any keyed-table collision kept BOTH sides, main's first, so no row is dropped. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
GitHub does not run the `merge=union` driver `.gitattributes:7` sets, so a branch touching `.agents/issue-index.md` re-conflicts whenever main appends a row. Resolved locally where the driver runs: the union auto-merge was DISCARDED, main's file taken wholesale, only rows whose KEY main lacks re-appended, and main's file asserted a strict PREFIX with zero duplicate keys. Any keyed-table collision kept BOTH sides, main's first, so no row is dropped. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
GitHub does not run the `merge=union` driver `.gitattributes:7` sets, so a branch touching `.agents/issue-index.md` re-conflicts whenever main appends a row. Resolved locally where the driver runs: the union auto-merge was DISCARDED, main's file taken wholesale, only rows whose KEY main lacks re-appended, and main's file asserted a strict PREFIX with zero duplicate keys. Any keyed-table collision kept BOTH sides, main's first, so no row is dropped. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
GitHub does not run the `merge=union` driver `.gitattributes:7` sets, so a branch touching `.agents/issue-index.md` re-conflicts whenever main appends a row. Resolved locally where the driver runs: the union auto-merge was DISCARDED, main's file taken wholesale, only rows whose KEY main lacks re-appended, and main's file asserted a strict PREFIX with zero duplicate keys. Any keyed-table collision kept BOTH sides, main's first, so no row is dropped. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
GitHub does not run the `merge=union` driver `.gitattributes:7` sets, so a branch touching `.agents/issue-index.md` re-conflicts whenever main appends a row. Resolved locally where the driver runs: the union auto-merge was DISCARDED, main's file taken wholesale, only rows whose KEY main lacks re-appended, and main's file asserted a strict PREFIX with zero duplicate keys. Any keyed-table collision kept BOTH sides, main's first, so no row is dropped. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…llable under the name it was staged with (#1416) `oracle-wheel-in-lease.md` calls the wheel on `/workspace/oracle-vllm` the durable artifact of the in-lease oracle build. It is, and `pip` cannot install it. The staged name vllm-0.1.dev1+g555967922-FLASHINFER-ONLY-cp312-cp312-linux_aarch64.whl has seven `-`-separated parts where PEP 427 allows five, six with a build tag, so `packaging.utils.parse_wheel_filename` raises `InvalidWheelFilename` before pip opens the file. Verified against that parser directly: the conforming `vllm-0.1.dev1+g555967922-cp312-cp312-linux_aarch64.whl` parses to `('vllm', Version('0.1.dev1+g555967922'))`, which is what the wheel's own `METADATA` carries. Only the marker is unrepresentable. The bytes, the commit and the FlashInfer-only property are untouched by a copy to that name. WHERE the failure lands is what made it expensive, and it is why this is recorded rather than absorbed. `pip install -q torch==2.13.0` has to run first and takes about thirteen minutes, so `RC[pip wheel]=1` arrives long after the job looks healthy; a driver that does not stop there then reports the whole thing as `ModuleNotFoundError: No module named 'vllm'` at the measurement step. That is an infrastructure failure presenting as a verdict about the model, on a box where a wrong verdict costs another lease. A driver installing this wheel therefore asserts the identity and exits on failure. Nothing else in that job was in doubt, and the spec says so rather than leaving a reader to wonder: `RC[apt-get install]=0`, `nvcc` already present at `cuda_13.0.r13.0` needing no install, `RC[pip torch]=0`, `cuda True NVIDIA GB10`. Found while measuring the oracle's own top-2 margin for #1388, and fixed in the same flow because it is small, clear, and blocks that measurement. The shared `README-WHEELS.md` on the NAS carries the same correction, since its `pip install <the FLASHINFER-ONLY wheel>` line is the instruction that cannot work. Evidence: `/workspace/a2q1-neartie/20260819T215514Z` on `dgx:gpu0`, 2026-08-19. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…he measurement that would decide it is blocked (#1388, #1431) §10.4 already owed the oracle's top-2 margin at the moved tokens. This commits the instrument that takes it and records why it has not been taken, so the next session spends its lease on the blocker rather than on rediscovering it. `scripts/nemotron-h-a2q1-neartie-gap.py` teacher-forces the pinned oracle on the golden prefix and reads its own top-K at the divergent position, in the shape this repository already uses for near-tie triage. It has never produced a number: five runs on `dgx:gpu0` were each killed by the host-memory watchdog during engine START-UP, before a token existed. That is #1431, and its header says so rather than presenting itself as a working recipe. The spec gains §10.5, which separates what the discriminator settled from what it did not. It settled that the moved token belongs to the device arm. It did not settle that the arm is wrong, and three things already in the record point the other way: The two device legs run DIFFERENT recurrent kernels -- `state_update_rows=23 chunk_scan_calls=0 gathers=0 scatters=0` against `state_update_rows=0 chunk_scan_calls=23 gathers=46 scatters=46` -- and emit a byte-identical wrong token. That excludes the recurrent reduction order and the gather/scatter state indexing, because neither could be invariant across that pair. The polarity flips with silicon, within one binary on each box: the device arm is 96/96 on Thor and 95/96 on GB10, the host arm 93/96 on Thor and 96/96 on GB10. A systematic FP8 defect in the device arm cannot be 96/96 on Thor. And the more precise arm is the worse tracker. The host arm is W8A16 and never quantizes the activation, so it is strictly more precise than vLLM, which runs these projections W8A8 -- confirmed from the oracle's own startup line, `Selected FlashInferFP8ScaledMMLinearKernel for ModelOptFp8LinearMethod`. If precision loss drove this the host arm would track the golden best; on Thor it is the worst, and diverges two positions EARLIER. Accumulating quantization error predicts the opposite ordering. The prompt itself is also not one the oracle is stable on. Same box, same staged checkpoint, same greedy sampling, changing only the engine configuration, the pinned oracle reproduced its own committed golden 32/32 on prompts 0 and 1 and 26/32 on prompt 2. Our device arm moves one token there; the oracle moves six. None of that replaces the margin in nats, which is what decides the FORM of the replacement gate. It is the reason the margin is worth a lease. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
This was referenced Aug 20, 2026
…it argues against cannot be reproduced (#1388, #926) The bottom line was right and the argument was not. §10.5 was headed "three independent lines say it is a tie" and read each line as exculpatory. Every fact it stated is correct and is kept. The reading is withdrawn. Line 1 is INCRIMINATING. The two device legs run different recurrent kernels and lose the same token, which excludes the recurrent reduction order and the gather/scatter indexing. §10.5 never used the third leg: `a3_hostmamba` reads 96/96 STRICT PASS. So the token is inside the DEVICE MAMBA BLOCK, the recurrent kernel is excluded INSIDE it, and what both failing legs still share and the passing leg does not is the FP8 W8A8 projection path -- which is what #1289 adds. Line 2 excludes only a gross systematic defect, which nobody proposed. A marginal perturbation whose token flip depends on the rest of the tower survives the silicon flip untouched, and that is the mechanism #1290 already applies to the HOST arm. Applying it to one arm and refusing it to the other is a choice about which arm to protect. Line 3 is refuted twice. The golden IS a W8A8 computation, so a W8A16 arm differs from it by the full activation-quantization error the reference applied and the arm did not; "more precise" therefore predicts the ordering that was observed, not the opposite. And on GB10 the ordering REVERSES -- host 96/96 beats device 95/96 -- while §10.5 cited Thor only. The section now records the state as UNDETERMINED. That is a stronger case for holding #1289 than the one it replaces. §10.4 two paragraphs above still said "no GB10 run of the DEVICE arm exists yet", which the discriminator superseded, so the file contradicted itself at its own head. Fixed with the sentence that says which run superseded it. The discriminator had no provenance at all: no run id, date, box, binary hash or evidence path. It has them now, and it carries the disclosure that matters most -- the six counters it quotes come from an UNCOMMITTED edit. None of `state_update_rows`, `chunk_scan_calls`, `conv_update_rows`, `conv_fwd_calls`, `gathers` or `scatters` occurs anywhere in `src/`, and the committed `[NH-DIAG]` facility prints a different line (`nemotron_h_device.cpp:1764`). The logs are real and readable; the instrument cannot be rebuilt from a checkout. Committing it is now owed. #926 already owned the golden's unreproducibility and was not linked from anywhere. `oracle.json` records no engine configuration and `af8170154` committed no generator, so the 26/32 on prompt 2 compares a known configuration against an unrecorded one. It is now indexed, listed under `## Owed`, and marked as PRECEDING #1431, because it needs no GPU lease. A candidate mechanism is added to the issue: `kv_cache_dtype=fp8_e4m3` is auto-selected here and the checkpoint carries no q scale, so vLLM imputes one from `k_scale`. The gate question is settled the other way from how the record read. `oracle.a.out` ran one configuration twice in one process, `ORACLE_LEG 1` and `ORACLE_LEG 2`, with identical results. That is determinism at fixed configuration, so a ratified distributional gate is INADMISSIBLE. What is licensed is re-deriving or re-pinning the golden against a NAMED configuration. Two watchdog floors, not one. Runs 1, 4 and 5 used `< 15000MB`; runs 2 and 3 used `< 20000MB` and were killed ~5 GB earlier in the drawdown. Their outcome at 15000 is now marked INFERRED rather than measured. The index row also stops quoting `skipped memory profiling` as a log line: it is a SOURCE string, it appears in no log, and that is the stronger result -- `gpu_worker.py:465-468` runs `profile_run()` before `:470-482` logs it, so run 5 died inside that forward. "byte-for-byte" is narrowed to the engine kwargs, the 5x host-footprint number is added (~17 GB against ~83 GB), and the torch/flashinfer delta between the two venvs is named as an unexcluded candidate. `nemotron-h-a2q1-neartie-gap.py` carried a stale layer contradicting its own honest header: "nothing below deviates from it" beside "ONE variable changed", "change the ENGINE not at all" beside an engine kwarg, "THREE ATTEMPTS" where there were five, and a conclusion inside KV cache allocation that the row's own result excludes. It now states what was measured, including that run 4 ran the reference configuration unchanged and died. The line-5 disclosure stays. §1.1 records what the reviewer's conv-dtype falsification did not: the guard at `nemotron_h_device.cpp:1582-1584` explicitly admits `kF32`, so it is not what holds the page at bf16. The allocation is. A `conv_dtype` mutation would sail past it. Pre-existing on `main`, listed rather than repaired. Upstream anchor drift is recorded for #1289's reviewer rather than fixed here, separated into what #1289 introduced and what `main` already carried. Records and one script. No product code, no checker semantics, no gate thresholds. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
The branch was behind `origin/main` 63d8780, which SKIPPED the `commit-trailers` and `commit-style` gates -- a skip is not a green -- and left the pull request `CONFLICTING`/`DIRTY` on GitHub, so `refs/pull/1432/merge` 404s and no workflow has ever been scheduled for this change. `git merge-tree` reports the tree clean, because the conflict is the `.agents/issue-index.md` append that GitHub resolves without the `merge=union` driver. Merged rather than rebased: a rebase needs a force-push, and `.agents/developer-preferences.md` says to ask first before any force-push or ref rewrite outside `main`. A merge fixes the same two problems with a plain push. The keyed record is verified rather than trusted. `.agents/issue-index.md`: 451 rows on `origin/main`, 445 on the branch, 455 merged; 4 branch-only keys (#926, #1290, #1416, #1431), zero duplicated keys, zero lost keys, and every `origin/main` row byte-for-byte equal in the result. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
`origin/main` advanced from 63d8780 to 94972c0 (#1436) between the preflight run and the push, so `check-commit-style.py` refused the new base as a non-ancestor. That is the rejected-push loop, taken before the push rather than after it: fetch, merge again, rerun the gate, push. The keyed record is re-verified against the newer base. `.agents/issue-index.md`: 451 rows on `origin/main` 94972c0, 455 merged, zero duplicated keys, zero lost keys, every `origin/main` row byte-for-byte equal, and the four branch-only keys (#926, #1290, #1416, #1431) all present. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
`origin/main` advanced again (#1428) while the previous merge was being gated, and its `.agents/issue-index.md` append put the pull request back into `CONFLICTING`/`DIRTY` -- GitHub does not apply the `merge=union` driver, so every concurrent append to that file re-opens the same conflict. Merged again so that `refs/pull/1432/merge` can be computed and a workflow can finally be scheduled for this change. The keyed record is re-verified against this base. `.agents/issue-index.md`: zero duplicated keys, zero lost keys, every `origin/main` row byte-for-byte equal, and the four branch-only keys (#926, #1290, #1416, #1431) all present. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
… of what the section says (#1388) Two defects in the section committed one commit earlier, both caught on a render read of the result rather than of the diff. §10.5.1 was headed "and what regenerates it" above a paragraph whose whole point is that NOTHING in this tree regenerates it: the counters come from an uncommitted edit on a lease box. A heading that promises a recipe over text that withdraws one is the same defect the rest of this change is repairing, one level down. It now says "why nothing in this tree can regenerate it". §10.5.5 said "the three continuations" over a table that grew a fourth row when #926's rebuilt oracle was added to it. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
`origin/main` advanced again (#1427). Merged so the pull request stays `MERGEABLE` and the scheduled workflow runs against a computable merge ref. The keyed record is verified against this base: 454 rows on `origin/main`, 458 merged, zero duplicated keys, zero lost keys, every `origin/main` row byte-for-byte equal, and the four branch-only keys (#926, #1290, #1416, #1431) all present. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…nd the anchors beside it hide that (#1388) A review of #1432 raised `nemotron_h_weights.cpp:855` as a too-wide dtype: it reads `mamba_ssm_cache_dtype` from the HF config and defaults it to `float32`, while vLLM's `CacheConfig` defaults the same name to `auto` (`vllm/config/cache.py:135`), which `mamba_utils.py:101-108` resolves to the model dtype, bf16. That reads as 2x the temporal-state bytes taken from `transformers` in place of the primary oracle. It is wrong, and this records why rather than leaving the next reader to re-derive it from the same anchors. vLLM does not stop at `CacheConfig`. `vllm/model_executor/models/config.py:605-631` declares `NemotronHForCausalLMConfig` with `DEFAULT_MAMBA_SSM_CACHE_DTYPE = "float32"` -- its docstring says "Only `float32` is known to have no accuracy issues by default" -- and `update_mamba_ssm_cache_dtype` reads the key FROM THE HF CONFIG whenever the CLI setting is `auto`, writing the result back. It is registered for `NemotronHForCausalLM` at `:879`. Same key, same source, same default as ours. Observed as well as read: the pinned oracle logs `config.py:621 Updating mamba_ssm_cache_dtype to 'float32' for NemotronH model` in five separate lease runs, including the 2026-08-18 run behind `oracle.a.out`, and this checkpoint's `config.json` carries `mamba_ssm_cache_dtype: "float32"` so the default is never reached. The real defect is the citation. `nemotron_h_registry.cpp:256-263` and `nemotron_h_device.cpp:1586` cite `mamba_utils.py` alone, which is exactly the half of the chain that makes the value look unmirrored, so the anchors invite the wrong conclusion. §10.6 records the refutation and §11 owes the missing anchor. A negative result is a result. This one would otherwise be re-filed against #1289's owner as a dtype bug. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…hange that documents off-by-one anchors (#1388) §1.1 cited the guard that admits `kF32` as `nemotron_h_device.cpp:1582-1584`. The `VT_CHECK` runs 1581 to 1583: 1581 opens it with `kBF16 || kF16 ||`, 1582 carries the `kF32` term, 1583 closes it with the message. The cited range started one line inside the statement and ran one line past it, into the NEXT `VT_CHECK`. Verified by reading the range rather than by trusting it, and the opening line is unique in the file. Corrected in §1.1 and in the pull request body. Filed as its own commit because the same change lists six anchor defects for #1289's reviewer. An off-by-one inside that list would be read as noise in the list itself. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
`origin/main` advanced again (#1408) during the preflight run. Merged so the pull request stays `MERGEABLE` and its workflow runs against a computable merge ref. The keyed record is verified against this base: zero duplicated keys, zero lost keys, every `origin/main` row byte-for-byte equal, and the four branch-only keys (#926, #1290, #1416, #1431) all present. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
… red this branch inherited #1392 merged at 07:26Z and closed #1371, so `CpuAttentionBackend` is registered and `cpu_attn.cpp` is on `main`. This branch was gating against a `main` without that fix, and #1409 measured what that costs: `build-test-cpu` and BOTH `sanitize-cpu` jobs failed on the identical test, `175 - test_nemotron_h_paged_forward`, against a diff of zero code files. Picking up this `main` is what lets those three jobs report on THIS change rather than on #1371, so it is taken before the CI verdict is read rather than after. Whether the three clear is a prediction until the run says so. If a `sanitize-cpu` job still fails, the failing test gets named rather than the job being waved through as a known red. The keyed record is verified BY KEY, not by a clean automatic merge. `.agents/issue-index.md`: 461 rows on `origin/main`, 465 merged, zero duplicated keys, zero lost keys, every `origin/main` row byte-for-byte equal, `origin/main`'s order preserved as a subsequence, and the four branch-only rows (#926, #1290, #1416, #1431) byte-for-byte equal to the branch versions. The append stays an append. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
`origin/main` advanced again (#1444) during the gate run that picked up 9ecaf1b. Merged so the pull request stays MERGEABLE and its workflow runs against a base that carries the #1371 fix. The keyed record is verified BY KEY: 462 rows on `origin/main`, 466 merged, zero duplicated keys, zero lost keys on either side, every `origin/main` row and every branch row byte-for-byte equal in the result, `origin/main`'s order preserved as a subsequence, and the four branch-only rows (#926, #1290, #1416, #1431) byte-for-byte equal to the branch versions. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
record(A2-Q1): §10.5 read its own evidence backwards, and the golden it argues against cannot be reproduced (#1388, #926)
#1289is held because the device mamba arm moves one token in 96. The bottomline of §10.5 was right -- the token is not yet shown to be a defect -- and the
argument for it was not. This rewrites the argument, gives the discriminator the
provenance it never had, and links the issue that already owned the golden's
unreproducibility.
§10.5, rewritten: the state is UNDETERMINED
The section was headed "three independent lines say it is a tie" and read each
line as exculpatory. Every fact it stated is correct and is kept. The reading is
withdrawn.
Line 1 is INCRIMINATING, not exculpatory. The two device legs run different
recurrent kernels --
state_update_rows=23 chunk_scan_calls=0 gathers=0 scatters=0againststate_update_rows=0 chunk_scan_calls=23 gathers=46 scatters=46-- and emit a byte-identical wrong token. That excludes therecurrent reduction order and the gather/scatter state indexing. §10.5 never
used the third leg:
a3_hostmambareads 96/96 STRICT PASS. So the token islocalized to the DEVICE MAMBA BLOCK, and with the recurrent kernel excluded
inside it, what both failing legs still share and the passing leg does not is the
FP8 W8A8 projection path -- which is exactly what #1289 adds. The elimination
points at the change.
Line 2 excludes only a GROSS systematic defect, which nobody proposed. A
marginal perturbation whose token flip depends on the rest of the tower survives
the silicon flip untouched, and that is the mechanism #1290 already applies to
the HOST arm. Applying it to one arm and refusing it to the other is a choice
about which arm to protect, not a reading of the evidence.
Line 3 is refuted twice. The facts hold: our host arm is W8A16
(
nemotron_h.cpp:416-422, "Weight-only:input_scaleis carried, not applied")and the oracle is W8A8
(
vllm/model_executor/kernels/linear/scaled_mm/flashinfer.py:67-85). But thegolden IS a W8A8 computation, so the quantity measured is
difference-from-reference, not absolute accuracy: a W8A16 arm differs from a W8A8
reference by the FULL activation-quantization error, so the observed ordering is
what is predicted rather than what contradicts it. And on GB10 the ordering
REVERSES -- host 96/96 beats device 95/96 -- while the section cited Thor only.
That is a stronger case for holding #1289 than the one it replaces.
The discriminator now has provenance, and one disclosure it needed
The run was cited with no run id, date, box, binary hash or evidence path. It is
20260819T200231Zondgx:gpu0under anrclease, 2026-08-19, evidence at/mnt/nas_share/rc/a2d1-discriminate/20260819T200231Z/, CUDA 13.0.88, GCC13.3.0,
121a,libvllm 0.0.3+cuda (ABI 21, header 21), checkpoint revision29f2d174,contention.txtempty.★ The counters come from an UNCOMMITTED instrument. None of
state_update_rows,chunk_scan_calls,conv_update_rows,conv_fwd_calls,gathersorscattersoccurs anywhere insrc/, and the committed[NH-DIAG]facility prints a different line (
nemotron_h_device.cpp:1764). The logs arereal and readable at that path; nobody can rebuild the instrument from a
checkout, and no binary hash was taken. §10.5 says so, and committing the
instrument is now owed.
§10.4 contradicted §10.5 at HEAD. Two paragraphs above, it still said "no
GB10 run of the DEVICE arm exists yet" and still owed "the only place a
DEVICE-arm A3 result does not yet exist at all". The discriminator superseded
both; the sentences outlived it. Fixed, naming the run that superseded them.
#926: the golden cannot be reproduced, and it PRECEDES the lease
oracle.jsonrecords the model, revision, sampling and thevllm/transformers/flashinfer versions. It records no engine configuration, and
af8170154committed no generator -- that capture ran from$HOME/venvs/vllm-oracle-next. So the26/32 on prompt 2result compares aknown configuration against an unrecorded one, and the difference cannot be
attributed to anything.
This was already filed. #926 has been open since 2026-08-15, was in no index
row, and was linked from nowhere. Rather than file a duplicate, this indexes it,
lists it under the spec's
## Owed, and marks it as preceding #1431 -- it ischeap and it needs no GPU lease. A candidate mechanism is added as a comment
there:
kv_cache_dtype=fp8_e4m3is auto-selected on this checkpoint(
cache.py:296) and the checkpoint carries no q scale, so vLLM imputes one(
kv_cache.py:134, "Checkpoint does not provide a q scaling factor. Setting itto k_scale"), on the
flashinferbackend the warning names. Untested as a cause.The gate form: CONFIG-SENSITIVITY, not non-determinism
oracle.a.outran the same configuration twice in one process --ORACLE_LEG 1andORACLE_LEG 2-- with identical results: 32/32, 32/32, 26/32,ORACLE TOKEN MATCH: 180/192. That is determinism at fixed configuration; the 32/32golden came from a different configuration.
The earlier body said "the prompt is also not one the oracle is stable on",
which invites the reading that the oracle's greedy decode is non-deterministic.
It is not, and
the distinction decides the gate FORM: AGENTS.md permits a ratified
distributional gate only when the oracle's greedy decode is non-deterministic,
so a distributional gate is INADMISSIBLE on this evidence. What is licensed
is re-deriving or re-pinning the golden against a NAMED engine configuration.
Nothing here is a case for weakening the token gate.
#1431: two watchdog floors, and a source quote that read as a log quote
The floor was not constant. Runs 1, 4 and 5 used
< 15000MB; runs 2 and 3 used< 20000MBand were killed ~5 GB earlier in the drawdown (12597, 19433, 19797,13941, 14846 MB). That runs 2 and 3 would also have crossed 15000 is an
inference, not a measurement -- still falling at ~1.18 GB/s, no arrest -- and
the row now says so.
job.shhard-codes one floor on unversioned CIFS and wasoverwritten between runs, so the artifact could not reconstruct three of the five
runs it produced; it now writes
watchdog.floorinto each run directory."skipped memory profiling"was quoted as if observed. It is a source stringand it appears in no log -- which is the stronger result. Under an absolute KV
budget
v1/worker/gpu_worker.py:465-468runsprofile_run()FIRST and only thenlogs
Initial free memory … skipped memory profiling(:470-482). Run 5 reachesneither string, so the process died inside that forward. The row is corrected
to say that.
"byte-for-byte" is narrowed to the engine kwargs, which is what was actually
identical. The venvs differ (
/tmp/nhspeed-oracleagainst/tmp/a2q1-oracle,with a fresh
pip install torch==2.13.0and a reinstalled wheel), and an identityassert on the vLLM COMMIT cannot see a toolchain delta, so the torch/flashinfer
difference is an unexcluded candidate for the regression. The number that
makes the regression vivid is now used: 08-18 reported
Available RAM: 66.97 GiBand peaked ~17 GB of host use; run 4 reported
94.86 GiBand burned ~83 GB --same box, same engine kwargs, 5x the host footprint.
The instrument's stale layer
scripts/nemotron-h-a2q1-neartie-gap.pylines 61-101 contradicted its own honestheader: "Nothing below deviates from it" beside "ONE variable changed", "change
the ENGINE not at all" beside an engine kwarg, "THREE ATTEMPTS" where there were
five, and a conclusion inside KV cache allocation that the row's own result
excludes. It now records what was measured, including that run 4 ran the
reference configuration unchanged and died, so the reference is a regression to
explain rather than a safe base to copy. The line-5 disclosure ("★ THIS HAS NEVER
PRODUCED A MEASUREMENT") stays, because it is accurate.
§10.6 -- one review finding checked at the pin and REFUTED
Review raised
nemotron_h_weights.cpp:855as a too-wide dtype: it readsmamba_ssm_cache_dtypefrom the HF config and defaults it tofloat32, whilevLLM's
CacheConfigdefaults the same name toauto(vllm/config/cache.py:135)which
mamba_utils.py:101-108resolves to bf16. That reads as 2x thetemporal-state bytes, mirrored from
transformersover the primary oracle --which AGENTS.md forbids and a token gate cannot see. It is a good hypothesis and
it is wrong.
vLLM does not stop at
CacheConfig.vllm/model_executor/models/config.py:605-631declares
NemotronHForCausalLMConfigwithDEFAULT_MAMBA_SSM_CACHE_DTYPE = "float32"-- docstring: "Onlyfloat32is known to have no accuracy issues bydefault" -- and
update_mamba_ssm_cache_dtypereads the key from the HFconfig whenever the CLI setting is
auto. Registered forNemotronHForCausalLMat:879. Same key, same source, same default as ours.Observed as well as read: the pinned oracle logs
config.py:621 Updating mamba_ssm_cache_dtype to 'float32' for NemotronH modelin five lease runs,including the 2026-08-18 run behind
oracle.a.out, and this checkpoint'sconfig.jsoncarriesmamba_ssm_cache_dtype: "float32".The real defect is the citation.
nemotron_h_registry.cpp:256-263andnemotron_h_device.cpp:1586citemamba_utils.pyalone -- exactly the half ofthe chain that makes the value look unmirrored -- so the anchors invite the wrong
conclusion. §10.6 records the refutation; §11 owes the missing anchor. Nothing is
filed, because there is no defect to file; a negative result that is not recorded
gets re-derived.
Recorded, not repaired
kF32while its own comment forbids it. Thefalsification that the persistent page is bf16 is correct
(
nemotron_h_registry.cpp:264allocateskBF16;runner.cpp:885-893sizesfrom it; only the transient gather row at
nemotron_h_device.cpp:1954is f32).But the guard at
:1581-1583admitskBF16 || kF16 || kF32, so it is notwhat holds the page there -- the allocation is, and a
conv_dtypemutationwould sail past it. Pre-existing on
main; §1.1 records it.split by ownership. Introduced by feat(#810 A2-Q1): NemotronH's 23 Mamba2 blocks reach the device on the FP8 W8A8 seam (#517) #1289:
mamba_mixer2.py:550→554,:586→585,:583/:583-585for the gated norm (self.norm(...)is at582). Already on
main, and therefore wider than feat(#810 A2-Q1): NemotronH's 23 Mamba2 blocks reach the device on the FP8 W8A8 seam (#517) #1289:nemotron_h.py:440→441 (three sites),:627-631forresidual is None(the four occurrences are 307/350/399/521;
residual = Noneis 618), andgdn_attn.py:405, which names the assigning branch where the comment describesthe
elseat 406-407.Verification
scripts/agent-preflight.shat5a58a6d63: 85 ok, 0 fail, 0 SKIP, "Allgates green.",
RANGE_COUNT = 25againstorigin/main94972c0.merge reported
SKIP commit-trailers/SKIP commit-styleand "NOT a greenpreflight" because the branch was behind
origin/main; after the merge bothreport
ok.check-commit-style.pythen refused the newer base as anon-ancestor when
origin/mainmoved from 63d8780 to 94972c0 mid-run,which is why there is a second merge commit.
.agents/developer-preferences.mdsays to ask first before any force-push orref rewrite outside
main. The merge fixes theCONFLICTING/DIRTYstate --a
.agents/issue-index.mdappend GitHub resolves without themerge=uniondriver -- with a plain push, which is what unblocks CI scheduling for this PR.
origin/main, 455merged, zero duplicated keys, zero lost keys, every
origin/mainrowbyte-for-byte equal, four branch-only keys (The Nemotron goldens' capture configuration is unrecorded, so they can be approximated but not re-derived: a rebuilt oracle reproduces 2 of 3 token-exactly #926, NemotronH: the HOST mamba arm reads 93/96 on Thor sm_110 and 96/96 on GB10 sm_121a, against the same golden #1290, The staged pinned-oracle wheel cannot be pip-installed: its filename violates PEP 427, and the failure surfaces 13 minutes later as ModuleNotFoundError #1416, The pinned oracle cannot complete engine start-up for the NemotronH NVFP4 checkpoint on dgx:gpu0: ~76 GB of host RAM disappears at the first forward, and a configuration that passed on 2026-08-18 no longer reproduces #1431).
Scope
Records and one script. No product code, no checker semantics, no gate
thresholds.
docs/is untouched by this change because no row changed lifecyclestate. Two files edited on the unversioned CIFS share cannot be diffed by a
reviewer and are described here instead:
job.shnow records its watchdog floor,and
README-WHEELS.md's stalepip install <the FLASHINFER-ONLY wheel>line --which stood ten lines above its own correction -- now points at that correction.
Issues: #926 (indexed and linked, not filed twice), #1431 and #1416. Spec:
.agents/specs/nemotron-h-a2q1-fp8-mamba.md§1.1, §10.4, §10.5, §11.FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]