feat(#1311 A2-D1): NemotronH decodes on the single-step recurrent kernels; token-neutral on both hosts, equivalence gated at n_groups=8, speed hypothesis refuted - #1362
Open
localai-bot wants to merge 47 commits into
Conversation
…e FP8 W8A8 seam (#517) The 23 Mamba2 layers were the last host bounce of a decode step, and they were the expensive one. The host arm reaches its two projections through `Linear(..., const NemotronHOwned&)`, whose `DenseFor` calls `NemotronHOwned::DenseBf16()` -- a FULL dequant of the fp8 tower into a fresh bf16 buffer ON EVERY CALL. That is 23 x (10304x2688 + 2688x4096) = 890e6 elements re-expanded per token, plus a download of the normed hidden and an upload of the mixer output per layer. The block moves whole or not at all: `mixer.in_proj` produces the fused `zxbcdt` that the causal conv and the SSD scan both consume (mamba_mixer2.py:550, split :692-696), so there is no intermediate landing in which the conv is on the device and `in_proj` is not. That is why the shared FP8 W8A8 linear seam had to be extracted first (#940, `dense_fp8_gemm.h`), and this change routes through it rather than re-typing its entry points. `NemotronHMamba2MixerDevice` mirrors the host arm statement for statement -- same vt:: ops, same order, same dtypes, different backend, which is the property A2-R established and the numeric gate reads: in_proj (FP8 W8A8) -> QkvSplit(z | xBC | dt) -> CausalConv1dFwd(silu) -> QkvSplit(x | B | C) -> Mamba2ChunkScan -> RmsNormGatedGroup(n_groups) -> out_proj (FP8 W8A8) The one substitution is the split: the host arm copies columns with `SliceCols` because `vt::Mamba2ChunkScan` validates every operand contiguous, and `vt::QkvSplit` is exactly that copy on the device. Both forwards select it at runtime, never by preprocessor: `MambaIsFp8` names only `NemotronHWeightForm` and `vt::OpRegistered` is the op table's own answer, so a dense NemotronH or a device without the fp8 pair keeps the host bounce. The paged arm hands the mixer the recurrent rows `vt::GdnStateGather` already gathered and zeroed and it advances them IN PLACE, so the two downloads, the host mixer and the two uploads per layer are gone; A2-P's gather/scatter pair and its `has_initial = true` reasoning are unchanged. The `ssm_dtype == f32` term in the paged selection is not decoration. The gather widens the page into an f32 working buffer by op contract and the host arm then narrows it back to `ssm_dtype`, so on a checkpoint whose `mamba_ssm_cache_dtype` is not f32 the two arms would round differently and the per-block numeric gate would be comparing two different computations. The device residency is built on first use and keyed on a `ResidentSlot` the weights own, never on an address (#237): across two engine builds in one process an address-keyed cache returns the previous engine's device pointers, which is plausible wrong values rather than a crash. The e4m3 staging copy is released as soon as `ResidentFp8` has uploaded it, so the conversion peaks at one projection rather than at the whole 890 MB tower. `dense_fp8::ResidentFp8` still does not account its own upload -- that is #974, and this change does not touch the shared header -- so A2-Q1 accounts what IT uploads at the site that causes it, through the same `load_stats::AddDeviceUpload` `ResidentWeight` and `ResidentNvfp4` call. That counter is also the instrument the new residency case reads: an arm that re-uploaded the tower every step returns identical numbers to one that uploads it once. G-SAFE is untouched. All three clauses of the interlock stand, this change creates no paging, no carried state and no batching that A2-P did not already create, and `num_reqs <= 1` remains A2-B's to remove. `tests/vllm/models/test_nemotron_h_mamba_device.cpp` is the cheap arm in front of the real-checkpoint gate, for the same reason A2-Q2a's device MoE file exists: `BuildTiny` is all-`kDense`, so nothing in the existing device suite could reach the FP8 W8A8 arm at all. The two arms are NOT bit-comparable by construction -- the host reference is W8A16 and the device arm is W8A8 as vLLM is -- so every band is measured in the run against a defect the fixture separates, and the element count is asserted against the geometry so a maximum over zero elements cannot read as agreement. `scripts/nemotron-h-a2q1-dgx-gate.sh` records the GB10 recipe rather than leaving it to be retyped, including the three environment facts that have each voided a run before: the CUDA lane is `sbsa`, `nvcc --version` is not a sufficient postcondition, and `/workspace` is CIFS and refuses symlinks. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…ms it divided, or refuses `examples/nemotron_h_gen` prints neither a rate nor a duration. It prints the engine load time and a `TOKEN MATCH: m/n` line, so the per-output-token number this row is measured against has to be derived from the wall clock, and a derived number that hides its terms is exactly how a rate over an unknown denominator comes to be quoted as if it had been measured. `scripts/nemotron-h-a2q1-per-token.py` prints the wall time, the load it subtracts and the token count it divides by on separate lines, and it refuses in the two cases where a plausible zero would otherwise be printed: when either term is absent from the log, and when the wall clock is not greater than the load it contains. The second case is not hypothetical -- the terms coming from different runs yields a NEGATIVE rate, which still formats like a measurement. The gate script calls it once per arm, so the device-on and device-off runs report the same shape. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…e it had to accept, so it gates the STATE Thor (sm_110) ran the focused suite and one case failed. The device arm itself passed everything else -- 49 assertions where a GPU-less box runs 4, the fresh block matched the host reference at T=1/8/12 (0.164, 0.282, 0.309 against a band of 0.5), both refusals threw, and the fp8 tower uploaded 61760 B on the first call and 0 B on the second. The failure is the instrument. The carry case banded the SECOND LEG'S OUTPUT against the separation of a dropped carry. On Thor the second leg agreed to 0.705 while a dropped carry separated by only 0.205, so the derived band (0.102) sat BELOW the deviation a FRESH leg already shows on this fixture. The two arms are W8A8 against W8A16, so a fresh leg already disagrees by the e4m3 activation quantization and a second leg compounds that with the same disagreement propagated through the carried state. A defect whose separation is smaller than the noise the comparison must accept is not resolvable from that comparison, and widening the band until it passes is what the spec's §8.1 says to stop for. So the assertion moves to what the carry actually IS. A dropped carry hands the next leg zeros, so the separation between the advanced state and a zeroed one is 1.0 by construction -- about six times the noise floor, which this fixture can genuinely resolve. The conv window and the SSM state are now banded separately, each against its own zeroed twin, each with its element count asserted against the geometry. The noise floor is MEASURED in the run, at the same width, from a fresh device leg against a fresh host leg, and printed beside the separation. The second leg's output is still measured, and it carries an assertion only when the separation exceeds twice that floor; the condition is printed either way, so "no assertion here" is a stated measurement rather than a silent hole. What this cannot see -- a carry advanced but wrong by less than the band -- is written down, and the real-checkpoint per-block gate is named as the instrument for it. Two things the failure exposed beside the band. The fixture left `mamba_ssm_cache_dtype` unset and so resolved bf16, which is NOT the configuration the paged forward selects the device arm for (`ssm_dtype == f32`); it now says float32 as the released checkpoint does, and the case asserts it, because a state silently halved is invisible to every comparison. And the whole file was a skip on a GPU-less box, so a new CPU-runnable case pins the op contract the split depends on -- three outputs of widths (I, conv_dim, num_heads) taking the head, middle and tail of an in_proj row. That case states plainly that it does not pin the production call site, which only the device numeric case covers. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
left open The spec's §3 measured Thor (sm_110) as having HALF an fp8 arm: the GEMM present through the unconditionally registered `kMatmulFp8CublasLt`, the activation quant trapped in a CUTLASS-gated translation unit that no sm_110 build compiles. #991 moved the registration out. Whether that actually reaches this model's arm was never run, and the spec said the conclusion had inverted twice during scoping, so it is worth writing down that it now has an answer. It does. On a build whose feature table reads `ENABLED for [110]: 1 ; DISABLED cells: 7` -- only `marlin-nvfp4`, with `cutlass-fp8` and both `scaledmm-c3x` cells off -- the FP8 W8A8 mamba arm executes: 49 assertions where a GPU-less box reports 4, the fresh block matching the host reference at three widths, and the fp8 tower uploading exactly once. The section also records the carry gate's failure and its repair, because the failure is the more instructive half: the case banded a defect whose separation (0.205) was SMALLER than the noise the comparison had to accept (0.164 on a fresh leg), which is §5.2's lesson arriving from the direction the spec did not anticipate. And it states which host can answer the occupancy question at all -- the 6.31% baseline is a GB10 number, so a Thor busy fraction would measure different silicon and answer something else. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…side a Thor number The A3 gate ran on Thor and printed `GPU busy in 358 of 2336 samples = 15.33% busy (baseline 6.31%)`. The denominator is right and the sample count is there, but the parenthesis is wrong: 6.31% was measured on GB10, and printing it beside an sm_110 number invites a comparison across two different pieces of silicon, two different clocks and two different memory systems. It answers a different question than the one it appears to answer. That is the failure this repository already knows by name -- a number quoted often enough comes to be treated as measured -- and a gate script is exactly where it starts, because the script's output is what gets pasted into a report. The baseline is now quoted only when `ARCH` is `121a`. On any other arch the line says so and names the comparison that IS valid there: the same-binary `VT_NEMOTRON_H_DEVICE_MAMBA` ON/OFF A/B on that same box. All three arms are self-tested, including the empty-sample arm, which reports UNMEASURED rather than 0% -- a busy fraction over zero samples formats identically to a genuinely idle GPU. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…line it cannot be read against The A3 gate ran on Thor (sm_110) with the FP8 W8A8 mamba arm on the device and returned 96/96, mode=decode, STRICT PASS, exit 0, with zero reference-tier lines. That last number is the one worth recording beside the verdict: the portable reference tier is numerically CORRECT, so a pass obtained on it is invisible in every other figure, and its absence is the only thing that separates the two. The busy fraction was 358 of 2336 samples, 15.33%. Both surfaces say explicitly that this is NOT comparable to the 6.31% baseline the unit is accepted against, because that baseline was taken on GB10 and this is different silicon. Recording the number without that sentence would have been the more useful-looking and less true entry, and the GB10 leg is still queued. BENCHMARKS also records what the run did NOT produce: no per-token time, because the Thor build predates the timing helper and its script line needed `bc`, which the container lacks. STATUS moves from one passing gate to two and names which arm the second one exercised. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…ered the wrong question (#1290) The Thor A/B ran both arms of the same binary against the same checkpoint and golden, differing only by VT_NEMOTRON_H_DEVICE_MAMBA. Two things came out of it and only one of them is a number this row may keep. THE ARM IS TOKEN-EXACT WHERE THE HOST ARM IS NOT. With the device arm on, the A3 gate read 96/96 mode=decode STRICT PASS, exit 0. With it off -- which is what main does on a GPU today -- it read 93/96 DIVERGENCE, exit 1. Zero reference-tier lines on both, so neither run took the portable tier. The mechanism is named rather than guessed: the golden comes from an oracle that computes these projections W8A8, and `DenseBf16` states outright at nemotron_h.cpp:419-422 that the host arm carries `input_scale` and does not apply it. That is #1290, filed while landing this row and fixed by it. It is n=1 per arm, and the row says so. THE BUSY FRACTION IS VOID, AND IT IS THIS UNIT'S ACCEPTANCE TEST. It read 15.33% on and 14.73% off, and neither figure means what it appears to: the sampler started with the process, so the multi-minute GPU-IDLE 20.1 GiB engine load sat inside the same window as the decode and diluted both arms toward each other. That is the same defect as summing prefill and decode into one profile, and a fraction over the wrong window is worse than no fraction, because it still formats like a measurement. No occupancy claim is made from that run in either direction. `run_gate` now starts the driver first, waits for `engine loaded in Ns`, and samples only after it; when that line never appears it reports the fraction as NOT REPORTED rather than sampling a window whose meaning is unknown. The per-token helper takes the decode window directly and subtracts nothing, since the load is already outside its brackets, and it refuses on a non-positive window -- the brackets failing to span a decode would otherwise print a negative or infinite rate that still looks like a measurement. What survives from the run without an instrument caveat is the wall-time ratio from the sample counts, one sampler at one nominal interval: 2.44x shorter with the arm on. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
`origin/main` moved while this row's Thor A/B was running, so the branch is merged rather than rebased and the gate is re-run over the merged tree. AGENTS.md R4: merge an immutable SHA and re-run the full gate after merging rather than reading the diff. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…a counter-measurement The issue as filed said the host mamba arm is what main does "on a GPU today" and therefore that main is token-wrong on a GPU. That is established for sm_110 ONLY. On GB10 (sm_121a) the SAME host arm reads 96/96 STRICT PASS -- the A3 run that closed #1157, /usr/local/nas_share/rc/nh1157/gate_fixed.out, same golden and same oracle revision. Left as written, the issue would get dismissed the first time somebody checked on GB10 and got a pass, and a silicon-specific defect would go with it. So the row, the spec and the benchmark entry now all say what was measured WHERE: host arm 93/96 on sm_110 with fa2 and cutlass-fp8 both DISABLED, host arm 96/96 on sm_121a with both ENABLED, n=1 per arm. What is established is narrower and still worth having: on ONE box, holding the entire rest of the tower constant, flipping only the mamba arm moves 93/96 to 96/96. The `DenseBf16` reading stays the leading mechanism -- the golden's oracle is W8A8 and the host arm is W8A16 because `input_scale` is carried and not applied (nemotron_h.cpp:419-422) -- but it now carries the obligation to explain why GB10 is clean, and the honest reading is that the perturbation is marginal rather than grossly wrong. One proposed candidate is excluded rather than chased. "The two arms resolve different fp8 GEMMs" cannot be the differentiator, because that configuration runs NO fp8 GEMM on either box: `DenseFor` dequantizes and hands the result to vt::MatmulBT on the CPU queue (nemotron_h_device.cpp:2027). The device-side difference that IS checkable is attention -- CudaArchFeatures.cmake:349 provides fa2 for 12.1a and not for 11.0, so GB10 runs FlashAttention-2 over the 6 GQA layers where Thor runs the portable fallback. Next step named rather than the question closed: the oracle's top-2 margin at the three moved tokens, because this family has already produced a "divergence" that turned out to be a bit-exact near-tie; then a NemotronHTrace layer bisect; then a repeat to lift n=1. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…of its four behaviours ARE refusals The helper exists because `examples/nemotron_h_gen` reports neither a rate nor a duration, so the per-output-token number this row is measured on has to be derived. A derived number that prints 0, or a negative, or a rate over an unknown denominator is indistinguishable from a measurement once it reaches a report, so most of the helper is guards -- and nothing in the tree pinned them. Six cases now do: the rate is printed with the window, the excluded load and the token count beside it; the load is excluded rather than subtracted a second time; a non-positive window refuses; a missing TOKEN MATCH line refuses; zero compared tokens refuses; and the vLLM denominator is the pinned 0.014369 s rather than a constant that could drift and silently restate the gap. Armed, not merely present. Deleting the non-positive-window guard reds the suite; making the missing-token path print `per output token 0.000000 s` reds it. The tree was restored byte-for-byte between mutations and after, sha256 be32e5f5c786ccb283c88b88e256f9c0e5d79af87eeef5615a3c10a615d3ce8c both times, and the suite is green at that sha. The whole `run_gate` window change was also driven end to end against a fake driver and a fake nvidia-smi before it is allowed near the scarce GB10 lease: a 3 s load plus a 4 s decode yields 39 samples over a 3.994 s window rather than ~70 over ~7 s, so the load really is outside the brackets; a driver that dies during load reports `NO busy fraction is sampled` and still surfaces its exit code 3; and a 93/96 divergence surfaces exit 1 with `matched 93` beside the rate. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…se doctest stringifies char* as bool
The Thor re-run confirmed the repaired carry gate -- 5 cases, 63 assertions, 0
failed -- and printed the numbers that justify it: a noise floor of 0.2465 at
T=1, a carried conv window agreeing to 0.1746 against a band of 0.5, and a
carried SSM state agreeing to 0.0614 against the same band, at 2.9x and 8.1x
margins.
It also proved the diagnosis that drove the repair. A dropped carry separates the
second leg's output by 0.2045 while the noise the comparison must accept is
0.2465, so the defect really is smaller than the noise, `separation > 2 *
noise_floor` is false, and the case correctly asserts nothing there.
Correctly, but not legibly. The line that reports WHY no assertion was made
printed `1`. doctest stringifies a `const char*` as a bool, and the message
streamed a `char*` ternary, so it printed the truthiness of a non-null pointer
whichever branch was live -- an unconditional `1` that says nothing about the
condition it names.
That line exists to make "no assertion was made here" a STATED result rather than
a silent hole, so a version of it that cannot say what it means is the same class
of defect as the band it reports on: an instrument that formats like information.
It now builds a std::string. Reproduced against doctest 2.5.2 in both forms
first: `MESSAGE("x " << (false ? "yes" : "prose"))` prints `x 1`, and the
std::string form prints `x prose`.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
…ide a Thor decode The decode-window sampler works. With the arm on, Thor read 42.55% busy over 564 DECODE samples where the old conflated window read 15.33%, and the reason is now measured rather than argued: the engine load is 500.9 s and the decode is 75.4 s, so the old window was 87% load. The same run also printed `per output token 0.785606 s (vLLM 0.014369 s; ratio 54.7x)`. That ratio is wrong in the way the busy fraction was wrong an hour ago: 0.014369 s is a GB10 figure and this decode happened on Thor, so the quotient compares two different pieces of silicon. Fixing the busy-fraction surface and leaving its twin is how a bad comparison survives a correction, and 54.7x is exactly the kind of number that gets quoted once it exists. The rate still prints, because it is a real measurement of that box. Only the comparison is withheld, and it is withheld BY NAME rather than silently dropped, so the reader knows a ratio was declined rather than forgotten. The gate script now passes ARCH through. Two cases pin both arms, and the guard is armed: quoting the ratio unconditionally reds the suite. Tree restored byte-for-byte after the mutation, sha256 3d94497009c384bcb36709a48b3e1c5da203fea23db020ecb097b79af035cae1, green at that sha. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…n Thor, and the divergence reproduced The acceptance test of this unit was never a ratio: the GPU busy fraction had to rise, measured with its denominator. On the corrected instrument it does. Second Thor lease, fresh build and clone, same binary, same checkpoint, same golden, differing only by VT_NEMOTRON_H_DEVICE_MAMBA: arm ON 96/96 STRICT PASS rc=0 240/564 = 42.55% busy 0.785606 s/token arm OFF 93/96 DIVERGENCE rc=1 700/3808 = 18.38% busy 5.633442 s/token +24.17 points, a 2.31x rise in occupancy, and 7.17x less time per decode token. Peak host 44070 MiB. Zero reference-tier lines on either arm, so neither ran on the portable tier. READ ON THE BOX IT WAS TAKEN ON. These are sm_110 figures, and BOTH references this row is normally quoted against -- the 6.31% baseline and the 0.014369 s per token -- are GB10's. Neither supports a ratio against these numbers, the instrument withholds both by name off 121a, and the stale `ratio 54.7x` and `ratio 392.1x` strings in that run's log predate the per-token arch gate and must not be quoted. The per-token VALUES are sound; the ratios beside them are not. The divergence also reproduced, which lifts the n=1 caveat for Thor: two independent leases with separate builds agree exactly, 96/96 on and 93/96 off both times. GB10 stays n=1 in the other direction, and no GB10 run of the DEVICE arm exists at all yet -- so the hypothesis is SUPPORTED on sm_110 and UNMEASURED on sm_121a, and the owed list now says so rather than implying one covers the other. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
`origin/main` moved again while the second Thor lease was running. Merged rather than rebased, and the gate is re-run over the merged tree: AGENTS.md R4 says to merge an immutable SHA and re-run the full gate rather than reading the diff. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
`main` advanced to f06b9e9 carrying #1297, which routes `test_dspark_block_size_guard.cpp` through `tests/support/process_id.h`. This branch's `build-newest-gcc` failure was that inherited red, so it is merged forward to pick the fix up rather than re-run against a stale base. `docs/BENCHMARKS.md` was a GENUINE content conflict, not a driver artifact: this branch inserts the A2-Q1 row and main inserted the `ENG-RECORD-ANCHOR-RATCHET` row at the same point. They are different keys, so BOTH are kept -- main's first, then this row's. Neither was dropped, and no ratchet was raised to fit them. The union auto-merge of `.agents/issue-index.md` was DISCARDED: main's file taken wholesale, only rows whose KEY main lacks re-appended, main's file asserted a strict PREFIX. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…gates This branch is based on PR #1289 (`row/A2-Q1-fp8-mamba`), which is where `NemotronHMamba2MixerDevice` lives, so it did not start from `main`. The preflight's trailer and style gates SKIP when `origin/main` is not an ancestor of HEAD, and a skipped gate has reported nothing about this tree. `docs/BENCHMARKS.md` was a GENUINE content conflict, not a driver artifact: main updated the `ENG-RECORD-ANCHOR-RATCHET` row while this line carries the A2-Q1 row inserted at the same point. BOTH are kept, main's version of its own row first, then this line's. Neither was dropped and no ratchet was raised. `.agents/issue-index.md` auto-merged and was verified rather than trusted: main's file is a strict PREFIX of the result, with only rows whose KEY main lacks appended after it. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
vLLM branches its Mamba2 mixer on `has_decode` (`mamba_mixer2.py:981`) and runs `causal_conv1d_update` and `selective_state_update` on the decode rows. Both take state indices and update the cache in place at the slot, so upstream's decode half performs no gather and no scatter. A2-Q1 ran the chunked prefill pair over both halves. `vt::Mamba2StateUpdate` is the port of `selective_state_update`. It has been registered on CUDA and CPU and gated since mamba2-ssd.md W2 landed, and it has zero callers under `src/vllm/`, which is a live "Nothing lands dead" case. The spec also records the risk that made this unit more than a swap: the equivalence the swap rests on was gated only at `heads_per_group = 2`, and NemotronH runs 8. No product code is written by this commit. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…1311) `vt::Mamba2StateUpdate` is the port of `selective_state_update`. It has been registered on CUDA and CPU and gated since mamba2-ssd.md W2 landed, and until this commit nothing under `src/vllm/` called it. The device mamba arm ran the chunked prefill pair over the decode rows too, wrapped in a gather and a scatter that upstream's decode half does not perform. `NemotronHPagedForward` now splits the recurrent half the way `mamba_mixer2.py:754-767` does. Decode rows take `vt::CausalConv1dUpdate` and `vt::Mamba2StateUpdate` at their cache slots, in place, with no gather and no scatter. Prefill rows keep the chunk scan, and their gather is narrowed to those rows. The selection is the metadata's `num_decodes`, never the token count: a one-token prefill of a fresh request is also T == 1 and must keep the chunk scan. The equivalence this swap rests on was not gated where NemotronH runs. The only decode-vs-prefill case ran heads_per_group = 2; NemotronH runs 8. Two new cases gate it at the driver group count on CPU and CUDA, at the multi-chunk shape and at the production T=1 chunk=128 shape, and they report the comparison's scale so the inherited 5e-3 tolerance cannot bound nothing. Mutation A2D1-M1, clamping the group index to min(h/hpg, 1), passes the pre-existing case 8/8 assertions RC=0 and fails the new one 4 of 58 assertions RC=1. The two arms compute the same recurrence and so produce the same tokens, which makes a token gate blind to which one ran. `NemotronHMambaArmCounts` counts launches at the vt:: call sites, and a CPU case drives it through `ModelRegistry::Forward` and asserts non-zero counts against the fixture's geometry, so a recorder that stopped recording reds before a GPU window is spent reading its zero as a result. `VT_NEMOTRON_H_MAMBA_DECODE_STEP=0` opts out, so the two arms can be A/B'd in one binary. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
The decode and prefill arms compute the same recurrence, so they emit the same tokens and no token gate can say which one ran. `NemotronHPagedForward` now prints the per-step counter delta under the existing `VT_NEMOTRON_H_DIAG` guard, and `scripts/nemotron-h-a2d1-gpu-gate.sh` runs the A3 gate on both legs of one binary and greps those lines. The script is a sibling of the A2-Q1 gate rather than an edit of it: that one owns the device-vs-host A/B and is live on another branch. This one holds the device mamba arm ON in both legs so the only difference is which kernels the decode rows take. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
`main` moved again while this row was in flight, so it is merged forward rather than gated against a stale base. The union auto-merge of `.agents/issue-index.md` was DISCARDED, not accepted. The automatic result INTERLEAVED main's two new rows with this line's, so main's file was no longer a prefix of it. Repaired the way AGENTS.md Records asks: main's file taken wholesale, then only the rows whose KEY main lacks (#1290 and #1311) re-appended verbatim at the end, and main's file asserted a strict PREFIX afterwards. `check-issue-index-append-only` agrees. The merged tree was then BUILT before this merge was pushed: a clean merge is not a compiling merge. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…#1311) The decode arm reads and rolls the conv page in place at its slot and has no use for the per-call window the non-paged arm allocates. Leaving the DBuf at function scope took a pool block per mamba layer per token on exactly the path this row exists to make cheap. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
… it (#1311) The row gains an explicitly PENDING measurement, which BENCHMARKS owes a line for, and the forensics live in benchmark-record.md where the keyed-table budget sends them. The launch and allocation counts are re-derived here from the kernel rather than carried over from the issue, and where the re-derivation disagrees -- the gather/scatter churn is about 283 MiB/token by this count, not the 414 the issue estimated -- the disagreement is stated rather than reconciled to the larger figure. Neither is measured. docs/STATUS.md moves in the SAME commit, because `check-doc-checkpoint` reads per commit: a change writing benchmark-record.md without moving STATUS reds on the grounds that a measurement is a claim about the project and STATUS is how the claim reaches a reader. The measurement here is explicitly PENDING, which is still a claim, so the line says so and claims no number. The STATUS cell is COMPRESSED rather than extended. Appending the clause took it to 408 chars against a 220 limit and moved the oversized-cell ratchet from 44 to 45, which is the page growing. The superseded A2-Q1 narrative is folded into the binding result instead. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…oring it (#1311) The flag has no meaning there -- a decode continues a sequence by definition, which is why upstream leaves has_initial_state None on a decode-only step. The arm was silently ignoring it, which is how the two cases the parameter exists to distinguish become indistinguishable again. A fresh request is a prefill and keeps the chunk scan. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…1311) Two defects in this row's own instrument, both found before the GPU window was spent and both of the shape where a broken instrument reads as a clean result. The counter line was gated on `VT_NEMOTRON_H_DIAG`, which downloads the carry and the residual per layer per step. A timed A/B leg under that flag measures the diagnostic, not the arm. The line now has its own `VT_NEMOTRON_H_ARM_TRACE` -- one fprintf of six resident counters per step -- so the reachability evidence and the timing come from one run instead of two that might differ. The gate script selected decode steps with `awk '$4=="nd=1"'`, but nd is field five. The test matched nothing, printed nothing, and would have read exactly like "no decode steps ran". It now matches on the field TEXT and reports the line counts, and says so explicitly when the decode count is zero. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
) The driver-group equivalence is the one piece of evidence that needs real CUDA and does not need the 20.1 GiB checkpoint. OPS_ONLY=1 builds and runs that suite and stops, so it does not queue behind a box that can host the whole gate, and it says plainly that nothing else was measured. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
`main` moved a third time while this row was in flight. Merged forward so the trailer, style and doc-checkpoint gates run against this tree rather than skipping on a stale base. `.agents/issue-index.md` auto-merged this time WITHOUT interleaving: main's file is already a strict prefix of the result and only #1290 and #1311 follow it, so the auto-merge was verified and kept rather than discarded. `check-issue-index-append-only` agrees. No other file conflicted. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
`main` moved a fourth time. Merged forward so the gates run against this tree instead of skipping on a stale base. `.agents/benchmark-record.md` conflicted, and it is an APPEND-ONLY log, so BOTH appended sections are kept: main's `BENCH-QWEN38-27B-BF16` re-measure first, then this row's `A2-D1` entry. Neither was dropped and neither was rewritten. Picking a winner on an append-only log is how a landed measurement disappears. `docs/STATUS.md` auto-merged: main edited a different row of the same table, so this row's A2-D1 line survives untouched. `.agents/issue-index.md` auto-merged with main's file a strict PREFIX of the result, verified rather than assumed. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
The fifth merge forward. `main` is landing several changes an hour today, and a row branch that is behind it makes the trailer, style and doc-checkpoint gates SKIP rather than fail, which reports nothing about this tree. Only `.agents/issue-index.md` touched, and the union auto-merge was DISCARDED rather than trusted: it INTERLEAVED main's new #1353 row with this line's, so main's file was no longer a prefix of the result. Repaired the way AGENTS.md Records asks -- main's file taken wholesale, then only the rows whose KEY main lacks (#1290 and #1311) re-appended verbatim -- and the prefix property asserted afterwards. This is the second time today the auto-merge on this file looked clean and was not, which is why the property is checked and not assumed. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
localai-bot
force-pushed
the
row/A2-D1-mamba-decode-step
branch
from
August 19, 2026 13:20
68a0ff3 to
1fd9b4a
Compare
…es a head size Found while landing #1311, and it is NOT this row's. `369ea7fd4` (#1332, #1344) landed a faithful mirror of `flash_attn.py:170-178` requiring `head_size % 8 == 0`. What it exposed is that the CPU selector no longer falls back when FA declines: `test_nemotron_h_paged_forward` throws 11 of 13 cases with "No valid attention backend for device type 0", while `cpu.cpp:45` offers two candidates and only FLASH_ATTN is named in the refusal. Attributed by counter-measurement rather than by reading the diff: the same suite on the same row at 68a0ff3, which lacks that commit, is 13 of 13 green with 3272 assertions on thor sm_110, and that row changes no attention or platform file at all. FILED not fixed in flow. The repair is in the attention-backend selector, which is BACKEND-ATTN-REGISTRY's surface and affects every CPU consumer rather than one model's suite; raising this fixture's head_dim to a multiple of eight would green the suite and leave the defect. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…e box (#1311) The comment said "this box has no FP8 GEMM". Thor has one; the test's queue does not, because `Q()` is a CPU queue and `kMatmulFp8CublasLt` is registered only for CUDA, and the paged forward selects the device mamba arm by asking the op table for the queue's device. The case therefore reads the same on a GPU box as on a CPU one, which is the property that makes it a floor. Verified: the suite is 13 of 13 with 3272 assertions on both. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…p arm (#1311) `thor:gpu0` sm_110 in an rc lease, real 20.1 GiB checkpoint, device mamba arm on, the single-step decode arm on its default: RC[a3 on]=0 TOKEN MATCH: 96/96 over 3 prompt(s) (full rows=3, short rows=0, mode=decode) STRICT PASS reference-tier lines in on: 0 That last line is not decoration: the portable reference tier is numerically correct, so a pass obtained on it is invisible in the tokens. Every decode step of that run, read off the vt:: call sites rather than off the branch condition: ARM step T=1 nd=1 np=0 state_update_rows=23 chunk_scan_calls=0 conv_update_rows=23 conv_fwd_calls=0 gathers=0 scatters=0 ARM lines total 96, decode 93, prefill 3 23 is the mamba layer count, so that is one state-update row per mamba layer and zero chunk scans, zero gathers and zero scatters on a decode step. vt::Mamba2StateUpdate went from zero callers under src/vllm/ to 23 launches per decoded token through a production entry point. The decode window is 74.511 s over 93 steps at 0.776159 s per output token, with the 654.7 s engine load OUTSIDE it. NO vLLM ratio is quoted for arch 110: the 0.014369 s reference is GB10's, and a ratio against it would compare two pieces of silicon. The sm_121a leg and the same-binary A/B delta stay owed. STATUS moves in this commit because check-doc-checkpoint reads per commit, and it moves in the STATUS column rather than the notes column: appending to the notes took that cell past 220 chars and the page's oversized-cell ratchet from 44 to 45. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
The sixth merge forward; `main` is landing several changes an hour today and GitHub had marked this pull request CONFLICTING. `.agents/benchmark-record.md` conflicted again, and it is an APPEND-ONLY log, so BOTH appended sections are kept: main's `LTX25-TEXT-LINEAR-MEM` entry first, then this row's `A2-D1` entry. Picking a winner on an append-only log is how a landed measurement disappears. `.agents/issue-index.md`: the union auto-merge was discarded and rebuilt as main's file plus only the rows whose KEY main lacks (#1290, #1311, #1371), with the prefix property asserted afterwards. That is the third time today the auto-merge on this file looked clean and interleaved. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…s what that closes (#1311) Both legs of one binary on thor:gpu0 sm_110, device mamba arm on, only the decode arm differing. ON RC=0 96/96 mode=decode STRICT PASS 0.776159 s/token busy 244/555 OFF RC=0 96/96 mode=decode STRICT PASS 0.773156 s/token busy 230/562 Per output token moved +0.388%, in the SLOWER direction. #1311 pre-registered "Refuted if per-token time moves less than 3%", so the speed hypothesis is REFUTED on that box and this change records it rather than reporting the two counter columns as though they were a result. A criterion that only ever confirms is not one. What the refutation does not close. The counters are not a prediction, they are what the run launched: 92 of 115 SSD kernel launches, all 230 driver alloc/frees, all 46 memsets, 104.9 MiB of per-token scratch and all 92 gather/scatter launches are gone, and the time did not move. So the c1 decode step on sm_110 is not bound by launch count, driver allocation or state-copy traffic. That is a finding about where the time is NOT. Three regimes were never entered and none is refuted: GB10 sm_121a, whose 6.31% and 0.014369 s references are not comparable to Thor and against which no ratio is quoted; concurrency above one, which is where qwen3_5.cpp:4730-4746 says the gather/scatter tax shows and which G-SAFE pins out; and n greater than one, since the two legs saw engine loads of 654.7 s and 778.2 s, an 18.8% spread that 0.388% cannot be separated from. No ceiling is declared. The next traceable step is an nsys trace of the decode window alone on both legs: the counters say what the step stopped launching, only a trace says what the 0.776 s is spent on. The row still lands. Correctness is the acceptance condition and it is met, the "Nothing lands dead" violation is closed by measurement, and the arm is what vLLM runs. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
The seventh merge forward; `main` has landed roughly one change an hour through this row's whole flight, and a branch behind it makes the trailer, style and doc-checkpoint gates SKIP rather than fail, which reports nothing about the tree. No file conflicted. `.agents/issue-index.md` auto-merged and was rebuilt anyway as main's file plus only the rows whose KEY main lacks (#1290, #1311, #1371), with the prefix property asserted afterwards -- the union auto-merge on this file has silently interleaved three times today, so it is verified rather than trusted whether or not git reports a conflict. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
localai-bot
changed the title
…claimed (#1311) It was written before any lease was obtained and outlived its own truth: the A3 gate is met on both legs and the A/B has a number, and the header still told a reader the measurement was owed. The header now carries the verdict -- correctness accepted, speed refuted on the box measured -- and names what is still owed. The superseded sentence says it was superseded rather than being silently dropped, because the order the evidence arrived in is part of the record. BENCHMARKS and STATUS move in the same commit, which check-doc-checkpoint reads per commit, and they gain the caveat that was only in the record: the two legs saw engine loads 18.8% apart, so a 0.388% move is not separable from noise. A reader of the headline number needs that beside it, not one file away. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…in-side red
The eighth merge forward, and this one is a REPAIR rather than hygiene. At
`601b576c6`, the commit this branch previously carried, `check-gate-commands`'s
ratchet suite was RED with four failures -- `test_dropping_{cudagraph_dedup,
mtp_depth,residency_config,serve_recipe_args}_from_the_pin_breaks_it` -- each
reporting `ENG-CUDAGRAPH-BREAK` leaving the gated population alongside the row
the case drops.
Attributed by counter-measurement rather than by reading the diff. A detached
worktree at `601b576c6` runs the suite RED at rc=1; the same suite at
`c9724b5ee` runs `Ran 46 tests` `OK` at rc=0. So it is a transient main-side
red, already repaired upstream, and merging forward is the whole fix. This row
touches no gate-command surface.
`.agents/benchmark-record.md` conflicted again and both appended sections are
kept, main's first. `.agents/issue-index.md` was rebuilt as main's file plus
only #1290, #1311 and #1371, with the prefix property asserted.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]
Collaborator
Author
⛔ DO NOT MERGE — GB10 sm_121a reads
|
…ED on attributing it (#1311) thor:gpu0 sm_110 passes 96/96 STRICT PASS on both legs. dgx:gpu0 sm_121a, same code, same checkpoint, reads: RC[a3 on]=1 TOKEN MATCH: 95/96 over 3 prompt(s) (full rows=3, short rows=0, mode=decode) DIVERGENCE The arm ran as designed there -- state_update_rows=23, chunk_scan_calls=0, gathers=0, scatters=0 on all 93 decode steps, reference-tier lines 0, and cutlass-fp8 ENABLED for [121a] -- so this is neither a routing failure nor a void run. One token in 96 differs. Whether this change CAUSES it is NOT established, and this commit does not guess. The chunk-scan OFF leg on the same binary, box and checkpoint is the attribution and is running: OFF at 96/96 means the change causes it and the row does not land as written; OFF at 95/96 or worse means it is pre-existing on sm_121a and owes its own issue. Prior GB10 trouble on this row -- STATUS's "GB10 read 4/24; cause and fix #1157", and A2-Q1's 93/96 DIVERGENCE arm under #1290 -- makes the second branch plausible and does not establish it. A single divergent token is exactly what a wrong recurrent carry and a benign bf16 near-tie both look like. So the records stop claiming an unqualified pass. BENCHMARKS, STATUS and the spec now say sm_110 accepted, sm_121a in question, and the spec gains a stop condition saying plainly that a pass on one gated host does not substitute for a failure on another. Also recorded and not attributed: GB10 runs this arm at 1.513958 s per output token against Thor's 0.776159, at 11.06% decode busy against 43.96%. That is not this row's claim and is written down so it is not read as one. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
The ninth merge forward, so the trailer, style and doc-checkpoint gates run against this tree instead of skipping on a stale base. No file conflicted. `.agents/issue-index.md` was rebuilt as main's file plus only the rows whose KEY main lacks, with the prefix property asserted, which is this branch's standing treatment of that file after three silent interleaves. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
GB10 returned `95/96 DIVERGENCE` and the log carried no way to tell a wrong recurrent carry from a benign bf16 near-tie. The driver had already printed the `got:` and `exp:` ids for the mismatching row (nemotron_h_gen/main.cpp:385); this script's verdict grep matched only the summary lines and threw them away, so triaging the failure would have cost a second lease on a contended box. A gate that reports a failure it cannot triage has not finished reporting. On a DIVERGENCE it now prints the per-prompt row lines and the got/exp ids beside the verdict. Found by needing it, which is the wrong time to find it. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…is the host's (#1311, #1388) The chunk-scan leg on dgx:gpu0 returns the same verdict as the single-step leg: RC[a3 on]=1 TOKEN MATCH: 95/96 full rows=3 short rows=0 mode=decode RC[a3 off]=1 TOKEN MATCH: 95/96 full rows=3 short rows=0 mode=decode Same binary, same box, same checkpoint, and the counters confirm the legs ran different kernels -- 23 state-update rows with no chunk scan and no gather/scatter against 23 chunk scans with 46 gathers and 46 scatters. The arm that predates this change diverges identically to the arm that replaces it, so this change does not cause it. Filed as #1388: it is the sm_121a re-run STATUS has carried as pending, it fails on both arms, and it is arch- or host-specific because sm_110 passes 96/96 on the same code and checkpoint. That gives the acceptance statement per host rather than in aggregate. sm_110 reads 96/96 on both arms and sm_121a reads 95/96 on both arms, so each host returns the SAME verdict on both arms and the change is TOKEN-NEUTRAL. The per-token move is +0.388% on sm_110 and -1.991% on sm_121a, both under #1311's 3% bar and pointing in OPPOSITE directions, which is itself evidence that neither is signal. The stop condition that fired is recorded as DISCHARGED BY THE A/B, not by the argument that the other host passed. Reading a pass across silicon is the inference this row refuses everywhere else. Still not established, and said so: whether both legs lose the SAME token. The counts and row shape match, but the got/exp ids were discarded by the script's own verdict grep, which is now repaired. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…1311 It is the sm_121a re-run docs/STATUS.md has carried as pending since A2-P, and it fails on BOTH recurrent decode arms, so it belongs to the host rather than to whichever arm is selected. Attributed by the same-binary A/B rather than by reading the diff: the arm that predates #1311 diverges identically to the arm that replaces it, and thor sm_110 passes 96/96 on the same code and checkpoint. The row also records what the run could NOT answer -- whether both legs lose the same token -- and why, since the gate script's verdict grep discarded the driver's got/exp ids. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
Collaborator
Author
✅ UNBLOCKED — the OFF leg attributes the divergence to the HOST, not this changeThe GB10 chunk-scan leg has returned. It reads the same verdict as the single-step leg: Same binary, same box, same checkpoint, and the counters confirm the two legs really did run different kernels:
The arm that predates this PR diverges identically to the arm that replaces it, so this PR does not cause it. That was the branch I pre-registered before the leg ran, and it is the A/B that discharges it — not the argument that Thor passed. Reading a pass across silicon is the inference this row refuses everywhere else. Filed as #1388: it is the "sm_121a re-run pending" The acceptance statement, per host rather than in aggregate
Each host returns the same verdict on both arms ⇒ this change is token-neutral. Both moves sit under #1311's own 3% bar ⇒ speed refuted on both, and they point in opposite directions, which is itself evidence that neither is signal. Still not established, and said soWhether both legs lose the same token. The counts and row shape match, which is what supports "neutral", but the driver's
|
localai-bot
changed the title
…ojections (#1388) The GB10 A/B varied the mamba KERNEL and concluded the 95/96 divergence was the host's. That conclusion is under-determined, and my own artefacts show why: the four kernel counters are non-zero in BOTH legs, and they are only reachable from the `mamba_on_device` branch, so both legs ran A2-Q1's FP8 W8A8 projections. The script hardcoded VT_NEMOTRON_H_DEVICE_MAMBA=1 in both. `main` carries no device mamba arm at all -- NemotronHMamba2MixerDevice, MambaIsFp8 and even the VT_NEMOTRON_H_DEVICE_MAMBA knob are absent there -- so a 95/96 on both legs is equally consistent with the divergence being the architecture's or being A2-Q1's projections, which neither leg turns off. Leg 3 routes the whole mamba block back to the host reference on the SAME binary and box, which is the only difference. Its own counters are the check that it took that path: four kernel counters at 0 with gathers and scatters non-zero, because the host branch still gathers and scatters but never enters the instrumented device mixer. A leg that cannot prove which path it took cannot discriminate anything. 96/96 on leg 3 means the divergence is A2-Q1's FP8 arm and not the architecture, which would re-scope #1388 and matters for #1289. 95/96 means the host owns it and the current framing stands. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
#1311) The GB10 A/B varied the mamba kernel and I recorded its cause as the host's. That is under-determined, and my own counters are what show it: the four kernel counters are non-zero in BOTH legs and are reachable only from the mamba_on_device branch, so both legs carried A2-Q1's FP8 W8A8 projections. The script pinned VT_NEMOTRON_H_DEVICE_MAMBA=1 in both. main carries no device mamba arm at all -- NemotronHMamba2MixerDevice, MambaIsFp8 and the VT_NEMOTRON_H_DEVICE_MAMBA knob are all absent, verified by git show on origin/main. So 95/96 on both legs fits "the host diverges" and "A2-Q1's FP8 projections diverge" equally, and this A/B separates neither. #1312's 96/96 on a main-based GB10 tree corroborates the second without settling it, being a different branch and binary. So the records stop asserting the conclusion. BENCHMARKS, the benchmark record and the spec now say the cause is undecided and name the queued discriminator: leg 3, VT_NEMOTRON_H_DEVICE_MAMBA=0 on the same binary and box, with its own counters checking it really took the host path. If it reads 96/96 the finding belongs to A2-Q1 (#1289), which carries the only real speed win measured this session, and that is a larger result than #1388 as first framed. Recording the arch reading as settled would have been the same error as reading the Thor pass across to GB10, which this row refuses everywhere else. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
localai-bot
force-pushed
the
row/A2-D1-mamba-decode-step
branch
from
August 19, 2026 18:00
62867bb to
3c169d9
Compare
Collaborator
Author
Correction, and this PR stays UNMERGED while it runsMy "the divergence is the host's, not this row's" needs qualifying. The token-neutrality of this PR still holds — that rests on the two legs returning the same verdict, which they do. What does not hold is my further claim that #1388 is arch-specific. Both legs carried A2-Q1's FP8 W8A8 projections. The four kernel counters are non-zero in each, and they are reachable only from the What is unaffected
What changed
If it reads Do not merge this PR until that reports. The DO-NOT-MERGE state stands. |
GitHub does not run the `merge=union` driver `.gitattributes:7` sets, so a branch touching `.agents/issue-index.md` re-conflicts whenever main appends a row. Resolved locally where the driver runs: the union auto-merge was DISCARDED, main's file taken wholesale, only rows whose KEY main lacks re-appended, and main's file asserted a strict PREFIX with zero duplicate keys. Any keyed-table collision kept BOTH sides, main's first, so no row is dropped. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
The tenth merge forward, so the trailer, style and doc-checkpoint gates run against this tree instead of skipping on a stale base. No file conflicted. `.agents/issue-index.md` was rebuilt as main's file plus only the rows whose KEY main lacks, with the prefix property asserted, which is this branch's standing treatment of that file. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…ices that arm at 6.64x (#1311, #1388, #810) The GB10 divergence had two candidate causes and the two-leg A/B could separate neither, because both legs sat on A2-Q1's device mamba arm. A third leg removes that arm on the same binary and box: on device mamba 1 95/96 DIVERGENCE 1.584694 s/tok 110.3x off device mamba 1 95/96 DIVERGENCE 1.554233 s/tok 108.2x hostmamba device mamba 0 96/96 STRICT PASS 10.318897 s/tok 718.1x So the cause is A2-Q1's arm, not the architecture. The earlier arch-specific reading was retracted before this ran; this is the experiment that tested it. Leg 3's own counters are what make it a discrimination rather than a correlation between a flag and an outcome: state_update_rows, chunk_scan_calls, conv_update_rows and conv_fwd_calls all zero while gathers and scatters stay at 46, which is the host path's signature and nothing else's. The divergence is one token and both device legs lose the SAME one -- prompt 2, position 32 of 32, 11286 against the oracle's 3468, positions 1-31 byte-identical. The repaired verdict grep captured it; the first GB10 run threw it away. Whether that is a defect or a bf16 near-tie is NOT settled here and a fresh implementer owns it. The second result is the larger one and is not this row's: A2-Q1's arm is worth 6.64x per output token on GB10, closing 718.1x to 108.2x versus vLLM with decode busy 7.86% to 10.18%. It agrees in direction and rough magnitude with #1289's independent Thor A/B. One run per leg on a contended box, engine load excluded; the 6.64x survives that and 108.2x remains an OPEN GAP, not parity. And the +-2% on/off delta is retired as noise by a sign flip: the first GB10 run put on 1.991% FASTER, this one puts it 1.960% SLOWER, same comparison, same box. A quantity that reverses direction between two runs is not a measurement of it. That strengthens #1311's refutation, which was argued from the 3% bar alone. This row's acceptance is unchanged: each host returns the same verdict on both KERNEL arms, so the swap is token-neutral and the sm_121a token belongs to the arm underneath it. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…his branch A plain push was REJECTED as non-fast-forward: the remote carried `dddae7f3f`, a merge of `origin/main` pushed by the submitter that resolved the `.agents/issue-index.md` re-conflict GitHub creates by not running the `merge=union` driver. That rejection is git doing its job, so this merges the remote in rather than forcing over it. No file conflicted. Verified rather than assumed afterwards: main's `issue-index.md` is still a strict PREFIX of the result, there are zero duplicate keys, and this row's A2-D1 records survived intact across all four surfaces. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
NemotronH ran the chunked prefill mamba kernels on its decode rows, and
vt::Mamba2StateUpdate— the port ofselective_state_update, registered onCUDA (
cuda_gdn.cu:6672) and CPU (cpu_ops.cpp:3408) and gated sincemamba2-ssd.mdW2 landed on 2026-08-13 — had zero callers undersrc/vllm/.That is a live
AGENTS.md"Nothing lands dead" case, and closing it is half ofwhy this unit exists.
vLLM branches its Mamba2 mixer on
has_decode(mamba_mixer2.py:981) and runscausal_conv1d_update:1012andselective_state_update:1087on thedecode rows. Both take state indices and update the cache in place at the slot,
so upstream's decode half performs neither a gather nor a scatter. A2-Q1 ran
vt::CausalConv1dFwd+vt::Mamba2ChunkScanwrapped inGatherNemotronHState/ScatterNemotronHStateover both halves. The treealready carried the warning:
qwen3_5.cpp:4730-4746says passing the stateindices to the op removes "the two host<->device copies per sequence per layer
that dominate the decode memcpy tax".
NemotronHPagedForwardnow splits the recurrent half the waymamba_mixer2.py:754-767does. The selection is the metadata'snum_decodes,never the token count: a one-token prefill of a fresh request is also
T == 1,carries no state in, and can have
prefill_has_initial_state == 0, whichneither single-step kernel can express. Prefill keeps the chunk scan, and its
gather is narrowed to the prefill rows.
The premise was not gated where this model runs
The swap rests on the state update and the chunk scan agreeing. The only
decode-vs-prefill case ran
H=4 G=2, i.e.heads_per_group = 2. NemotronH runsheads_per_group = 8.mamba2-ssd.md§8.3 declares an equivalence contract,but §8.3 is device-vs-host and says nothing about this.
Two new cases gate it at the driver group count, on CPU and CUDA, at a
multi-chunk shape and at the production
T=1 chunk=128shape, and they reportthe comparison's scale unconditionally so the inherited 5e-3 tolerance cannot
bound nothing.
Mutation A2D1-M1, clamping the group index to
min(h/hpg, 1):The mutation is invisible to the case that existed and caught by the case this
PR adds, on both shapes. Tree restored byte-for-byte afterwards (md5
753ba5c3d0869396c20f2205eb2617d7).Which arm ran is recorded, because tokens cannot say
The two arms compute the same recurrence, so they emit the same tokens and a
token gate is blind to which one produced them.
NemotronHMambaArmCountscounts launches at the
vt::call sites, andVT_NEMOTRON_H_DIAGprints theper-step delta. A CPU case drives the counters through
ModelRegistry::Forwardand asserts non-zero counts against the fixture's geometry, so a recorder that
stopped recording reds in seconds rather than on a GPU window where its zero
would read as a result.
VT_NEMOTRON_H_MAMBA_DECODE_STEP=0opts out, so both arms run in one binary.Gates
CPU, all bare exit codes:
Driver-group equivalence, measured in the run rather than asserted:
CUDA evidence, measured on a leased GPU
thor:gpu0(sm_110), inside anrclease,ARCH=110, tree68a0ff378--which
git diffs EMPTY against this PR's head over every file this row owns,so it is this code.
scripts/nemotron-h-a2d1-gpu-gate.shis the recipe.The CUDA driver-group equivalence -- both arms on device, through entirely
separate kernels:
orin:gpu0gave NO CUDA evidence and is reported as such. The suite read12 cases, 12 passed, SUCCESSthere, but at 2527 assertions -- exactly theCPU-only count -- and none of the
CUDA driver-grouplines appeared. The fiveCUDA cases took their skip path. A green with the CUDA cases muted is not a CUDA
result.
An inherited red on
main, filed as #1371Merging
mainin makestest_nemotron_h_paged_forwardthrow 11 of 13 cases ona CPU box:
369ea7fd4(#1332, #1344) landedhead_size % 8 == 0on the FA backend, afaithful mirror of
flash_attn.py:170-178; this fixture useskHeadDim = 6,and the CPU selector no longer falls back to
CPU_ATTN. It is not thisrow's: the same suite on the same row at
68a0ff378, which lacks thatcommit, is 13 of 13 green with 3272 assertions on Thor, and this row
git diffs empty oversrc/vllm/v1/attention/,include/vllm/v1/attention/andsrc/vllm/platforms/. Filed as#1371 rather than fixed in
flow, because the repair is in the selector and affects every CPU consumer.
Note the run prints
assertions: 18, 18 passed, 0 failedbeside11 failedcases: a thrown case runs no assertions, so a gate grepping only
assertions:reads it as clean.
The A3 e2e token gate, on BOTH gated hosts
Same-binary A/B on each box, device mamba arm ON in both legs, decode window
only,
reference-tier lines: 0everywhere:thor:gpu0sm_11096/96 mode=decode STRICT PASS96/96 mode=decode STRICT PASSdgx:gpu0sm_121a95/96 DIVERGENCE95/96 DIVERGENCEEach host returns the SAME verdict on both arms, so this change is
token-neutral. The sm_121a divergence is the host's, not this row's — the arm
that predates the change diverges identically to the arm that replaces it. Filed
as #1388; it is the "sm_121a
re-run pending"
STATUShas carried since A2-P and it fails on both arms.That attribution is the A/B, not the argument that Thor passed. Reading a
pass across silicon is the inference this row refuses everywhere else.
Both speed moves are under #1311's own 3% bar, so the speed hypothesis is
REFUTED on both hosts — and they point in opposite directions, which is itself
evidence that neither is signal.
Which kernels the decode steps launched
Read off the
vt::call sites, not off the branch condition, on every one ofthe 93 decode steps of each leg:
23 is the mamba layer count.
vt::Mamba2StateUpdatewent from zero callersunder
src/vllm/to 23 launches per decoded token through a production entrypoint, which closes the "Nothing lands dead" case by measurement. Prefill
still reports
np=1and keeps the chunk scan, which is the point.The counters also make the A/B meaningful: they prove the two legs really ran
different kernels, which is what lets the identical token verdicts attribute the
divergence to the host.
What the refutation does and does not close
The counters are not a prediction; they are what the run launched. The arm
removed 92 of 115 SSD kernel launches, all 230 driver alloc/frees, all 46
memsets, 104.9 MiB of per-token scratch and all 92 gather/scatter launches per
token — and the time did not move on either host. So the c1 decode step is
not bound by launch count, driver allocation or state-copy traffic. That is
a finding about where the time is not.
Never entered, so not refuted: concurrency > 1, where
qwen3_5.cpp:4730-4746says the tax shows and G-SAFE pinsnum_reqs <= 1; andn > 1 — one run per leg, with the sm_110 legs seeing engine loads 18.8%
apart and the two hosts' deltas having opposite signs.
No ceiling is declared. Next traceable step: an
nsystrace of the decodewindow alone. Also unexplained and recorded so it is not read as an A2-D1
result: GB10 runs this arm ~2x slower than Thor (1.514 vs 0.776 s/token, 11.06%
vs 43.96% busy).
Why this lands
AGENTS.md"Nothing lands dead" violation is closed by measurement.mamba_mixer2.py:981), which is repositorypolicy independent of a local speed delta.
n_groups=8equivalence the swap rests on is now gated and was notbefore, with a mutation the pre-existing case cannot see.
Still OWED (spec
## Owed)dgxlease: re-run the repaired gate and read thegot:/exp:ids, so a wrong carry and a benign bf16 near-tie are separated.This PR's script discarded them — a defect in my recipe, found by needing it
and fixed here.
nsysdecode-window trace.Based on #1289 (
row/A2-Q1-fp8-mamba), which addsNemotronHMamba2MixerDevice, the function this PR changes, and is mergedforward onto
main.Closes #1311
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]