spec(GDN-MOE-BF16-OUT): the GDN recurrence output and the z gate are f32 on every MoE checkpoint, and both references keep them bf16

.agents/issue-index.md

@@ -351,3 +351,6 @@ rather than merged. `scripts/check-agent-record.py` gates both.
 | [#1162](https://github.com/mudler/vllm.cpp/issues/1162) | `ENG-CUDAGRAPH-DEDUP` | We instantiate **one `cudaGraphExec` per padded decode bucket, per model**, and `grep -rn "cudaGraphExecUpdate" src include` returns nothing. `src/vt/cuda/cuda_backend.cu:222-232` instantiates a fresh exec per capture; `include/vllm/model_executor/models/decode_graph_sizes.h:32-41` yields 7 buckets at `max_num_seqs=32` and 11 at 64; eight drivers each build their own set. SGLang folds compatible captures onto one executable by hashing graph topology and calling `cudaGraphExecUpdate` on a signature hit (`cuda_graph_dedup_mixin.py:219-242`, logging "captured %d CUDA graphs, deduped to %d execs" at `:358`). Portable to us unchanged, because it is driver-level rather than PyTorch-level. It is a **memory and capture-time** change, NOT a throughput change — a deduped replay launches the same nodes — and it matters because on GB10 unified memory an OOM reboots the box, capture time is startup latency (a recorded gate axis), and bucket count is exactly what widening graph coverage would raise. Owed: the dedup registry behind the `vt` seam, a capture-count/exec-count log line, and a same-binary A/B proving a deduped replay is byte-identical rather than asserting it. Hazard already recorded: capture bakes host source addresses and a clean `compute-sanitizer` run is NOT evidence a capture path is safe (`specs/decode-graph-scratch-uaf-2026-07-18.md`). Spec [`sglang-breakable-cuda-graph.md`](specs/sglang-breakable-cuda-graph.md) `## Owed`. Analysis: [#1161](https://github.com/mudler/vllm.cpp/issues/1161) | perf |
 | [#1163](https://github.com/mudler/vllm.cpp/issues/1163) | `ENG-CUDAGRAPH-BREAK` | Two defects, stated separately. **(1) Capture has no break points:** `src/vllm/v1/worker/gpu/runner.cpp:1338-1341` routes only `pure_decode` batches to a graph, so prefill, mixed batches, and anything whose metadata is computed on the host stay eager for the WHOLE step. There is no way to express "capture this forward except for these three calls", so coverage is a cliff rather than a slope; [#1020](https://github.com/mudler/vllm.cpp/issues/1020) is one instance (a spec verify step whose actual draft depth differs from the configured `k` silently falls out to eager). **(2) Eight hand-rolled drivers:** `Qwen3_5DecodeGraph` (`qwen3_5.h:275`), `Qwen3_5DenseDecodeGraph` (`qwen3_5_dense.h:391`), `Qwen3MoeDecodeGraph` (`qwen3_moe.h:117`), `Qwen3DenseDecodeGraph` (`qwen3.h:243`), `DeepseekV2DecodeGraph` (`deepseek_v2.h:324`), `VoxtralDecodeGraph` (`voxtral.h:126`), plus graph code in `deepseek_v4.cpp` and `laguna.cpp` — each re-deriving capture, bucket padding, persistent-input threading and the pure-decode predicate. AGENTS.md names this shape: a parallel path written by hand instead of a shared seam, and every new model that wants a decode graph writes a ninth one. Mirror vLLM's `CUDAGraphMode.PIECEWISE` boundary (`splitting_ops`, i.e. the attention ops); take the CONSTRUCTION from SGLang BCG, because vLLM gets its split from `torch.compile` and we have no compiler. **Explicitly NOT a throughput row and must not be sold as one** — our prefill has 3.8% host idle and is >96% GPU-busy, and decode already banked its launch-overhead win; the value is coverage and one seam instead of eight. Any speed claim must first name and measure a path that is currently eager AND currently host-bound. Owed: the seam, break-point registration, a reachability mutation (delete the production call site, rerun the focused gate), and bit-exactness vs eager on every migrated model over MORE than one replay. Large and structural; spike first. Spec [`sglang-breakable-cuda-graph.md`](specs/sglang-breakable-cuda-graph.md) `## Owed`. Analysis: [#1161](https://github.com/mudler/vllm.cpp/issues/1161) | feature |
 | [#1164](https://github.com/mudler/vllm.cpp/issues/1164) | `ENG-CUDAGRAPH-DIFFUSION` | Our diffusion path captures **no CUDA graphs at all** — `grep` for capture across `src/vllm/model_executor/models/ltx2*.cpp` returns nothing — and a denoise loop is the ideal capture target (fixed shapes, tens of identical iterations). SGLang enabled BCG on exactly this shape AFTER our pin and measured LTX-2 two-stage H200 e2e 10.75 s → 6.90 s (`d4be483efb`), SANA 1024px e2e -26% (`6c7498113f`), SANA denoise 0.73 → 0.457 s (`56ef810cad`), and Z-Image BCG made bit-exact vs eager (`f5f0c3ee7a`) — cited as DATED UPSTREAM EVENTS, never as pinned evidence, and to be discounted hard because their win is mostly PyTorch host tax our C++ loop does not pay. **BLOCKED, and the blocker is ours:** the premise a denoise-loop graph needs is a denoise loop that RUNS ON THE DEVICE, and three open issues already measured that ours does not. [#1024](https://github.com/mudler/vllm.cpp/issues/1024): a `--device cuda` render stages 35.54 GiB onto the GPU and then does no compute on it — GPU utilization **exactly 0 in 321 of 347 samples**, all 26 non-zero samples inside the staging window, and from t=251 s onward over 17 minutes of 0 while the process holds exactly 1.00 core of 20. [#1007](https://github.com/mudler/vllm.cpp/issues/1007): the video VAE decode has no device arm. [#1087](https://github.com/mudler/vllm.cpp/issues/1087): after [#1041](https://github.com/mudler/vllm.cpp/issues/1041), **57-66% of render wall** is one contiguous single-threaded phase measuring 1731 s and 1732 s across two rungs whose voxel counts differ 2.75x, i.e. resolution-CONSTANT, and [#1010](https://github.com/mudler/vllm.cpp/issues/1010) is open because nothing timestamps a phase boundary so the phase is not even named. A graph collapses host launch dispatch; there is close to no device dispatch here to collapse and the dominant cost is a serial host phase a graph cannot touch, so **capturing now would measure nothing**. Unblock order: [#1010](https://github.com/mudler/vllm.cpp/issues/1010) phase log → [#1087](https://github.com/mudler/vllm.cpp/issues/1087) name and fix the serial phase → [#1024](https://github.com/mudler/vllm.cpp/issues/1024)/[#1007](https://github.com/mudler/vllm.cpp/issues/1007) device residency → THEN measure GPU-busy vs wall. That fourth step is the decision point and it is a MEASUREMENT, not an implementation: if the loop is device-resident and host-dispatch-bound, capture it; if GPU-bound, close it the way [#1161](https://github.com/mudler/vllm.cpp/issues/1161) closed prefill. Gate when it runs: bit-exact PIXELS vs eager over more than one replay, never an exit code (see [#1149](https://github.com/mudler/vllm.cpp/issues/1149)). Spec [`sglang-breakable-cuda-graph.md`](specs/sglang-breakable-cuda-graph.md) `## Owed` | perf |
+| [#1168](https://github.com/mudler/vllm.cpp/issues/1168) | `GDN-MOE-BF16-OUT` | `GdnOutDType(bool dense_model)` (`src/vllm/model_executor/models/qwen3_5.cpp:3190` @ `dd8a3b0e1`) resolves `bf16` for a dense checkpoint and `f32` for a MoE one, and all three call sites pass `cfg.num_experts == 0` (`:3749`, `:4155`, `:4386`). So on every MoE checkpoint the GDN recurrence output `dcore` `[T,Hv,Dv]` (`:3807`, `:4253`, `:4587`), the `z` output gate (`:3584`, `:3688-3690`) and the gated-RMSNorm weight that must match them (`:3817-3818`, `:4301-4302`, `:4733-4734`) all stay f32. Each of `dcore` and `z` is stored once and loaded once, so f32 doubles four passes over the two largest per-layer GDN activations. vLLM keeps both at the bf16 model dtype and does not branch on model shape: `core_attn_out = torch.zeros(..., dtype=hidden_states.dtype)` (`vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py:870-873` @ `5559679`), `z` is a split of the bf16 `in_proj_qkvz` output (`:843`, `:859-860`), and `RMSNormGated` is built with no dtype override (`:459-465`); the shared `packed_modules_mapping` on `Qwen3_5ForCausalLMBase` (`vllm/model_executor/models/qwen3_5.py:280-297`) covers the dense and MoE arms alike. SGLang agrees — `RMSNormGated(..., dtype=config.torch_dtype)` (`python/sglang/srt/models/qwen3_5.py:522-536` @ `f63458b5be`) — so this is a deviation from the PRIMARY oracle, not an SGLang-only difference. No gate could see it: AGENTS.md's "a token gate cannot detect a dtype that is too wide" describes this exactly, and `test_qwen36_paged_engine` is 315/315 BECAUSE f32 is the more precise deviation. The tree already derived the whole finding at `qwen3_5.cpp:3172-3189` and deferred it — "Keep every unmeasured 35B arm, including GGUF, on its prior f32 default; the explicit env override remains available for its later independently gated campaign" — and this row is that campaign. Coupled: `detail::ShouldUsePackedGdnDecode` (`:76-84`) carries an `e.dense_model` term from `f344decf4`'s day-one "real-model safety gates", which neither reference has (`vllm/envs.py:124` defaults `VLLM_ENABLE_FLA_PACKED_RECURRENT_DECODE` True with no shape term; `gdn_triton.py:43` keys on the platform alone); removing it BEFORE the dtype change is inert because `GdnPackedDecodeDTypesCompatible` (`:115-122`) already pins `core_out` to bf16, so it is removed WITH the dtype, not before. Affected arms are the MoE checkpoints only — `nvidia/Qwen3.6-35B-A3B-NVFP4` (gateable here) and `Qwen/Qwen3.8-2.4T-A95B` (not: ~4.8 TB bf16 against 128 GB unified memory); the dense `Qwen3.8-27B` is already bf16 and gains nothing. `VT_GDN_OUT_BF16` already exists as a same-binary A/B override. Spec [`gdn-moe-bf16-out.md`](specs/gdn-moe-bf16-out.md) | perf |
+| [#1169](https://github.com/mudler/vllm.cpp/issues/1169) | — | Packed GDN decode is unreachable on every MoE checkpoint for a second, independent reason: `detail::ShouldUsePackedGdnDecode` (`src/vllm/model_executor/models/qwen3_5.cpp:76-84` @ `dd8a3b0e1`) requires `e.has_packed_ba`, populated at `:4410` as `!w.in_proj_ba.Empty()`, and `in_proj_ba` is written at exactly ONE site in the tree — `src/vllm/model_executor/models/qwen3_5_dense_weights.cpp:436`, the DENSE loader. The MoE loader loads the shards split (`qwen3_5_weights.cpp:563-564`) and so does the GGUF loader (`qwen3_5_gguf_weights.cpp:1067-1070`); `qwen3_5.cpp:3220` states it in the tree ("the only path that populates `in_proj_ba`"). Consequence: removing the `dense_model` term and narrowing `GdnOutDType` (#1168) is NOT sufficient to reach packed decode on a MoE model, which corrects the premise row `GDN-MOE-BF16-OUT` started from. It matters because the 35B's geometry (`Hv=32`, `Hg=16`, `Dk=Dv=128`) has a vendored Triton AOT decode cubin for every supported architecture (`src/vt/cuda/triton_aot_vendored/*/gdn_decode_h32.*`) that is exactly the FLA kernel vLLM runs, and `.agents/kernel-matrix.md` records the contrast the vendoring exists to capture — Triton REG:205 / 0 spill against the hand CUDA port at REG:255 + STACK:48. Upstream has one physical `in_proj_ba` on every Qwen3.5/3.6 GDN layer, dense and MoE alike (`qwen_gdn_linear_attn.py:843`; `packed_modules_mapping` on the shared `Qwen3_5ForCausalLMBase`, `vllm/model_executor/models/qwen3_5.py:287-297` @ `5559679`), so the merged owner is the upstream topology and our split MoE load is the deviation. Same class of gap on `in_proj_qkvz`, which no MoE or GGUF loader builds either. Closing it also owes a repair to the stale `KERNEL-GDN-PACKED-DECODE` sentence in `.agents/kernel-matrix.md` claiming "the launcher guard rejects its `Hv=32` shape anyway": `TryTritonPackedDecode` accepts `Hv=32` and dispatches `gdn_decode_h32_default` (`src/vt/cuda/cuda_gdn.cu:5207`, `:5239`). Filed, not fixed — it is a loader change with its own resident-weight lifetime and byte-exactness argument, and the finding row is spec-only. Listed under `## Owed` in [`gdn-moe-bf16-out.md`](specs/gdn-moe-bf16-out.md) | gap |
+| [#1170](https://github.com/mudler/vllm.cpp/issues/1170) | — | All four GDN Triton AOT fast paths reject any geometry whose linear V-head count is not 48 or 32 — `TryTritonPackedDecode` (`src/vt/cuda/cuda_gdn.cu:5207` @ `dd8a3b0e1`), `TryTritonDeltaH` (`:5264`), `TryTritonChunkO` (`:5298`) and `TryTritonWU` (`:5361`), each reading `if (hv_n != 48 && hv_n != 32) return false;` on top of `dk == 128 && dv == 128 && hk_n == 16`. Those two are the only vendored specializations (`src/vt/cuda/triton_aot_vendored/*/gdn_{decode,deltah,chunko,wu}_h{48,32}.*`): 48 is the dense 27B and 32 is `Qwen3.6-35B-A3B`. `Qwen/Qwen3.8-2.4T-A95B` has 128 linear V-heads ([`qwen38-text-only.md`](specs/qwen38-text-only.md)) and clears every other term, so it is rejected on `hv_n` alone and runs the hand CUDA kernels on all four legs — the ones `.agents/kernel-matrix.md` measured by cuobjdump at REG:255 + STACK:48 (spilling) against the vLLM FLA cubin's REG:205 / 0 spill, which is the whole reason the vendored cubins exist and are default-on. `Qwen3.8-27B` is NOT affected: it is the `Qwen3.6-27B` geometry retrained, 48 V-heads, and hits every AOT arm. Neither reference restricts the head count — SGLang's `TritonGDNKernel` sets `supports_packed_decode` from the platform alone and takes `num_v_heads` as a runtime argument (`python/sglang/srt/layers/attention/linear/kernels/gdn_triton.py:43` @ `f63458b5be`), and `VLLM_ENABLE_FLA_PACKED_RECURRENT_DECODE` has no shape term (`vllm/envs.py:124` @ `5559679`); both JIT-compile per shape, which is the property the AOT vendoring trades away for a Python-free runtime. Closing it needs `h128` specializations vendored across the supported architectures, or a stated rule for which head counts get an AOT arm plus a visible fallback cost at the call site. Filed, not fixed: either close needs the checkpoint that motivates it, and this hardware cannot run the 2.4T (~4.8 TB bf16 against 128 GB unified memory), so the fallback cannot be measured here today. Listed under `## Owed` in [`gdn-moe-bf16-out.md`](specs/gdn-moe-bf16-out.md) | perf |