fix(ENG-EXPERT-STREAM): the grouped CUDA dispatch launched nothing and called it success (#1029)

.agents/issue-index.md

@@ -290,3 +290,4 @@ rather than merged. `scripts/check-agent-record.py` gates both.
 | [#1021](https://github.com/mudler/vllm.cpp/issues/1021) | — | LTX-2.5 **DiT device staging takes 450 s — 7.5 minutes — and no record names the phase**. MEASURED on `dgx.casa` (GB10, boot id `03717c9d-63c8-4652-a8fe-a63d012c5718`, build `0e1bee42f`, under `$HOME/gpu.lock`), per-PID at a nominal 2 s over 192 samples: CUDA compute-app footprint 4.22 -> **35.20 GiB**, GPU utilization mean **0.2%** with zero in **164/192** samples, CPU **0.15 cores of 20**, `Anonymous` +0.01 GiB. Neither GPU-bound nor CPU-bound. **The RATE is not one number and this row records the spread rather than the slower figure.** Recomputed from the counters: rung 1 stages 31723 MiB in 450 s = **70.5 MiB/s**; rung 2 stages the same ~32 GiB (`capp_mib` 4322 -> 36396 = 32074 MiB) in **251 s = 127.8 MiB/s** — same host, same boot id, same build, a **1.81x spread** that is itself unattributed, because the sampler recorded no system-wide load column. An earlier draft said "~52 MiB/s", which is the PLATEAU divided by the WHOLE 700 s run rather than by the staging window, and which contradicts its own inputs in both directions (450 s x 52 MiB/s = 22.9 GiB against a recorded 35.54 GiB plateau). Any row taking this lever must measure the rate itself rather than inherit either figure. The shape is `src/vllm/model_executor/models/ltx2_loader.cpp:738-756 @ 332aed738`: ~3,504 tensors, each a raw `cudaMalloc` (`backend.Alloc` at `:747`, `src/vt/cuda/cuda_backend.cu:77-81 @ 332aed738`) followed by a full `backend.Synchronize(queue)` at `:749`, serialized against the host read. The plateau at 36396 MiB = 35.54 GiB lands within 1% of the 35.32 GiB the loader contract predicts. Sampler CSV not retrievable ([#1040](https://github.com/mudler/vllm.cpp/issues/1040)). Same loop as [#1016](https://github.com/mudler/vllm.cpp/issues/1016); a row should take both. Listed under `## Owed` in [`ltx25-decode-speed.md`](specs/ltx25-decode-speed.md) | feature |
 | [#1024](https://github.com/mudler/vllm.cpp/issues/1024) | — | An LTX-2.5 render invoked with `--device cuda` **stages 35.54 GiB onto the GPU and then never uses it**. MEASURED on `dgx.casa` (GB10, boot id `03717c9d-63c8-4652-a8fe-a63d012c5718`, build `0e1bee42f`), 320x192/25f, per-PID over **347 samples**: GPU utilization never exceeded 2%, was **exactly 0 in 321/347**, and **every one of the 26 non-zero samples falls inside the DiT staging window** (t<=251 s) — so **every post-staging sample reads 0**, while the compute-app footprint sits flat at 36396 MiB and the process holds **exactly 1.00 core of 20** across the two `utime` windows this figure covers (+116.7 s over 117 s, +142.2 s over 142 s = **259 s of measurement**; §1.3 of the spec records a third window in the same regime, +648.9 s over 649 s, so 259 s is what these two cover and NOT the sampler's total, which the record cannot settle — [#1040](https://github.com/mudler/vllm.cpp/issues/1040)); 0 frames written. **This row states sample counts and states NO minute figure, because no consistent one exists.** Drafts variously said "17+ minutes" here and "over 15 minutes" over "the first 1192 s" in the spec; 347 samples at a 2 s cadence span 694 s, so neither fits, and no dropped-sample rate was ever recorded. The GPU-zero half is over EVERY sample; the core-count half rests on 259 s. Staging took the device path (`Anonymous` stayed at 0.01 GiB, so no f32 widening — `src/vllm/multimodal/ltx2_video.cpp:786 @ 332aed738`). Leaves a disjunction this issue cannot yet split, because nothing timestamps a phase boundary ([#1010](https://github.com/mudler/vllm.cpp/issues/1010)): either the denoise is not taking `Ltx2DitForwardDevice` (`ltx2_video.cpp:2946` vs `:2948 @ 332aed738`), or it is and something before it burns the rest of the run single-threaded. Broader than [#1007](https://github.com/mudler/vllm.cpp/issues/1007): the 0% GPU is a property of the whole post-load render, not of the decode phase, so `docs/USAGE.md:873-874 @ 332aed738` understates it. **Owed: a positive control that `utilization.gpu` reads high for a real kernel on GB10** — this box already returns `[N/A]` for `--query-gpu=memory.used`. Sampler CSV not retrievable ([#1040](https://github.com/mudler/vllm.cpp/issues/1040)). Listed under `## Owed` in [`ltx25-decode-speed.md`](specs/ltx25-decode-speed.md) | bug |
 | [#1040](https://github.com/mudler/vllm.cpp/issues/1040) | — | `LTX25-DECODE-SPEED` ([#1006](https://github.com/mudler/vllm.cpp/issues/1006)) records three sets of PASSING measurements and **none has a retrievable evidence artifact**, which `.agents/verification.md` requires alongside the SHA, command, environment and exit status. Rung 1's 248 per-PID sampler rows, rung 2's 1082 rows and 347 side-car per-PID samples, rung 2's `run.log`, and §1.4's `~/work/ltx25-e2e/render8-console.log` all live only on `dgx.casa` — `ping -c 2 -W 3 dgx.casa` exits 1 with 100% packet loss and `Destination Host Unreachable` (checked 2026-08-16), this box's documented unified-memory OOM-reboot mode, which needs a physical power cycle. The spec's `REMOTE_UNVERIFIED` mark was correctly scoped to rung 2's EXIT REASON; this issue extends the same honesty to the passing numbers. **Second and more corrosive: neither rung's sampler cadence closes.** Rung 1 states 248 samples at 2 s over a 701 s run split 192+56 across 450 s and 164 s windows — at 2 s those windows hold 225 and 82, and 248 samples cover 496 s of 701. A draft of rung 2's §5 stated 347 per-PID samples at 2 s "over the first 1192 s" — at 2 s that window holds 596. No dropped-sample rate is recorded anywhere, so the wall each sample set covers is NOT derivable, and the spec now states sample counts and fractions rather than minute figures. The raw CSVs settle both in one pass. Filed while repairing the fresh review of [PR #1038](https://github.com/mudler/vllm.cpp/pull/1038); not fixable in that flow, because no edit to the tree produces a file on a host that does not answer. Listed under `## Owed` in [`ltx25-decode-speed.md`](specs/ltx25-decode-speed.md) | bug |
+| [#1029](https://github.com/mudler/vllm.cpp/issues/1029) | `ENG-EXPERT-STREAM` | `IsCudaKeepQuantSupported` (`src/vt/cuda/cuda_quant_dot.cu`) returns true for `kIQ1_S` and `kIQ1_XXXS` since [#967](https://github.com/mudler/vllm.cpp/pull/967), and THREE dispatch switches consume that predicate while #967 extended only the dense one. `MatmulBTQuantGroupedKernelCuda` uses the predicate to SKIP its CPU fallback and then dispatches through a `switch (w)` with no case for either dtype and no `default:`: it quantizes the activation, launches NOTHING, returns, and `CheckCuda(cudaGetLastError())` reports success because there was no launch to fail, so the output tensor is never written. Measured on GB10 by an independent review through a poisoned output buffer: both IQ1 encodings leave `-12345` in place at NMSE `4.58e6` / `9.96e6` against the CPU oracle, with `iq2_s` passing as the control. Reachable by DEFAULT (`qwen3_5_gguf_weights.cpp` -> `qwen3_5.cpp` `KqGrouped` -> `vt::MatmulBTQuantGrouped`, `VT_QWEN35_GROUPED_MOE` on) on the 96.92 % of `Qwen3.8-2.4T` those two encodings make up, so #967 converted correct-but-slow into silently wrong. Same omission in the fused `MoeGateUpSwiGLUGroupedCuda` seam, where it turned a NAMED refusal into silence. It landed green because the CUDA grouped dispatch had NO test: `grep -rl MatmulBTQuantGrouped tests/` found two files and neither mentioned `kCUDA`. Repaired by adding both arms to both grouped switches, a `default:` that THROWS and names the dtype on all three (the general fix: past that predicate there is no fallback left), and a grouped + fused CUDA gate over the dense gate's case table driven through a POISONED output buffer. Also seals the device codebooks against the CPU tables, which `cuda_quant_iq_tables.cuh` claimed from the day it landed and no test did: 266 of 2048 `d_iq1s_grid` entries (13.0 %) are never addressed by the gate's own `mt19937(0x5EED)` stream. Device evidence is OWED and the issue stays open for it: `dgx.casa` was unreachable throughout and this box has no CUDA device, so only the CPU arm ran. Spec [`expert-streaming.md`](specs/expert-streaming.md) | bug |

.agents/specs/expert-streaming.md

@@ -946,6 +946,95 @@ TTFT improving 4.5x while decode did not is consistent with all of this: prefill
 touches a wide expert set once, where sequential order and slot reuse both help,
 while decode re-reads a fresh 6.9 GB every step.
 
+## One predicate, three switches, and only one of them grew (issue #1029)
+
+`#967` taught `IsCudaKeepQuantSupported` (`src/vt/cuda/cuda_quant_dot.cu`) to
+return true for `kIQ1_S` and `kIQ1_XXXS`. Three dispatch switches consume that
+predicate and `#967` extended one of them, the dense `MatmulBTQuantKernelCuda`.
+
+The grouped GEMM used the same predicate to SKIP its CPU fallback and then
+dispatched through a `switch (w)` that had no case for either dtype and no
+`default:`. It quantized the activation, launched nothing, and returned;
+`CheckCuda(cudaGetLastError())` reported success, because a launch that never
+happened cannot fail; and the output tensor kept whatever it already held. An
+independent review measured this on GB10 through a poisoned output buffer: both
+IQ1 encodings left `-12345` in place, at NMSE `4.58e6` and `9.96e6` against the
+CPU oracle, while the `iq2_s` control passed. Before `#967` these dtypes took
+the CPU fallback and emitted correct tokens slowly, so `#967` converted
+correct-but-slow into silently wrong, which inverts its stated purpose.
+
+That path is the DEFAULT routed-expert path of the target checkpoint:
+`qwen3_5_gguf_weights.cpp` accepts any `KeepQuantDType`, which now includes ggml
+19 and 66, and `qwen3_5.cpp` `KqGrouped` reaches `vt::MatmulBTQuantGrouped` with
+`VT_QWEN35_GROUPED_MOE` on by default. The two encodings are 96.92 % of the
+model. The fused `MoeGateUpSwiGLUGroupedCuda` seam had the same hole, where the
+consequence was worse in kind: its guard previously threw a NAMED refusal, and
+`#967` turned that named refusal into silence.
+
+The repair is three things, and the third is the one that matters most.
+
+1. Both grouped switches gained the `kIQ1_S` and `kIQ1_XXXS` arms. No new kernel
+   code was needed: `QuantDotGemmGroupedKernel` and
+   `QuantDotGemmGroupedFusedSwiGLUKernel` are generic in `W` and depend only on
+   `DotSuperblock<W>` and `FinalFactor<W>`, both already specialized for these
+   two dtypes by the dense path.
+2. All three switches gained a `default:` that THROWS and names the dtype. This
+   is the general repair rather than the specific one: past the
+   `IsCudaKeepQuantSupported` gate there is no CPU fallback left, so any future
+   missing case is a silent no-op unless something refuses out loud.
+3. The grouped seams are now gated. They had NO test: `grep -rl
+   MatmulBTQuantGrouped tests/` found two files and neither mentioned `kCUDA`,
+   which is exactly why F1 and F2 landed green. `tests/vt/test_cuda_quant_dot.cpp`
+   now drives both grouped seams over the same case table the dense gate uses,
+   including the two new dtypes, against the CPU grouped golden, THROUGH A
+   POISONED OUTPUT BUFFER.
+
+The poison is not decoration and it is not redundant with the value comparison.
+Mutating `MatmulBTQuantKernel` to write nothing makes the golden AND the
+independent reconstruction both stay at the poison value, so the byte comparison
+passes with zero failures and only the poison assertion fires: 55 failed
+assertions, all of them `poisoned == 0`, and `memcmp` failures zero. Every value
+gate in that file would have read an unwritten buffer as a merely inaccurate
+result.
+
+Chosen against `-Wswitch` for CUDA (`cmake/CompilerWarnings.cmake`), which the
+issue proposed. CUDA gets `-Werror=all-warnings` and no `-Wall`, so `-Wswitch`
+never runs, but adding it would not have caught this defect either: `-Wswitch`
+is silent whenever a `default:` label exists, and a `default:` is exactly what
+the repair adds. The flag that would fire is `-Wswitch-enum`, which warns on
+every enum switch in the tree that omits any enumerator even with a default, and
+that is a tree-wide change with no measurement behind it here, on a lane this
+box cannot compile. Recorded rather than done.
+
+**Owed on this repair.** The CUDA arms of the new gate have NOT run on a device.
+`dgx.casa` was unreachable for the whole of this work (`No route to host`, ping
+100 % loss), this box has no CUDA toolkit and no NVIDIA device, and there is no
+second CUDA host. What ran is the CPU arm, which is real: it drives the same
+poisoned-buffer instrument through the CPU grouped golden on every host, and the
+mutations above are its evidence. What is still owed is the GB10 run of the three
+CUDA cases, including the confirmation that removing either new `case` turns the
+grouped gate red. Issue #1029 stays open until that runs.
+
+Also sealed here: `cuda_quant_iq_tables.cuh` claimed from the day it landed that
+"a runtime test memcmps these tables against the CPU host tables". No such test
+existed. The CPU tests digest the HOST symbols, and nothing read
+`vt::cuda::d_iq1s_grid` at all, so a device transcription slip was visible only
+when a weight sample happened to address the drifted entry. Replaying the CUDA
+gate's own `std::mt19937(0x5EED)` stream, 266 of the 2048 `d_iq1s_grid` entries
+(13.0 %) are never addressed, which is why drifting entry 0 is caught and
+drifting entry 3 stays green at 150032/150032 assertions. That figure came from
+the review and was re-derived here rather than quoted: replaying the stream over
+the dense gate's widest weight (16 rows times 8 super-blocks, 128 blocks, 4096
+grid draws) gives 1782 distinct entries and 266 never addressed.
+
+The new grouped gate widens the same weight to 64 rows (E=4, N=16), which is 512
+blocks and 16384 draws, and that reaches all 2048 entries. Recorded because it is
+true, not because it closes anything. It is coverage by accident of shape rather
+than by contract, one shape change away from shrinking again, and it says nothing
+at all about the other seven tables. The seal now exists
+(`src/vt/cuda/cuda_iq_table_seal.h` plus the gate case), it covers all eight
+device codebooks byte for byte, and both false comments were corrected.
+
 ## Risks/decisions
 
 | Risk / decision | Call |

src/vt/cuda/cuda_iq_table_seal.h

@@ -0,0 +1,43 @@
+// Device-codebook drift seal. `cuda_quant_iq_tables.cuh` is a HAND TRANSCRIPTION
+// of `src/vt/cpu/cpu_quant_iq_tables.h`, and until this header existed nothing
+// compared the two: the CPU tests digest the HOST symbols, which the device
+// arrays are not, so a slipped literal in the `.cuh` was invisible to every gate
+// that did not happen to address that entry. Replaying the CUDA gate's own
+// std::mt19937(0x5EED) weight stream, 266 of the 2048 `d_iq1s_grid` entries
+// (13.0 %) are never addressed at all, so a drift there is green by luck.
+//
+// A `__device__` array has no host address, so a plain C++ translation unit
+// cannot take its address and cannot call `cudaMemcpyFromSymbol` on it. The copy
+// therefore has to happen inside the CUDA TU that defines the tables. This
+// header is the CUDA-free declaration of that copy, so the gate in
+// `tests/vt/test_cuda_quant_dot.cpp` can memcmp the result against the CPU
+// tables without pulling `<cuda_runtime.h>` into a host build.
+#ifndef VT_CUDA_IQ_TABLE_SEAL_H_
+#define VT_CUDA_IQ_TABLE_SEAL_H_
+
+#include <cstdint>
+
+namespace vt::cuda {
+
+// One host-side copy of every codebook `cuda_quant_iq_tables.cuh` defines. The
+// extents are restated here rather than derived, and `cuda_quant_dot.cu`
+// static_asserts each one against `sizeof(d_<table>)`, so a device array that
+// changes length fails to compile instead of silently truncating the seal.
+struct IqTableSnapshot {
+  uint8_t kmask_iq2xs[8];
+  uint8_t ksigns_iq2xs[128];
+  uint64_t iq1s_grid[2048];
+  uint64_t iq1xxxs_grid[256];
+  uint64_t iq2xxs_grid[256];
+  uint32_t iq3xxs_grid[256];
+  uint64_t iq2s_grid[1024];
+  int8_t kvalues_mxfp4[16];
+};
+
+// Copies the device codebooks into `out`. Requires a live CUDA context; throws
+// std::runtime_error if any copy fails. Defined in cuda_quant_dot.cu.
+void SnapshotIqTablesFromDevice(IqTableSnapshot* out);
+
+}  // namespace vt::cuda
+
+#endif  // VT_CUDA_IQ_TABLE_SEAL_H_