BLOCKER: CUDA grouped MoE writes NOTHING for IQ1_S/IQ1_XXXS - silent wrong output on the default path

[localai-bot]

Found by independent review of PR #967, which was self-reviewed and merged with --admin. Live on main today. Row ENG-EXPERT-STREAM, issue #912.

The defect

IsCudaKeepQuantSupported (src/vt/cuda/cuda_quant_dot.cu:1600-1601) now returns true for kIQ1_S and kIQ1_XXXS. Three dispatch switches consume that predicate; #967 extended only the dense one.

MatmulBTQuantGroupedKernelCuda (:1948-1957) uses the predicate at :1916 to SKIP its CPU fallback (:1917-1921), then dispatches through a switch (w) that has no case for either new dtype and no default:. It quantizes the activation, launches nothing, and returns. CheckCuda(cudaGetLastError()) at :1958 sees no error because there was no launch. The output tensor is never written.

Before #967 these dtypes returned false and took the CPU fallback: correct, just slow. #967 converted correct-but-slow into silently wrong, which is the exact inversion of its stated purpose.

Measured (GB10, reviewer probe, poisoned output buffer)

case	device out[0]	CPU oracle out[0]	nmse vs CPU
iq1_s	-12345 (poison, untouched)	-2.18233	4.58e6
iq1_xxxs	-12345 (poison, untouched)	-4.11988	9.96e6
iq2_s (control, has a case)	matches		pass

Why it is reachable by default

qwen3_5_gguf_weights.cpp:62 accepts any KeepQuantDType, which now includes ggml 19 and 66 -> qwen3_5.cpp:6091 -> :5186 KqGrouped -> vt::MatmulBTQuantGrouped (ops.cpp:214) -> the broken switch. VT_QWEN35_GROUPED_MOE defaults ON.

#985 ("loads and generates") ran on --device cpu, so nothing has exercised the CUDA arm end to end yet. The defect is waiting at exactly the next step of this row.

Why it compiled silently

cmake/CompilerWarnings.cmake:28 gives CUDA only -Werror=all-warnings and no -Wall, so the host pass never runs -Wswitch. A missing enum case in a switch without a default is therefore not an error here.

F2 (HIGH): the same omission in the fused MoE SwiGLU seam

MoeGateUpSwiGLUGroupedCuda (:2117-2126) guards with the same predicate at :2092-2096 and previously threw a NAMED error. #967 made the guard pass while the switch still matches nothing, so a loud refusal became silence. Latent for IQ1 today (reachable via cuda_deepseek_v4.cu:2082 / vt::MergedGemm), but any keep-quant MoE arch binding the shared seam inherits it.

F3 (HIGH): the CUDA grouped dispatch has no test at all

grep -rl MatmulBTQuantGrouped tests/ finds two files, neither with any kCUDA reference. The whole grouped dispatch is ungated on device, which is why F1 and F2 landed green.

F5 (MEDIUM): device grid tables are unsealed, and 13% of one is untested

cuda_quant_iq_tables.cuh:41-43 claims "the digests the CPU tests re-derive cover these bytes too". They do not: the tests digest the HOST symbols vt::cpu::kIq1sGrid / kIq1xxxsGrid; nothing reads vt::cuda::d_iq1s_grid. The header at :4 also claims a runtime memcmp test against the CPU tables that does not exist.

Replaying the gate's fixed-seed generator: 266 of 2048 d_iq1s_grid entries (13.0%) are never addressed by any test shape. Drifting entry 0 is caught; drifting entry 3 is GREEN, 150032/150032.

Remediation

1. Add the two LaunchGroupedGemm<> cases (the templates are generic in W; DotSuperblock/FinalFactor already specialize, so no new kernel code) and a default: that THROWS naming the dtype, in both grouped switches.
2. Extend the CUDA gate to the grouped seam over the same case table.
3. Add the cudaMemcpyFromSymbol + memcmp seal the header already claims, and fix both comments.
4. Consider giving CUDA -Wswitch so the compiler catches the next one.

Owning row: ENG-EXPERT-STREAM.

[localai-bot]

PR #1035 repairs F1, F2, F3 and F5 in code. Leaving this open, because the device half of the evidence does not exist yet.

Fixed

• Both grouped switches gain the kIQ1_S / kIQ1_XXXS arms. No kernel code was needed: QuantDotGemmGroupedKernel and QuantDotGemmGroupedFusedSwiGLUKernel are generic in W and reach only DotSuperblock<W> and FinalFactor<W>, both already specialized by the dense path.
• All three switches gain a default: that throws and names the dtype. Past IsCudaKeepQuantSupported there is no fallback left, so this is the general repair and the two added arms are the specific one.
• F3: both grouped seams are now gated over the dense gate's case table, against the CPU grouped golden, through a poisoned output buffer.
• F5: SnapshotIqTablesFromDevice + a byte-exact memcmp of all eight device codebooks against the CPU tables, which cuda_quant_iq_tables.cuh had claimed since it landed. Both false comments corrected.

On F4 (-Wswitch for CUDA): recorded, not done. It would not have caught this. -Wswitch is silent whenever a default: label exists, and a default: is exactly what the repair adds. The flag that fires is -Wswitch-enum, which warns on every enum switch in the tree that omits an enumerator even with a default. That is a tree-wide change with no measurement behind it, on a compile lane that cannot be run from here.

Owed, and why this issue stays open. The CUDA arms of the new gate have never run. dgx.casa was unreachable for the whole of this work (No route to host, ping 100 % loss, retried through the session), the working box has no CUDA toolkit and no NVIDIA device, and there is no second CUDA host. What is owed on GB10:

1. the three new CUDA cases green;
2. the confirmation that deleting either new case turns the grouped gate red.

What did run: the full 492-test suite green, and a CPU arm of the same poisoned-buffer probe. That arm answers one question the value gates cannot. Mutating MatmulBTQuantKernel to write nothing leaves the golden and the independent reconstruction both at the poison value, so memcmp passes with zero failures and only the poison assertion fires (55 failed assertions, all poisoned == 0). Every value gate in that file would have read an unwritten buffer as a merely inaccurate result.

The 13.0 % figure was re-derived rather than quoted: replaying std::mt19937(0x5EED) over the dense gate's widest weight (128 blocks, 4096 grid draws) gives 1782 distinct entries and 266 untouched, matching the review exactly. The new grouped gate's widest weight is 512 blocks and 16384 draws and reaches all 2048, which is coverage by accident of shape rather than by contract, and says nothing about the seven other tables.

[localai-bot]

Fixed by #1035 (b493f4981).

Both grouped switches gained the two arms, plus default: labels so the next missing case is loud instead of silent. The device grid tables gained the cudaMemcpyFromSymbol seal the header had been claiming.

Operator verification on GB10, under flock, independent of the implementer report:

baseline test_cuda_quant_dot	13 cases, 13 passed, RC=0 (9 before)
delete the grouped kIQ1_S arm (re-inject this defect)	applied (md5 65546e0c to 968e1e8d), compiled rc=0, 1 case FAILED, RC=1
tree after	restored to 65546e0c

So the test CATCHES the defect rather than merely accompanying the fix, which is the property #967 lacked and the reason a silent-wrong-output bug landed green.

Full gate on the rebased branch: 495/495, 0 failed, exit 0.

Two instrument failures nearly produced a false verdict and are worth recording: git diff on the dgx copy printed git's USAGE text (that tree is a tar extract with no .git), and rm -rf of the previous scratch tree silently failed on root-owned files the container had created, so the first mutation ran against a stale revision. It aborted with anchor count 0 rather than reporting a pass, which is the only reason this is a real result.