ENG-EXPERT-STREAM: the CUDA arm for IQ1_S and IQ1_XXXS, verified on GB10 by localai-bot · Pull Request #967 · mudler/vllm.cpp

localai-bot · 2026-08-16T00:53:50Z

W10 and W13 of .agents/specs/expert-streaming.md, the owed half of #946. Tracked by #912.

Why this is not optional polish

cuda_quant_dot.cu maps an unrecognised weight dtype to return false, which is a silent CPU fallback. Without these kernels the Qwen3.8-2.4T routed experts, 96.92 % of either checkpoint, decode on the CPU while the model emits CORRECT tokens and looks healthy. No token gate can see that, so it would surface only as "the GPU is mysteriously slow".

Hardware result

Run on dgx.casa (GB10, sm_121a, nvcc 13.0.1), test_cuda_quant_dot: 9 cases, 150032 assertions, 0 failed.

Bound	Meaning	Measured
`nmse_cpu <= 1e-6`	device kernel vs the CPU oracle	5.23e-14
`nmse_ref <= ceiling`	vs an independent f64 dequantize	5.24e-4

The first is the assertion this port lives by, and it passes by eight orders of magnitude.

The one bound that moved, and why

nmse_ref for iq1_s came in at 5.23981e-4 against a 5e-4 default, at m=4, k=2048, n=1, where n=1 and m=4 is four dot products. That comparison holds a Q8_K-quantized ACTIVATION against an f32 reference, so it measures activation error, not the kernel: Q8_K carries one scale per 256 elements while these encodings carry a per-32 scale.

What identifies it is the CPU test measuring 5.23975e-4 at the same shape. The two agree to six significant figures, so this is the activation term reproducing itself, not anything device-specific. The ceiling is per-CASE, applies only to that comparison, and is set to 2e-3: about 4x the residual and an order of magnitude below the weakest decode defect measured by mutation on the CPU arm (2.7e-2 for a dropped delta term). The CUDA-vs-CPU bound is untouched at 1e-6 and shared by every encoding.

Design note

The device grids deliberately keep the SAME u64 layout as the CPU tables rather than upstream's packed u32 *_grid_gpu variants: one encoding to validate instead of two, the device dot reads as a literal transcription of the CPU one, and the provenance digests the CPU tests re-derive cover these bytes too.

Both dots read bsums, which no other IQ dot here needs, because IQ1_S and IQ1_XXXS reconstruct dl*(grid + delta): the dot separates into an integer grid term plus delta times the activation's per-16 sums, which Q8_K already caches.

Evidence

Device parity, above, on real GB10 hardware.
Before it ran, both dots were diffed line by line against the CPU kernels. Every arithmetic line matches except GRID + i vs &GRID[i] and y[i].bsums vs yb->bsums, the same expressions written for a block loop and for a single block pointer. A desk check is not a hardware result and did not stand in for one.
CPU gate: 485/485 ctest, 0 failed, on the rebased tree.
The synthetic scale narrowing mirrors the CPU test for the same measured reason; the delta sign bit stays random so a kernel that dropped it is still caught.

The GPU lock was 3h38m stale with zero compute processes when the device run happened, past the 2h TTL. It was reclaimed for the duration and released after.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]

Without these two the Qwen3.8-2.4T routed experts, which are 96.92 % of either checkpoint, fall through `IsCudaKeepQuantSupported` to the CPU arm. That path emits CORRECT tokens at CPU speed, so no token gate can see it and the model simply looks slow. Naming it as owed was right; leaving it owed while claiming a device benchmark would not be. Both device dots are transcriptions of the CPU kernels they must agree with bit for bit, including the `bsums` split that neither of the other IQ dots here needs: IQ1_S and IQ1_XXXS reconstruct dl*(grid + delta), so the dot separates into an integer grid term plus delta times the activation's per-16 sums, which Q8_K already caches. The device grids deliberately keep the SAME u64 layout as the CPU tables rather than upstream's packed u32 `*_grid_gpu` variants. One encoding to validate instead of two, the device dot reads as a literal transcription of the CPU one, and the provenance digests the CPU tests re-derive therefore cover these bytes as well. Parity is gated by the existing CUDA-vs-CPU case, which is the assertion that matters here: `nmse_cpu <= 1e-6` against the CPU oracle, plus `nmse_ref` against an independent f64 dequantize. The synthetic scale narrowing mirrors the CPU test for the same measured reason, and the delta sign bit stays random so a kernel that dropped it is still caught. STATUS, stated plainly: this COMPILES and LINKS on GB10 (nvcc 13.0.1, sm_121a) and is NOT yet verified on device. dgx's GPU is held by another session, so the parity test has not run. Do not read this as a working device path until that test reports. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

…ef bound measured The device kernels now have a hardware result rather than a compile. On dgx.casa (GB10, sm_121a, nvcc 13.0.1) the parity case reports nmse_cpu = 5.2e-14 against the CPU oracle, on a 1e-6 bound. That is the assertion this port lives or dies by, and it passes by eight orders of magnitude. One case needed a ceiling, and the reason is measured rather than assumed. The f64-dequant comparison holds a Q8_K-quantized ACTIVATION against an f32 reference, so it bounds activation error, not the kernel. iq1_s reported 5.23981e-4 against the 5e-4 default at m=4, k=2048, n=1, where n=1 and m=4 is four dot products. The CPU test measures 5.23975e-4 at the SAME shape: the two agree to six significant figures, which is what identifies the activation term rather than anything device-specific. The ceiling is per-CASE and applies only to that comparison; the CUDA-vs-CPU bound is untouched at 1e-6 and shared by every encoding. Set to 2e-3: about 4x the residual, and an order of magnitude below the weakest decode defect the CPU mutation run measured (2.7e-2 for a dropped delta term). Also recorded before this ran: the two device dots were diffed line by line against the CPU kernels they transcribe. Every arithmetic line matches except `GRID + i` versus `&GRID[i]` and `y[i].bsums` versus `yb->bsums`, which are the same expressions written for a loop over blocks and for a single block pointer. A desk check is not a hardware result, which is why it did not stand in for one. The GPU lock was 3h38m stale with zero compute processes when this ran, past the 2h TTL, and was reclaimed for the duration and released after. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

localai-bot · 2026-08-16T01:10:17Z

Gated locally, CI saturated

Check	Result
CUDA parity on GB10 (`test_cuda_quant_dot`)	9/9 cases, 150032 assertions, 0 failed
`nmse_cpu` vs the CPU oracle	5.23e-14 (bound 1e-6)
CPU `ctest`, post-rebase	485/485, 0 failed
`check-commit-trailers` / `check-commit-style`	pass

The device run was on dgx.casa (GB10, sm_121a, nvcc 13.0.1) inside vllmcpp-build:gb10. --gpus all is required even to load the binary, because libcuda.so.1 is injected by the container toolkit.

Not run here: windows-msvc-*, arm64, vulkan. This change is a CUDA kernel plus its test, so none of them exercise it.

…the first, and it came from main (#968, #672) With #965's `C4456 'loaded' shadow` removed, `windows-msvc-cpu` and `windows-msvc-vulkan` failed again on this row's pull request — and on a different cause: include\vector(1461,29): error C2220: the following warning is treated as an error include\vector(1461,29): warning C4244: '=': conversion from 'const double' to 'float', possible loss of data raised from `src/vllm/multimodal/ltx2_video.cpp:203,214`, the two narrowing `positions.assign` calls that `c7cb59fbb` (#964, LTX25-TOKEN-APPEND) landed on `main` while this row was in flight. `StreamState::positions` and `Ltx2LatentState::positions` differ in element type; GCC and Clang narrow silently, MSVC diagnoses and the build treats it as an error. NOT FIXED HERE, deliberately. #964's own comment at `ltx2_video.cpp:129-132` reasons that "double -> float -> double reproduces the bits", so the narrowing is intentional and a silencing `static_cast` would be a claim about that reasoning rather than a formatting repair. It belongs to the lane that owns the round trip. Filed as #968 with the evidence rather than papered over. THE MATCHED ARM SPLITS EXACTLY ON THE MERGE BASE, which is what makes it inherited rather than mine. Grepping each `windows-msvc-cpu` job log for the warning: #966 and #951, both on `c7cb59fbb`, hit it twice each; #967, #956, #950, #939 and #938, all based before it, do not hit it at all. This row's diff touches zero LTX2 files. THE FINDING WORTH CARRYING is not either warning. It is that TWO INDEPENDENT CAUSES WERE STACKED BEHIND ONE HABITUALLY-RED JOB NAME, and the first hid the second — and that neither was #645, the `M_PI` regression both jobs are usually attributed to. A known-red list tells you a job is often red. It never tells you that today's red is the same one. Only reading the log does. Issue: #968 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]

FOLLOWING_AGENTS_PROTOCOL Keeps the branch current before review. The incoming commit is the ENG-EXPERT-STREAM CUDA arm for IQ1_S/IQ1_XXXS (#967); it touches no MiniMax-Music3 surface. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]

…gating it (#931) FOLLOWING_AGENTS_PROTOCOL `origin/main` moved from `c7cb59fbb` to `10813dfeb` (#967) WHILE the post-merge preflight was running. The remote-tracking ref is shared by every worktree of this checkout, so a sibling worktree's fetch advances it under a run in progress. The visible symptom was a preflight that printed "All gates green." without printing its "Commit trailers vs origin/main" block at all: the block is guarded by `git merge-base --is-ancestor origin/main HEAD`, which had been true when the run started and was false by the time the guard was evaluated. A green banner is therefore not evidence that the guarded blocks ran. Read the block headings, not the banner. This merge restores the ancestry so the trailer block is armed again. Automatic merge, no conflict, and the incoming change touches only `src/vt/cuda/cuda_quant_dot.cu`, `src/vt/cuda/cuda_quant_iq_tables.cuh` and `tests/vt/test_cuda_quant_dot.cpp`. Nothing it carries overlaps this row's files, and no keyed record is involved. Nothing is rebased, amended or forced. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]

… heard, and the five keys upstream refuses that we dropped (#672, #953, #965) (#966) feat(MODEL-MUSIC-MUSIC3): a music-only server, an example that can be heard, and the five keys upstream refuses that we dropped (#672, #953) Developer directive: parity on what upstream supports — "we want to be a good reference" — usage docs for MiniMax-Music3, and in those docs the weights. Then, mid-flight: "we should allow to load only the music model" and "we need to have an e2e test working". FOLLOWING_AGENTS_PROTOCOL ## The upstream surface, enumerated rather than summarized SGLang-Omni `748a0b43` `models/minimax_music3/` and diffusers `c6da9936` `modular_pipelines/minimax_music3/`, read field by field and recorded with `file:line` in spec §10.1 so the next reader re-derives nothing. **Closed here:** the music-only server, the missing example, and five refusals. **Owed and named:** the non-`wav` response formats, request batching and `/v1/audio/speech/batch`, the 32 kHz delivery resample. **Permanently refused rather than owed:** streaming — neither upstream arm has it (`supports_streaming_vocoder=False`). **One place we are ahead of both arms:** `guidance_scale` is a real per-request control here, where diffusers freezes it at 1.7 into the guider component (`denoise.py:180`) and SGLang exposes it only as a serve-time knob. ## `--model` is optional when `--speech-model` is given Serving a 28.5 GB music model also forced loading an unrelated text model, and on this box the smallest text checkpoint is 35B — so the recipe this project documented was effectively unrunnable. Upstream's own is `sgl-omni serve --model MiniMaxAI/MiniMax-Music3`, no text tower anywhere. vllm-server --speech-model /path/to/minimax-music3 Third instance of a shape already in `server_main.cpp`: a pooling checkpoint serves `/v1/embeddings` alone, a Parakeet checkpoint serves `/v1/audio/transcriptions` alone. It mirrors vLLM's task-conditional registration (`api_server.py:255-265`). **Additive, and proved rather than argued.** The only case whose verdict changes is BOTH flags absent, which was an error and remains one with a message naming both options. The route table is gated in both directions over a real socket, because a handler-dispatch test cannot see route registration at all. ## The example the music family did not have `examples/minimax_music3_gen` — a thin client of `include/vllm.h` and nothing else, like `parakeet-transcribe` and `vllm-cli`. Hearing this model previously needed a running server plus a `curl`, or a C ABI caller nobody had written. ## Five keys upstream refuses by name were SILENT here (#953) `temperature`, `top_p`, `top_k`, `repetition_penalty` — refused upstream at `request_builders.py:14-19,109-114`, because this model's autoregressive stage has ONE sampler, a fixed top-50 draw (`encoders.py:48,94-103`). And `max_new_tokens`, upstream's LENGTH spelling in 25 Hz frames rather than seconds (`request_builders.py:56-68`), so a 250-frame request silently became the family's 60 s default. That is the #925 class exactly, in the same file that already carries #925's refusal one paragraph above. Fixed in flow. ## The e2e gate no longer reports a skip wearing a pass It read 5 cases / 5 passed with **`assertions: 0`** whenever the checkpoint was absent — the same shape that fooled this project on `test_qwen3_paged_engine`. Split into a checkpoint-free half that runs unconditionally in CI (request contract, both ceilings, the speech-only route table over a real socket with a stub synthesizer) and the env-gated half, whose HTTP case now drives the real engine over a real socket against the music-only server shape. A coverage-report case prints which arms ran, every run. The full arm was run: `POST /v1/audio/speech -> 200 audio/wav, 12332 bytes in 518 s wall`, 2 AR frames -> 6 latent frames -> 3072 samples/channel, 6144 int16 samples all non-zero, 0 clipped, 2818 of 3072 positions differing between L and R, and `/v1/completions` + `/v1/chat/completions` both 404 from the route table. `checkpoint_arms_run=5`. | arm | cases | assertions | |---|---|---| | `test_minimax_music3_e2e_real`, no env vars | 9 | 37 (was 5 / **0**) | | `test_minimax_music3_e2e_real`, checkpoint only | 9 | 86 | | `test_minimax_music3_e2e_real`, checkpoint + `VLLM_CPP_MUSIC3_DIT=1` | 9 | **582** | | `test_speech_api` | 6 | 67 | | `test_openai_api_server` | 62 | 733 | | `test_openai_conformance` | 23 | 252 | | `test_minimax_h3` (unchanged) | 79 | 57395 | | server flag ctest cases | 7 passed | | ## The weights are documented (porting-a-model.md §2.1) `docs/USAGE.md` gains component-by-component tables: the diffusers arm at `MiniMaxAI/MiniMax-Music3`@`fbdf52fbaaca799592917417eb05f1899f1255ec`, **28.5 GB resident** (28 517 617 303 B, measured) out of a 57.4 GB repository and why they differ; the native `.pth` arm we refuse and SGLang-Omni serves; the one implemented GGUF Q4_K artifact with its sha256; and the fourteen third-party quantized repositories in five formats, each marked refused. The revision is verified rather than copied — `condition_encoder/diffusion_pytorch_model.safetensors` on disk hashes to that revision's own LFS record. ## A sample a human can hear 2.0 s of 44100 Hz stereo from this engine in 3286 s of wall clock: RMS 0.03169, peak 0.97437 with 0 clipped samples, 84 073 of 88 064 positions differing between left and right. **Its samples are compared to nothing** — §5 withdrew the token gate — so it shows the pipeline runs, not that the music is right. It is not committed: `check-pr-size.py` classifies every path and none takes a `.wav` outside `tests/`, where a file compared to nothing would sit beside the goldens and imply it was one. ## The four asks, answered directly **1. Music-only server.** `vllm-server --speech-model <dir>` with NO `--model` starts and serves, observed live rather than inferred: server: speech/music-only model (family=minimax-music3, 44100 Hz, text-only synthesis, family DETECTED); serving /v1/audio/speech server: listening on http://0.0.0.0:18923 (model 'minimax-music3') `--model` alone and `--model` + `--speech-model` are **byte-identical in behaviour**. The whole change is one new early branch, `if (args.model_dir.empty())`, which loads the speech engine and `return 0`s before reaching a single line of the existing path; nothing downstream of it was touched. The only case whose verdict changes is BOTH flags absent, which was an error and remains one. Server suites: **7 of 7** ctest cases (4 pre-existing + 3 new — neither flag is still an error and now names both options; `--speech-model` alone reaches the speech LOAD; `--speech-family` alone still demands a checkpoint), `test_openai_api_server` **62 cases / 733 assertions** (+1 case / +24 assertions, the speech-only route table over a real socket), `test_openai_conformance` **23 / 252** unchanged. **2. e2e, three arms.** What a bare CI run executes unconditionally: the request contract on the exact body the real case posts, the near-miss and sampling refusals, the duration arithmetic including both ceilings, and the speech-only route table over a real socket with a stub synthesizer. What stays env-gated: everything needing the 28.5 GB checkpoint, plus the two 2.4B-DiT arms behind `VLLM_CPP_MUSIC3_DIT`. | arm | cases | assertions | checkpoint arms run | |---|---|---|---| | no env vars (what CI runs) | 9 | **37** | 0 — was 5 cases / **0 assertions** | | `VLLM_CPP_MUSIC3_CHECKPOINT` | 9 | **86** | 3 | | + `VLLM_CPP_MUSIC3_DIT=1` | 9 | **582** | 5 | **3. The five keys.** All five were **accepted and silently dropped**; all five are **now refused by name**. | key | upstream anchor | why it cannot be honoured | |---|---|---| | `temperature` | `request_builders.py:14-19,109-114` | the AR stage's only sampler is a fixed top-50 draw, `encoders.py:48,94-103` | | `top_p` | same | no nucleus branch exists | | `top_k` | same | `_AR_SAMPLING_TOP_K` is a module constant of 50 | | `repetition_penalty` | same | no penalty is applied anywhere in the loop | | `max_new_tokens` | `request_builders.py:56-68`, `constants.py:4-5` | upstream's LENGTH, in 25 Hz frames not seconds; the refusal names `audio_duration` and the /25 conversion | **4. The weights table** (`docs/USAGE.md`, "MiniMax-Music3: the exact weights"). It carries: repo **and** revision — `MiniMaxAI/MiniMax-Music3`@`fbdf52fbaaca799592917417eb05f1899f1255ec`, verified rather than copied, since `condition_encoder/diffusion_pytorch_model.safetensors` on disk hashes to `83179c5e…a202c2a4d`, that revision's own LFS record; the Q4_K artifact's sha256 `4c5d41b2…c70cbdd0` at revision `c36aaeed…` with its exact byte count; **28.5 GB resident (28 517 617 303 B, measured) versus 57.4 GB repository**, with the reason they differ; the refused native `.pth` arm (`qwen_7B/`, `flowmatching_vae.pth`, `dav.pth`) and that SGLang-Omni serves it; and all fourteen community quant repositories across five formats, each marked refused and each marked **third-party** rather than first-party. This is the first application of `.agents/porting-a-model.md` §2.1 (landing as #951). ## Two reds stacked behind one habitually-red job name `windows-msvc-cpu`/`windows-msvc-vulkan` are usually attributed to #645 (`M_PI` in three LTX2 sources). **Neither of the two causes here was #645**, and the first hid the second. **#965, fixed in flow.** `C4456: declaration of 'loaded' hides previous local declaration` at `server_main.cpp:1315` — W6's own speech-attach block declaring `loaded` inside the text engine's `loaded`. The only warning in the job, on `main` since W6 landed. Matched arm: #956, #950 and #939, none touching the speech surface, fail identically. Renamed; nothing suppressed. **#968, filed and NOT fixed here.** With the shadow gone the same jobs failed again on `C4244: conversion from 'const double' to 'float'`, raised inside MSVC's `<vector>` from `ltx2_video.cpp:203,214` — two narrowing `positions.assign` calls that `c7cb59fbb` (#964) landed on `main` while this row was in flight. **This branch touches zero LTX2 files.** The matched arm splits exactly on the merge base: #966 and #951 (on `c7cb59fbb`) hit it, #967/#956/ #950/#939/#938 (before it) do not. It is deliberately left to the LTX-2.5 lane — #964's own comment reasons that "double -> float -> double reproduces the bits", so a silencing cast is a claim about that reasoning rather than a formatting fix. **The finding, which outlives both:** a known-red list tells you a job is often red. It never tells you that today's red is the same one. Only reading the log does — and here it took two readings, because removing the first cause is what made the second visible.  ### The first of the two, in detail (#965) `windows-msvc-cpu`/`windows-msvc-vulkan` failed here, and they are **not** #645 (`M_PI` in three LTX2 sources). They were W6's own `C4456: declaration of 'loaded' hides previous local declaration` at `server_main.cpp:1315` — the only warning in the job, on `main` since W6 landed. The matched-arm check is what separated it from my diff: #956, #950 and #939, all touching no speech surface, fail identically. `main` has no baseline because `windows-msvc-*` are PR-only (#584), so it presents to each author in turn as their own red. Filed and fixed in flow by renaming the inner declaration; nothing suppressed. ## Mutations Four run, four fire: sampling refusal neutered (5 assertions red), `max_new_tokens` refusal neutered (2 red), `--model` made mandatory again (2 ctest cases red), generate routes registered unconditionally (3 cases / 6 assertions red in the api-server suite, 1 / 2 in the e2e suite). Sources restored and verified sha256-identical. Supersedes #954 (untrailered merge commits) and #963 (a `server_main.cpp` commit that owed `docs/USAGE.md` under the per-commit documentation checkpoint). Same tree, linear history, every commit green on `check-doc-checkpoint`, `check-commit-trailers` and `check-commit-style` locally before pushing. Every source file is byte-identical to the one built and gated. Issue: #672 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>

…d called it success #967 taught `IsCudaKeepQuantSupported` to return true for `kIQ1_S` and `kIQ1_XXXS`. Three dispatch switches consume that predicate, and #967 extended one of them. `MatmulBTQuantGroupedKernelCuda` used the same predicate to skip its CPU fallback, then dispatched through a `switch (w)` with no case for either dtype and no `default:`. It quantized the activation, launched nothing, and returned. `CheckCuda(cudaGetLastError())` reported success, because a launch that never happened cannot fail, so the output tensor kept whatever it already held. An independent review measured that on GB10 through a poisoned output buffer: both IQ1 encodings left -12345 in place, at NMSE 4.58e6 and 9.96e6 against the CPU oracle, while the `iq2_s` control passed. That path is the default routed-expert path of Qwen3.8-2.4T, and the two encodings are 96.92 % of it, so #967 turned correct-but-slow into silently wrong. The fused MoE SwiGLU seam had the same hole, where a NAMED refusal became silence. Both grouped switches gain the two arms. No kernel code was needed: the grouped templates are generic in `W` and depend only on `DotSuperblock<W>` and `FinalFactor<W>`, which the dense path already specializes. The `default:` that now throws on all three is the more useful half. Past that predicate there is no fallback left, so any future missing case is a silent no-op unless something refuses out loud. It landed green because the grouped dispatch had no test at all: two files in `tests/` mention `MatmulBTQuantGrouped` and neither mentions `kCUDA`. Both grouped seams are now driven over the dense gate's case table, against the CPU grouped golden, through a poisoned output buffer. The poison is not redundant with the value comparison. Mutating `MatmulBTQuantKernel` to write nothing leaves the golden AND the independent reconstruction both at the poison value, so `memcmp` passes with zero failures and only the poison assertion fires: 55 failed assertions, all `poisoned == 0`. Every value gate in that file would have read an unwritten buffer as a merely inaccurate result. A CPU arm of the same probe runs on every host, because both CUDA cases return early without a device and doctest prints SUCCESS for a gate that asserted nothing. It takes the host from 0 assertions to 112. `cuda_quant_iq_tables.cuh` claimed from the day it landed that a runtime test memcmps the device tables against the CPU ones. No such test existed, and nothing read `vt::cuda::d_iq1s_grid` at all: the CPU tests digest the host symbols, which are different objects in a different address space. Replaying the CUDA gate's own mt19937(0x5EED) stream, 266 of 2048 `d_iq1s_grid` entries (13.0 %) are never addressed, so drifting entry 0 is caught and drifting entry 3 is green at 150032/150032. The seal now exists and covers all eight device codebooks byte for byte. Both false comments are corrected. Not done: `-Wswitch` for CUDA. It would not have caught this. `-Wswitch` is silent whenever a `default:` label exists, and a `default:` is what this change adds. The flag that fires is `-Wswitch-enum`, which warns on every enum switch in the tree that omits an enumerator even with a default, and that is a tree-wide change with no measurement behind it. Owed: the CUDA arms of the new gate have not run on a device. `dgx.casa` was unreachable throughout (`No route to host`, ping 100 % loss), this box has no CUDA toolkit and no NVIDIA device, and there is no second CUDA host. Issue #1029 stays open for the GB10 run. Closes nothing. Refs #1029, #912. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

…d called it success (#1029) (#1035) #967 taught `IsCudaKeepQuantSupported` to return true for `kIQ1_S` and `kIQ1_XXXS`. Three dispatch switches consume that predicate, and #967 extended one of them. `MatmulBTQuantGroupedKernelCuda` used the same predicate to skip its CPU fallback, then dispatched through a `switch (w)` with no case for either dtype and no `default:`. It quantized the activation, launched nothing, and returned. `CheckCuda(cudaGetLastError())` reported success, because a launch that never happened cannot fail, so the output tensor kept whatever it already held. An independent review measured that on GB10 through a poisoned output buffer: both IQ1 encodings left -12345 in place, at NMSE 4.58e6 and 9.96e6 against the CPU oracle, while the `iq2_s` control passed. That path is the default routed-expert path of Qwen3.8-2.4T, and the two encodings are 96.92 % of it, so #967 turned correct-but-slow into silently wrong. The fused MoE SwiGLU seam had the same hole, where a NAMED refusal became silence. Both grouped switches gain the two arms. No kernel code was needed: the grouped templates are generic in `W` and depend only on `DotSuperblock<W>` and `FinalFactor<W>`, which the dense path already specializes. The `default:` that now throws on all three is the more useful half. Past that predicate there is no fallback left, so any future missing case is a silent no-op unless something refuses out loud. It landed green because the grouped dispatch had no test at all: two files in `tests/` mention `MatmulBTQuantGrouped` and neither mentions `kCUDA`. Both grouped seams are now driven over the dense gate's case table, against the CPU grouped golden, through a poisoned output buffer. `cuda_quant_iq_tables.cuh` claimed from the day it landed that a runtime test memcmps the device tables against the CPU ones. No such test existed, and nothing read `vt::cuda::d_iq1s_grid` at all: the CPU tests digest the host symbols, which are different objects in a different address space. Replaying the CUDA gate's own mt19937(0x5EED) stream, 266 of 2048 `d_iq1s_grid` entries (13.0 %) are never addressed, so drifting entry 0 is caught and drifting entry 3 is green at 150032/150032. That figure came from the review and was re-derived here rather than quoted: the dense gate's widest weight is 128 blocks and 4096 grid draws, giving 1782 distinct entries and 266 untouched. The seal now exists and covers all eight device codebooks byte for byte. Both false comments are corrected. The same replay measured one thing in the other direction: the new grouped gate builds a 64-row weight, 512 blocks and 16384 draws, and that reaches all 2048 entries. Recorded as a fact rather than as a closure. It is coverage by accident of shape, one edit away from shrinking again, and it says nothing about the seven other tables. ## The poison earns its place It is not redundant with the value comparison. Mutating `MatmulBTQuantKernel` to write nothing leaves the golden AND the independent reconstruction both at the poison value, so `memcmp` passes with ZERO failures and only the poison assertion fires. | mutation | applied | compiled | result | |---|---|---|---| | baseline (unmutated) | n/a | yes | 1 case, 112 assertions, 0 failed, SUCCESS | | M1 grouped golden writes nothing | 2 lines | yes | FAILURE, 110 failed assertions | | M2 grouped golden ignores `eids[p]` | 2 lines | yes | FAILURE, 11 failed assertions | | M3 grouped golden ignores the broadcast row | 2 lines | yes | FAILURE, 11 failed assertions | | M4 `MatmulBTQuantKernel` writes nothing (golden AND reference) | 1 line | yes | FAILURE, exit 1, **memcmp failures 0**, poison failures 55 | Each mutation was restored byte-exact (`cmp -s` against a pre-mutation snapshot), and the tree is clean against `HEAD` afterwards. `applied` is measured against that snapshot rather than against `HEAD`, because the tree already carries the fix. A CPU arm of the same probe runs on every host, because both CUDA cases return early without a device and doctest prints SUCCESS for a gate that asserted nothing. It takes this box from 0 assertions to 112. ## What did NOT run, and why **The CUDA arms of the new gate have not run on a device.** `dgx.casa` was unreachable for the whole of this work (`ssh: connect to host dgx.casa port 22: No route to host`; `ping` 100 % packet loss, retried), this box has neither a CUDA toolkit nor an NVIDIA device (`nvcc not found`, `nvidia-smi` absent, no `/dev/nvidia*`), and there is no second CUDA host. So: - The two new CUDA test cases and the codebook seal are UNVERIFIED at runtime. - The confirmation that removing either new `case` turns the grouped gate red is UNVERIFIED at runtime. It follows statically from the `default:` that throws, and from the poison assertion for the pre-fix shape, but it has not been run. - The `cuda-fat-build` CI job compiles `cuda_quant_dot.cu` with nvcc over ten SM targets, so the production edit gets a compile check here. It configures with `-DVLLM_CPP_BUILD_TESTS=OFF`, so the CUDA-only test block does not. Issue #1029 stays OPEN for the GB10 run rather than being closed by this PR. ## Not done: `-Wswitch` for CUDA The issue proposed it. It would not have caught this defect. `-Wswitch` is silent whenever a `default:` label exists, and a `default:` is exactly what this change adds. The flag that fires is `-Wswitch-enum`, which warns on every enum switch in the tree that omits an enumerator even with a default. That is a tree-wide change with no measurement behind it, on a compile lane this box cannot run. Recorded here and in the spec rather than done. ## Gate - Full CPU build, CI configuration (`-DVLLM_CPP_BUILD_TESTS=ON`, no build type, so asserts are live): exit 0, zero `error:` lines. - Full `ctest`: **100 % tests passed, 0 failed out of 492**, exit 0, 710.36 s. Two did not run and both are pre-existing environment skips (`test_modelopt_mixed_precision_checkpoint`, `test_voxtral_e2e`). - Focused: `test_cuda_quant_dot` 10 cases, 112 assertions, 0 failed. The CUDA cases return early on this host, which is why the CPU arm exists: without it the same binary reports SUCCESS on 0 assertions. - This branch replaces #1034, whose history carried a `git merge` commit with git's default message and therefore no trailers. Rewriting it would have meant a force-push, so the work was cherry-picked onto `origin/main` instead. The resulting tree is byte-identical: `git rev-parse HEAD^{tree}` on both heads is `dbc2bb620d5a1b6da356a25218f1c850ea6fbbb1`, so the `ctest` run above is a run of exactly this tree. - `scripts/agent-preflight.sh`: **All gates green**, including `check-agent-record`, `commit-trailers` and `commit-style`. - New: `src/vt/cuda/cuda_iq_table_seal.h` declares the seal's copy. It is a gate instrument, called only from `tests/vt/test_cuda_quant_dot.cpp`, and it exists because a host translation unit cannot take the address of a `__device__` array. Named here rather than left to be discovered. Refs #1029, #912. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>

mudler added 2 commits August 16, 2026 00:49

localai-bot merged commit 10813df into main Aug 16, 2026
17 of 23 checks passed

localai-bot mentioned this pull request Aug 16, 2026

windows-msvc-cpu/vulkan red on every PR based on c7cb59fbb: C4244 double->float in ltx2_video.cpp's StreamState<->Ltx2LatentState positions copy (#964) #968

Closed

localai-bot deleted the row/ENG-EXPERT-STREAM-IQ1-CUDA branch August 16, 2026 01:10

localai-bot mentioned this pull request Aug 16, 2026

feat(MODEL-MUSIC-MUSIC3): a music-only server, an example that can be heard, and the five keys upstream refuses that we dropped (#672, #953, #965) #966

Merged

This was referenced Aug 21, 2026

feat(KERNEL-QUANT-CIQ-GEMM-ROCM): keep-quant GEMM providers on kROCM #1640

Closed

feat(KERNEL-QUANT-CIQ-GEMM-ROCM): keep-quant GEMM providers on kROCM ghazni101/vllm.cpp#1

Open

Navigation Menu

ENG-EXPERT-STREAM: the CUDA arm for IQ1_S and IQ1_XXXS, verified on GB10 - #967

ENG-EXPERT-STREAM: the CUDA arm for IQ1_S and IQ1_XXXS, verified on GB10#967
localai-bot merged 2 commits into
mainfrom
row/ENG-EXPERT-STREAM-IQ1-CUDA

localai-bot commented Aug 16, 2026

Uh oh!

localai-bot commented Aug 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

localai-bot commented Aug 16, 2026

Why this is not optional polish

Hardware result

The one bound that moved, and why

Design note

Evidence

Uh oh!

localai-bot commented Aug 16, 2026

Gated locally, CI saturated

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants