ENG-EXPERT-STREAM: decode the two encodings the Qwen3.8-2.4T checkpoints are 97% made of

.agents/issue-index.md

@@ -250,3 +250,4 @@ rather than merged. `scripts/check-agent-record.py` gates both.
 | [#873](https://github.com/mudler/vllm.cpp/issues/873) | `GATE-CI-CONCURRENCY` | `main` went RED on six release/registration gates after the #865 `ci.yml` rewrite: `check-release-binary-contract.py` and `check-test-registration.py` credit a checker to CI only through an UNCONDITIONAL job, and #865 gave `agent-record` an `if:`; the closed-PR skip is re-expressed through `needs:` and the byte-exact Windows PR proof schema restored (spec [`ci-concurrency.md`](specs/ci-concurrency.md)) | bug |
 | [#874](https://github.com/mudler/vllm.cpp/issues/874) | — | `windows-msvc-cpu`/`windows-msvc-vulkan` still start on a CLOSED pull request: `check-release-workflow.py::validate_pr_ci` compares their whole job mapping for equality, so neither an `if:` clause nor a `needs:` guard can be added. Listed under `## Owed` in [`ci-concurrency.md`](specs/ci-concurrency.md) | bug |
 | [#949](https://github.com/mudler/vllm.cpp/issues/949) | — | Nothing in the tree refuses a borrowed `vt::Tensor` that outlives the object owning its storage, and the ONLY instrument that catches one is `sanitize-cpu`, which is `continue-on-error` — that is how [#904](https://github.com/mudler/vllm.cpp/issues/904) landed. Measured in the #936 review rather than argued: with the #904 fix reverted, a plain Release build with no sanitizer runs the case 18/18 passed, 546 assertions, `rc=0`, because `dtype` lives in the `vt::Tensor` struct and not in the freed buffer, so no ordinary gate can see the dangling read. Three remedies are open and none is foregone: promote the lane once it has a `main` baseline, add a test that fails without a sanitizer, or reject the pattern statically — a prototype detector for a member access chained onto a call returning an owning type by value swept 1777 files with no hit but the defect. Anchors: the owning deleter `src/vllm/model_executor/models/ltx2_device.cpp:1088 @ 800dd082f`, the read `src/vt/cpu/cpu_layernorm.cpp:33 @ 800dd082f`. Listed under `## Owed` in [`ltx2-device-staged-view-uaf.md`](specs/ltx2-device-staged-view-uaf.md) | bug |
+| [#933](https://github.com/mudler/vllm.cpp/issues/933) | `ENG-EXPERT-STREAM` | Measure gateability of the `llama-cpp-unsloth` oracle by BUILDING it and RUNNING `Qwen3.8-2.4T-A95B UD-Q1_0` on it. The oracle is pinned at `36fe8e1cc` (branch `iq1-narrow`) and records `gateable = no`, because the IQ1_XXXS port is grounded in the fork's SOURCE, read and cited, which is weaker than a running comparison. It is the only place ggml type 66 is defined: the vllm.cpp pin `237ad9b96` ends at `Q1_0 = 41` and `ggml-org` master `ad1de39e0` at `Q2_0 = 42`, while type 66 carries 96.92 % of that checkpoint's parameters. Running it needs the full 370 GiB checkpoint and, per Unsloth's documentation, at least 450 GB of RAM. Until then the ported arm has no running oracle, which is what `gateable = no` makes visible | task |

.agents/oracles/llama-cpp-unsloth.md

@@ -0,0 +1,99 @@
+# Oracle: `unslothai/llama.cpp` (branch `iq1-narrow`)
+
+A FORK of `ggml-org/llama.cpp`, admitted for one narrow reason: it is the only
+place the sub-IQ1_S quant encodings are defined, and a published checkpoint this
+project must run uses one of them. It does not replace the `llama-cpp` oracle,
+and it never outranks vLLM.
+
+## Why a fork is admitted here
+
+`unsloth/Qwen3.8-2.4T-A95B-GGUF UD-Q1_0` stores its routed experts as ggml type
+**66**, which is about 97 % of that model's parameters. That type is in no
+upstream llama.cpp:
+
+| Tree | Revision | Highest type |
+|---|---|---|
+| vllm.cpp pin | `237ad9b96` | `GGML_TYPE_Q1_0 = 41` |
+| `ggml-org/llama.cpp` master, 2026-08-15 | `ad1de39e0` | `GGML_TYPE_Q2_0 = 42`, `COUNT = 43` |
+| this fork, branch `iq1-narrow` | `36fe8e1cc` | `GGML_TYPE_IQ1_XXXS = 66`, `COUNT = 67` |
+
+The fork declares three encodings below IQ1_S, all added by the branch head
+commit "IQ1_XS, IQ1_XXS, IQ1_XXXS: three quant types below IQ1_S" (2026-08-07):
+
+```
+GGML_TYPE_IQ1_XS   = 64, // 1.4375 bpw, 1024-entry grid
+GGML_TYPE_IQ1_XXS  = 65, // 1.3125 bpw,  512-entry grid
+GGML_TYPE_IQ1_XXXS = 66, // 1.1875 bpw,  256-entry grid
+```
+
+The identification is not taken on trust. The bits per weight were derived
+independently from the checkpoint's own GGUF offset deltas, before this fork was
+located, and came out at **1.1875**, matching the declaration exactly. The block
+layout confirms it a second way: `block_iq1_xxxs` is
+`{ ggml_half d; uint8_t qs[QK_K/8]; uint8_t sc[QK_K/64]; }`, so 2 + 32 + 4 = 38
+bytes per 256 elements, which is 1.1875 bpw.
+
+## Scope, and what this oracle may not do
+
+Use it ONLY for the sub-IQ1_S encodings it defines: block layout, codebook grid,
+scale and delta decode, and the reference `vec_dot` and `dequantize_row`. For
+everything else, including CPU and GGUF k-quant speed and memory floors, the
+oracle is [`llama-cpp`](llama-cpp.md) at its own pin.
+
+It is a mirror source for ONE encoding family, not a design reference. Where
+upstream llama.cpp or vLLM defines behavior, that behavior wins, exactly as
+`AGENTS.md` requires.
+
+## Gateability
+
+`gateable = no`, and issue #933 owes the measurement.
+
+`AGENTS.md` admits an oracle as gateable only once it demonstrably BUILDS and
+RUNS the model. Neither has been done here: the encoding was ported from the
+fork's SOURCE, read and cited, which is a different and weaker thing than a
+running comparison. Running it needs the full 370 GiB `UD-Q1_0` checkpoint and,
+per Unsloth's own documentation, at least 450 GB of RAM.
+
+So the ported arm has no running oracle to compare against yet. That is visible
+debt rather than a gap to be discovered later, which is what `gateable = no` is
+for.
+
+What HAS been done, so the gap is not overstated: the ported decoder was run on
+real `UD-Q1_0` bytes and compared against an independent transcription of the
+fork's own `dequantize_row_iq1_xxxs`, bit-identical over 1179648 weights across
+three tensor roles, two layers and two shards. See the census section of
+[`.agents/specs/expert-streaming.md`](../specs/expert-streaming.md). That removes
+transcription error from our side. It cannot detect a defect in the FORK itself,
+because both sides read the same source, which is exactly the residue #933 owes.
+
+## Pin
+
+```oracle-pin
+id = llama-cpp-unsloth
+role = secondary
+scope = the sub-IQ1_S quant encodings IQ1_XS (64), IQ1_XXS (65) and IQ1_XXXS (66), which no upstream llama.cpp defines
+upstream = https://github.com/unslothai/llama.cpp
+pin = 36fe8e1cc7f2b3b8c92fdda0ab07600141921786
+pin_label = iq1-narrow
+pinned_on = 2026-08-15
+gateable = no
+evidence = #933
+```
+
+## Anchors used by the port
+
+Read at the pin above. Re-verify before relying on them, because a fork branch
+can be rebased under a name.
+
+| Piece | Anchor |
+|---|---|
+| type ids 64, 65, 66 | `ggml/include/ggml.h` enum `ggml_type` |
+| `block_iq1_xxxs`, 38 bytes | `ggml/src/ggml-common.h:478-483` |
+| `NGRID_IQ1XXXS = 256` | `ggml/src/ggml-common.h:1181` |
+| `iq1_xxxs_grid`, uint64, 256 | `ggml/src/ggml-common.h:2095` |
+| `iq1_xxxs_grid_gpu`, uint32, 256 | `ggml/src/ggml-common.h:2620` |
+| `ggml_vec_dot_iq1_xxxs_q8_K_generic` | `ggml/src/ggml-cpu/quants.c:1281` |
+| `dequantize_row_iq1_xxxs` | `ggml/src/ggml-quants.c:2727` |
+
+The delta constant is upstream's own `IQ1S_DELTA` (0.125), reused unchanged by
+the fork rather than redefined.

.agents/specs/expert-streaming.md

@@ -434,6 +434,147 @@ corrects the record so the next person to cost that leaf starts from what is
 true. Re-verify these anchors before implementing: they were re-derived at HEAD
 on 2026-08-14 and this spec has already been wrong about them once.
 
+## Target checkpoint census and the quant encodings it needs (2026-08-15, issue #912)
+
+The GGUF reconciliation note above establishes that the GGUF lane needs no bank
+and only a residency policy. It did not ask a prior question: **can this tree
+decode the bytes of the checkpoint the row exists to run?** Measuring that
+changed the target.
+
+Every published GGUF of `unsloth/Qwen3.8-2.4T-A95B-GGUF` at revision
+`567d3e6ac26c5474b18311e619c04350fb9a5556` was censused by parsing tensor
+headers directly (HTTP range requests, no full download). Coverage is total, not
+sampled: **1702 tensor records parsed against 1702 declared in
+`split.tensors.count`.**
+
+`UD-Q1_0` (370 GiB), the smallest quant and the one first chosen for this row,
+is **not decodable by any admissible oracle** and was rejected:
+
+- Its expert tensors carry ggml type **66**, holding ~97 % of all parameters.
+- Type 66 exists in neither the pinned llama.cpp (`237ad9b96`, whose enum ends
+  at `GGML_TYPE_Q1_0 = 41`) nor upstream `ggml-org/llama.cpp` master
+  (`ad1de39e0`, 2026-08-15, ends at `GGML_TYPE_Q2_0 = 42`, `COUNT = 43`).
+- It is defined only on the fork branch `unslothai/llama.cpp @ iq1-narrow` as
+  `GGML_TYPE_IQ1_XXXS = 66, // 1.1875 bpw, 256-entry grid`. The bits-per-weight
+  derived independently here from GGUF offset deltas is **1.1875**, matching that
+  declaration exactly, which is what confirms the identification.
+
+**That refusal was overruled by developer direction on 15 August 2026, and the
+fork is now pinned.** The refusal rested on a policy premise, no upstream
+definition and no pinned oracle, rather than on a technical one. The fork is
+public and pinnable, so the developer directed anchoring the encoding to it, and
+that resolves the objection at its root: the encoding now HAS a recorded
+upstream and a fixed revision to cite.
+
+`unslothai/llama.cpp` is therefore admitted to the oracle table as
+`llama-cpp-unsloth`, pinned at `36fe8e1cc` on branch `iq1-narrow` and scoped to
+the sub-IQ1_S encodings alone. It records `gateable = no`, because the port was
+grounded in the fork's SOURCE rather than in a running comparison, and issue
+#933 owes the build-and-run measurement. See
+[`.agents/oracles/llama-cpp-unsloth.md`](../oracles/llama-cpp-unsloth.md).
+
+Both checkpoints are therefore targets, and the order is UD-Q1_0 first per
+developer direction, then UD-IQ1_S. The two are structurally IDENTICAL, censused
+the same way over all shards, 1702 records against 1702 declared in each: same
+276 expert tensors, same 96.92 %, same six other encodings with the same counts.
+Only the expert encoding differs, ggml 66 against ggml 19. So the same streaming
+lane serves both, and each needed exactly one new encoding.
+
+The census is the scope statement for both targets, `UD-Q1_0` (370 GiB) and
+`UD-IQ1_S` (473 GiB). Support status is read from the CODE rather than from
+comments (`BlockDTypeFromGgmlTypeId` in `src/vt/dtype.cpp`, `BlockVecDot` in
+`src/vt/cpu/cpu_quant_dot.cpp`, and the `DType` enum itself). That distinction
+earned itself: a stale header comment listing six encodings led this section to
+record Q2_K as missing, and reading the dispatch showed it has been served all
+along.
+
+Both censuses, side by side. The `UD-Q1_0` row set is identical apart from the
+expert encoding:
+
+| Encoding | ggml id | Tensors | Gparams | % params | Status |
+|---|---|---|---|---|---|
+| IQ1_XXXS (`UD-Q1_0` experts) | 66 | 276 | 2370.8 | **96.92** | added, fork-anchored |
+| IQ1_S (`UD-IQ1_S` experts) | 19 | 276 | 2370.8 | **96.92** | added |
+| Q5_K | 13 | 420 | 34.0 | 1.39 | served (`VecDotQ5_KQ8_K`) |
+| Q2_K | 10 | 3 | 25.8 | 1.05 | served (`VecDotQ2_KQ8_K`) |
+| Q6_K | 14 | 162 | 10.8 | 0.44 | served (`VecDotQ6_KQ8_K`) |
+| Q4_K | 12 | 2 | 4.1 | 0.17 | served (`VecDotQ4_KQ8_K`) |
+| Q8_0 | 8 | 1 | 0.1 | 0.01 | served |
+| F32 | 0 | 838 | 0.5 | 0.02 | served (not keep-quant) |
+
+**Each checkpoint was missing exactly ONE encoding, and it was 96.92 % of the
+model.** In both, the expert tensors are precisely the 92 non-MTP layers times
+three expert matrices (`ffn_down_exps`, `ffn_gate_exps`, `ffn_up_exps`), and the
+Q2_K trio is block 92, the `nextn` MTP layer. So one encoding per checkpoint
+makes a complete 92-layer forward pass loadable, and nothing else gates the
+first benchmark.
+
+IQ1_S is an ordinary upstream k-quant at the existing pin, and this tree already
+carries the whole grid-table pattern it needs, built for DeepSeek-V4 under
+`.agents/specs/gguf-iquant-dsv4.md`: `kIQ2_XXS`, `kIQ3_XXS`, `kIQ2_S` and
+`kMXFP4` each have a keep-quant `vec_dot` against the Q8_K activation encoding,
+CPU grids in `src/vt/cpu/cpu_quant_iq_tables.h` and device grids in
+`src/vt/cuda/cuda_quant_iq_tables.cuh`. IQ1_S adds one more row on those rails.
+
+Upstream anchors, re-derived at the pin `237ad9b96` (re-verify before
+implementing, per the reconciliation note's own warning):
+
+| Piece | Anchor |
+|---|---|
+| `block_iq1_s` (`ggml_half d; uint8_t qs[32]; uint16_t qh[8]`) | `ggml/src/ggml-common.h:414-419` |
+| block size assert, 50 bytes per 256 elements = 1.5625 bpw | `ggml/src/ggml-common.h:420` |
+| `ggml_vec_dot_iq1_s_q8_K_generic` | `ggml/src/ggml-cpu/quants.c:1099` |
+| `iq1s_grid`, uint64, `NGRID_IQ1S` = 2048 | `ggml/src/ggml-common.h:1124` |
+| `iq1s_grid_gpu`, uint32, 2048 | `ggml/src/ggml-common.h:1639` |
+
+### The IQ1_XXXS decode was checked against REAL checkpoint bytes
+
+Synthetic random blocks sweep bit patterns, but they cannot catch a
+misunderstanding of the FILE: a wrong field order, a wrong stride, or a codebook
+that turns real weights into legal-looking noise. So the decode was also run on
+the downloaded `UD-Q1_0` shards, 15 August 2026.
+
+Two independent things were compared on the SAME bytes: this tree's
+`DequantIQ1_XXXS`, and a separate transcription of the fork's
+`dequantize_row_iq1_xxxs` reading the fork's grid directly out of its own tree.
+
+| Tensor | Layer | K | Weights | Result |
+|---|---|---|---|---|
+| `ffn_gate_exps` | 0 | 8192 | 524288 | bit-identical |
+| `ffn_up_exps` | 0, 23 | 8192 | 262144 | bit-identical |
+| `ffn_down_exps` | 0, 23 | 2048 | 65536 | bit-identical |
+
+**1179648 real weights, max absolute difference 0.0.** The decoded values also
+look like weights rather than noise: mean -4.2e-7, sd 8.6e-4, symmetric tails,
+24 discrete levels, and no non-finite value. The file layout resolves exactly as
+the port assumes, `row_bytes = K/256*38`, and the implied tensor size of
+1275068416 bytes matches 512 experts times 2048x8192 at 1.1875 bpw.
+
+Be precise about what this does and does not establish. It removes transcription
+error from OUR C++, which is the failure this port was most exposed to, and it
+proves the reader addresses the real file correctly. It does NOT make the fork
+oracle gateable: both sides are transcriptions of the same source, so a defect in
+the FORK would be reproduced identically by both. Only building and running the
+fork closes that, which is what #933 owes.
+
+Why this is in this spec rather than its own row: the encoding is the load
+path's half of the same capability. A streamer that can address an expert slice
+it cannot decode moves bytes for nothing, so the row's own gate cannot be met
+without it. It is nonetheless useful beyond this row, since any low-bit GGUF
+gains it, which is the standing k-quant obligation in `AGENTS.md` rather than a
+detour for one model.
+
+This note does not re-scope W1-W7 or claim a date. It adds the leaves below and
+records why the target checkpoint changed.
+
+| Leaf | Scope | Depends on | Gate slice |
+|---|---|---|---|
+| W9 IQ1_S decode | `DType::kIQ1_S`, geometry `{256, 50, 19}`, `BlockDTypeFromGgmlTypeId` row, CPU `iq1s_grid` + `VecDotIQ1_SQ8_K` traits row, dequant path | - | CPU tests green against upstream-derived vectors; `KeepQuantDType(19)` true |
+| W10 IQ1_S device decode | `iq1s_grid_gpu` + CUDA `vec_dot`, so streamed slices dot on device. NOT optional polish: `cuda_quant_dot.cu:1531` maps an unknown weight dtype to `return false`, which is a SILENT CPU fallback. Tokens would still be correct, so no token gate can see it, and a 2.4 T model would simply run at CPU speed while looking healthy | W9 | CUDA tests; parity with the CPU arm; the fallback must be observable rather than inferred |
+| W11 checkpoint load | UD-Q1_0 first, then UD-IQ1_S, loads end to end on dgx.casa, refusing any encoding it cannot honour by name | W9, W10, W12 | model loads; token output captured |
+| W12 IQ1_XXXS decode | `DType::kIQ1_XXXS`, geometry `{256, 38, 66}`, the 256-entry fork codebook, `VecDotIQ1_XXXSQ8_K`, dequant, traits row and the reader's `case 66`. Grounded in the pinned fork oracle, cited per site | - | CPU tests green; `KeepQuantDType(66)` true; grid digest sealed |
+| W13 IQ1_XXXS device decode | the CUDA arm, for the same silent-fallback reason as W10 | W12 | CUDA tests; parity with the CPU arm |
+
 ## Risks/decisions
 
 | Risk / decision | Call |
@@ -443,6 +584,9 @@ on 2026-08-14 and this spec has already been wrong about them once.
 | Uniform-routing h≈f may undershoot G3 | W5 measures h(f), but G3 stays fixed at f=0.5/12 tok/s. A miss is an open gap; changing the fraction is a new recorded gate, never silent rebasing |
 | Expert bank = second copy of expert bytes on disk (~17 GiB) | Accepted: one-time build, keyed+versioned, evictable file; alternative (per-miss repack kernel) taxes every miss forever |
 | GGUF checkpoints (APEX 35B, and every Qwen3.8-class checkpoint that exists) not covered by W1-W6 | Still out of scope for W1-W6, but the stated blocker is GONE: the slicer landed 2026-07-22. What remains is the residency policy, and the GGUF lane needs NO bank at all (see the reconciliation note) |
+| Target checkpoint carries an encoding no UPSTREAM oracle defines (`UD-Q1_0`, ggml type 66) | First refused, then admitted on developer direction (15 August 2026) by pinning the fork as `llama-cpp-unsloth` at `36fe8e1cc`. The objection was policy, not feasibility, and a recorded pin answers it. The cost is honest and recorded rather than hidden: the pin is a BRANCH, which can be rebased under its own name, so the ported grid carries a digest seal, and `gateable = no` with issue #933 owing the build-and-run measurement. The oracle is scoped to the sub-IQ1_S encodings only and never outranks vLLM or upstream llama.cpp |
+| IQ1_S at 1.5625 bpw may cost accuracy versus the 1.1875 bpw quant originally chosen | Accepted, and it moves the other way too: IQ1_S is the HIGHER-fidelity encoding of the two. The cost is 473 GiB rather than 370 GiB on disk, which the 3.0 TB free on dgx.casa absorbs. Quality is not asserted here; the row's gate is token output against an oracle, and any accuracy claim needs its own measurement |
+| One encoding is 96.92 % of the target model | This is why W9 blocks W11 and why no partial-decode fallback is offered. A model that expands its experts to bf16 to avoid IQ1_S would need multiple TB of memory, so "unsupported encoding" here means refusal by name, never a silent widening |
 | tmpfs "tier" temptation on GB10 | Rejected with reasons (§Scope verdict): tmpfs is the same unified memory; documented so it is not re-proposed |
 | CUDA graphs vs data-dependent miss handling | Phase 1 explicitly disables graphs. Current Marlin has no address table; graph compatibility would require a separately spiked kernel/dispatch change after W3 profiling |
 | Full-layer prefill with C<E | A single unmodified Marlin launch cannot address experts absent from slots. W4 uses exact chunk filters + scatter accumulation and proves every routed pair once; it may not allocate a hidden E-sized buffer |

AGENTS.md

@@ -241,6 +241,7 @@ only when it appears in this table and has a recorded pin:
 | SGLang | `sglang` | a model or serving path that SGLang implements and vLLM does not |
 | SGLang-Omni | `sglang-omni` | omni, speech, TTS, and music models served by SGLang's pipeline runtime, in a third repository that is not SGLang |
 | llama.cpp | `llama-cpp` | CPU and GGUF k-quant floors |
+| `unslothai/llama.cpp` fork | `llama-cpp-unsloth` | the sub-IQ1_S quant encodings (IQ1_XS, IQ1_XXS, IQ1_XXXS) that no upstream llama.cpp defines, and that a published Qwen3.8-2.4T checkpoint stores its experts in |
 | Tenstorrent tt-forge | `tt-forge` | Tenstorrent hardware, for which vLLM has no backend |
 
 <!-- oracle-registry:end -->

include/vt/dtype.h

@@ -75,6 +75,8 @@ enum class DType : uint8_t {
   kIQ2_XXS,
   kIQ3_XXS,
   kIQ2_S,
+  kIQ1_S,
+  kIQ1_XXXS,
   kMXFP4,
 };