From 225a0ec344c209245a3d21cb3bd3e6f8da81639d Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 16 Aug 2026 11:33:46 +0000 Subject: [PATCH] fix(ENG-EXPERT-STREAM): seal the IQ1 decode against the oracles, not against itself The IQ1_S / IQ1_XXXS decode landed in #946 with its codebooks sealed and every other decode parameter pinned only by self-consistency. `ReferenceDotF64` and the G3 NMSE reference both decode the weight with `vt::cpu::BlockToFloat`, the function under test, so they are independent in the summation and nowhere else. An independent review injected three defects, each applied and compiled, and the whole gate stayed green with an unchanged assertion count: `kIq1sDelta` doubled to 0.25 (which hits both encodings), the IQ1_S delta sign inverted in dequant and vec_dot together, and the IQ1_S scale read from `qh` bits 13-15 instead of 12-14 in both paths. All three are reproduced here before the repair and all three now fail. What closes it is a golden-vector fixture whose expected values come from the oracles themselves rather than from this tree. Each blob is decoded by `ggml_get_type_traits(type)->to_float` in a build of ggml-org/llama.cpp @ 237ad9b96 for IQ1_S and IQ3_XXS, and of unslothai/llama.cpp @ 36fe8e1cc for IQ1_XXXS, and the IQ1 inputs are real `blk.0.ffn_gate_exps.weight` bytes from the two target checkpoints. Agreement is bit-exact on all 1024 values per encoding. That is stronger than the spec's 15 August run, which compared our C++ against a hand transcription of the fork: both sides were transcriptions there, so a defect in the fork would have been reproduced identically by each. The NMSE ceiling was 2e-3, described as about 4x the residual. Re-measured over all 12 shapes per type, the unmutated peak is 5.240e-4 for iq1_s and 3.109e-4 for iq1_xxxs, and the doubled delta takes iq1_s to 6.967e-4. So 2e-3 passed a defect that 6e-4 fails, and the ceiling is now set from that measurement. The iq1_xxxs half is the more instructive one: the same defect moves its NMSE the wrong way, to 1.420e-4, so no ceiling catches it there. A statistic whose reference decodes through the function under test cannot seal a decode parameter at all. It bounds quantization error, which is its own job. `DequantGgufRowToF32` listed no case 19 and no case 66, so the loader's expansion path threw "unsupported ggml type" for the two encodings the target checkpoints are 96.92 % made of. That is not a corner: `RouteGgufTensor` sends a tensor there whenever VT_CPU_REF is set, keep-quant is off, K is ragged, or the role is not verbatim, so the reference lane could not load the model. Case 18 (IQ3_XXS) carried the same omission since the DeepSeek-V4 port and is fixed in the same shared branch, gated on its own oracle-produced golden. Three smaller repairs from the same review. The IQ1_S vec_dot comment claimed the other kernels do not read `bsums` when Q2_K, Q4_K and Q5_K do. The i-quant table header called itself a 1:1 mirror of one revision while carrying one table from a different fork, and indexed neither new table. Both checkpoint-census cases claimed total coverage of 1702 tensor records while enumerating six of the seven encodings and summing to 864; they now carry F32's 838 tensors and assert that the buckets sum, which is what makes the claim checkable. Closes #1023. Row ENG-EXPERT-STREAM, issue #912. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/issue-index.md | 1 + .agents/specs/expert-streaming.md | 64 +++ .../model_loader/gguf_dequant.cpp | 11 + src/vt/cpu/cpu_quant_dot.cpp | 13 +- src/vt/cpu/cpu_quant_iq_tables.h | 12 +- tests/vllm/test_gguf_dequant.cpp | 55 ++ tests/vllm/test_gguf_keep_quant.cpp | 108 ++-- tests/vt/iq1_golden_vectors.h | 499 ++++++++++++++++++ tests/vt/test_ops_quant_dot.cpp | 125 ++++- 9 files changed, 827 insertions(+), 61 deletions(-) create mode 100644 tests/vt/iq1_golden_vectors.h diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 5e44e992f..1897927cf 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -270,3 +270,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#995](https://github.com/mudler/vllm.cpp/issues/995) | `ENG-EXPERT-STREAM` | `check-env-doc` and `test_check_env_doc` RED in `scripts/agent-preflight.sh` on **every** branch based on `3005447f8`: `VT_MOE_EXPERT_STREAM`, `VT_MOE_EXPERT_STREAM_SLOTS` and `VT_MOE_EXPERT_STREAM_SLOT_BYTES` are read from `src/vllm/model_executor/models/qwen3_5.cpp` (`:5145`, `:5164` @ `3005447f8`) and appear in neither `docs/ENVIRONMENT.md` (`grep -c` returns **0**) nor `scripts/env-doc-allowlist.txt`. Arrived with [#993](https://github.com/mudler/vllm.cpp/pull/993). Found while gating [#986](https://github.com/mudler/vllm.cpp/issues/986) and proved pre-existing with a matched-arm check rather than asserted: the three sites are in a file that branch does not touch, and `git diff origin/main...HEAD | grep '^+.*VT_MOE_EXPERT'` returns nothing. NOT fixed in flow: documenting a knob means stating its default and when to touch it, and the expert-streamer's slot accounting belongs to the row that added it, so a plausible-sounding entry written by a passer-by is how `docs/ENVIRONMENT.md` stops being trustworthy. It is a PRE-FLIGHT gate, so it fails before every edit and presents to each author as a red their own diff caused, which is the shape [#965](https://github.com/mudler/vllm.cpp/issues/965) and [#968](https://github.com/mudler/vllm.cpp/issues/968) both took | bug | | [#987](https://github.com/mudler/vllm.cpp/issues/987) | `LTX25-RETAKE` | Two `ltx-2.5` refusal messages state reasons that are no longer true. (a) `src/vllm/multimodal/ltx2_video.cpp:1608 @ 0e1bee42f` says "nothing reads `ref_video_dir` at all", and MiniMax-H3 has always consumed the directory in full (`ReadReferenceClipChw`, `src/vllm/multimodal/minimax_h3_video.cpp:135 @ 0e1bee42f`, called at `:650`); [#975](https://github.com/mudler/vllm.cpp/issues/975) inherited the wider claim from this message. The claim that holds is narrower: the LTX-2.5 engine never reads the directory's CONTENTS. (b) `ltx2_video.cpp:1636-1638 @ 0e1bee42f` says "there is no AUDIO_VAE_ENCODER key filter", and `c2019b0e3` landed `Ltx2AudioVaeEncoderKeyRules()` (`include/vllm/model_executor/models/ltx2_audio_input.h:73 @ 0e1bee42f`) with a live call through `Ltx2EncodeAudioToLatent`. Both rewritten in the `WHAT IS *NOT* THE REASON` shape in the same flow, with one assertion tied to the LOCAL fact that the LTX side now reads the directory | bug | | [#995](https://github.com/mudler/vllm.cpp/issues/995) | `ENG-EXPERT-STREAM` | `check-env-doc` and `test_check_env_doc` are RED on `origin/main`, so every branch cut from it inherits a preflight failure its own diff did not cause: `VT_MOE_EXPERT_STREAM`, `VT_MOE_EXPERT_STREAM_SLOTS` and `VT_MOE_EXPERT_STREAM_SLOT_BYTES` are read from `src/vllm/model_executor/models/qwen3_5.cpp` (`:5145`, `:5195`, `:5189` @ `4496ef196`) and appear in neither `docs/ENVIRONMENT.md` nor `scripts/env-doc-allowlist.txt`. They arrived with the `ENG-EXPERT-STREAM` W4 wiring commit `3005447f8` ([#993](https://github.com/mudler/vllm.cpp/pull/993)). Same shape as [#965](https://github.com/mudler/vllm.cpp/issues/965)/[#968](https://github.com/mudler/vllm.cpp/issues/968): a pre-existing red presenting to each author in turn as their own. Fixed by DOCUMENTING all three in `docs/ENVIRONMENT.md`, not by allowlisting them — a knob that changes the host memory a MoE decode reserves and disables the default-on grouped-MoE path is a deployment surface, not a kernel-internal micro-tuning switch | bug +| [#1023](https://github.com/mudler/vllm.cpp/issues/1023) | `ENG-EXPERT-STREAM` | The IQ1_S (ggml 19) / IQ1_XXXS (ggml 66) decode landed in [#946](https://github.com/mudler/vllm.cpp/pull/946) with every parameter EXCEPT the grid pinned only by self-consistency. The grid seal (FNV-1a digest + lane census) works and stops one table short: `ReferenceDotF64` (`tests/vt/test_ops_quant_dot.cpp:231-236`) and the G3 NMSE reference (`:915`) both decode the weight with `vt::cpu::BlockToFloat`, the function under test, so they are independent only in the SUMMATION. Three injected defects, each applied and compiled, left the suite green with an UNCHANGED assertion count: `kIq1sDelta` `0.125F`->`0.25F` (`cpu_quant_iq_tables.h:422`, affects BOTH encodings), the IQ1_S delta sign inverted in dequant AND vec_dot, and the IQ1_S scale read from `qh` bits 13-15 instead of 12-14 in both paths. Second defect, same PR: `gguf_dequant.cpp:107-116` lists no `case 19` and no `case 66`, so the expansion path throws `unsupported ggml type` for the two encodings the target checkpoints are 96.92 % made of, and `RouteGgufTensor` (`gguf_keep_quant.cpp:122`) sends a tensor there whenever `VT_CPU_REF` is on, keep-quant is off, K is ragged, or the role is not verbatim — a refusal to load on the reference lane. **ggml 18 (IQ3_XXS) carried the same omission**, pre-existing since the DeepSeek-V4 UD-IQ2_XXS port, and all three are one shared `switch` branch. Repaired by golden vectors whose EXPECTED values come from the ORACLES themselves (`ggml_get_type_traits(type)->to_float` built from `ggml-org/llama.cpp @ 237ad9b96` for 18/19 and `unslothai/llama.cpp @ 36fe8e1cc` for 66) over REAL checkpoint bytes, which is the first reference for this encoding family that is not this tree's own decoder. Also: the `2e-3` NMSE ceiling passed the doubled-delta defect that `6e-4` fails (measured 5.240e-4 unmutated, 6.967e-4 mutated on `iq1_s`), and on `iq1_xxxs` that defect moves the statistic the WRONG WAY (3.109e-4 -> 1.420e-4), so no ceiling can catch it and only the goldens can; and both new census cases claimed TOTAL coverage of 1702 records while summing to 864, F32's 838 tensors omitted. Spec [`expert-streaming.md`](specs/expert-streaming.md) | bug | diff --git a/.agents/specs/expert-streaming.md b/.agents/specs/expert-streaming.md index 5c808bdea..860418b99 100644 --- a/.agents/specs/expert-streaming.md +++ b/.agents/specs/expert-streaming.md @@ -557,6 +557,70 @@ oracle gateable: both sides are transcriptions of the same source, so a defect i the FORK would be reproduced identically by both. Only building and running the fork closes that, which is what #933 owes. +### The decode is now sealed against the ORACLES, not against itself (#1023) + +An independent review of #946 found that this seal covered the CODEBOOKS and +nothing else. `kIq1sGrid` and `kIq1xxxsGrid` each carry an FNV-1a digest and a +lane census, and corrupting a grid entry does fail. Every other decode parameter +was pinned only by self-consistency, because `ReferenceDotF64` and the G3 +`MatmulBTQuant` NMSE reference both decode the weight with +`vt::cpu::BlockToFloat`, the function under test. They are independent in the +SUMMATION and nowhere else. Three defects were injected, each applied and +compiled, and the whole gate stayed green with an unchanged assertion count: + +| Mutation | Before #1023 | After | +|---|---|---| +| `kIq1sDelta` `0.125F` to `0.25F`, which hits BOTH encodings | UNCAUGHT | 3 cases, 2049 assertions fail | +| IQ1_S delta sign inverted in dequant AND vec_dot | UNCAUGHT | 1 case, 1024 assertions fail | +| IQ1_S scale from `qh` bits 13-15 instead of 12-14, both paths | UNCAUGHT | 1 case, 960 assertions fail | + +What closes it is a committed golden-vector fixture, +`tests/vt/iq1_golden_vectors.h`, whose EXPECTED values are produced by the +oracles themselves rather than by this tree: +`ggml_get_type_traits(type)->to_float`, called in a build of +`ggml-org/llama.cpp @ 237ad9b96` for IQ1_S and IQ3_XXS, and of +`unslothai/llama.cpp @ 36fe8e1cc7f2b3b8c92fdda0ab07600141921786` for IQ1_XXXS. +The IQ1 inputs are real `blk.0.ffn_gate_exps.weight` bytes, four 256-element +blocks from shard 2 of each split, so this commits a reproducible slice of the +1179648-weight run above. Agreement is BIT-EXACT on all 1024 values per +encoding, and `kIq1sDelta` is additionally sealed by value against upstream +`IQ1S_DELTA` (`ggml-common.h:1121`). + +This is stronger than the 15 August run, and it is worth saying how. That run +compared our C++ against a hand transcription of the fork, so BOTH sides were +transcriptions and a defect in the FORK would have been reproduced identically +by each. These vectors are decoded by the fork's own compiled code. That still +does not make the fork gateable in the sense #933 owes, which is running the +MODEL, but the fork's decoder is no longer transcribed at all. + +The same review found the NMSE ceiling widened past the point where it +discriminates. It was `2e-3`, described as about 4x the residual. Re-measured 16 +August 2026 over all 12 shapes per type, with the ceiling forced to `1e-12` so +doctest prints every captured value: + +| Type | Unmutated max | With `kIq1sDelta = 0.25` | +|---|---|---| +| `iq1_s` | 5.240e-4 (m=4, n=1) | 6.967e-4 | +| `iq1_xxxs` | 3.109e-4 (m=1, n=1) | 1.420e-4 | + +So `2e-3` passed that defect and `6e-4`, the value now set, fails it. The +`iq1_xxxs` column is the more useful half: the defect moves that statistic the +WRONG WAY, so NO ceiling catches it there. An NMSE against a dequant-f32 +reference cannot seal a decode parameter at all when both sides decode through +the same function. The ceiling bounds quantization error, which is its own job; +the goldens carry the decode. + +Two further findings, repaired in the same flow. `DequantGgufRowToF32` listed no +`case 19` and no `case 66`, so the expansion path threw `unsupported ggml type` +for the two encodings the target checkpoints are 96.92 % made of, and +`RouteGgufTensor` routes a tensor there whenever `VT_CPU_REF` is set, keep-quant +is off, K is ragged, or the role is not verbatim: a refusal to load on the +reference lane. `case 18` (IQ3_XXS, the DeepSeek-V4 UD-IQ2_XXS `ffn_down` +encoding) carried the same omission and is fixed with it, gated on its own +oracle-produced golden. And both checkpoint-census cases claimed TOTAL coverage +of 1702 tensor records while enumerating six of the seven encodings and summing +to 864; they now carry F32's 838 tensors and assert that the buckets sum. + Why this is in this spec rather than its own row: the encoding is the load path's half of the same capability. A streamer that can address an expert slice it cannot decode moves bytes for nothing, so the row's own gate cannot be met diff --git a/src/vllm/model_executor/model_loader/gguf_dequant.cpp b/src/vllm/model_executor/model_loader/gguf_dequant.cpp index 0a8bd4eb1..e8d5423f0 100644 --- a/src/vllm/model_executor/model_loader/gguf_dequant.cpp +++ b/src/vllm/model_executor/model_loader/gguf_dequant.cpp @@ -113,11 +113,22 @@ std::vector DequantGgufRowToF32(uint32_t ggml_type, const uint8_t* data, case 14: // Q6_K case 22: // IQ2_S (~2.5-bit codebook; UD-IQ2_M ffn_gate/up experts) case 39: // MXFP4 (OCP micro-scaling fp4; UD-IQ2_M ffn_down experts) + case 18: // IQ3_XXS (~3-bit codebook; UD-IQ2_XXS ffn_down) + case 19: // IQ1_S (1.5625 bpw; Qwen3.8-2.4T-A95B UD-IQ1_S experts) + case 66: // IQ1_XXXS (1.1875 bpw; UD-Q1_0 experts, fork-anchored) case 16: { // IQ2_XXS (~2-bit codebook; UD-IQ2_XXS DeepSeek-V4 vehicle) // The block decoders moved to vt (src/vt/cpu/cpu_quant_dequant.cpp) so // the loader oracle and the compute-in-quant GEMM's generic fallback // share ONE implementation. The code is byte-identical to what lived // here, so numerics are unchanged; this test suite gates that. + // This list is the loader's EXPANSION path, and it has drifted behind + // `BlockDTypeFromGgmlTypeId` twice: 18, 19 and 66 were all decodable in + // vt while this switch still threw "unsupported ggml type" for them. The + // drift is not a corner case, because `RouteGgufTensor` routes a tensor + // here whenever the VT_CPU_REF oracle switch is on, keep-quant is off, K + // is ragged, or the role is not a verbatim one. A missing case is + // therefore a refusal to load the checkpoint on the reference lane. + // Keep this list in step with vt/dtype.cpp `kBlockDTypes`. // NOTE: MXFP4 (39) decodes here through the vt kMXFP4 block dtype — the // GGML micro-scaling form (kvalues_mxfp4 == 2*e2m1, scale 2^(byte-128)) — // NOT through the compressed-tensors E8M0ToF32 path used for NVFP4 (40) diff --git a/src/vt/cpu/cpu_quant_dot.cpp b/src/vt/cpu/cpu_quant_dot.cpp index e50aba275..82ca1cb02 100644 --- a/src/vt/cpu/cpu_quant_dot.cpp +++ b/src/vt/cpu/cpu_quant_dot.cpp @@ -588,11 +588,14 @@ void VecDotIQ3_XXSQ8_K(int n, float* s, size_t bs, const void* vx, size_t bx, // bits from `qh[ib]`; kIq1sGrid entries are packed TERNARY (-1/0/+1) bytes, so // there is no sign array to apply. // -// The delta term is why this kernel reads `bsums` and the others do not. IQ1_S -// reconstructs a weight as dl*(grid[j] + delta), so the dot splits into -// sum(dl*grid*q8) plus delta*dl*sum(q8), and that second sum over each group of -// 16 is exactly what Q8_K already caches in `bsums`. Recomputing it from `qs` -// would be arithmetically identical but would read the activation twice. +// The delta term is why this kernel reads `bsums`. IQ1_S reconstructs a weight +// as dl*(grid[j] + delta), so the dot splits into sum(dl*grid*q8) plus +// delta*dl*sum(q8), and that second sum over each group of 16 is exactly what +// Q8_K already caches in `bsums`. Recomputing it from `qs` would be +// arithmetically identical but would read the activation twice. +// +// The reason is specific to the delta; the FIELD is not. Q2_K, Q4_K and Q5_K +// read `bsums` too, for their own per-sub-block minimum term. void VecDotIQ1_SQ8_K(int n, float* s, size_t bs, const void* vx, size_t bx, const void* vy, size_t by, int nrc) { VT_CHECK(n % kQK_K == 0, "vec_dot_iq1_s_q8_K: n must be a multiple of 256"); diff --git a/src/vt/cpu/cpu_quant_iq_tables.h b/src/vt/cpu/cpu_quant_iq_tables.h index 9965cf341..0309d33f0 100644 --- a/src/vt/cpu/cpu_quant_iq_tables.h +++ b/src/vt/cpu/cpu_quant_iq_tables.h @@ -1,8 +1,16 @@ -// I-quant codebook + sign tables — a 1:1 mirror of llama.cpp @ 237ad9b96 -// `ggml/src/ggml-common.h`: +// I-quant codebook + sign tables. Every table here but one mirrors llama.cpp +// @ 237ad9b96 `ggml/src/ggml-common.h` 1:1: // :499 kmask_iq2xs (8) :503 ksigns_iq2xs (128) // :550 iq2xxs_grid (256) :1007 iq3xxs_grid (256) // :748 iq2s_grid (1024) :1116 kvalues_mxfp4 (16) +// :1121 IQ1S_DELTA :1124 iq1s_grid (2048) +// +// The exception is `kIq1xxxsGrid` (256), which comes from a DIFFERENT tree: +// the pinned fork oracle unslothai/llama.cpp @ 36fe8e1cc, +// `ggml-common.h:2095 iq1_xxxs_grid`, because ggml type 66 is declared by no +// upstream llama.cpp. See .agents/oracles/llama-cpp-unsloth.md and the note +// above that table. IQ1_XXXS reuses upstream's own IQ1S_DELTA unchanged, so the +// one delta constant serves both IQ1 encodings. // // These are the shared decode tables for the IQ2_XXS / IQ3_XXS / IQ2_S / MXFP4 // block encodings. They were originally private to cpu_quant_dequant.cpp (the diff --git a/tests/vllm/test_gguf_dequant.cpp b/tests/vllm/test_gguf_dequant.cpp index 6e5e0e5d0..6dc0020b0 100644 --- a/tests/vllm/test_gguf_dequant.cpp +++ b/tests/vllm/test_gguf_dequant.cpp @@ -2,6 +2,7 @@ #include #include +#include #include #include "vllm/model_executor/model_loader/gguf_dequant.h" @@ -448,3 +449,57 @@ TEST_CASE("DequantGgufRowToF32 rejects unsupported i-quant type") { std::vector b2(136, 0); CHECK_THROWS_AS(DequantGgufRowToF32(23, b2.data(), 256), std::runtime_error); } + +// --- IQ1_S (19) / IQ1_XXXS (66): the two encodings the Qwen3.8-2.4T-A95B +// checkpoints are 96.92 % made of. `vt::cpu::BlockToFloat` has decoded both +// since #946, but this expand-to-bf16 path did not list them, so it threw +// "unsupported ggml type". That is not a corner: `RouteGgufTensor` sends a +// tensor here whenever the VT_CPU_REF oracle switch is on, keep-quant is off, +// K is ragged, or the role is not a verbatim one, so the reference lane could +// not load the target checkpoint at all. +// +// Gated on the ORACLE-produced goldens rather than on "does not throw": the +// inputs are real checkpoint bytes and the expected values come from the pinned +// upstream and fork ggml themselves. Provenance in +// tests/vt/iq1_golden_vectors.h. +#include "../vt/iq1_golden_vectors.h" + +namespace { + +void CheckGgufDequantAgainstOracle(uint32_t ggml_type, const uint8_t* blocks, + const uint32_t* golden_bits, size_t n) { + const std::vector out = + DequantGgufRowToF32(ggml_type, blocks, static_cast(n)); + REQUIRE(out.size() == n); + for (size_t i = 0; i < n; ++i) { + CAPTURE(i); + uint32_t bits = 0; + std::memcpy(&bits, &out[i], sizeof(bits)); + CHECK(bits == golden_bits[i]); + } +} + +} // namespace + +TEST_CASE("DequantGgufRowToF32 IQ1_S row matches the pinned oracle") { + CheckGgufDequantAgainstOracle(19, vllm_test::kIq1sGoldenBlocks, + vllm_test::kIq1sGoldenBits, + std::size(vllm_test::kIq1sGoldenBits)); +} + +TEST_CASE("DequantGgufRowToF32 IQ1_XXXS row matches the pinned FORK oracle") { + CheckGgufDequantAgainstOracle(66, vllm_test::kIq1xxxsGoldenBlocks, + vllm_test::kIq1xxxsGoldenBits, + std::size(vllm_test::kIq1xxxsGoldenBits)); +} + +TEST_CASE("DequantGgufRowToF32 IQ3_XXS row matches the pinned oracle") { + // Same omission, found by the same review, in the same shared branch, so it + // is fixed in the same flow rather than filed and deferred (issue #1023). + // IQ3_XXS is the `ffn_down` encoding of the DeepSeek-V4 UD-IQ2_XXS vehicle, + // and `vt::cpu::BlockToFloat` has decoded it since that port; only this + // expansion path never listed it. + CheckGgufDequantAgainstOracle(18, vllm_test::kIq3xxsGoldenBlocks, + vllm_test::kIq3xxsGoldenBits, + std::size(vllm_test::kIq3xxsGoldenBits)); +} diff --git a/tests/vllm/test_gguf_keep_quant.cpp b/tests/vllm/test_gguf_keep_quant.cpp index 371d480d9..97713ebac 100644 --- a/tests/vllm/test_gguf_keep_quant.cpp +++ b/tests/vllm/test_gguf_keep_quant.cpp @@ -277,6 +277,69 @@ TEST_CASE("KeepQuantDType covers the executable encodings") { } } +namespace { + +// One row of a checkpoint census: an encoding, how many tensor records carry +// it, and whether the loader is expected to keep it in blocks or expand it. +// `keep_quant == false` is a real entry, not an omission: F32 is 838 of the +// 1702 records in both target checkpoints, it is served by the expansion path, +// and leaving it out is what let an incomplete list call itself total. +struct CensusRow { + uint32_t ggml_type; + const char* name; + int tensors; + bool keep_quant; +}; + +// `split.tensors.count` from the shard headers, the same number on both +// checkpoints. The list below must add up to exactly this. +constexpr int kQwen38DeclaredTensors = 1702; + +// unsloth/Qwen3.8-2.4T-A95B-GGUF @ 567d3e6ac2, quant UD-IQ1_S, all 12 shards. +constexpr CensusRow kUdIq1sCensus[] = { + {19, "iq1_s", 276, true}, // 96.92 % of params + {13, "q5_k", 420, true}, {10, "q2_k", 3, true}, + {14, "q6_k", 162, true}, {12, "q4_k", 2, true}, + {8, "q8_0", 1, true}, {0, "f32", 838, false}, +}; + +// The same census over UD-Q1_0, all 10 shards. Identical apart from the expert +// encoding: ggml 66 where UD-IQ1_S has ggml 19. +constexpr CensusRow kUdQ10Census[] = { + {66, "iq1_xxxs", 276, true}, // 96.92 % of params + {13, "q5_k", 420, true}, {10, "q2_k", 3, true}, + {14, "q6_k", 162, true}, {12, "q4_k", 2, true}, + {8, "q8_0", 1, true}, {0, "f32", 838, false}, +}; + +template +void CheckCheckpointCensus(const CensusRow (&census)[N], const char* quant) { + CAPTURE(std::string(quant)); + vt::DType dt = vt::DType::kF32; + int counted = 0; + for (const CensusRow& p : census) { + CAPTURE(p.name); + CAPTURE(p.ggml_type); + CAPTURE(p.tensors); + counted += p.tensors; + // Assert the ROUTING both ways. A row expected to keep its blocks must + // have a dot kernel, and a row expected to expand must NOT silently be + // reclassified as keep-quant by a later change. + if (p.keep_quant) { + CHECK(KeepQuantDType(p.ggml_type, &dt)); + CHECK(vt::cpu::HasQuantDotKernel(dt)); + } else { + CHECK_FALSE(KeepQuantDType(p.ggml_type, &dt)); + } + } + // The list accounts for every declared record, so "this checkpoint needs + // nothing else" is a measurement rather than a claim. + CAPTURE(counted); + CHECK(counted == kQwen38DeclaredTensors); +} + +} // namespace + TEST_CASE("every encoding in the Qwen3.8-2.4T UD-IQ1_S checkpoint decodes") { // Census of unsloth/Qwen3.8-2.4T-A95B-GGUF @ 567d3e6ac2, quant UD-IQ1_S, // taken by parsing the tensor header of all 12 shards: 1702 tensor records @@ -289,24 +352,14 @@ TEST_CASE("every encoding in the Qwen3.8-2.4T UD-IQ1_S checkpoint decodes") { // bytes for nothing. IQ1_S alone is 96.92 % of the parameters (the // ffn_{down,gate,up}_exps of all 92 non-MTP layers), so an unsupported // encoding here is a refusal to run the model, never a slow path. - struct Present { - uint32_t ggml_type; - const char* name; - int tensors; - }; - constexpr Present kCheckpoint[] = { - {19, "iq1_s", 276}, // 96.92 % of params - {13, "q5_k", 420}, {10, "q2_k", 3}, {14, "q6_k", 162}, - {12, "q4_k", 2}, {8, "q8_0", 1}, - }; - vt::DType dt = vt::DType::kF32; - for (const Present& p : kCheckpoint) { - CAPTURE(p.name); - CAPTURE(p.ggml_type); - CAPTURE(p.tensors); - CHECK(KeepQuantDType(p.ggml_type, &dt)); - CHECK(vt::cpu::HasQuantDotKernel(dt)); - } + // + // The list has to SUM to the declared record count, and this case asserts + // that it does. An earlier revision claimed total coverage while enumerating + // six of the seven encodings: F32 was left out and the counts summed to 864, + // not 1702. A census that cannot say how many records it accounted for has + // not reported a census, and a list that is short by 838 tensors while + // calling itself TOTAL is the more misleading of the two states. + CheckCheckpointCensus(kUdIq1sCensus, "UD-IQ1_S"); } TEST_CASE("every encoding in the Qwen3.8-2.4T UD-Q1_0 checkpoint decodes") { @@ -320,24 +373,7 @@ TEST_CASE("every encoding in the Qwen3.8-2.4T UD-Q1_0 checkpoint decodes") { // oracle `llama-cpp-unsloth` (.agents/oracles/llama-cpp-unsloth.md), which is // admitted for this encoding family alone and is `gateable = no` until #933 // measures it. - struct Present { - uint32_t ggml_type; - const char* name; - int tensors; - }; - constexpr Present kCheckpoint[] = { - {66, "iq1_xxxs", 276}, // 96.92 % of params - {13, "q5_k", 420}, {10, "q2_k", 3}, {14, "q6_k", 162}, - {12, "q4_k", 2}, {8, "q8_0", 1}, - }; - vt::DType dt = vt::DType::kF32; - for (const Present& p : kCheckpoint) { - CAPTURE(p.name); - CAPTURE(p.ggml_type); - CAPTURE(p.tensors); - CHECK(KeepQuantDType(p.ggml_type, &dt)); - CHECK(vt::cpu::HasQuantDotKernel(dt)); - } + CheckCheckpointCensus(kUdQ10Census, "UD-Q1_0"); } TEST_CASE("routing table is TOTAL: every role x every encoding is explicit") { diff --git a/tests/vt/iq1_golden_vectors.h b/tests/vt/iq1_golden_vectors.h new file mode 100644 index 000000000..a4db8cca7 --- /dev/null +++ b/tests/vt/iq1_golden_vectors.h @@ -0,0 +1,499 @@ +// Golden decode vectors for the two IQ1 encodings, produced by the ORACLES +// THEMSELVES rather than by this tree. +// +// Why this file exists. Every other check on the IQ1_S / IQ1_XXXS decode is +// self-consistent: `ReferenceDotF64` and the G3 NMSE reference both decode the +// weight with `vt::cpu::BlockToFloat`, which is the function under test, so +// they are independent only in the summation. The grid tables are sealed by +// digest, but a seal on the codebook says nothing about the SCALE field, the +// DELTA magnitude, or the DELTA SIGN. Three separate defects were injected into +// those three parameters and the whole suite stayed green. This fixture is the +// external reference that closes that hole. +// +// Provenance of the INPUTS. These are real bytes from the checkpoints the row +// targets, read out of the GGUF tensor data with a standalone header parser +// (`unsloth/Qwen3.8-2.4T-A95B-GGUF`, shard 2 of each split, tensor +// `blk.0.ffn_gate_exps.weight`, first four 256-element blocks): +// +// IQ1_S UD-IQ1_S shard 00002-of-00012, ggml type 19, 4 x 50 = 200 bytes +// IQ1_XXXS UD-Q1_0 shard 00002-of-00010, ggml type 66, 4 x 38 = 152 bytes +// +// Real bytes, not synthetic ones, because a synthetic block cannot catch a +// misreading of the FILE. The four blocks carry both delta-sign polarities and +// sub-block scales 2 through 7, so every field the decode reads varies inside +// the fixture. This also reproduces, as a committed slice, the 1179648-weight +// bit-identity run the spec records. +// +// Provenance of the EXPECTED VALUES. Each blob was decoded by the pinned +// oracle's own `ggml_get_type_traits(type)->to_float`, in a build of that +// oracle, and the f32 results are stored here as raw bit patterns: +// +// IQ1_S ggml-org/llama.cpp @ 237ad9b96 (.agents/upstream-sync.md) +// IQ1_XXXS unslothai/llama.cpp @ 36fe8e1cc7f2b3b8c92fdda0ab07600141921786 +// (.agents/oracles/llama-cpp-unsloth.md) +// +// Reproduce with: +// git -C checkout && cmake -B build -DGGML_CUDA=OFF && +// cmake --build build --target ggml -j8 +// then call ggml_get_type_traits(19|66)->to_float on the bytes below. +// +// The comparison is BIT-EXACT, not approximate. `dl * (grid[j] + delta)` is +// the same f32 expression on both sides, so any difference at all means a +// decode parameter diverged rather than that rounding moved. +#pragma once + +#include + +namespace vllm_test { + +inline constexpr uint8_t kIq1sGoldenBlocks[200] = { + 0xe8, 0x0d, 0xf1, 0xef, 0x07, 0xa5, 0xca, 0x16, 0xd8, 0x76, 0xb5, 0xd2, + 0xef, 0x13, 0x2c, 0xfe, 0x63, 0xf8, 0xa5, 0x5c, 0x1b, 0x8f, 0xce, 0x8a, + 0x97, 0x75, 0xa7, 0xf3, 0x1f, 0x6f, 0x67, 0xa3, 0x0f, 0xe2, 0x02, 0xf9, + 0xfe, 0xca, 0x1a, 0x32, 0x89, 0x25, 0x6e, 0x29, 0xd6, 0x28, 0xe8, 0xcb, + 0xb2, 0xaf, 0xc9, 0x0c, 0xca, 0x97, 0xf6, 0xd1, 0x7e, 0x50, 0x2d, 0x4d, + 0xf5, 0x55, 0xb8, 0x7d, 0x74, 0x87, 0x0b, 0x20, 0xa1, 0xfe, 0xea, 0x68, + 0x95, 0x19, 0xe1, 0xd6, 0x1b, 0xad, 0x2c, 0xcd, 0x2f, 0xf5, 0x19, 0x4c, + 0xd6, 0x7c, 0x83, 0xbc, 0x0a, 0xbd, 0xa9, 0xb9, 0xff, 0xb2, 0xb1, 0x39, + 0x65, 0x46, 0xc0, 0x35, 0xb7, 0x0c, 0xab, 0x33, 0x8b, 0x08, 0xff, 0x03, + 0x70, 0x30, 0x9b, 0x29, 0x05, 0x41, 0x2b, 0xf5, 0xc3, 0x26, 0x9b, 0xce, + 0x30, 0xe7, 0x63, 0xcf, 0x48, 0x6c, 0xfe, 0xdf, 0x7f, 0x85, 0xb9, 0x50, + 0x91, 0x68, 0x66, 0x4b, 0xe7, 0xb3, 0x2b, 0xf9, 0x13, 0xbd, 0x01, 0x31, + 0x58, 0xbf, 0x7b, 0x40, 0x15, 0xc9, 0xfc, 0x0b, 0xe0, 0x8e, 0x6c, 0x94, + 0x50, 0x95, 0x30, 0x79, 0xf2, 0xbc, 0x13, 0x2c, 0x10, 0xcc, 0xe1, 0xf7, + 0xf8, 0x49, 0x15, 0xab, 0x17, 0x44, 0xb3, 0x4d, 0xd3, 0x7e, 0x8d, 0x41, + 0x06, 0xb7, 0x0a, 0x90, 0xd8, 0x67, 0x95, 0x48, 0xd5, 0x58, 0x18, 0xb1, + 0xae, 0xd0, 0xec, 0x40, 0xcf, 0xcf, 0xec, 0x76, +}; + +inline constexpr uint32_t kIq1sGoldenBits[1024] = { + 0xba313000U, 0xba313000U, 0xba313000U, 0x3b9b0a00U, 0xba313000U, 0xba313000U, + 0xbbc75600U, 0xba313000U, 0xba313000U, 0xba313000U, 0x3b9b0a00U, 0xba313000U, + 0xba313000U, 0xbbc75600U, 0xba313000U, 0xbbc75600U, 0xbbc75600U, 0x3b9b0a00U, + 0xba313000U, 0xba313000U, 0xba313000U, 0xba313000U, 0xba313000U, 0xba313000U, + 0xba313000U, 0xba313000U, 0xbbc75600U, 0xba313000U, 0x3b9b0a00U, 0x3b9b0a00U, + 0xba313000U, 0xba313000U, 0xb9d4a000U, 0xb9d4a000U, 0xbb6f3400U, 0xb9d4a000U, + 0xb9d4a000U, 0xb9d4a000U, 0xb9d4a000U, 0x3b3a0c00U, 0xb9d4a000U, 0x3b3a0c00U, + 0xb9d4a000U, 0x3b3a0c00U, 0xbb6f3400U, 0x3b3a0c00U, 0xb9d4a000U, 0x3b3a0c00U, + 0x3b3a0c00U, 0xb9d4a000U, 0x3b3a0c00U, 0xb9d4a000U, 0xbb6f3400U, 0xb9d4a000U, + 0xb9d4a000U, 0xb9d4a000U, 0xb9d4a000U, 0x3b3a0c00U, 0xb9d4a000U, 0xb9d4a000U, + 0xb9d4a000U, 0x3b3a0c00U, 0x3b3a0c00U, 0xb9d4a000U, 0x39a56000U, 0xbb10b400U, + 0xbb10b400U, 0xbb10b400U, 0xbb10b400U, 0x39a56000U, 0xbb10b400U, 0x39a56000U, + 0x39a56000U, 0x3b3a0c00U, 0x39a56000U, 0x39a56000U, 0xbb10b400U, 0x39a56000U, + 0x39a56000U, 0x39a56000U, 0x39a56000U, 0x39a56000U, 0x3b3a0c00U, 0x39a56000U, + 0x39a56000U, 0xbb10b400U, 0x39a56000U, 0xbb10b400U, 0xbb10b400U, 0x3b3a0c00U, + 0xbb10b400U, 0x39a56000U, 0xbb10b400U, 0x39a56000U, 0x39a56000U, 0xbb10b400U, + 0x396c4000U, 0xbaceb800U, 0x396c4000U, 0xbaceb800U, 0x396c4000U, 0x396c4000U, + 0x396c4000U, 0xbaceb800U, 0x3b04e400U, 0x396c4000U, 0x396c4000U, 0xbaceb800U, + 0xbaceb800U, 0x396c4000U, 0x3b04e400U, 0xbaceb800U, 0x396c4000U, 0x396c4000U, + 0x396c4000U, 0x396c4000U, 0xbaceb800U, 0xbaceb800U, 0x396c4000U, 0x3b04e400U, + 0x396c4000U, 0xbaceb800U, 0xbaceb800U, 0xbaceb800U, 0x3b04e400U, 0x396c4000U, + 0xbaceb800U, 0x396c4000U, 0x3b04e400U, 0x396c4000U, 0x3b04e400U, 0xbaceb800U, + 0xbaceb800U, 0x396c4000U, 0x396c4000U, 0x3b04e400U, 0x3b04e400U, 0xbaceb800U, + 0x396c4000U, 0x396c4000U, 0xbaceb800U, 0x3b04e400U, 0x3b04e400U, 0x396c4000U, + 0x3b04e400U, 0x396c4000U, 0x396c4000U, 0xbaceb800U, 0x396c4000U, 0x396c4000U, + 0x3b04e400U, 0x396c4000U, 0xbaceb800U, 0x396c4000U, 0x3b04e400U, 0x396c4000U, + 0x396c4000U, 0x3b04e400U, 0x396c4000U, 0x396c4000U, 0x396c4000U, 0xbaceb800U, + 0x396c4000U, 0x396c4000U, 0x396c4000U, 0x396c4000U, 0x396c4000U, 0x3b04e400U, + 0x3b04e400U, 0xbaceb800U, 0x396c4000U, 0xbaceb800U, 0x396c4000U, 0xbaceb800U, + 0xbaceb800U, 0x396c4000U, 0x396c4000U, 0x3b04e400U, 0x3b04e400U, 0x396c4000U, + 0x396c4000U, 0xbaceb800U, 0x396c4000U, 0x396c4000U, 0x396c4000U, 0x3b04e400U, + 0x3b04e400U, 0x3b04e400U, 0xbaceb800U, 0x3b04e400U, 0x396c4000U, 0x396c4000U, + 0xbb6f3400U, 0xb9d4a000U, 0x3b3a0c00U, 0xb9d4a000U, 0xb9d4a000U, 0x3b3a0c00U, + 0xbb6f3400U, 0xbb6f3400U, 0xb9d4a000U, 0xb9d4a000U, 0xb9d4a000U, 0xb9d4a000U, + 0xb9d4a000U, 0xb9d4a000U, 0xbb6f3400U, 0x3b3a0c00U, 0x3b3a0c00U, 0x3b3a0c00U, + 0xb9d4a000U, 0xbb6f3400U, 0xb9d4a000U, 0x3b3a0c00U, 0xb9d4a000U, 0x3b3a0c00U, + 0xb9d4a000U, 0xb9d4a000U, 0xbb6f3400U, 0xb9d4a000U, 0xb9d4a000U, 0x3b3a0c00U, + 0x3b3a0c00U, 0xb9d4a000U, 0xb96c4000U, 0xb96c4000U, 0xbb04e400U, 0xb96c4000U, + 0x3aceb800U, 0x3aceb800U, 0x3aceb800U, 0xbb04e400U, 0xb96c4000U, 0xb96c4000U, + 0xb96c4000U, 0xbb04e400U, 0xbb04e400U, 0xb96c4000U, 0xb96c4000U, 0x3aceb800U, + 0x3aceb800U, 0x3aceb800U, 0xb96c4000U, 0xb96c4000U, 0x3aceb800U, 0xb96c4000U, + 0xbb04e400U, 0x3aceb800U, 0xb96c4000U, 0xb96c4000U, 0x3aceb800U, 0xbb04e400U, + 0xb96c4000U, 0x3aceb800U, 0x3aceb800U, 0x3aceb800U, 0x3a0f8e00U, 0x3a0f8e00U, + 0xbb7b3880U, 0x3a0f8e00U, 0x3a0f8e00U, 0x3a0f8e00U, 0x3a0f8e00U, 0x3ba17fc0U, + 0x3a0f8e00U, 0x3a0f8e00U, 0x3a0f8e00U, 0x3a0f8e00U, 0x3a0f8e00U, 0xbb7b3880U, + 0xbb7b3880U, 0x3a0f8e00U, 0x3a0f8e00U, 0x3a0f8e00U, 0x3ba17fc0U, 0xbb7b3880U, + 0x3a0f8e00U, 0x3a0f8e00U, 0x3a0f8e00U, 0x3a0f8e00U, 0x3a0f8e00U, 0x3a0f8e00U, + 0x3a0f8e00U, 0x3a0f8e00U, 0x3a0f8e00U, 0x3a0f8e00U, 0x3a0f8e00U, 0x3ba17fc0U, + 0xbb16bb80U, 0x3aea7900U, 0xb985fc00U, 0xbb16bb80U, 0xb985fc00U, 0xbb16bb80U, + 0xb985fc00U, 0xb985fc00U, 0x3aea7900U, 0x3aea7900U, 0xb985fc00U, 0xb985fc00U, + 0xbb16bb80U, 0xb985fc00U, 0xbb16bb80U, 0xbb16bb80U, 0xb985fc00U, 0xbb16bb80U, + 0xbb16bb80U, 0xb985fc00U, 0x3aea7900U, 0xb985fc00U, 0x3aea7900U, 0xbb16bb80U, + 0xb985fc00U, 0xb985fc00U, 0xbb16bb80U, 0xb985fc00U, 0x3aea7900U, 0x3aea7900U, + 0xbb16bb80U, 0x3aea7900U, 0xb985fc00U, 0xb985fc00U, 0x3aea7900U, 0x3aea7900U, + 0xb985fc00U, 0xb985fc00U, 0xbb16bb80U, 0xb985fc00U, 0xb985fc00U, 0xb985fc00U, + 0x3aea7900U, 0x3aea7900U, 0xb985fc00U, 0xb985fc00U, 0xb985fc00U, 0xbb16bb80U, + 0xb985fc00U, 0xbb16bb80U, 0x3aea7900U, 0x3aea7900U, 0x3aea7900U, 0x3aea7900U, + 0xb985fc00U, 0xb985fc00U, 0xb985fc00U, 0xbb16bb80U, 0xb985fc00U, 0xb985fc00U, + 0xb985fc00U, 0xbb16bb80U, 0xb985fc00U, 0x3aea7900U, 0xb985fc00U, 0xb985fc00U, + 0xb985fc00U, 0x3aea7900U, 0x3aea7900U, 0xb985fc00U, 0xb985fc00U, 0xbb16bb80U, + 0xb985fc00U, 0x3aea7900U, 0xbb16bb80U, 0xb985fc00U, 0x3aea7900U, 0x3aea7900U, + 0x3aea7900U, 0xb985fc00U, 0x3aea7900U, 0x3aea7900U, 0xbb16bb80U, 0xb985fc00U, + 0x3aea7900U, 0xb985fc00U, 0xbb16bb80U, 0x3aea7900U, 0xb985fc00U, 0xb985fc00U, + 0x3aea7900U, 0x3aea7900U, 0xb985fc00U, 0xb985fc00U, 0xb985fc00U, 0xb985fc00U, + 0x3aea7900U, 0xbb16bb80U, 0xbb16bb80U, 0xb985fc00U, 0xb985fc00U, 0xb985fc00U, + 0x3aea7900U, 0x3aea7900U, 0xbb16bb80U, 0x3aea7900U, 0x3aea7900U, 0x3aea7900U, + 0x3aea7900U, 0x3aea7900U, 0x3aea7900U, 0x3aea7900U, 0x3aea7900U, 0xb985fc00U, + 0xbb16bb80U, 0xbb16bb80U, 0xb985fc00U, 0xb985fc00U, 0xb985fc00U, 0xb985fc00U, + 0x3aea7900U, 0xb985fc00U, 0xb985fc00U, 0xb985fc00U, 0x3aea7900U, 0xb985fc00U, + 0xb985fc00U, 0xbb16bb80U, 0x3985fc00U, 0xbaea7900U, 0x3985fc00U, 0x3985fc00U, + 0x3985fc00U, 0x3b16bb80U, 0x3985fc00U, 0xbaea7900U, 0xbaea7900U, 0x3b16bb80U, + 0xbaea7900U, 0xbaea7900U, 0xbaea7900U, 0x3b16bb80U, 0xbaea7900U, 0x3b16bb80U, + 0x3985fc00U, 0x3985fc00U, 0x3985fc00U, 0x3b16bb80U, 0x3985fc00U, 0x3985fc00U, + 0x3985fc00U, 0x3b16bb80U, 0x3985fc00U, 0x3b16bb80U, 0xbaea7900U, 0x3985fc00U, + 0x3985fc00U, 0xbaea7900U, 0x3b16bb80U, 0x3985fc00U, 0x3b41cc80U, 0x39ac4400U, + 0x39ac4400U, 0xbb16bb80U, 0x39ac4400U, 0x39ac4400U, 0x3b41cc80U, 0x39ac4400U, + 0x3b41cc80U, 0xbb16bb80U, 0x3b41cc80U, 0x39ac4400U, 0x3b41cc80U, 0x3b41cc80U, + 0x39ac4400U, 0x39ac4400U, 0x39ac4400U, 0xbb16bb80U, 0x39ac4400U, 0xbb16bb80U, + 0x39ac4400U, 0x39ac4400U, 0x39ac4400U, 0xbb16bb80U, 0x39ac4400U, 0xbb16bb80U, + 0x39ac4400U, 0x39ac4400U, 0xbb16bb80U, 0x39ac4400U, 0x39ac4400U, 0x39ac4400U, + 0x3985fc00U, 0x3985fc00U, 0x3b16bb80U, 0x3b16bb80U, 0x3985fc00U, 0xbaea7900U, + 0xbaea7900U, 0xbaea7900U, 0x3985fc00U, 0xbaea7900U, 0x3985fc00U, 0xbaea7900U, + 0x3b16bb80U, 0xbaea7900U, 0x3985fc00U, 0xbaea7900U, 0x3b16bb80U, 0x3985fc00U, + 0xbaea7900U, 0xbaea7900U, 0x3985fc00U, 0x3b16bb80U, 0x3985fc00U, 0x3b16bb80U, + 0x3985fc00U, 0x3b16bb80U, 0xbaea7900U, 0xbaea7900U, 0x3985fc00U, 0x3b16bb80U, + 0x3b16bb80U, 0xbaea7900U, 0x3b3ef380U, 0x39a9bc00U, 0x39a9bc00U, 0x39a9bc00U, + 0xbb148480U, 0x39a9bc00U, 0x39a9bc00U, 0x3b3ef380U, 0x39a9bc00U, 0x39a9bc00U, + 0x3b3ef380U, 0xbb148480U, 0x3b3ef380U, 0x39a9bc00U, 0x39a9bc00U, 0x39a9bc00U, + 0x3b3ef380U, 0x3b3ef380U, 0x39a9bc00U, 0x39a9bc00U, 0x3b3ef380U, 0x3b3ef380U, + 0x3b3ef380U, 0x39a9bc00U, 0x39a9bc00U, 0x39a9bc00U, 0x3b3ef380U, 0x39a9bc00U, + 0xbb148480U, 0x39a9bc00U, 0x3b3ef380U, 0x39a9bc00U, 0x3ae70700U, 0x3ae70700U, + 0x3ae70700U, 0x3ae70700U, 0x3ae70700U, 0x3ae70700U, 0x3ae70700U, 0x3ae70700U, + 0x3ae70700U, 0xbb148480U, 0xb9840400U, 0xb9840400U, 0xb9840400U, 0xb9840400U, + 0xb9840400U, 0xb9840400U, 0xbb148480U, 0xbb148480U, 0xb9840400U, 0x3ae70700U, + 0xb9840400U, 0xbb148480U, 0x3ae70700U, 0x3ae70700U, 0xb9840400U, 0x3ae70700U, + 0xb9840400U, 0xbb148480U, 0xb9840400U, 0xb9840400U, 0xb9840400U, 0xbb148480U, + 0xbb9f2040U, 0xba0d7200U, 0xba0d7200U, 0x3b778780U, 0xba0d7200U, 0xbb9f2040U, + 0xba0d7200U, 0xba0d7200U, 0xba0d7200U, 0xba0d7200U, 0x3b778780U, 0xba0d7200U, + 0xba0d7200U, 0xba0d7200U, 0x3b778780U, 0xba0d7200U, 0xba0d7200U, 0xba0d7200U, + 0xba0d7200U, 0xba0d7200U, 0xba0d7200U, 0xba0d7200U, 0xba0d7200U, 0xba0d7200U, + 0xba0d7200U, 0xba0d7200U, 0xba0d7200U, 0xba0d7200U, 0x3b778780U, 0xba0d7200U, + 0xba0d7200U, 0xba0d7200U, 0xb9840400U, 0xbb148480U, 0xbb148480U, 0xbb148480U, + 0xb9840400U, 0x3ae70700U, 0xbb148480U, 0xb9840400U, 0xb9840400U, 0xb9840400U, + 0x3ae70700U, 0x3ae70700U, 0xb9840400U, 0xb9840400U, 0xbb148480U, 0xb9840400U, + 0x3ae70700U, 0x3ae70700U, 0xb9840400U, 0xb9840400U, 0xbb148480U, 0xbb148480U, + 0x3ae70700U, 0xb9840400U, 0xb9840400U, 0x3ae70700U, 0x3ae70700U, 0xb9840400U, + 0xbb148480U, 0x3ae70700U, 0xbb148480U, 0x3ae70700U, 0xbae70700U, 0xbae70700U, + 0x3b148480U, 0x39840400U, 0x39840400U, 0x3b148480U, 0x39840400U, 0xbae70700U, + 0xbae70700U, 0xbae70700U, 0x39840400U, 0x39840400U, 0xbae70700U, 0xbae70700U, + 0x39840400U, 0xbae70700U, 0x39840400U, 0xbae70700U, 0x3b148480U, 0xbae70700U, + 0x3b148480U, 0x39840400U, 0x39840400U, 0x39840400U, 0xbae70700U, 0xbae70700U, + 0x39840400U, 0x39840400U, 0x39840400U, 0xbae70700U, 0x39840400U, 0xbae70700U, + 0xbb148480U, 0xb9840400U, 0xb9840400U, 0xb9840400U, 0xb9840400U, 0xb9840400U, + 0xbb148480U, 0xbb148480U, 0xb9840400U, 0xb9840400U, 0xb9840400U, 0xb9840400U, + 0xbb148480U, 0xb9840400U, 0xb9840400U, 0xb9840400U, 0x3ae70700U, 0x3ae70700U, + 0xb9840400U, 0xb9840400U, 0x3ae70700U, 0xb9840400U, 0x3ae70700U, 0xb9840400U, + 0x3ae70700U, 0x3ae70700U, 0xb9840400U, 0xb9840400U, 0xb9840400U, 0xbb148480U, + 0x3ae70700U, 0x3ae70700U, 0x3b3ef380U, 0x39a9bc00U, 0xbb148480U, 0x39a9bc00U, + 0x39a9bc00U, 0x39a9bc00U, 0x39a9bc00U, 0x39a9bc00U, 0xbb148480U, 0x39a9bc00U, + 0x39a9bc00U, 0xbb148480U, 0x39a9bc00U, 0x3b3ef380U, 0x3b3ef380U, 0x3b3ef380U, + 0x39a9bc00U, 0x39a9bc00U, 0x39a9bc00U, 0x39a9bc00U, 0xbb148480U, 0x3b3ef380U, + 0x39a9bc00U, 0xbb148480U, 0x39a9bc00U, 0x3b3ef380U, 0x39a9bc00U, 0x3b3ef380U, + 0x3b3ef380U, 0x39a9bc00U, 0xbb148480U, 0xbb148480U, 0xbb3ef380U, 0x3b148480U, + 0xb9a9bc00U, 0xb9a9bc00U, 0xb9a9bc00U, 0xbb3ef380U, 0xbb3ef380U, 0x3b148480U, + 0xb9a9bc00U, 0xb9a9bc00U, 0x3b148480U, 0xbb3ef380U, 0xb9a9bc00U, 0x3b148480U, + 0x3b148480U, 0xbb3ef380U, 0x3b148480U, 0xb9a9bc00U, 0x3b148480U, 0xb9a9bc00U, + 0xb9a9bc00U, 0x3b148480U, 0xb9a9bc00U, 0xb9a9bc00U, 0xbb3ef380U, 0x3b148480U, + 0xb9a9bc00U, 0xb9a9bc00U, 0xbb3ef380U, 0x3b148480U, 0xb9a9bc00U, 0xb9a9bc00U, + 0x39cf9800U, 0x39cf9800U, 0x39cf9800U, 0xbb35a500U, 0x39cf9800U, 0xbb35a500U, + 0x39cf9800U, 0xbb35a500U, 0x39cf9800U, 0x39cf9800U, 0x39cf9800U, 0x39cf9800U, + 0x39cf9800U, 0xbb35a500U, 0x39cf9800U, 0x39cf9800U, 0x3b698b00U, 0x3b698b00U, + 0x39cf9800U, 0x39cf9800U, 0x39cf9800U, 0xbb35a500U, 0x3b698b00U, 0x3b698b00U, + 0xbb35a500U, 0x39cf9800U, 0x3b698b00U, 0x39cf9800U, 0x39cf9800U, 0xbb35a500U, + 0x39cf9800U, 0x39cf9800U, 0xbafb8200U, 0x398fb800U, 0x398fb800U, 0x3b21af00U, + 0x3b21af00U, 0x398fb800U, 0x3b21af00U, 0x398fb800U, 0x398fb800U, 0xbafb8200U, + 0x398fb800U, 0x398fb800U, 0x398fb800U, 0xbafb8200U, 0xbafb8200U, 0x398fb800U, + 0xbafb8200U, 0xbafb8200U, 0x3b21af00U, 0x398fb800U, 0x3b21af00U, 0x398fb800U, + 0x3b21af00U, 0xbafb8200U, 0x3b21af00U, 0x3b21af00U, 0xbafb8200U, 0xbafb8200U, + 0x398fb800U, 0x3b21af00U, 0x398fb800U, 0x398fb800U, 0xbb19b300U, 0x39afa800U, + 0x39afa800U, 0x39afa800U, 0x39afa800U, 0x39afa800U, 0xbb19b300U, 0x3b459d00U, + 0x39afa800U, 0xbb19b300U, 0x39afa800U, 0x39afa800U, 0xbb19b300U, 0x39afa800U, + 0xbb19b300U, 0x39afa800U, 0x39afa800U, 0x39afa800U, 0x39afa800U, 0x3b459d00U, + 0x3b459d00U, 0x39afa800U, 0xbb19b300U, 0x39afa800U, 0x39afa800U, 0x39afa800U, + 0x39afa800U, 0xbb19b300U, 0x3b459d00U, 0x39afa800U, 0x39afa800U, 0x39afa800U, + 0xbafb8200U, 0xbafb8200U, 0xbafb8200U, 0x3ac39e00U, 0xbafb8200U, 0xbafb8200U, + 0xbafb8200U, 0xbafb8200U, 0xbafb8200U, 0xbafb8200U, 0xb95f9000U, 0xb95f9000U, + 0xbafb8200U, 0xb95f9000U, 0xb95f9000U, 0xb95f9000U, 0xb95f9000U, 0xb95f9000U, + 0xb95f9000U, 0x3ac39e00U, 0xb95f9000U, 0xbafb8200U, 0x3ac39e00U, 0xb95f9000U, + 0x3ac39e00U, 0xb95f9000U, 0xb95f9000U, 0xbafb8200U, 0x3ac39e00U, 0xbafb8200U, + 0xb95f9000U, 0xbafb8200U, 0x3b19b300U, 0xb9afa800U, 0xb9afa800U, 0xb9afa800U, + 0x3b19b300U, 0xb9afa800U, 0xb9afa800U, 0x3b19b300U, 0xbb459d00U, 0xbb459d00U, + 0x3b19b300U, 0xb9afa800U, 0x3b19b300U, 0xb9afa800U, 0x3b19b300U, 0xb9afa800U, + 0xb9afa800U, 0xb9afa800U, 0xbb459d00U, 0xb9afa800U, 0xb9afa800U, 0xb9afa800U, + 0x3b19b300U, 0xbb459d00U, 0x3b19b300U, 0xb9afa800U, 0xb9afa800U, 0x3b19b300U, + 0xb9afa800U, 0x3b19b300U, 0xbb459d00U, 0xbb459d00U, 0xbafb8200U, 0xbafb8200U, + 0x398fb800U, 0x3b21af00U, 0x398fb800U, 0x398fb800U, 0x398fb800U, 0x398fb800U, + 0xbafb8200U, 0x398fb800U, 0x398fb800U, 0x398fb800U, 0x3b21af00U, 0x398fb800U, + 0x3b21af00U, 0x398fb800U, 0x398fb800U, 0x398fb800U, 0xbafb8200U, 0x3b21af00U, + 0x3b21af00U, 0xbafb8200U, 0x398fb800U, 0x398fb800U, 0x398fb800U, 0xbafb8200U, + 0xbafb8200U, 0x398fb800U, 0xbafb8200U, 0x398fb800U, 0xbafb8200U, 0xbafb8200U, + 0x3afb8200U, 0xb98fb800U, 0xb98fb800U, 0xb98fb800U, 0xbb21af00U, 0x3afb8200U, + 0x3afb8200U, 0x3afb8200U, 0xbb21af00U, 0xbb21af00U, 0xb98fb800U, 0xb98fb800U, + 0xbb21af00U, 0x3afb8200U, 0xb98fb800U, 0xbb21af00U, 0xbb21af00U, 0xb98fb800U, + 0xbb21af00U, 0xb98fb800U, 0xbb21af00U, 0xb98fb800U, 0x3afb8200U, 0x3afb8200U, + 0xb98fb800U, 0xb98fb800U, 0xb98fb800U, 0xb98fb800U, 0x3afb8200U, 0x3afb8200U, + 0xb98fb800U, 0x3afb8200U, 0x3b86bc80U, 0x39ef8800U, 0x39ef8800U, 0x39ef8800U, + 0x39ef8800U, 0x39ef8800U, 0x39ef8800U, 0x39ef8800U, 0x39ef8800U, 0x39ef8800U, + 0x39ef8800U, 0x39ef8800U, 0x39ef8800U, 0xbb519700U, 0xbb519700U, 0x3b86bc80U, + 0x3b86bc80U, 0x39ef8800U, 0x39ef8800U, 0x39ef8800U, 0x3b86bc80U, 0x39ef8800U, + 0xbb519700U, 0x39ef8800U, 0xbb519700U, 0x3b86bc80U, 0x39ef8800U, 0x39ef8800U, + 0x39ef8800U, 0xbb519700U, 0x39ef8800U, 0x39ef8800U, +}; + +inline constexpr uint8_t kIq1xxxsGoldenBlocks[152] = { + 0xf1, 0x0d, 0x3f, 0x10, 0x83, 0xb0, 0xe1, 0xa1, 0x72, 0xcc, 0x3a, 0x6d, + 0x10, 0x08, 0x15, 0x2b, 0xdb, 0x90, 0x68, 0xa0, 0xc0, 0x85, 0xe4, 0x75, + 0x88, 0xad, 0x23, 0xd4, 0xf0, 0xca, 0x27, 0x66, 0xd6, 0xe6, 0xcf, 0x23, + 0x23, 0xac, 0xb8, 0x0c, 0xe1, 0x35, 0x86, 0xde, 0x61, 0x39, 0x2e, 0xd3, + 0x48, 0x2f, 0xf4, 0xd9, 0x22, 0xb0, 0xe2, 0x8f, 0x7a, 0xee, 0x91, 0x22, + 0x28, 0xa2, 0xea, 0xb5, 0xbc, 0x4a, 0x15, 0x6b, 0x04, 0x93, 0xa3, 0xbb, + 0xb7, 0xbb, 0x4b, 0x35, 0x0e, 0x0c, 0x39, 0x9a, 0xce, 0xcd, 0xee, 0xa7, + 0xb4, 0x1b, 0xda, 0xb6, 0x81, 0x97, 0xa2, 0x48, 0xb5, 0xe7, 0x84, 0x51, + 0x94, 0x11, 0x06, 0x69, 0x99, 0xb5, 0x7d, 0xfb, 0x12, 0x0b, 0x41, 0x2b, + 0xcd, 0x9f, 0xb5, 0xbf, 0xc4, 0xc5, 0x49, 0x0c, 0x0c, 0x5b, 0xeb, 0x5d, + 0xc9, 0x34, 0x23, 0xa5, 0xd4, 0x3e, 0x44, 0x93, 0x02, 0x6b, 0xf7, 0x2b, + 0xed, 0xc7, 0x2e, 0x07, 0x8a, 0xc9, 0x5e, 0x26, 0xfc, 0x19, 0xb8, 0xec, + 0x82, 0x35, 0x40, 0x5c, 0x45, 0xb5, 0x4d, 0x7d, +}; + +inline constexpr uint32_t kIq1xxxsGoldenBits[1024] = { + 0xba323e00U, 0xba323e00U, 0xba323e00U, 0xba323e00U, 0xba323e00U, 0xba323e00U, + 0xbbc885c0U, 0xba323e00U, 0xba323e00U, 0xba323e00U, 0x3b9bf640U, 0xba323e00U, + 0xba323e00U, 0xbbc885c0U, 0xba323e00U, 0xbbc885c0U, 0xba323e00U, 0x3b9bf640U, + 0xba323e00U, 0xba323e00U, 0xba323e00U, 0xba323e00U, 0xba323e00U, 0xba323e00U, + 0xba323e00U, 0xba323e00U, 0xbbc885c0U, 0xba323e00U, 0x3b9bf640U, 0x3b9bf640U, + 0xba323e00U, 0xba323e00U, 0xb9d5e400U, 0xb9d5e400U, 0xbb70a080U, 0xb9d5e400U, + 0xb9d5e400U, 0xb9d5e400U, 0xb9d5e400U, 0x3b3b2780U, 0xb9d5e400U, 0xb9d5e400U, + 0xb9d5e400U, 0x3b3b2780U, 0xbb70a080U, 0x3b3b2780U, 0xb9d5e400U, 0xb9d5e400U, + 0x3b3b2780U, 0xb9d5e400U, 0xb9d5e400U, 0x3b3b2780U, 0xbb70a080U, 0xb9d5e400U, + 0xb9d5e400U, 0xb9d5e400U, 0xb9d5e400U, 0x3b3b2780U, 0xb9d5e400U, 0xb9d5e400U, + 0xb9d5e400U, 0x3b3b2780U, 0x3b3b2780U, 0xb9d5e400U, 0x39a65c00U, 0x39a65c00U, + 0xbb119080U, 0xbb119080U, 0x39a65c00U, 0x39a65c00U, 0xbb119080U, 0x39a65c00U, + 0x39a65c00U, 0x3b3b2780U, 0x39a65c00U, 0x39a65c00U, 0xbb119080U, 0x39a65c00U, + 0x39a65c00U, 0x39a65c00U, 0x39a65c00U, 0x39a65c00U, 0x3b3b2780U, 0x39a65c00U, + 0x39a65c00U, 0xbb119080U, 0x39a65c00U, 0xbb119080U, 0x39a65c00U, 0x3b3b2780U, + 0x39a65c00U, 0x39a65c00U, 0x39a65c00U, 0x39a65c00U, 0xbb119080U, 0xbb119080U, + 0x396da800U, 0xbacff300U, 0x396da800U, 0xbacff300U, 0x396da800U, 0x396da800U, + 0x396da800U, 0xbacff300U, 0x3b05ae80U, 0x396da800U, 0x396da800U, 0xbacff300U, + 0xbacff300U, 0x396da800U, 0x3b05ae80U, 0xbacff300U, 0x396da800U, 0x396da800U, + 0x396da800U, 0x396da800U, 0x396da800U, 0xbacff300U, 0x396da800U, 0x3b05ae80U, + 0x396da800U, 0xbacff300U, 0xbacff300U, 0xbacff300U, 0x3b05ae80U, 0x396da800U, + 0x396da800U, 0x396da800U, 0x39a65c00U, 0x39a65c00U, 0x3b3b2780U, 0xbb119080U, + 0xbb119080U, 0x39a65c00U, 0x39a65c00U, 0x39a65c00U, 0x3b3b2780U, 0x39a65c00U, + 0x39a65c00U, 0x39a65c00U, 0xbb119080U, 0x3b3b2780U, 0x39a65c00U, 0x39a65c00U, + 0x3b3b2780U, 0x39a65c00U, 0x39a65c00U, 0x39a65c00U, 0x39a65c00U, 0x39a65c00U, + 0x3b3b2780U, 0x39a65c00U, 0xbb119080U, 0x39a65c00U, 0x3b3b2780U, 0x39a65c00U, + 0x39a65c00U, 0x39a65c00U, 0x39a65c00U, 0x39a65c00U, 0x396da800U, 0xbacff300U, + 0x396da800U, 0x396da800U, 0x396da800U, 0x396da800U, 0x396da800U, 0x3b05ae80U, + 0x3b05ae80U, 0xbacff300U, 0x396da800U, 0xbacff300U, 0x396da800U, 0x396da800U, + 0x396da800U, 0x396da800U, 0xbacff300U, 0x3b05ae80U, 0x3b05ae80U, 0x396da800U, + 0x396da800U, 0x396da800U, 0x396da800U, 0x396da800U, 0x396da800U, 0x3b05ae80U, + 0x396da800U, 0x3b05ae80U, 0x396da800U, 0x3b05ae80U, 0x396da800U, 0x396da800U, + 0xbb70a080U, 0xb9d5e400U, 0x3b3b2780U, 0xb9d5e400U, 0x3b3b2780U, 0xb9d5e400U, + 0xb9d5e400U, 0xbb70a080U, 0xb9d5e400U, 0xb9d5e400U, 0xb9d5e400U, 0xb9d5e400U, + 0xb9d5e400U, 0xb9d5e400U, 0xbb70a080U, 0x3b3b2780U, 0xb9d5e400U, 0xb9d5e400U, + 0xb9d5e400U, 0xbb70a080U, 0xb9d5e400U, 0x3b3b2780U, 0xb9d5e400U, 0x3b3b2780U, + 0xb9d5e400U, 0xb9d5e400U, 0xbb70a080U, 0xb9d5e400U, 0xb9d5e400U, 0x3b3b2780U, + 0x3b3b2780U, 0xb9d5e400U, 0xbb05ae80U, 0x3acff300U, 0xbb05ae80U, 0xb96da800U, + 0xb96da800U, 0x3acff300U, 0xb96da800U, 0xbb05ae80U, 0x3acff300U, 0xb96da800U, + 0xb96da800U, 0xbb05ae80U, 0xbb05ae80U, 0xb96da800U, 0xb96da800U, 0xb96da800U, + 0xb96da800U, 0x3acff300U, 0xb96da800U, 0xbb05ae80U, 0x3acff300U, 0xb96da800U, + 0xbb05ae80U, 0x3acff300U, 0xb96da800U, 0xb96da800U, 0x3acff300U, 0xb96da800U, + 0xb96da800U, 0xb96da800U, 0xb96da800U, 0x3acff300U, 0x3a0d9000U, 0x3a0d9000U, + 0xbb77bc00U, 0x3a0d9000U, 0x3a0d9000U, 0x3a0d9000U, 0x3a0d9000U, 0x3b9f4200U, + 0x3a0d9000U, 0x3a0d9000U, 0x3a0d9000U, 0x3a0d9000U, 0x3a0d9000U, 0xbb77bc00U, + 0xbb77bc00U, 0x3a0d9000U, 0x3a0d9000U, 0x3a0d9000U, 0x3b9f4200U, 0x3a0d9000U, + 0x3a0d9000U, 0x3a0d9000U, 0x3a0d9000U, 0x3a0d9000U, 0x3a0d9000U, 0x3a0d9000U, + 0x3a0d9000U, 0x3a0d9000U, 0xbb77bc00U, 0x3a0d9000U, 0x3a0d9000U, 0x3b9f4200U, + 0xbb14a400U, 0xb9842000U, 0xb9842000U, 0xb9842000U, 0x3ae73800U, 0xbb14a400U, + 0xb9842000U, 0xb9842000U, 0x3ae73800U, 0xb9842000U, 0xb9842000U, 0xb9842000U, + 0xbb14a400U, 0xb9842000U, 0xbb14a400U, 0xb9842000U, 0xb9842000U, 0xb9842000U, + 0xbb14a400U, 0xb9842000U, 0xb9842000U, 0xb9842000U, 0x3ae73800U, 0xbb14a400U, + 0xb9842000U, 0xbb14a400U, 0xbb14a400U, 0xb9842000U, 0xb9842000U, 0xb9842000U, + 0xbb14a400U, 0x3ae73800U, 0xb9842000U, 0xb9842000U, 0x3ae73800U, 0x3ae73800U, + 0xb9842000U, 0xb9842000U, 0xbb14a400U, 0xb9842000U, 0xb9842000U, 0xb9842000U, + 0x3ae73800U, 0xb9842000U, 0xb9842000U, 0xb9842000U, 0x3ae73800U, 0xbb14a400U, + 0xb9842000U, 0xbb14a400U, 0x3ae73800U, 0xb9842000U, 0x3ae73800U, 0x3ae73800U, + 0xb9842000U, 0x3ae73800U, 0xb9842000U, 0xbb14a400U, 0xbb14a400U, 0xbb14a400U, + 0xb9842000U, 0xbb14a400U, 0xb9842000U, 0x3ae73800U, 0xb9842000U, 0xb9842000U, + 0xb9842000U, 0xb9842000U, 0x3ae73800U, 0xb9842000U, 0xb9842000U, 0xbb14a400U, + 0xb9842000U, 0xb9842000U, 0xbb14a400U, 0xb9842000U, 0x3ae73800U, 0x3ae73800U, + 0xb9842000U, 0xb9842000U, 0x3ae73800U, 0xb9842000U, 0xbb14a400U, 0xb9842000U, + 0xb9842000U, 0xb9842000U, 0xb9842000U, 0x3ae73800U, 0xb9842000U, 0x3ae73800U, + 0x3ae73800U, 0x3ae73800U, 0xb9842000U, 0xb9842000U, 0xb9842000U, 0xb9842000U, + 0xb9842000U, 0xbb14a400U, 0xbb14a400U, 0xb9842000U, 0xb9842000U, 0xb9842000U, + 0xb9842000U, 0xb9842000U, 0xbb14a400U, 0xb9842000U, 0x3ae73800U, 0x3ae73800U, + 0x3ae73800U, 0xb9842000U, 0xb9842000U, 0x3ae73800U, 0x3ae73800U, 0xb9842000U, + 0xbb14a400U, 0xbb14a400U, 0x3ae73800U, 0xb9842000U, 0xb9842000U, 0xb9842000U, + 0xb9842000U, 0xb9842000U, 0xb9842000U, 0xb9842000U, 0x3ae73800U, 0xb9842000U, + 0xb9842000U, 0xbb14a400U, 0x39a9e000U, 0x39a9e000U, 0x39a9e000U, 0x39a9e000U, + 0x39a9e000U, 0x3b3f1c00U, 0x39a9e000U, 0xbb14a400U, 0x39a9e000U, 0x39a9e000U, + 0xbb14a400U, 0xbb14a400U, 0x39a9e000U, 0x3b3f1c00U, 0x39a9e000U, 0x39a9e000U, + 0x39a9e000U, 0x39a9e000U, 0x39a9e000U, 0x3b3f1c00U, 0x39a9e000U, 0x39a9e000U, + 0x39a9e000U, 0x3b3f1c00U, 0x39a9e000U, 0x3b3f1c00U, 0x39a9e000U, 0x39a9e000U, + 0x39a9e000U, 0xbb14a400U, 0x3b3f1c00U, 0x39a9e000U, 0x39cfa000U, 0x39cfa000U, + 0x39cfa000U, 0xbb35ac00U, 0x39cfa000U, 0x39cfa000U, 0x3b699400U, 0x39cfa000U, + 0x3b699400U, 0xbb35ac00U, 0x39cfa000U, 0x39cfa000U, 0x3b699400U, 0x39cfa000U, + 0xbb35ac00U, 0x39cfa000U, 0x39cfa000U, 0xbb35ac00U, 0x39cfa000U, 0xbb35ac00U, + 0x39cfa000U, 0x39cfa000U, 0x39cfa000U, 0xbb35ac00U, 0x39cfa000U, 0xbb35ac00U, + 0x39cfa000U, 0x39cfa000U, 0xbb35ac00U, 0x39cfa000U, 0x39cfa000U, 0x39cfa000U, + 0x39842000U, 0x39842000U, 0x3b14a400U, 0x39842000U, 0xbae73800U, 0x39842000U, + 0xbae73800U, 0xbae73800U, 0x39842000U, 0x39842000U, 0x39842000U, 0xbae73800U, + 0x3b14a400U, 0x39842000U, 0x39842000U, 0x39842000U, 0x3b14a400U, 0x39842000U, + 0x39842000U, 0xbae73800U, 0x39842000U, 0x3b14a400U, 0x39842000U, 0x39842000U, + 0x39842000U, 0x39842000U, 0xbae73800U, 0xbae73800U, 0x39842000U, 0x39842000U, + 0x3b14a400U, 0x39842000U, 0x3b48b500U, 0x39b26800U, 0x39b26800U, 0x39b26800U, + 0xbb1c1b00U, 0x39b26800U, 0xbb1c1b00U, 0x39b26800U, 0x39b26800U, 0x39b26800U, + 0x3b48b500U, 0x39b26800U, 0x3b48b500U, 0x39b26800U, 0x39b26800U, 0x39b26800U, + 0x39b26800U, 0x39b26800U, 0x39b26800U, 0x39b26800U, 0x3b48b500U, 0x3b48b500U, + 0x3b48b500U, 0x39b26800U, 0x39b26800U, 0x39b26800U, 0x3b48b500U, 0x39b26800U, + 0x39b26800U, 0x3b48b500U, 0x3b48b500U, 0x39b26800U, 0xbaff7200U, 0xb9631000U, + 0x3ac6ae00U, 0x3ac6ae00U, 0x3ac6ae00U, 0xb9631000U, 0xb9631000U, 0x3ac6ae00U, + 0x3ac6ae00U, 0xbaff7200U, 0xb9631000U, 0xb9631000U, 0xb9631000U, 0x3ac6ae00U, + 0xb9631000U, 0xb9631000U, 0xb9631000U, 0xbaff7200U, 0xb9631000U, 0xb9631000U, + 0xb9631000U, 0xbaff7200U, 0x3ac6ae00U, 0xb9631000U, 0xb9631000U, 0x3ac6ae00U, + 0xb9631000U, 0xb9631000U, 0xb9631000U, 0xb9631000U, 0xb9631000U, 0xbaff7200U, + 0xbb88d880U, 0xb9f34800U, 0xb9f34800U, 0xb9f34800U, 0xb9f34800U, 0xbb88d880U, + 0xb9f34800U, 0x3b54df00U, 0x3b54df00U, 0xb9f34800U, 0x3b54df00U, 0xb9f34800U, + 0xb9f34800U, 0xbb88d880U, 0x3b54df00U, 0xb9f34800U, 0xb9f34800U, 0xb9f34800U, + 0xb9f34800U, 0xb9f34800U, 0xb9f34800U, 0xb9f34800U, 0xb9f34800U, 0xb9f34800U, + 0xb9f34800U, 0xb9f34800U, 0xb9f34800U, 0xb9f34800U, 0x3b54df00U, 0xb9f34800U, + 0xb9f34800U, 0xb9f34800U, 0xb9631000U, 0xb9631000U, 0xbaff7200U, 0xbaff7200U, + 0xb9631000U, 0x3ac6ae00U, 0xb9631000U, 0xb9631000U, 0xb9631000U, 0xb9631000U, + 0x3ac6ae00U, 0x3ac6ae00U, 0xb9631000U, 0xb9631000U, 0xbaff7200U, 0xb9631000U, + 0xb9631000U, 0x3ac6ae00U, 0xb9631000U, 0xb9631000U, 0xb9631000U, 0xbaff7200U, + 0x3ac6ae00U, 0xb9631000U, 0xb9631000U, 0x3ac6ae00U, 0x3ac6ae00U, 0xb9631000U, + 0xb9631000U, 0xb9631000U, 0xb9631000U, 0x3ac6ae00U, 0xbaff7200U, 0xbaff7200U, + 0x3b243700U, 0x3991f800U, 0x3991f800U, 0x3991f800U, 0x3991f800U, 0x3991f800U, + 0xbaff7200U, 0x3991f800U, 0x3991f800U, 0x3991f800U, 0xbaff7200U, 0xbaff7200U, + 0x3991f800U, 0x3991f800U, 0x3991f800U, 0xbaff7200U, 0x3b243700U, 0xbaff7200U, + 0x3b243700U, 0x3991f800U, 0x3991f800U, 0x3991f800U, 0x3991f800U, 0xbaff7200U, + 0x3991f800U, 0x3b243700U, 0x3b243700U, 0xbaff7200U, 0x3991f800U, 0xbaff7200U, + 0xbb243700U, 0xb991f800U, 0xb991f800U, 0xb991f800U, 0xb991f800U, 0xb991f800U, + 0xbb243700U, 0xbb243700U, 0xb991f800U, 0xb991f800U, 0xbb243700U, 0xb991f800U, + 0xbb243700U, 0xb991f800U, 0xb991f800U, 0xb991f800U, 0x3aff7200U, 0x3aff7200U, + 0xb991f800U, 0xb991f800U, 0x3aff7200U, 0xb991f800U, 0xb991f800U, 0xb991f800U, + 0xb991f800U, 0x3aff7200U, 0xb991f800U, 0xb991f800U, 0xb991f800U, 0xbb243700U, + 0x3aff7200U, 0xb991f800U, 0x3b48b500U, 0x39b26800U, 0xbb1c1b00U, 0x39b26800U, + 0x39b26800U, 0x39b26800U, 0x39b26800U, 0x39b26800U, 0xbb1c1b00U, 0x3b48b500U, + 0x39b26800U, 0xbb1c1b00U, 0x39b26800U, 0x39b26800U, 0x3b48b500U, 0x3b48b500U, + 0x39b26800U, 0x39b26800U, 0x39b26800U, 0x39b26800U, 0xbb1c1b00U, 0x39b26800U, + 0x39b26800U, 0xbb1c1b00U, 0x39b26800U, 0x39b26800U, 0x39b26800U, 0x3b48b500U, + 0x3b48b500U, 0x3b48b500U, 0xbb1c1b00U, 0xbb1c1b00U, 0xbb243700U, 0x3aff7200U, + 0xb991f800U, 0xb991f800U, 0xb991f800U, 0xb991f800U, 0xbb243700U, 0xb991f800U, + 0x3aff7200U, 0xb991f800U, 0xb991f800U, 0xbb243700U, 0xbb243700U, 0xb991f800U, + 0x3aff7200U, 0xbb243700U, 0xb991f800U, 0xb991f800U, 0x3aff7200U, 0xb991f800U, + 0xb991f800U, 0x3aff7200U, 0x3aff7200U, 0xb991f800U, 0xbb243700U, 0xb991f800U, + 0xb991f800U, 0xb991f800U, 0xbb243700U, 0x3aff7200U, 0xb991f800U, 0xb991f800U, + 0xbb24fa80U, 0x3b541d80U, 0x39bc8c00U, 0xbb24fa80U, 0x39bc8c00U, 0xbb24fa80U, + 0x39bc8c00U, 0xbb24fa80U, 0x39bc8c00U, 0x39bc8c00U, 0x39bc8c00U, 0x39bc8c00U, + 0x39bc8c00U, 0xbb24fa80U, 0x39bc8c00U, 0x39bc8c00U, 0x3b541d80U, 0x3b541d80U, + 0x39bc8c00U, 0x3b541d80U, 0x39bc8c00U, 0x39bc8c00U, 0x39bc8c00U, 0x3b541d80U, + 0xbb24fa80U, 0x39bc8c00U, 0x3b541d80U, 0x39bc8c00U, 0x39bc8c00U, 0xbb24fa80U, + 0x39bc8c00U, 0x39bc8c00U, 0xbb06fb80U, 0x399a4400U, 0x399a4400U, 0x3b2d8c80U, + 0x3b2d8c80U, 0x399a4400U, 0x3b2d8c80U, 0x399a4400U, 0x399a4400U, 0xbb06fb80U, + 0x399a4400U, 0x399a4400U, 0x399a4400U, 0xbb06fb80U, 0xbb06fb80U, 0x399a4400U, + 0xbb06fb80U, 0x399a4400U, 0x3b2d8c80U, 0x399a4400U, 0x3b2d8c80U, 0x399a4400U, + 0x399a4400U, 0xbb06fb80U, 0x3b2d8c80U, 0x399a4400U, 0xbb06fb80U, 0x399a4400U, + 0x399a4400U, 0x3b2d8c80U, 0x399a4400U, 0x399a4400U, 0x39bc8c00U, 0x39bc8c00U, + 0x39bc8c00U, 0x39bc8c00U, 0x39bc8c00U, 0x39bc8c00U, 0xbb24fa80U, 0x3b541d80U, + 0xbb24fa80U, 0xbb24fa80U, 0x39bc8c00U, 0x39bc8c00U, 0x39bc8c00U, 0x39bc8c00U, + 0xbb24fa80U, 0x39bc8c00U, 0x39bc8c00U, 0x39bc8c00U, 0xbb24fa80U, 0x3b541d80U, + 0x39bc8c00U, 0x39bc8c00U, 0xbb24fa80U, 0x39bc8c00U, 0x39bc8c00U, 0x39bc8c00U, + 0x39bc8c00U, 0xbb24fa80U, 0x3b541d80U, 0x39bc8c00U, 0x39bc8c00U, 0x39bc8c00U, + 0xb96ff800U, 0xb96ff800U, 0xb96ff800U, 0x3ad1f900U, 0xbb06fb80U, 0xbb06fb80U, + 0xbb06fb80U, 0xbb06fb80U, 0xb96ff800U, 0xbb06fb80U, 0xb96ff800U, 0xb96ff800U, + 0xbb06fb80U, 0xb96ff800U, 0xb96ff800U, 0xb96ff800U, 0xb96ff800U, 0xb96ff800U, + 0xb96ff800U, 0x3ad1f900U, 0xbb06fb80U, 0xbb06fb80U, 0x3ad1f900U, 0x3ad1f900U, + 0x3ad1f900U, 0xb96ff800U, 0xb96ff800U, 0xbb06fb80U, 0xbb06fb80U, 0xb96ff800U, + 0x3ad1f900U, 0xbb06fb80U, 0x3b24fa80U, 0xb9bc8c00U, 0xb9bc8c00U, 0xb9bc8c00U, + 0x3b24fa80U, 0xb9bc8c00U, 0xb9bc8c00U, 0x3b24fa80U, 0xb9bc8c00U, 0xbb541d80U, + 0xb9bc8c00U, 0xb9bc8c00U, 0x3b24fa80U, 0xb9bc8c00U, 0x3b24fa80U, 0xb9bc8c00U, + 0xb9bc8c00U, 0xb9bc8c00U, 0xbb541d80U, 0xb9bc8c00U, 0xb9bc8c00U, 0xb9bc8c00U, + 0x3b24fa80U, 0xbb541d80U, 0x3b24fa80U, 0xb9bc8c00U, 0xb9bc8c00U, 0xb9bc8c00U, + 0xb9bc8c00U, 0xb9bc8c00U, 0xbb541d80U, 0xbb541d80U, 0xbb06fb80U, 0xbb06fb80U, + 0x399a4400U, 0x3b2d8c80U, 0x399a4400U, 0x399a4400U, 0x399a4400U, 0x399a4400U, + 0xbb06fb80U, 0x399a4400U, 0x399a4400U, 0x3b2d8c80U, 0x3b2d8c80U, 0x399a4400U, + 0x3b2d8c80U, 0x399a4400U, 0x399a4400U, 0x399a4400U, 0xbb06fb80U, 0x3b2d8c80U, + 0x399a4400U, 0xbb06fb80U, 0x399a4400U, 0x399a4400U, 0x399a4400U, 0xbb06fb80U, + 0xbb06fb80U, 0x399a4400U, 0xbb06fb80U, 0x3b2d8c80U, 0x399a4400U, 0xbb06fb80U, + 0xb9bc8c00U, 0xb9bc8c00U, 0xb9bc8c00U, 0xb9bc8c00U, 0xb9bc8c00U, 0xb9bc8c00U, + 0x3b24fa80U, 0x3b24fa80U, 0xbb541d80U, 0xb9bc8c00U, 0xb9bc8c00U, 0xb9bc8c00U, + 0xb9bc8c00U, 0xb9bc8c00U, 0xb9bc8c00U, 0xbb541d80U, 0xbb541d80U, 0xbb541d80U, + 0xbb541d80U, 0xb9bc8c00U, 0xbb541d80U, 0xb9bc8c00U, 0x3b24fa80U, 0xb9bc8c00U, + 0xb9bc8c00U, 0xb9bc8c00U, 0xb9bc8c00U, 0xb9bc8c00U, 0x3b24fa80U, 0xb9bc8c00U, + 0xb9bc8c00U, 0x3b24fa80U, 0x3b909fc0U, 0x3a008e00U, 0x3a008e00U, 0x3a008e00U, + 0x3a008e00U, 0x3a008e00U, 0x3a008e00U, 0x3a008e00U, 0x3a008e00U, 0x3a008e00U, + 0x3a008e00U, 0x3a008e00U, 0x3a008e00U, 0xbb60f880U, 0xbb60f880U, 0x3a008e00U, + 0x3b909fc0U, 0x3a008e00U, 0x3a008e00U, 0x3a008e00U, 0x3a008e00U, 0x3a008e00U, + 0xbb60f880U, 0x3a008e00U, 0x3a008e00U, 0x3b909fc0U, 0x3a008e00U, 0x3a008e00U, + 0x3a008e00U, 0xbb60f880U, 0x3a008e00U, 0x3a008e00U, +}; + + +// IQ3_XXS (ggml 18) has no committed checkpoint slice here, so its input is +// one legal SYNTHETIC block instead: an f16 d of 0.0125 followed by 96 +// payload bytes from a fixed LCG, which varies both the 4 grid-index bytes +// and the packed sign/scale word of every group. The EXPECTED values are +// still the oracle's own, from the same 237ad9b96 build. It is here because +// `DequantGgufRowToF32` omitted `case 18` for the same reason it omitted 19 +// and 66, and the three are fixed together in one shared branch. +inline constexpr uint8_t kIq3xxsGoldenBlocks[98] = { + 0x66, 0x22, 0x53, 0x89, 0x85, 0xed, 0x76, 0xe8, 0xd9, 0x88, 0x4a, 0x8b, + 0x02, 0xcf, 0xf5, 0x45, 0xd1, 0x95, 0x1c, 0x39, 0x63, 0xb6, 0x31, 0x3c, + 0x51, 0xd3, 0x80, 0xe8, 0x9d, 0xca, 0xf9, 0x7f, 0x57, 0xf8, 0xbb, 0x86, + 0x12, 0x2e, 0x65, 0x1e, 0x0a, 0x74, 0x1b, 0x21, 0x12, 0x80, 0x5a, 0x00, + 0x3f, 0xf4, 0x7c, 0x5f, 0x57, 0x5b, 0x0a, 0x6d, 0xf7, 0xe1, 0xc6, 0x05, + 0x86, 0xd6, 0x73, 0x86, 0xdd, 0xe1, 0x6a, 0x72, 0xb3, 0x04, 0xde, 0xc9, + 0xca, 0x53, 0xea, 0x23, 0xdb, 0x74, 0x60, 0x90, 0x42, 0xd5, 0x50, 0x30, + 0x69, 0xb0, 0x5a, 0x91, 0xf6, 0xc0, 0xb4, 0xcb, 0xb4, 0xc0, 0xfa, 0x5d, + 0x42, 0xa8, +}; + +inline constexpr uint32_t kIq3xxsGoldenBits[256] = { + 0x3c4cc000U, 0xbd7ff000U, 0x3db32800U, 0xbd199000U, 0x3db32800U, 0xbc4cc000U, + 0xbe465a00U, 0x3d7ff000U, 0x3e465a00U, 0x3d7ff000U, 0xbe0cc400U, 0x3d7ff000U, + 0x3e465a00U, 0xbc4cc000U, 0xbc4cc000U, 0xbe465a00U, 0xbd199000U, 0x3d199000U, + 0xbd7ff000U, 0xbd7ff000U, 0x3db32800U, 0x3db32800U, 0xbe265c00U, 0x3e265c00U, + 0xbde65800U, 0x3d7ff000U, 0xbe465a00U, 0x3e0cc400U, 0x3d199000U, 0xbc4cc000U, + 0x3e465a00U, 0xbd7ff000U, 0x3ed32600U, 0xbe0cc400U, 0xbf2ff500U, 0xbed32600U, + 0xbfc18d80U, 0x3f9e5c80U, 0xc0085de0U, 0xbf2ff500U, 0xbf9e5c80U, 0xbe0cc400U, + 0x3e0cc400U, 0x3e0cc400U, 0xbf2ff500U, 0x40085de0U, 0x3f2ff500U, 0xbfc18d80U, + 0xbf2ff500U, 0xbfc18d80U, 0x3f2ff500U, 0xc0085de0U, 0x3e0cc400U, 0xbe0cc400U, + 0x3ed32600U, 0x3ed32600U, 0x3f765700U, 0xbfc18d80U, 0xbf765700U, 0xbfc18d80U, + 0xbf2ff500U, 0x3e0cc400U, 0x3ed32600U, 0x3f765700U, 0x3f6ff100U, 0xbf6ff100U, + 0x3f6ff100U, 0xbe3ff400U, 0x3f0ff700U, 0xc01bf640U, 0xc039f460U, 0x3e3ff400U, + 0xbf0ff700U, 0xbf0ff700U, 0xbe3ff400U, 0x3f6ff100U, 0x4039f460U, 0x3fa7f580U, + 0xbf6ff100U, 0x3fd7f280U, 0x3f6ff100U, 0x3fa7f580U, 0xc003f7c0U, 0xbe3ff400U, + 0x3e3ff400U, 0xbf0ff700U, 0xbe3ff400U, 0x3f0ff700U, 0x3e3ff400U, 0xbe3ff400U, + 0xbfa7f580U, 0x3f0ff700U, 0x3fa7f580U, 0xbf6ff100U, 0x3fd7f280U, 0xc003f7c0U, + 0x3eacc200U, 0x40a75bf0U, 0x401729c0U, 0x3fd7f280U, 0x401729c0U, 0xc01729c0U, + 0xc08c5da0U, 0x408c5da0U, 0x406d8ac0U, 0x3fd7f280U, 0x3fd7f280U, 0x401729c0U, + 0x408c5da0U, 0xbeacc200U, 0x3f819180U, 0xc06d8ac0U, 0x3eacc200U, 0xc08c5da0U, + 0x401729c0U, 0xc0a75bf0U, 0x40425a40U, 0x3f819180U, 0x401729c0U, 0x3fd7f280U, + 0x3f819180U, 0xbeacc200U, 0x40425a40U, 0xbf819180U, 0x401729c0U, 0xc01729c0U, + 0x401729c0U, 0xc0a75bf0U, 0x404a59c0U, 0x40258f40U, 0x4000c4c0U, 0x40258f40U, + 0xbf5cbf00U, 0x40258f40U, 0xc04a59c0U, 0x3fb7f480U, 0x3e932a00U, 0x4000c4c0U, + 0x3f5cbf00U, 0x3e932a00U, 0x404a59c0U, 0xc08e90b0U, 0xc0258f40U, 0x3e932a00U, + 0x3e932a00U, 0x3fb7f480U, 0xbe932a00U, 0x3fb7f480U, 0x4000c4c0U, 0xc000c4c0U, + 0x3fb7f480U, 0x3e932a00U, 0xc000c4c0U, 0xc08e90b0U, 0x3e932a00U, 0x3e932a00U, + 0x3e932a00U, 0x3e932a00U, 0x3fb7f480U, 0x3fb7f480U, 0x3e9ff600U, 0xbfc7f380U, + 0x3fc7f380U, 0xbe9ff600U, 0xc09af650U, 0x405bf240U, 0xbfc7f380U, 0x3e9ff600U, + 0x3e9ff600U, 0xc00bf740U, 0x3f6ff100U, 0x3e9ff600U, 0x3e9ff600U, 0xc09af650U, + 0x400bf740U, 0x3fc7f380U, 0x3e9ff600U, 0xbfc7f380U, 0x405bf240U, 0xbf6ff100U, + 0xbe9ff600U, 0x3e9ff600U, 0xbe9ff600U, 0x3e9ff600U, 0xc00bf740U, 0xbfc7f380U, + 0xbe9ff600U, 0x3f6ff100U, 0x405bf240U, 0x4033f4c0U, 0x3fc7f380U, 0xc09af650U, + 0x405bf240U, 0x405bf240U, 0xbfc7f380U, 0x3fc7f380U, 0xbfc7f380U, 0xbfc7f380U, + 0x409af650U, 0xbf6ff100U, 0xbf6ff100U, 0xbe9ff600U, 0xc033f4c0U, 0x3f6ff100U, + 0xc033f4c0U, 0x3fc7f380U, 0x405bf240U, 0x3f6ff100U, 0xc00bf740U, 0xc09af650U, + 0x3e9ff600U, 0x3e9ff600U, 0xbe9ff600U, 0x3f6ff100U, 0xbf6ff100U, 0x3fc7f380U, + 0xc05bf240U, 0x3f6ff100U, 0xc00bf740U, 0x409af650U, 0x409af650U, 0x3f6ff100U, + 0x3fc7f380U, 0x4081f7e0U, 0x3fa7f580U, 0xbf498d00U, 0x3e865e00U, 0xc038c140U, + 0xc0822b10U, 0xbf498d00U, 0xbe865e00U, 0xbe865e00U, 0xbf498d00U, 0xc01729c0U, + 0x4038c140U, 0xbfa7f580U, 0xbfa7f580U, 0xbe865e00U, 0x4038c140U, 0xc038c140U, + 0xbe865e00U, 0x40822b10U, 0x3f498d00U, 0xbfa7f580U, 0x3f498d00U, 0x401729c0U, + 0x4038c140U, 0x3fa7f580U, 0x405a58c0U, 0xc01729c0U, 0x3e865e00U, 0x405a58c0U, + 0x40822b10U, 0x3f498d00U, 0xbfa7f580U, 0x405a58c0U, +}; + +} // namespace vllm_test diff --git a/tests/vt/test_ops_quant_dot.cpp b/tests/vt/test_ops_quant_dot.cpp index 73ee865fc..986ff2eb3 100644 --- a/tests/vt/test_ops_quant_dot.cpp +++ b/tests/vt/test_ops_quant_dot.cpp @@ -42,6 +42,7 @@ #include +#include "iq1_golden_vectors.h" // oracle-produced IQ1_S / IQ1_XXXS goldens #include "vt/cpu/cpu_threadpool.h" // Threadpool::SwapForTesting (via -I src) #include "vt/cpu/cpu_quant_iq_tables.h" // kIq1sGrid provenance check #include "vt/device.h" @@ -133,24 +134,39 @@ const WeightCase kWeightCases[] = { {vt::DType::kIQ3_XXS, 256, 98, 0, -1, -1, "iq3_xxs"}, {vt::DType::kIQ2_S, 256, 82, 0, -1, -1, "iq2_s"}, // IQ1_S (1.5625 bpw) is 96.92 % of the parameters of the - // Qwen3.8-2.4T-A95B UD-IQ1_S checkpoint ENG-EXPERT-STREAM targets. - // IQ1_S needs its own ceiling, measured not guessed. Its weights are - // ternary times a per-32 scale, so a super-block spans a wider dynamic - // range than a 4-6 bit codebook does, while Q8_K gives the activation ONE - // scale per 256 elements. The residual sits at ~5.2e-4 on the thinnest - // shape here (m=4, n=1 is FOUR dot products, so the statistic is noisy), - // against 5e-4. Set to 2e-3: ~4x the observed residual, and still an order - // of magnitude below the WEAKEST decode defect measured by mutation. - // Dropping the delta term reads 2.7e-2 (the smallest signal of the nine), - // the wrong delta sign bit 3.0e-1, ignoring the qh scale 9.4e-1, dropping - // the delta sign 5.7, and ignoring the 3 high grid-index bits 5.7e+1. So - // the ceiling sits in the empty band between noise and every defect. - {vt::DType::kIQ1_S, 256, 50, 0, -1, -1, "iq1_s", 2e-3}, - // IQ1_XXXS (1.1875 bpw) is 96.92 % of the Qwen3.8-2.4T-A95B UD-Q1_0 - // checkpoint. Type 66, from the pinned FORK oracle `llama-cpp-unsloth`. - // Same ceiling and the same reason as IQ1_S: ternary lanes times a per-32 - // scale against a Q8_K activation that gets one scale per 256. - {vt::DType::kIQ1_XXXS, 256, 38, 0, -1, -1, "iq1_xxxs", 2e-3}, + // Qwen3.8-2.4T-A95B UD-IQ1_S checkpoint ENG-EXPERT-STREAM targets, and + // IQ1_XXXS (1.1875 bpw) is 96.92 % of the UD-Q1_0 one. Both need a ceiling + // above upstream's 5e-4: their weights are ternary times a per-32 scale, so + // a super-block spans a wider dynamic range than a 4-6 bit codebook does, + // while Q8_K gives the activation ONE scale per 256 elements. + // + // 6e-4, set from the measurement rather than chosen for headroom. Every + // shape in this case, all 12 per type, re-measured 16 August 2026 with the + // ceiling forced to 1e-12 so doctest prints each captured value: + // + // iq1_s max 5.240e-4 at m=4 n=1, then 3.498e-4 at m=1 n=1 + // iq1_xxxs max 3.109e-4 at m=1 n=1, then 2.637e-5 at m=1 n=7 + // + // The n=1 column is where both the signal and the noise live: the NMSE + // denominator there is a single dot product, so it neither averages nor + // cancels. Every n=7 and n=16 shape sits below 3e-5. + // + // An earlier revision set this to 2e-3, described as ~4x the residual. That + // was a relaxation past the point where the statistic can discriminate. + // Doubling kIq1sDelta to 0.25 moves iq1_s to 6.967e-4, which 6e-4 fails and + // 2e-3 passes. Measure before widening a bound. + // + // Be precise about what this bound is worth, because the same mutation + // moves iq1_xxxs the WRONG WAY: 3.109e-4 unmutated against 1.420e-4 with the + // doubled delta, so NO ceiling catches that defect on this statistic. An + // NMSE against a dequant-f32 reference cannot seal a decode parameter at + // all, since both sides decode through the same `BlockToFloat`. What seals + // it is the pair of golden-vector cases below, which compare against the + // ORACLES bit for bit. This ceiling bounds quantization error, which is its + // own job, and is kept tight enough to stay a second signal where it can be + // one. + {vt::DType::kIQ1_S, 256, 50, 0, -1, -1, "iq1_s", 6e-4}, + {vt::DType::kIQ1_XXXS, 256, 38, 0, -1, -1, "iq1_xxxs", 6e-4}, {vt::DType::kMXFP4, 32, 17, -1, -1, 0, "mxfp4"}, }; @@ -665,6 +681,79 @@ TEST_CASE("kIq1xxxsGrid is the PINNED FORK table, not a look-alike") { CHECK(lanes[2] == 397); // +1 } +TEST_CASE("kIq1sDelta is upstream IQ1S_DELTA, not a value this tree chose") { + // `ggml/src/ggml-common.h:1121` at the pinned 237ad9b96 is + // `#define IQ1S_DELTA 0.125f`. The FORK reuses that same macro for IQ1_XXXS + // (`dequantize_row_iq1_xxxs` and `ggml_vec_dot_iq1_xxxs_q8_K` both spell + // IQ1S_DELTA), so one constant serves BOTH encodings here and one wrong value + // corrupts both at once. + // + // Sealed by value for exactly the reason the two grids are sealed by digest: + // `DequantIQ1_S`, `DequantIQ1_XXXS`, `VecDotIQ1_SQ8_K` and + // `VecDotIQ1_XXXSQ8_K` all read THIS definition, so every one of them agrees + // with every other whatever it holds. Consistency is not correctness. The + // grid seal stopped one table short of this constant, and a doubled delta + // survived the whole suite until the golden vectors below were added. + CHECK(vt::cpu::kIq1sDelta == 0.125F); +} + +// Compare a decode against the oracle-produced goldens, BIT for BIT. Both sides +// evaluate the same f32 expression `dl * (grid[j] + delta)`, so equality is +// exact and any difference means a decode PARAMETER diverged, not that rounding +// moved. Asserting per element (rather than on a digest or a mismatch count) +// makes the assertion total say how many values were actually examined. +void CheckAgainstOracle(vt::DType dtype, const uint8_t* blocks, + const uint32_t* golden_bits, size_t n) { + std::vector got(n); + vt::cpu::BlockToFloat(dtype)(blocks, got.data(), static_cast(n)); + for (size_t i = 0; i < n; ++i) { + CAPTURE(i); + uint32_t bits = 0; + std::memcpy(&bits, &got[i], sizeof(bits)); + CHECK(bits == golden_bits[i]); + } +} + +TEST_CASE("IQ1_S decodes REAL checkpoint bytes as the PINNED ORACLE does") { + // The first independent reference this encoding has. Everything else in this + // file decodes the weight with `vt::cpu::BlockToFloat`, the function under + // test, so it is independent only in the summation: a wrong scale field, a + // wrong delta magnitude or a flipped delta sign moves the reference and the + // kernel together and they still agree. + // + // Here the expected values come from ggml-org/llama.cpp @ 237ad9b96 running + // its OWN `ggml_get_type_traits(GGML_TYPE_IQ1_S)->to_float`, and the inputs + // are real `blk.0.ffn_gate_exps.weight` bytes from the UD-IQ1_S checkpoint. + // Provenance and the reproduction recipe are in iq1_golden_vectors.h. + // + // This seals the DEQUANT arm. The vec_dot arm is tied to it by the G3 + // "vec_dot matches f64 dequantize-then-dot" case above, so a defect injected + // into either one alone now fails somewhere, and a defect injected into both + // fails here. + CHECK(std::size(vllm_test::kIq1sGoldenBlocks) == 4 * 50); // 4 blocks + CHECK(std::size(vllm_test::kIq1sGoldenBits) == 4 * 256); + CheckAgainstOracle(vt::DType::kIQ1_S, vllm_test::kIq1sGoldenBlocks, + vllm_test::kIq1sGoldenBits, + std::size(vllm_test::kIq1sGoldenBits)); +} + +TEST_CASE("IQ1_XXXS decodes REAL checkpoint bytes as the PINNED FORK does") { + // Same argument as the IQ1_S case above. The oracle is the fork, + // unslothai/llama.cpp @ 36fe8e1cc, because no upstream llama.cpp defines ggml + // type 66 at all; the inputs are real UD-Q1_0 bytes. + // + // What this does and does not establish is the same as the spec's + // 1179648-weight run, of which these four blocks are the committed slice: it + // removes transcription error from OUR C++ and proves the block layout is + // read correctly. It does NOT make the fork gateable, because the fork's own + // decode is still unvalidated against anything but itself (#933). + CHECK(std::size(vllm_test::kIq1xxxsGoldenBlocks) == 4 * 38); // 4 blocks + CHECK(std::size(vllm_test::kIq1xxxsGoldenBits) == 4 * 256); + CheckAgainstOracle(vt::DType::kIQ1_XXXS, vllm_test::kIq1xxxsGoldenBlocks, + vllm_test::kIq1xxxsGoldenBits, + std::size(vllm_test::kIq1xxxsGoldenBits)); +} + TEST_CASE("G3 vec_dot is bit-exact run to run (fixed reduction order)") { for (const WeightCase& c : kWeightCases) { CAPTURE(std::string(c.name));