diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 5e44e992f..1897927cf 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -270,3 +270,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#995](https://github.com/mudler/vllm.cpp/issues/995) | `ENG-EXPERT-STREAM` | `check-env-doc` and `test_check_env_doc` RED in `scripts/agent-preflight.sh` on **every** branch based on `3005447f8`: `VT_MOE_EXPERT_STREAM`, `VT_MOE_EXPERT_STREAM_SLOTS` and `VT_MOE_EXPERT_STREAM_SLOT_BYTES` are read from `src/vllm/model_executor/models/qwen3_5.cpp` (`:5145`, `:5164` @ `3005447f8`) and appear in neither `docs/ENVIRONMENT.md` (`grep -c` returns **0**) nor `scripts/env-doc-allowlist.txt`. Arrived with [#993](https://github.com/mudler/vllm.cpp/pull/993). Found while gating [#986](https://github.com/mudler/vllm.cpp/issues/986) and proved pre-existing with a matched-arm check rather than asserted: the three sites are in a file that branch does not touch, and `git diff origin/main...HEAD | grep '^+.*VT_MOE_EXPERT'` returns nothing. NOT fixed in flow: documenting a knob means stating its default and when to touch it, and the expert-streamer's slot accounting belongs to the row that added it, so a plausible-sounding entry written by a passer-by is how `docs/ENVIRONMENT.md` stops being trustworthy. It is a PRE-FLIGHT gate, so it fails before every edit and presents to each author as a red their own diff caused, which is the shape [#965](https://github.com/mudler/vllm.cpp/issues/965) and [#968](https://github.com/mudler/vllm.cpp/issues/968) both took | bug | | [#987](https://github.com/mudler/vllm.cpp/issues/987) | `LTX25-RETAKE` | Two `ltx-2.5` refusal messages state reasons that are no longer true. (a) `src/vllm/multimodal/ltx2_video.cpp:1608 @ 0e1bee42f` says "nothing reads `ref_video_dir` at all", and MiniMax-H3 has always consumed the directory in full (`ReadReferenceClipChw`, `src/vllm/multimodal/minimax_h3_video.cpp:135 @ 0e1bee42f`, called at `:650`); [#975](https://github.com/mudler/vllm.cpp/issues/975) inherited the wider claim from this message. The claim that holds is narrower: the LTX-2.5 engine never reads the directory's CONTENTS. (b) `ltx2_video.cpp:1636-1638 @ 0e1bee42f` says "there is no AUDIO_VAE_ENCODER key filter", and `c2019b0e3` landed `Ltx2AudioVaeEncoderKeyRules()` (`include/vllm/model_executor/models/ltx2_audio_input.h:73 @ 0e1bee42f`) with a live call through `Ltx2EncodeAudioToLatent`. Both rewritten in the `WHAT IS *NOT* THE REASON` shape in the same flow, with one assertion tied to the LOCAL fact that the LTX side now reads the directory | bug | | [#995](https://github.com/mudler/vllm.cpp/issues/995) | `ENG-EXPERT-STREAM` | `check-env-doc` and `test_check_env_doc` are RED on `origin/main`, so every branch cut from it inherits a preflight failure its own diff did not cause: `VT_MOE_EXPERT_STREAM`, `VT_MOE_EXPERT_STREAM_SLOTS` and `VT_MOE_EXPERT_STREAM_SLOT_BYTES` are read from `src/vllm/model_executor/models/qwen3_5.cpp` (`:5145`, `:5195`, `:5189` @ `4496ef196`) and appear in neither `docs/ENVIRONMENT.md` nor `scripts/env-doc-allowlist.txt`. They arrived with the `ENG-EXPERT-STREAM` W4 wiring commit `3005447f8` ([#993](https://github.com/mudler/vllm.cpp/pull/993)). Same shape as [#965](https://github.com/mudler/vllm.cpp/issues/965)/[#968](https://github.com/mudler/vllm.cpp/issues/968): a pre-existing red presenting to each author in turn as their own. Fixed by DOCUMENTING all three in `docs/ENVIRONMENT.md`, not by allowlisting them — a knob that changes the host memory a MoE decode reserves and disables the default-on grouped-MoE path is a deployment surface, not a kernel-internal micro-tuning switch | bug +| [#1023](https://github.com/mudler/vllm.cpp/issues/1023) | `ENG-EXPERT-STREAM` | The IQ1_S (ggml 19) / IQ1_XXXS (ggml 66) decode landed in [#946](https://github.com/mudler/vllm.cpp/pull/946) with every parameter EXCEPT the grid pinned only by self-consistency. The grid seal (FNV-1a digest + lane census) works and stops one table short: `ReferenceDotF64` (`tests/vt/test_ops_quant_dot.cpp:231-236`) and the G3 NMSE reference (`:915`) both decode the weight with `vt::cpu::BlockToFloat`, the function under test, so they are independent only in the SUMMATION. Three injected defects, each applied and compiled, left the suite green with an UNCHANGED assertion count: `kIq1sDelta` `0.125F`->`0.25F` (`cpu_quant_iq_tables.h:422`, affects BOTH encodings), the IQ1_S delta sign inverted in dequant AND vec_dot, and the IQ1_S scale read from `qh` bits 13-15 instead of 12-14 in both paths. Second defect, same PR: `gguf_dequant.cpp:107-116` lists no `case 19` and no `case 66`, so the expansion path throws `unsupported ggml type` for the two encodings the target checkpoints are 96.92 % made of, and `RouteGgufTensor` (`gguf_keep_quant.cpp:122`) sends a tensor there whenever `VT_CPU_REF` is on, keep-quant is off, K is ragged, or the role is not verbatim — a refusal to load on the reference lane. **ggml 18 (IQ3_XXS) carried the same omission**, pre-existing since the DeepSeek-V4 UD-IQ2_XXS port, and all three are one shared `switch` branch. Repaired by golden vectors whose EXPECTED values come from the ORACLES themselves (`ggml_get_type_traits(type)->to_float` built from `ggml-org/llama.cpp @ 237ad9b96` for 18/19 and `unslothai/llama.cpp @ 36fe8e1cc` for 66) over REAL checkpoint bytes, which is the first reference for this encoding family that is not this tree's own decoder. Also: the `2e-3` NMSE ceiling passed the doubled-delta defect that `6e-4` fails (measured 5.240e-4 unmutated, 6.967e-4 mutated on `iq1_s`), and on `iq1_xxxs` that defect moves the statistic the WRONG WAY (3.109e-4 -> 1.420e-4), so no ceiling can catch it and only the goldens can; and both new census cases claimed TOTAL coverage of 1702 records while summing to 864, F32's 838 tensors omitted. Spec [`expert-streaming.md`](specs/expert-streaming.md) | bug | diff --git a/.agents/specs/expert-streaming.md b/.agents/specs/expert-streaming.md index 5c808bdea..860418b99 100644 --- a/.agents/specs/expert-streaming.md +++ b/.agents/specs/expert-streaming.md @@ -557,6 +557,70 @@ oracle gateable: both sides are transcriptions of the same source, so a defect i the FORK would be reproduced identically by both. Only building and running the fork closes that, which is what #933 owes. +### The decode is now sealed against the ORACLES, not against itself (#1023) + +An independent review of #946 found that this seal covered the CODEBOOKS and +nothing else. `kIq1sGrid` and `kIq1xxxsGrid` each carry an FNV-1a digest and a +lane census, and corrupting a grid entry does fail. Every other decode parameter +was pinned only by self-consistency, because `ReferenceDotF64` and the G3 +`MatmulBTQuant` NMSE reference both decode the weight with +`vt::cpu::BlockToFloat`, the function under test. They are independent in the +SUMMATION and nowhere else. Three defects were injected, each applied and +compiled, and the whole gate stayed green with an unchanged assertion count: + +| Mutation | Before #1023 | After | +|---|---|---| +| `kIq1sDelta` `0.125F` to `0.25F`, which hits BOTH encodings | UNCAUGHT | 3 cases, 2049 assertions fail | +| IQ1_S delta sign inverted in dequant AND vec_dot | UNCAUGHT | 1 case, 1024 assertions fail | +| IQ1_S scale from `qh` bits 13-15 instead of 12-14, both paths | UNCAUGHT | 1 case, 960 assertions fail | + +What closes it is a committed golden-vector fixture, +`tests/vt/iq1_golden_vectors.h`, whose EXPECTED values are produced by the +oracles themselves rather than by this tree: +`ggml_get_type_traits(type)->to_float`, called in a build of +`ggml-org/llama.cpp @ 237ad9b96` for IQ1_S and IQ3_XXS, and of +`unslothai/llama.cpp @ 36fe8e1cc7f2b3b8c92fdda0ab07600141921786` for IQ1_XXXS. +The IQ1 inputs are real `blk.0.ffn_gate_exps.weight` bytes, four 256-element +blocks from shard 2 of each split, so this commits a reproducible slice of the +1179648-weight run above. Agreement is BIT-EXACT on all 1024 values per +encoding, and `kIq1sDelta` is additionally sealed by value against upstream +`IQ1S_DELTA` (`ggml-common.h:1121`). + +This is stronger than the 15 August run, and it is worth saying how. That run +compared our C++ against a hand transcription of the fork, so BOTH sides were +transcriptions and a defect in the FORK would have been reproduced identically +by each. These vectors are decoded by the fork's own compiled code. That still +does not make the fork gateable in the sense #933 owes, which is running the +MODEL, but the fork's decoder is no longer transcribed at all. + +The same review found the NMSE ceiling widened past the point where it +discriminates. It was `2e-3`, described as about 4x the residual. Re-measured 16 +August 2026 over all 12 shapes per type, with the ceiling forced to `1e-12` so +doctest prints every captured value: + +| Type | Unmutated max | With `kIq1sDelta = 0.25` | +|---|---|---| +| `iq1_s` | 5.240e-4 (m=4, n=1) | 6.967e-4 | +| `iq1_xxxs` | 3.109e-4 (m=1, n=1) | 1.420e-4 | + +So `2e-3` passed that defect and `6e-4`, the value now set, fails it. The +`iq1_xxxs` column is the more useful half: the defect moves that statistic the +WRONG WAY, so NO ceiling catches it there. An NMSE against a dequant-f32 +reference cannot seal a decode parameter at all when both sides decode through +the same function. The ceiling bounds quantization error, which is its own job; +the goldens carry the decode. + +Two further findings, repaired in the same flow. `DequantGgufRowToF32` listed no +`case 19` and no `case 66`, so the expansion path threw `unsupported ggml type` +for the two encodings the target checkpoints are 96.92 % made of, and +`RouteGgufTensor` routes a tensor there whenever `VT_CPU_REF` is set, keep-quant +is off, K is ragged, or the role is not verbatim: a refusal to load on the +reference lane. `case 18` (IQ3_XXS, the DeepSeek-V4 UD-IQ2_XXS `ffn_down` +encoding) carried the same omission and is fixed with it, gated on its own +oracle-produced golden. And both checkpoint-census cases claimed TOTAL coverage +of 1702 tensor records while enumerating six of the seven encodings and summing +to 864; they now carry F32's 838 tensors and assert that the buckets sum. + Why this is in this spec rather than its own row: the encoding is the load path's half of the same capability. A streamer that can address an expert slice it cannot decode moves bytes for nothing, so the row's own gate cannot be met diff --git a/src/vllm/model_executor/model_loader/gguf_dequant.cpp b/src/vllm/model_executor/model_loader/gguf_dequant.cpp index 0a8bd4eb1..e8d5423f0 100644 --- a/src/vllm/model_executor/model_loader/gguf_dequant.cpp +++ b/src/vllm/model_executor/model_loader/gguf_dequant.cpp @@ -113,11 +113,22 @@ std::vector DequantGgufRowToF32(uint32_t ggml_type, const uint8_t* data, case 14: // Q6_K case 22: // IQ2_S (~2.5-bit codebook; UD-IQ2_M ffn_gate/up experts) case 39: // MXFP4 (OCP micro-scaling fp4; UD-IQ2_M ffn_down experts) + case 18: // IQ3_XXS (~3-bit codebook; UD-IQ2_XXS ffn_down) + case 19: // IQ1_S (1.5625 bpw; Qwen3.8-2.4T-A95B UD-IQ1_S experts) + case 66: // IQ1_XXXS (1.1875 bpw; UD-Q1_0 experts, fork-anchored) case 16: { // IQ2_XXS (~2-bit codebook; UD-IQ2_XXS DeepSeek-V4 vehicle) // The block decoders moved to vt (src/vt/cpu/cpu_quant_dequant.cpp) so // the loader oracle and the compute-in-quant GEMM's generic fallback // share ONE implementation. The code is byte-identical to what lived // here, so numerics are unchanged; this test suite gates that. + // This list is the loader's EXPANSION path, and it has drifted behind + // `BlockDTypeFromGgmlTypeId` twice: 18, 19 and 66 were all decodable in + // vt while this switch still threw "unsupported ggml type" for them. The + // drift is not a corner case, because `RouteGgufTensor` routes a tensor + // here whenever the VT_CPU_REF oracle switch is on, keep-quant is off, K + // is ragged, or the role is not a verbatim one. A missing case is + // therefore a refusal to load the checkpoint on the reference lane. + // Keep this list in step with vt/dtype.cpp `kBlockDTypes`. // NOTE: MXFP4 (39) decodes here through the vt kMXFP4 block dtype — the // GGML micro-scaling form (kvalues_mxfp4 == 2*e2m1, scale 2^(byte-128)) — // NOT through the compressed-tensors E8M0ToF32 path used for NVFP4 (40) diff --git a/src/vt/cpu/cpu_quant_dot.cpp b/src/vt/cpu/cpu_quant_dot.cpp index e50aba275..82ca1cb02 100644 --- a/src/vt/cpu/cpu_quant_dot.cpp +++ b/src/vt/cpu/cpu_quant_dot.cpp @@ -588,11 +588,14 @@ void VecDotIQ3_XXSQ8_K(int n, float* s, size_t bs, const void* vx, size_t bx, // bits from `qh[ib]`; kIq1sGrid entries are packed TERNARY (-1/0/+1) bytes, so // there is no sign array to apply. // -// The delta term is why this kernel reads `bsums` and the others do not. IQ1_S -// reconstructs a weight as dl*(grid[j] + delta), so the dot splits into -// sum(dl*grid*q8) plus delta*dl*sum(q8), and that second sum over each group of -// 16 is exactly what Q8_K already caches in `bsums`. Recomputing it from `qs` -// would be arithmetically identical but would read the activation twice. +// The delta term is why this kernel reads `bsums`. IQ1_S reconstructs a weight +// as dl*(grid[j] + delta), so the dot splits into sum(dl*grid*q8) plus +// delta*dl*sum(q8), and that second sum over each group of 16 is exactly what +// Q8_K already caches in `bsums`. Recomputing it from `qs` would be +// arithmetically identical but would read the activation twice. +// +// The reason is specific to the delta; the FIELD is not. Q2_K, Q4_K and Q5_K +// read `bsums` too, for their own per-sub-block minimum term. void VecDotIQ1_SQ8_K(int n, float* s, size_t bs, const void* vx, size_t bx, const void* vy, size_t by, int nrc) { VT_CHECK(n % kQK_K == 0, "vec_dot_iq1_s_q8_K: n must be a multiple of 256"); diff --git a/src/vt/cpu/cpu_quant_iq_tables.h b/src/vt/cpu/cpu_quant_iq_tables.h index 9965cf341..0309d33f0 100644 --- a/src/vt/cpu/cpu_quant_iq_tables.h +++ b/src/vt/cpu/cpu_quant_iq_tables.h @@ -1,8 +1,16 @@ -// I-quant codebook + sign tables — a 1:1 mirror of llama.cpp @ 237ad9b96 -// `ggml/src/ggml-common.h`: +// I-quant codebook + sign tables. Every table here but one mirrors llama.cpp +// @ 237ad9b96 `ggml/src/ggml-common.h` 1:1: // :499 kmask_iq2xs (8) :503 ksigns_iq2xs (128) // :550 iq2xxs_grid (256) :1007 iq3xxs_grid (256) // :748 iq2s_grid (1024) :1116 kvalues_mxfp4 (16) +// :1121 IQ1S_DELTA :1124 iq1s_grid (2048) +// +// The exception is `kIq1xxxsGrid` (256), which comes from a DIFFERENT tree: +// the pinned fork oracle unslothai/llama.cpp @ 36fe8e1cc, +// `ggml-common.h:2095 iq1_xxxs_grid`, because ggml type 66 is declared by no +// upstream llama.cpp. See .agents/oracles/llama-cpp-unsloth.md and the note +// above that table. IQ1_XXXS reuses upstream's own IQ1S_DELTA unchanged, so the +// one delta constant serves both IQ1 encodings. // // These are the shared decode tables for the IQ2_XXS / IQ3_XXS / IQ2_S / MXFP4 // block encodings. They were originally private to cpu_quant_dequant.cpp (the diff --git a/tests/vllm/test_gguf_dequant.cpp b/tests/vllm/test_gguf_dequant.cpp index 6e5e0e5d0..6dc0020b0 100644 --- a/tests/vllm/test_gguf_dequant.cpp +++ b/tests/vllm/test_gguf_dequant.cpp @@ -2,6 +2,7 @@ #include #include +#include #include #include "vllm/model_executor/model_loader/gguf_dequant.h" @@ -448,3 +449,57 @@ TEST_CASE("DequantGgufRowToF32 rejects unsupported i-quant type") { std::vector b2(136, 0); CHECK_THROWS_AS(DequantGgufRowToF32(23, b2.data(), 256), std::runtime_error); } + +// --- IQ1_S (19) / IQ1_XXXS (66): the two encodings the Qwen3.8-2.4T-A95B +// checkpoints are 96.92 % made of. `vt::cpu::BlockToFloat` has decoded both +// since #946, but this expand-to-bf16 path did not list them, so it threw +// "unsupported ggml type". That is not a corner: `RouteGgufTensor` sends a +// tensor here whenever the VT_CPU_REF oracle switch is on, keep-quant is off, +// K is ragged, or the role is not a verbatim one, so the reference lane could +// not load the target checkpoint at all. +// +// Gated on the ORACLE-produced goldens rather than on "does not throw": the +// inputs are real checkpoint bytes and the expected values come from the pinned +// upstream and fork ggml themselves. Provenance in +// tests/vt/iq1_golden_vectors.h. +#include "../vt/iq1_golden_vectors.h" + +namespace { + +void CheckGgufDequantAgainstOracle(uint32_t ggml_type, const uint8_t* blocks, + const uint32_t* golden_bits, size_t n) { + const std::vector out = + DequantGgufRowToF32(ggml_type, blocks, static_cast(n)); + REQUIRE(out.size() == n); + for (size_t i = 0; i < n; ++i) { + CAPTURE(i); + uint32_t bits = 0; + std::memcpy(&bits, &out[i], sizeof(bits)); + CHECK(bits == golden_bits[i]); + } +} + +} // namespace + +TEST_CASE("DequantGgufRowToF32 IQ1_S row matches the pinned oracle") { + CheckGgufDequantAgainstOracle(19, vllm_test::kIq1sGoldenBlocks, + vllm_test::kIq1sGoldenBits, + std::size(vllm_test::kIq1sGoldenBits)); +} + +TEST_CASE("DequantGgufRowToF32 IQ1_XXXS row matches the pinned FORK oracle") { + CheckGgufDequantAgainstOracle(66, vllm_test::kIq1xxxsGoldenBlocks, + vllm_test::kIq1xxxsGoldenBits, + std::size(vllm_test::kIq1xxxsGoldenBits)); +} + +TEST_CASE("DequantGgufRowToF32 IQ3_XXS row matches the pinned oracle") { + // Same omission, found by the same review, in the same shared branch, so it + // is fixed in the same flow rather than filed and deferred (issue #1023). + // IQ3_XXS is the `ffn_down` encoding of the DeepSeek-V4 UD-IQ2_XXS vehicle, + // and `vt::cpu::BlockToFloat` has decoded it since that port; only this + // expansion path never listed it. + CheckGgufDequantAgainstOracle(18, vllm_test::kIq3xxsGoldenBlocks, + vllm_test::kIq3xxsGoldenBits, + std::size(vllm_test::kIq3xxsGoldenBits)); +} diff --git a/tests/vllm/test_gguf_keep_quant.cpp b/tests/vllm/test_gguf_keep_quant.cpp index 371d480d9..97713ebac 100644 --- a/tests/vllm/test_gguf_keep_quant.cpp +++ b/tests/vllm/test_gguf_keep_quant.cpp @@ -277,6 +277,69 @@ TEST_CASE("KeepQuantDType covers the executable encodings") { } } +namespace { + +// One row of a checkpoint census: an encoding, how many tensor records carry +// it, and whether the loader is expected to keep it in blocks or expand it. +// `keep_quant == false` is a real entry, not an omission: F32 is 838 of the +// 1702 records in both target checkpoints, it is served by the expansion path, +// and leaving it out is what let an incomplete list call itself total. +struct CensusRow { + uint32_t ggml_type; + const char* name; + int tensors; + bool keep_quant; +}; + +// `split.tensors.count` from the shard headers, the same number on both +// checkpoints. The list below must add up to exactly this. +constexpr int kQwen38DeclaredTensors = 1702; + +// unsloth/Qwen3.8-2.4T-A95B-GGUF @ 567d3e6ac2, quant UD-IQ1_S, all 12 shards. +constexpr CensusRow kUdIq1sCensus[] = { + {19, "iq1_s", 276, true}, // 96.92 % of params + {13, "q5_k", 420, true}, {10, "q2_k", 3, true}, + {14, "q6_k", 162, true}, {12, "q4_k", 2, true}, + {8, "q8_0", 1, true}, {0, "f32", 838, false}, +}; + +// The same census over UD-Q1_0, all 10 shards. Identical apart from the expert +// encoding: ggml 66 where UD-IQ1_S has ggml 19. +constexpr CensusRow kUdQ10Census[] = { + {66, "iq1_xxxs", 276, true}, // 96.92 % of params + {13, "q5_k", 420, true}, {10, "q2_k", 3, true}, + {14, "q6_k", 162, true}, {12, "q4_k", 2, true}, + {8, "q8_0", 1, true}, {0, "f32", 838, false}, +}; + +template +void CheckCheckpointCensus(const CensusRow (&census)[N], const char* quant) { + CAPTURE(std::string(quant)); + vt::DType dt = vt::DType::kF32; + int counted = 0; + for (const CensusRow& p : census) { + CAPTURE(p.name); + CAPTURE(p.ggml_type); + CAPTURE(p.tensors); + counted += p.tensors; + // Assert the ROUTING both ways. A row expected to keep its blocks must + // have a dot kernel, and a row expected to expand must NOT silently be + // reclassified as keep-quant by a later change. + if (p.keep_quant) { + CHECK(KeepQuantDType(p.ggml_type, &dt)); + CHECK(vt::cpu::HasQuantDotKernel(dt)); + } else { + CHECK_FALSE(KeepQuantDType(p.ggml_type, &dt)); + } + } + // The list accounts for every declared record, so "this checkpoint needs + // nothing else" is a measurement rather than a claim. + CAPTURE(counted); + CHECK(counted == kQwen38DeclaredTensors); +} + +} // namespace + TEST_CASE("every encoding in the Qwen3.8-2.4T UD-IQ1_S checkpoint decodes") { // Census of unsloth/Qwen3.8-2.4T-A95B-GGUF @ 567d3e6ac2, quant UD-IQ1_S, // taken by parsing the tensor header of all 12 shards: 1702 tensor records @@ -289,24 +352,14 @@ TEST_CASE("every encoding in the Qwen3.8-2.4T UD-IQ1_S checkpoint decodes") { // bytes for nothing. IQ1_S alone is 96.92 % of the parameters (the // ffn_{down,gate,up}_exps of all 92 non-MTP layers), so an unsupported // encoding here is a refusal to run the model, never a slow path. - struct Present { - uint32_t ggml_type; - const char* name; - int tensors; - }; - constexpr Present kCheckpoint[] = { - {19, "iq1_s", 276}, // 96.92 % of params - {13, "q5_k", 420}, {10, "q2_k", 3}, {14, "q6_k", 162}, - {12, "q4_k", 2}, {8, "q8_0", 1}, - }; - vt::DType dt = vt::DType::kF32; - for (const Present& p : kCheckpoint) { - CAPTURE(p.name); - CAPTURE(p.ggml_type); - CAPTURE(p.tensors); - CHECK(KeepQuantDType(p.ggml_type, &dt)); - CHECK(vt::cpu::HasQuantDotKernel(dt)); - } + // + // The list has to SUM to the declared record count, and this case asserts + // that it does. An earlier revision claimed total coverage while enumerating + // six of the seven encodings: F32 was left out and the counts summed to 864, + // not 1702. A census that cannot say how many records it accounted for has + // not reported a census, and a list that is short by 838 tensors while + // calling itself TOTAL is the more misleading of the two states. + CheckCheckpointCensus(kUdIq1sCensus, "UD-IQ1_S"); } TEST_CASE("every encoding in the Qwen3.8-2.4T UD-Q1_0 checkpoint decodes") { @@ -320,24 +373,7 @@ TEST_CASE("every encoding in the Qwen3.8-2.4T UD-Q1_0 checkpoint decodes") { // oracle `llama-cpp-unsloth` (.agents/oracles/llama-cpp-unsloth.md), which is // admitted for this encoding family alone and is `gateable = no` until #933 // measures it. - struct Present { - uint32_t ggml_type; - const char* name; - int tensors; - }; - constexpr Present kCheckpoint[] = { - {66, "iq1_xxxs", 276}, // 96.92 % of params - {13, "q5_k", 420}, {10, "q2_k", 3}, {14, "q6_k", 162}, - {12, "q4_k", 2}, {8, "q8_0", 1}, - }; - vt::DType dt = vt::DType::kF32; - for (const Present& p : kCheckpoint) { - CAPTURE(p.name); - CAPTURE(p.ggml_type); - CAPTURE(p.tensors); - CHECK(KeepQuantDType(p.ggml_type, &dt)); - CHECK(vt::cpu::HasQuantDotKernel(dt)); - } + CheckCheckpointCensus(kUdQ10Census, "UD-Q1_0"); } TEST_CASE("routing table is TOTAL: every role x every encoding is explicit") { diff --git a/tests/vt/iq1_golden_vectors.h b/tests/vt/iq1_golden_vectors.h new file mode 100644 index 000000000..a4db8cca7 --- /dev/null +++ b/tests/vt/iq1_golden_vectors.h @@ -0,0 +1,499 @@ +// Golden decode vectors for the two IQ1 encodings, produced by the ORACLES +// THEMSELVES rather than by this tree. +// +// Why this file exists. Every other check on the IQ1_S / IQ1_XXXS decode is +// self-consistent: `ReferenceDotF64` and the G3 NMSE reference both decode the +// weight with `vt::cpu::BlockToFloat`, which is the function under test, so +// they are independent only in the summation. The grid tables are sealed by +// digest, but a seal on the codebook says nothing about the SCALE field, the +// DELTA magnitude, or the DELTA SIGN. Three separate defects were injected into +// those three parameters and the whole suite stayed green. This fixture is the +// external reference that closes that hole. +// +// Provenance of the INPUTS. These are real bytes from the checkpoints the row +// targets, read out of the GGUF tensor data with a standalone header parser +// (`unsloth/Qwen3.8-2.4T-A95B-GGUF`, shard 2 of each split, tensor +// `blk.0.ffn_gate_exps.weight`, first four 256-element blocks): +// +// IQ1_S UD-IQ1_S shard 00002-of-00012, ggml type 19, 4 x 50 = 200 bytes +// IQ1_XXXS UD-Q1_0 shard 00002-of-00010, ggml type 66, 4 x 38 = 152 bytes +// +// Real bytes, not synthetic ones, because a synthetic block cannot catch a +// misreading of the FILE. The four blocks carry both delta-sign polarities and +// sub-block scales 2 through 7, so every field the decode reads varies inside +// the fixture. This also reproduces, as a committed slice, the 1179648-weight +// bit-identity run the spec records. +// +// Provenance of the EXPECTED VALUES. Each blob was decoded by the pinned +// oracle's own `ggml_get_type_traits(type)->to_float`, in a build of that +// oracle, and the f32 results are stored here as raw bit patterns: +// +// IQ1_S ggml-org/llama.cpp @ 237ad9b96 (.agents/upstream-sync.md) +// IQ1_XXXS unslothai/llama.cpp @ 36fe8e1cc7f2b3b8c92fdda0ab07600141921786 +// (.agents/oracles/llama-cpp-unsloth.md) +// +// Reproduce with: +// git -C checkout && cmake -B build -DGGML_CUDA=OFF && +// cmake --build build --target ggml -j8 +// then call ggml_get_type_traits(19|66)->to_float on the bytes below. +// +// The comparison is BIT-EXACT, not approximate. `dl * (grid[j] + delta)` is +// the same f32 expression on both sides, so any difference at all means a +// decode parameter diverged rather than that rounding moved. +#pragma once + +#include + +namespace vllm_test { + +inline constexpr uint8_t kIq1sGoldenBlocks[200] = { + 0xe8, 0x0d, 0xf1, 0xef, 0x07, 0xa5, 0xca, 0x16, 0xd8, 0x76, 0xb5, 0xd2, + 0xef, 0x13, 0x2c, 0xfe, 0x63, 0xf8, 0xa5, 0x5c, 0x1b, 0x8f, 0xce, 0x8a, + 0x97, 0x75, 0xa7, 0xf3, 0x1f, 0x6f, 0x67, 0xa3, 0x0f, 0xe2, 0x02, 0xf9, + 0xfe, 0xca, 0x1a, 0x32, 0x89, 0x25, 0x6e, 0x29, 0xd6, 0x28, 0xe8, 0xcb, + 0xb2, 0xaf, 0xc9, 0x0c, 0xca, 0x97, 0xf6, 0xd1, 0x7e, 0x50, 0x2d, 0x4d, + 0xf5, 0x55, 0xb8, 0x7d, 0x74, 0x87, 0x0b, 0x20, 0xa1, 0xfe, 0xea, 0x68, + 0x95, 0x19, 0xe1, 0xd6, 0x1b, 0xad, 0x2c, 0xcd, 0x2f, 0xf5, 0x19, 0x4c, + 0xd6, 0x7c, 0x83, 0xbc, 0x0a, 0xbd, 0xa9, 0xb9, 0xff, 0xb2, 0xb1, 0x39, + 0x65, 0x46, 0xc0, 0x35, 0xb7, 0x0c, 0xab, 0x33, 0x8b, 0x08, 0xff, 0x03, + 0x70, 0x30, 0x9b, 0x29, 0x05, 0x41, 0x2b, 0xf5, 0xc3, 0x26, 0x9b, 0xce, + 0x30, 0xe7, 0x63, 0xcf, 0x48, 0x6c, 0xfe, 0xdf, 0x7f, 0x85, 0xb9, 0x50, + 0x91, 0x68, 0x66, 0x4b, 0xe7, 0xb3, 0x2b, 0xf9, 0x13, 0xbd, 0x01, 0x31, + 0x58, 0xbf, 0x7b, 0x40, 0x15, 0xc9, 0xfc, 0x0b, 0xe0, 0x8e, 0x6c, 0x94, + 0x50, 0x95, 0x30, 0x79, 0xf2, 0xbc, 0x13, 0x2c, 0x10, 0xcc, 0xe1, 0xf7, + 0xf8, 0x49, 0x15, 0xab, 0x17, 0x44, 0xb3, 0x4d, 0xd3, 0x7e, 0x8d, 0x41, + 0x06, 0xb7, 0x0a, 0x90, 0xd8, 0x67, 0x95, 0x48, 0xd5, 0x58, 0x18, 0xb1, + 0xae, 0xd0, 0xec, 0x40, 0xcf, 0xcf, 0xec, 0x76, +}; + +inline constexpr uint32_t kIq1sGoldenBits[1024] = { + 0xba313000U, 0xba313000U, 0xba313000U, 0x3b9b0a00U, 0xba313000U, 0xba313000U, + 0xbbc75600U, 0xba313000U, 0xba313000U, 0xba313000U, 0x3b9b0a00U, 0xba313000U, + 0xba313000U, 0xbbc75600U, 0xba313000U, 0xbbc75600U, 0xbbc75600U, 0x3b9b0a00U, + 0xba313000U, 0xba313000U, 0xba313000U, 0xba313000U, 0xba313000U, 0xba313000U, + 0xba313000U, 0xba313000U, 0xbbc75600U, 0xba313000U, 0x3b9b0a00U, 0x3b9b0a00U, + 0xba313000U, 0xba313000U, 0xb9d4a000U, 0xb9d4a000U, 0xbb6f3400U, 0xb9d4a000U, + 0xb9d4a000U, 0xb9d4a000U, 0xb9d4a000U, 0x3b3a0c00U, 0xb9d4a000U, 0x3b3a0c00U, + 0xb9d4a000U, 0x3b3a0c00U, 0xbb6f3400U, 0x3b3a0c00U, 0xb9d4a000U, 0x3b3a0c00U, + 0x3b3a0c00U, 0xb9d4a000U, 0x3b3a0c00U, 0xb9d4a000U, 0xbb6f3400U, 0xb9d4a000U, + 0xb9d4a000U, 0xb9d4a000U, 0xb9d4a000U, 0x3b3a0c00U, 0xb9d4a000U, 0xb9d4a000U, + 0xb9d4a000U, 0x3b3a0c00U, 0x3b3a0c00U, 0xb9d4a000U, 0x39a56000U, 0xbb10b400U, + 0xbb10b400U, 0xbb10b400U, 0xbb10b400U, 0x39a56000U, 0xbb10b400U, 0x39a56000U, + 0x39a56000U, 0x3b3a0c00U, 0x39a56000U, 0x39a56000U, 0xbb10b400U, 0x39a56000U, + 0x39a56000U, 0x39a56000U, 0x39a56000U, 0x39a56000U, 0x3b3a0c00U, 0x39a56000U, + 0x39a56000U, 0xbb10b400U, 0x39a56000U, 0xbb10b400U, 0xbb10b400U, 0x3b3a0c00U, + 0xbb10b400U, 0x39a56000U, 0xbb10b400U, 0x39a56000U, 0x39a56000U, 0xbb10b400U, + 0x396c4000U, 0xbaceb800U, 0x396c4000U, 0xbaceb800U, 0x396c4000U, 0x396c4000U, + 0x396c4000U, 0xbaceb800U, 0x3b04e400U, 0x396c4000U, 0x396c4000U, 0xbaceb800U, + 0xbaceb800U, 0x396c4000U, 0x3b04e400U, 0xbaceb800U, 0x396c4000U, 0x396c4000U, + 0x396c4000U, 0x396c4000U, 0xbaceb800U, 0xbaceb800U, 0x396c4000U, 0x3b04e400U, + 0x396c4000U, 0xbaceb800U, 0xbaceb800U, 0xbaceb800U, 0x3b04e400U, 0x396c4000U, + 0xbaceb800U, 0x396c4000U, 0x3b04e400U, 0x396c4000U, 0x3b04e400U, 0xbaceb800U, + 0xbaceb800U, 0x396c4000U, 0x396c4000U, 0x3b04e400U, 0x3b04e400U, 0xbaceb800U, + 0x396c4000U, 0x396c4000U, 0xbaceb800U, 0x3b04e400U, 0x3b04e400U, 0x396c4000U, + 0x3b04e400U, 0x396c4000U, 0x396c4000U, 0xbaceb800U, 0x396c4000U, 0x396c4000U, + 0x3b04e400U, 0x396c4000U, 0xbaceb800U, 0x396c4000U, 0x3b04e400U, 0x396c4000U, + 0x396c4000U, 0x3b04e400U, 0x396c4000U, 0x396c4000U, 0x396c4000U, 0xbaceb800U, + 0x396c4000U, 0x396c4000U, 0x396c4000U, 0x396c4000U, 0x396c4000U, 0x3b04e400U, + 0x3b04e400U, 0xbaceb800U, 0x396c4000U, 0xbaceb800U, 0x396c4000U, 0xbaceb800U, + 0xbaceb800U, 0x396c4000U, 0x396c4000U, 0x3b04e400U, 0x3b04e400U, 0x396c4000U, + 0x396c4000U, 0xbaceb800U, 0x396c4000U, 0x396c4000U, 0x396c4000U, 0x3b04e400U, + 0x3b04e400U, 0x3b04e400U, 0xbaceb800U, 0x3b04e400U, 0x396c4000U, 0x396c4000U, + 0xbb6f3400U, 0xb9d4a000U, 0x3b3a0c00U, 0xb9d4a000U, 0xb9d4a000U, 0x3b3a0c00U, + 0xbb6f3400U, 0xbb6f3400U, 0xb9d4a000U, 0xb9d4a000U, 0xb9d4a000U, 0xb9d4a000U, + 0xb9d4a000U, 0xb9d4a000U, 0xbb6f3400U, 0x3b3a0c00U, 0x3b3a0c00U, 0x3b3a0c00U, + 0xb9d4a000U, 0xbb6f3400U, 0xb9d4a000U, 0x3b3a0c00U, 0xb9d4a000U, 0x3b3a0c00U, + 0xb9d4a000U, 0xb9d4a000U, 0xbb6f3400U, 0xb9d4a000U, 0xb9d4a000U, 0x3b3a0c00U, + 0x3b3a0c00U, 0xb9d4a000U, 0xb96c4000U, 0xb96c4000U, 0xbb04e400U, 0xb96c4000U, + 0x3aceb800U, 0x3aceb800U, 0x3aceb800U, 0xbb04e400U, 0xb96c4000U, 0xb96c4000U, + 0xb96c4000U, 0xbb04e400U, 0xbb04e400U, 0xb96c4000U, 0xb96c4000U, 0x3aceb800U, + 0x3aceb800U, 0x3aceb800U, 0xb96c4000U, 0xb96c4000U, 0x3aceb800U, 0xb96c4000U, + 0xbb04e400U, 0x3aceb800U, 0xb96c4000U, 0xb96c4000U, 0x3aceb800U, 0xbb04e400U, + 0xb96c4000U, 0x3aceb800U, 0x3aceb800U, 0x3aceb800U, 0x3a0f8e00U, 0x3a0f8e00U, + 0xbb7b3880U, 0x3a0f8e00U, 0x3a0f8e00U, 0x3a0f8e00U, 0x3a0f8e00U, 0x3ba17fc0U, + 0x3a0f8e00U, 0x3a0f8e00U, 0x3a0f8e00U, 0x3a0f8e00U, 0x3a0f8e00U, 0xbb7b3880U, + 0xbb7b3880U, 0x3a0f8e00U, 0x3a0f8e00U, 0x3a0f8e00U, 0x3ba17fc0U, 0xbb7b3880U, + 0x3a0f8e00U, 0x3a0f8e00U, 0x3a0f8e00U, 0x3a0f8e00U, 0x3a0f8e00U, 0x3a0f8e00U, + 0x3a0f8e00U, 0x3a0f8e00U, 0x3a0f8e00U, 0x3a0f8e00U, 0x3a0f8e00U, 0x3ba17fc0U, + 0xbb16bb80U, 0x3aea7900U, 0xb985fc00U, 0xbb16bb80U, 0xb985fc00U, 0xbb16bb80U, + 0xb985fc00U, 0xb985fc00U, 0x3aea7900U, 0x3aea7900U, 0xb985fc00U, 0xb985fc00U, + 0xbb16bb80U, 0xb985fc00U, 0xbb16bb80U, 0xbb16bb80U, 0xb985fc00U, 0xbb16bb80U, + 0xbb16bb80U, 0xb985fc00U, 0x3aea7900U, 0xb985fc00U, 0x3aea7900U, 0xbb16bb80U, + 0xb985fc00U, 0xb985fc00U, 0xbb16bb80U, 0xb985fc00U, 0x3aea7900U, 0x3aea7900U, + 0xbb16bb80U, 0x3aea7900U, 0xb985fc00U, 0xb985fc00U, 0x3aea7900U, 0x3aea7900U, + 0xb985fc00U, 0xb985fc00U, 0xbb16bb80U, 0xb985fc00U, 0xb985fc00U, 0xb985fc00U, + 0x3aea7900U, 0x3aea7900U, 0xb985fc00U, 0xb985fc00U, 0xb985fc00U, 0xbb16bb80U, + 0xb985fc00U, 0xbb16bb80U, 0x3aea7900U, 0x3aea7900U, 0x3aea7900U, 0x3aea7900U, + 0xb985fc00U, 0xb985fc00U, 0xb985fc00U, 0xbb16bb80U, 0xb985fc00U, 0xb985fc00U, + 0xb985fc00U, 0xbb16bb80U, 0xb985fc00U, 0x3aea7900U, 0xb985fc00U, 0xb985fc00U, + 0xb985fc00U, 0x3aea7900U, 0x3aea7900U, 0xb985fc00U, 0xb985fc00U, 0xbb16bb80U, + 0xb985fc00U, 0x3aea7900U, 0xbb16bb80U, 0xb985fc00U, 0x3aea7900U, 0x3aea7900U, + 0x3aea7900U, 0xb985fc00U, 0x3aea7900U, 0x3aea7900U, 0xbb16bb80U, 0xb985fc00U, + 0x3aea7900U, 0xb985fc00U, 0xbb16bb80U, 0x3aea7900U, 0xb985fc00U, 0xb985fc00U, + 0x3aea7900U, 0x3aea7900U, 0xb985fc00U, 0xb985fc00U, 0xb985fc00U, 0xb985fc00U, + 0x3aea7900U, 0xbb16bb80U, 0xbb16bb80U, 0xb985fc00U, 0xb985fc00U, 0xb985fc00U, + 0x3aea7900U, 0x3aea7900U, 0xbb16bb80U, 0x3aea7900U, 0x3aea7900U, 0x3aea7900U, + 0x3aea7900U, 0x3aea7900U, 0x3aea7900U, 0x3aea7900U, 0x3aea7900U, 0xb985fc00U, + 0xbb16bb80U, 0xbb16bb80U, 0xb985fc00U, 0xb985fc00U, 0xb985fc00U, 0xb985fc00U, + 0x3aea7900U, 0xb985fc00U, 0xb985fc00U, 0xb985fc00U, 0x3aea7900U, 0xb985fc00U, + 0xb985fc00U, 0xbb16bb80U, 0x3985fc00U, 0xbaea7900U, 0x3985fc00U, 0x3985fc00U, + 0x3985fc00U, 0x3b16bb80U, 0x3985fc00U, 0xbaea7900U, 0xbaea7900U, 0x3b16bb80U, + 0xbaea7900U, 0xbaea7900U, 0xbaea7900U, 0x3b16bb80U, 0xbaea7900U, 0x3b16bb80U, + 0x3985fc00U, 0x3985fc00U, 0x3985fc00U, 0x3b16bb80U, 0x3985fc00U, 0x3985fc00U, + 0x3985fc00U, 0x3b16bb80U, 0x3985fc00U, 0x3b16bb80U, 0xbaea7900U, 0x3985fc00U, + 0x3985fc00U, 0xbaea7900U, 0x3b16bb80U, 0x3985fc00U, 0x3b41cc80U, 0x39ac4400U, + 0x39ac4400U, 0xbb16bb80U, 0x39ac4400U, 0x39ac4400U, 0x3b41cc80U, 0x39ac4400U, + 0x3b41cc80U, 0xbb16bb80U, 0x3b41cc80U, 0x39ac4400U, 0x3b41cc80U, 0x3b41cc80U, + 0x39ac4400U, 0x39ac4400U, 0x39ac4400U, 0xbb16bb80U, 0x39ac4400U, 0xbb16bb80U, + 0x39ac4400U, 0x39ac4400U, 0x39ac4400U, 0xbb16bb80U, 0x39ac4400U, 0xbb16bb80U, + 0x39ac4400U, 0x39ac4400U, 0xbb16bb80U, 0x39ac4400U, 0x39ac4400U, 0x39ac4400U, + 0x3985fc00U, 0x3985fc00U, 0x3b16bb80U, 0x3b16bb80U, 0x3985fc00U, 0xbaea7900U, + 0xbaea7900U, 0xbaea7900U, 0x3985fc00U, 0xbaea7900U, 0x3985fc00U, 0xbaea7900U, + 0x3b16bb80U, 0xbaea7900U, 0x3985fc00U, 0xbaea7900U, 0x3b16bb80U, 0x3985fc00U, + 0xbaea7900U, 0xbaea7900U, 0x3985fc00U, 0x3b16bb80U, 0x3985fc00U, 0x3b16bb80U, + 0x3985fc00U, 0x3b16bb80U, 0xbaea7900U, 0xbaea7900U, 0x3985fc00U, 0x3b16bb80U, + 0x3b16bb80U, 0xbaea7900U, 0x3b3ef380U, 0x39a9bc00U, 0x39a9bc00U, 0x39a9bc00U, + 0xbb148480U, 0x39a9bc00U, 0x39a9bc00U, 0x3b3ef380U, 0x39a9bc00U, 0x39a9bc00U, + 0x3b3ef380U, 0xbb148480U, 0x3b3ef380U, 0x39a9bc00U, 0x39a9bc00U, 0x39a9bc00U, + 0x3b3ef380U, 0x3b3ef380U, 0x39a9bc00U, 0x39a9bc00U, 0x3b3ef380U, 0x3b3ef380U, + 0x3b3ef380U, 0x39a9bc00U, 0x39a9bc00U, 0x39a9bc00U, 0x3b3ef380U, 0x39a9bc00U, + 0xbb148480U, 0x39a9bc00U, 0x3b3ef380U, 0x39a9bc00U, 0x3ae70700U, 0x3ae70700U, + 0x3ae70700U, 0x3ae70700U, 0x3ae70700U, 0x3ae70700U, 0x3ae70700U, 0x3ae70700U, + 0x3ae70700U, 0xbb148480U, 0xb9840400U, 0xb9840400U, 0xb9840400U, 0xb9840400U, + 0xb9840400U, 0xb9840400U, 0xbb148480U, 0xbb148480U, 0xb9840400U, 0x3ae70700U, + 0xb9840400U, 0xbb148480U, 0x3ae70700U, 0x3ae70700U, 0xb9840400U, 0x3ae70700U, + 0xb9840400U, 0xbb148480U, 0xb9840400U, 0xb9840400U, 0xb9840400U, 0xbb148480U, + 0xbb9f2040U, 0xba0d7200U, 0xba0d7200U, 0x3b778780U, 0xba0d7200U, 0xbb9f2040U, + 0xba0d7200U, 0xba0d7200U, 0xba0d7200U, 0xba0d7200U, 0x3b778780U, 0xba0d7200U, + 0xba0d7200U, 0xba0d7200U, 0x3b778780U, 0xba0d7200U, 0xba0d7200U, 0xba0d7200U, + 0xba0d7200U, 0xba0d7200U, 0xba0d7200U, 0xba0d7200U, 0xba0d7200U, 0xba0d7200U, + 0xba0d7200U, 0xba0d7200U, 0xba0d7200U, 0xba0d7200U, 0x3b778780U, 0xba0d7200U, + 0xba0d7200U, 0xba0d7200U, 0xb9840400U, 0xbb148480U, 0xbb148480U, 0xbb148480U, + 0xb9840400U, 0x3ae70700U, 0xbb148480U, 0xb9840400U, 0xb9840400U, 0xb9840400U, + 0x3ae70700U, 0x3ae70700U, 0xb9840400U, 0xb9840400U, 0xbb148480U, 0xb9840400U, + 0x3ae70700U, 0x3ae70700U, 0xb9840400U, 0xb9840400U, 0xbb148480U, 0xbb148480U, + 0x3ae70700U, 0xb9840400U, 0xb9840400U, 0x3ae70700U, 0x3ae70700U, 0xb9840400U, + 0xbb148480U, 0x3ae70700U, 0xbb148480U, 0x3ae70700U, 0xbae70700U, 0xbae70700U, + 0x3b148480U, 0x39840400U, 0x39840400U, 0x3b148480U, 0x39840400U, 0xbae70700U, + 0xbae70700U, 0xbae70700U, 0x39840400U, 0x39840400U, 0xbae70700U, 0xbae70700U, + 0x39840400U, 0xbae70700U, 0x39840400U, 0xbae70700U, 0x3b148480U, 0xbae70700U, + 0x3b148480U, 0x39840400U, 0x39840400U, 0x39840400U, 0xbae70700U, 0xbae70700U, + 0x39840400U, 0x39840400U, 0x39840400U, 0xbae70700U, 0x39840400U, 0xbae70700U, + 0xbb148480U, 0xb9840400U, 0xb9840400U, 0xb9840400U, 0xb9840400U, 0xb9840400U, + 0xbb148480U, 0xbb148480U, 0xb9840400U, 0xb9840400U, 0xb9840400U, 0xb9840400U, + 0xbb148480U, 0xb9840400U, 0xb9840400U, 0xb9840400U, 0x3ae70700U, 0x3ae70700U, + 0xb9840400U, 0xb9840400U, 0x3ae70700U, 0xb9840400U, 0x3ae70700U, 0xb9840400U, + 0x3ae70700U, 0x3ae70700U, 0xb9840400U, 0xb9840400U, 0xb9840400U, 0xbb148480U, + 0x3ae70700U, 0x3ae70700U, 0x3b3ef380U, 0x39a9bc00U, 0xbb148480U, 0x39a9bc00U, + 0x39a9bc00U, 0x39a9bc00U, 0x39a9bc00U, 0x39a9bc00U, 0xbb148480U, 0x39a9bc00U, + 0x39a9bc00U, 0xbb148480U, 0x39a9bc00U, 0x3b3ef380U, 0x3b3ef380U, 0x3b3ef380U, + 0x39a9bc00U, 0x39a9bc00U, 0x39a9bc00U, 0x39a9bc00U, 0xbb148480U, 0x3b3ef380U, + 0x39a9bc00U, 0xbb148480U, 0x39a9bc00U, 0x3b3ef380U, 0x39a9bc00U, 0x3b3ef380U, + 0x3b3ef380U, 0x39a9bc00U, 0xbb148480U, 0xbb148480U, 0xbb3ef380U, 0x3b148480U, + 0xb9a9bc00U, 0xb9a9bc00U, 0xb9a9bc00U, 0xbb3ef380U, 0xbb3ef380U, 0x3b148480U, + 0xb9a9bc00U, 0xb9a9bc00U, 0x3b148480U, 0xbb3ef380U, 0xb9a9bc00U, 0x3b148480U, + 0x3b148480U, 0xbb3ef380U, 0x3b148480U, 0xb9a9bc00U, 0x3b148480U, 0xb9a9bc00U, + 0xb9a9bc00U, 0x3b148480U, 0xb9a9bc00U, 0xb9a9bc00U, 0xbb3ef380U, 0x3b148480U, + 0xb9a9bc00U, 0xb9a9bc00U, 0xbb3ef380U, 0x3b148480U, 0xb9a9bc00U, 0xb9a9bc00U, + 0x39cf9800U, 0x39cf9800U, 0x39cf9800U, 0xbb35a500U, 0x39cf9800U, 0xbb35a500U, + 0x39cf9800U, 0xbb35a500U, 0x39cf9800U, 0x39cf9800U, 0x39cf9800U, 0x39cf9800U, + 0x39cf9800U, 0xbb35a500U, 0x39cf9800U, 0x39cf9800U, 0x3b698b00U, 0x3b698b00U, + 0x39cf9800U, 0x39cf9800U, 0x39cf9800U, 0xbb35a500U, 0x3b698b00U, 0x3b698b00U, + 0xbb35a500U, 0x39cf9800U, 0x3b698b00U, 0x39cf9800U, 0x39cf9800U, 0xbb35a500U, + 0x39cf9800U, 0x39cf9800U, 0xbafb8200U, 0x398fb800U, 0x398fb800U, 0x3b21af00U, + 0x3b21af00U, 0x398fb800U, 0x3b21af00U, 0x398fb800U, 0x398fb800U, 0xbafb8200U, + 0x398fb800U, 0x398fb800U, 0x398fb800U, 0xbafb8200U, 0xbafb8200U, 0x398fb800U, + 0xbafb8200U, 0xbafb8200U, 0x3b21af00U, 0x398fb800U, 0x3b21af00U, 0x398fb800U, + 0x3b21af00U, 0xbafb8200U, 0x3b21af00U, 0x3b21af00U, 0xbafb8200U, 0xbafb8200U, + 0x398fb800U, 0x3b21af00U, 0x398fb800U, 0x398fb800U, 0xbb19b300U, 0x39afa800U, + 0x39afa800U, 0x39afa800U, 0x39afa800U, 0x39afa800U, 0xbb19b300U, 0x3b459d00U, + 0x39afa800U, 0xbb19b300U, 0x39afa800U, 0x39afa800U, 0xbb19b300U, 0x39afa800U, + 0xbb19b300U, 0x39afa800U, 0x39afa800U, 0x39afa800U, 0x39afa800U, 0x3b459d00U, + 0x3b459d00U, 0x39afa800U, 0xbb19b300U, 0x39afa800U, 0x39afa800U, 0x39afa800U, + 0x39afa800U, 0xbb19b300U, 0x3b459d00U, 0x39afa800U, 0x39afa800U, 0x39afa800U, + 0xbafb8200U, 0xbafb8200U, 0xbafb8200U, 0x3ac39e00U, 0xbafb8200U, 0xbafb8200U, + 0xbafb8200U, 0xbafb8200U, 0xbafb8200U, 0xbafb8200U, 0xb95f9000U, 0xb95f9000U, + 0xbafb8200U, 0xb95f9000U, 0xb95f9000U, 0xb95f9000U, 0xb95f9000U, 0xb95f9000U, + 0xb95f9000U, 0x3ac39e00U, 0xb95f9000U, 0xbafb8200U, 0x3ac39e00U, 0xb95f9000U, + 0x3ac39e00U, 0xb95f9000U, 0xb95f9000U, 0xbafb8200U, 0x3ac39e00U, 0xbafb8200U, + 0xb95f9000U, 0xbafb8200U, 0x3b19b300U, 0xb9afa800U, 0xb9afa800U, 0xb9afa800U, + 0x3b19b300U, 0xb9afa800U, 0xb9afa800U, 0x3b19b300U, 0xbb459d00U, 0xbb459d00U, + 0x3b19b300U, 0xb9afa800U, 0x3b19b300U, 0xb9afa800U, 0x3b19b300U, 0xb9afa800U, + 0xb9afa800U, 0xb9afa800U, 0xbb459d00U, 0xb9afa800U, 0xb9afa800U, 0xb9afa800U, + 0x3b19b300U, 0xbb459d00U, 0x3b19b300U, 0xb9afa800U, 0xb9afa800U, 0x3b19b300U, + 0xb9afa800U, 0x3b19b300U, 0xbb459d00U, 0xbb459d00U, 0xbafb8200U, 0xbafb8200U, + 0x398fb800U, 0x3b21af00U, 0x398fb800U, 0x398fb800U, 0x398fb800U, 0x398fb800U, + 0xbafb8200U, 0x398fb800U, 0x398fb800U, 0x398fb800U, 0x3b21af00U, 0x398fb800U, + 0x3b21af00U, 0x398fb800U, 0x398fb800U, 0x398fb800U, 0xbafb8200U, 0x3b21af00U, + 0x3b21af00U, 0xbafb8200U, 0x398fb800U, 0x398fb800U, 0x398fb800U, 0xbafb8200U, + 0xbafb8200U, 0x398fb800U, 0xbafb8200U, 0x398fb800U, 0xbafb8200U, 0xbafb8200U, + 0x3afb8200U, 0xb98fb800U, 0xb98fb800U, 0xb98fb800U, 0xbb21af00U, 0x3afb8200U, + 0x3afb8200U, 0x3afb8200U, 0xbb21af00U, 0xbb21af00U, 0xb98fb800U, 0xb98fb800U, + 0xbb21af00U, 0x3afb8200U, 0xb98fb800U, 0xbb21af00U, 0xbb21af00U, 0xb98fb800U, + 0xbb21af00U, 0xb98fb800U, 0xbb21af00U, 0xb98fb800U, 0x3afb8200U, 0x3afb8200U, + 0xb98fb800U, 0xb98fb800U, 0xb98fb800U, 0xb98fb800U, 0x3afb8200U, 0x3afb8200U, + 0xb98fb800U, 0x3afb8200U, 0x3b86bc80U, 0x39ef8800U, 0x39ef8800U, 0x39ef8800U, + 0x39ef8800U, 0x39ef8800U, 0x39ef8800U, 0x39ef8800U, 0x39ef8800U, 0x39ef8800U, + 0x39ef8800U, 0x39ef8800U, 0x39ef8800U, 0xbb519700U, 0xbb519700U, 0x3b86bc80U, + 0x3b86bc80U, 0x39ef8800U, 0x39ef8800U, 0x39ef8800U, 0x3b86bc80U, 0x39ef8800U, + 0xbb519700U, 0x39ef8800U, 0xbb519700U, 0x3b86bc80U, 0x39ef8800U, 0x39ef8800U, + 0x39ef8800U, 0xbb519700U, 0x39ef8800U, 0x39ef8800U, +}; + +inline constexpr uint8_t kIq1xxxsGoldenBlocks[152] = { + 0xf1, 0x0d, 0x3f, 0x10, 0x83, 0xb0, 0xe1, 0xa1, 0x72, 0xcc, 0x3a, 0x6d, + 0x10, 0x08, 0x15, 0x2b, 0xdb, 0x90, 0x68, 0xa0, 0xc0, 0x85, 0xe4, 0x75, + 0x88, 0xad, 0x23, 0xd4, 0xf0, 0xca, 0x27, 0x66, 0xd6, 0xe6, 0xcf, 0x23, + 0x23, 0xac, 0xb8, 0x0c, 0xe1, 0x35, 0x86, 0xde, 0x61, 0x39, 0x2e, 0xd3, + 0x48, 0x2f, 0xf4, 0xd9, 0x22, 0xb0, 0xe2, 0x8f, 0x7a, 0xee, 0x91, 0x22, + 0x28, 0xa2, 0xea, 0xb5, 0xbc, 0x4a, 0x15, 0x6b, 0x04, 0x93, 0xa3, 0xbb, + 0xb7, 0xbb, 0x4b, 0x35, 0x0e, 0x0c, 0x39, 0x9a, 0xce, 0xcd, 0xee, 0xa7, + 0xb4, 0x1b, 0xda, 0xb6, 0x81, 0x97, 0xa2, 0x48, 0xb5, 0xe7, 0x84, 0x51, + 0x94, 0x11, 0x06, 0x69, 0x99, 0xb5, 0x7d, 0xfb, 0x12, 0x0b, 0x41, 0x2b, + 0xcd, 0x9f, 0xb5, 0xbf, 0xc4, 0xc5, 0x49, 0x0c, 0x0c, 0x5b, 0xeb, 0x5d, + 0xc9, 0x34, 0x23, 0xa5, 0xd4, 0x3e, 0x44, 0x93, 0x02, 0x6b, 0xf7, 0x2b, + 0xed, 0xc7, 0x2e, 0x07, 0x8a, 0xc9, 0x5e, 0x26, 0xfc, 0x19, 0xb8, 0xec, + 0x82, 0x35, 0x40, 0x5c, 0x45, 0xb5, 0x4d, 0x7d, +}; + +inline constexpr uint32_t kIq1xxxsGoldenBits[1024] = { + 0xba323e00U, 0xba323e00U, 0xba323e00U, 0xba323e00U, 0xba323e00U, 0xba323e00U, + 0xbbc885c0U, 0xba323e00U, 0xba323e00U, 0xba323e00U, 0x3b9bf640U, 0xba323e00U, + 0xba323e00U, 0xbbc885c0U, 0xba323e00U, 0xbbc885c0U, 0xba323e00U, 0x3b9bf640U, + 0xba323e00U, 0xba323e00U, 0xba323e00U, 0xba323e00U, 0xba323e00U, 0xba323e00U, + 0xba323e00U, 0xba323e00U, 0xbbc885c0U, 0xba323e00U, 0x3b9bf640U, 0x3b9bf640U, + 0xba323e00U, 0xba323e00U, 0xb9d5e400U, 0xb9d5e400U, 0xbb70a080U, 0xb9d5e400U, + 0xb9d5e400U, 0xb9d5e400U, 0xb9d5e400U, 0x3b3b2780U, 0xb9d5e400U, 0xb9d5e400U, + 0xb9d5e400U, 0x3b3b2780U, 0xbb70a080U, 0x3b3b2780U, 0xb9d5e400U, 0xb9d5e400U, + 0x3b3b2780U, 0xb9d5e400U, 0xb9d5e400U, 0x3b3b2780U, 0xbb70a080U, 0xb9d5e400U, + 0xb9d5e400U, 0xb9d5e400U, 0xb9d5e400U, 0x3b3b2780U, 0xb9d5e400U, 0xb9d5e400U, + 0xb9d5e400U, 0x3b3b2780U, 0x3b3b2780U, 0xb9d5e400U, 0x39a65c00U, 0x39a65c00U, + 0xbb119080U, 0xbb119080U, 0x39a65c00U, 0x39a65c00U, 0xbb119080U, 0x39a65c00U, + 0x39a65c00U, 0x3b3b2780U, 0x39a65c00U, 0x39a65c00U, 0xbb119080U, 0x39a65c00U, + 0x39a65c00U, 0x39a65c00U, 0x39a65c00U, 0x39a65c00U, 0x3b3b2780U, 0x39a65c00U, + 0x39a65c00U, 0xbb119080U, 0x39a65c00U, 0xbb119080U, 0x39a65c00U, 0x3b3b2780U, + 0x39a65c00U, 0x39a65c00U, 0x39a65c00U, 0x39a65c00U, 0xbb119080U, 0xbb119080U, + 0x396da800U, 0xbacff300U, 0x396da800U, 0xbacff300U, 0x396da800U, 0x396da800U, + 0x396da800U, 0xbacff300U, 0x3b05ae80U, 0x396da800U, 0x396da800U, 0xbacff300U, + 0xbacff300U, 0x396da800U, 0x3b05ae80U, 0xbacff300U, 0x396da800U, 0x396da800U, + 0x396da800U, 0x396da800U, 0x396da800U, 0xbacff300U, 0x396da800U, 0x3b05ae80U, + 0x396da800U, 0xbacff300U, 0xbacff300U, 0xbacff300U, 0x3b05ae80U, 0x396da800U, + 0x396da800U, 0x396da800U, 0x39a65c00U, 0x39a65c00U, 0x3b3b2780U, 0xbb119080U, + 0xbb119080U, 0x39a65c00U, 0x39a65c00U, 0x39a65c00U, 0x3b3b2780U, 0x39a65c00U, + 0x39a65c00U, 0x39a65c00U, 0xbb119080U, 0x3b3b2780U, 0x39a65c00U, 0x39a65c00U, + 0x3b3b2780U, 0x39a65c00U, 0x39a65c00U, 0x39a65c00U, 0x39a65c00U, 0x39a65c00U, + 0x3b3b2780U, 0x39a65c00U, 0xbb119080U, 0x39a65c00U, 0x3b3b2780U, 0x39a65c00U, + 0x39a65c00U, 0x39a65c00U, 0x39a65c00U, 0x39a65c00U, 0x396da800U, 0xbacff300U, + 0x396da800U, 0x396da800U, 0x396da800U, 0x396da800U, 0x396da800U, 0x3b05ae80U, + 0x3b05ae80U, 0xbacff300U, 0x396da800U, 0xbacff300U, 0x396da800U, 0x396da800U, + 0x396da800U, 0x396da800U, 0xbacff300U, 0x3b05ae80U, 0x3b05ae80U, 0x396da800U, + 0x396da800U, 0x396da800U, 0x396da800U, 0x396da800U, 0x396da800U, 0x3b05ae80U, + 0x396da800U, 0x3b05ae80U, 0x396da800U, 0x3b05ae80U, 0x396da800U, 0x396da800U, + 0xbb70a080U, 0xb9d5e400U, 0x3b3b2780U, 0xb9d5e400U, 0x3b3b2780U, 0xb9d5e400U, + 0xb9d5e400U, 0xbb70a080U, 0xb9d5e400U, 0xb9d5e400U, 0xb9d5e400U, 0xb9d5e400U, + 0xb9d5e400U, 0xb9d5e400U, 0xbb70a080U, 0x3b3b2780U, 0xb9d5e400U, 0xb9d5e400U, + 0xb9d5e400U, 0xbb70a080U, 0xb9d5e400U, 0x3b3b2780U, 0xb9d5e400U, 0x3b3b2780U, + 0xb9d5e400U, 0xb9d5e400U, 0xbb70a080U, 0xb9d5e400U, 0xb9d5e400U, 0x3b3b2780U, + 0x3b3b2780U, 0xb9d5e400U, 0xbb05ae80U, 0x3acff300U, 0xbb05ae80U, 0xb96da800U, + 0xb96da800U, 0x3acff300U, 0xb96da800U, 0xbb05ae80U, 0x3acff300U, 0xb96da800U, + 0xb96da800U, 0xbb05ae80U, 0xbb05ae80U, 0xb96da800U, 0xb96da800U, 0xb96da800U, + 0xb96da800U, 0x3acff300U, 0xb96da800U, 0xbb05ae80U, 0x3acff300U, 0xb96da800U, + 0xbb05ae80U, 0x3acff300U, 0xb96da800U, 0xb96da800U, 0x3acff300U, 0xb96da800U, + 0xb96da800U, 0xb96da800U, 0xb96da800U, 0x3acff300U, 0x3a0d9000U, 0x3a0d9000U, + 0xbb77bc00U, 0x3a0d9000U, 0x3a0d9000U, 0x3a0d9000U, 0x3a0d9000U, 0x3b9f4200U, + 0x3a0d9000U, 0x3a0d9000U, 0x3a0d9000U, 0x3a0d9000U, 0x3a0d9000U, 0xbb77bc00U, + 0xbb77bc00U, 0x3a0d9000U, 0x3a0d9000U, 0x3a0d9000U, 0x3b9f4200U, 0x3a0d9000U, + 0x3a0d9000U, 0x3a0d9000U, 0x3a0d9000U, 0x3a0d9000U, 0x3a0d9000U, 0x3a0d9000U, + 0x3a0d9000U, 0x3a0d9000U, 0xbb77bc00U, 0x3a0d9000U, 0x3a0d9000U, 0x3b9f4200U, + 0xbb14a400U, 0xb9842000U, 0xb9842000U, 0xb9842000U, 0x3ae73800U, 0xbb14a400U, + 0xb9842000U, 0xb9842000U, 0x3ae73800U, 0xb9842000U, 0xb9842000U, 0xb9842000U, + 0xbb14a400U, 0xb9842000U, 0xbb14a400U, 0xb9842000U, 0xb9842000U, 0xb9842000U, + 0xbb14a400U, 0xb9842000U, 0xb9842000U, 0xb9842000U, 0x3ae73800U, 0xbb14a400U, + 0xb9842000U, 0xbb14a400U, 0xbb14a400U, 0xb9842000U, 0xb9842000U, 0xb9842000U, + 0xbb14a400U, 0x3ae73800U, 0xb9842000U, 0xb9842000U, 0x3ae73800U, 0x3ae73800U, + 0xb9842000U, 0xb9842000U, 0xbb14a400U, 0xb9842000U, 0xb9842000U, 0xb9842000U, + 0x3ae73800U, 0xb9842000U, 0xb9842000U, 0xb9842000U, 0x3ae73800U, 0xbb14a400U, + 0xb9842000U, 0xbb14a400U, 0x3ae73800U, 0xb9842000U, 0x3ae73800U, 0x3ae73800U, + 0xb9842000U, 0x3ae73800U, 0xb9842000U, 0xbb14a400U, 0xbb14a400U, 0xbb14a400U, + 0xb9842000U, 0xbb14a400U, 0xb9842000U, 0x3ae73800U, 0xb9842000U, 0xb9842000U, + 0xb9842000U, 0xb9842000U, 0x3ae73800U, 0xb9842000U, 0xb9842000U, 0xbb14a400U, + 0xb9842000U, 0xb9842000U, 0xbb14a400U, 0xb9842000U, 0x3ae73800U, 0x3ae73800U, + 0xb9842000U, 0xb9842000U, 0x3ae73800U, 0xb9842000U, 0xbb14a400U, 0xb9842000U, + 0xb9842000U, 0xb9842000U, 0xb9842000U, 0x3ae73800U, 0xb9842000U, 0x3ae73800U, + 0x3ae73800U, 0x3ae73800U, 0xb9842000U, 0xb9842000U, 0xb9842000U, 0xb9842000U, + 0xb9842000U, 0xbb14a400U, 0xbb14a400U, 0xb9842000U, 0xb9842000U, 0xb9842000U, + 0xb9842000U, 0xb9842000U, 0xbb14a400U, 0xb9842000U, 0x3ae73800U, 0x3ae73800U, + 0x3ae73800U, 0xb9842000U, 0xb9842000U, 0x3ae73800U, 0x3ae73800U, 0xb9842000U, + 0xbb14a400U, 0xbb14a400U, 0x3ae73800U, 0xb9842000U, 0xb9842000U, 0xb9842000U, + 0xb9842000U, 0xb9842000U, 0xb9842000U, 0xb9842000U, 0x3ae73800U, 0xb9842000U, + 0xb9842000U, 0xbb14a400U, 0x39a9e000U, 0x39a9e000U, 0x39a9e000U, 0x39a9e000U, + 0x39a9e000U, 0x3b3f1c00U, 0x39a9e000U, 0xbb14a400U, 0x39a9e000U, 0x39a9e000U, + 0xbb14a400U, 0xbb14a400U, 0x39a9e000U, 0x3b3f1c00U, 0x39a9e000U, 0x39a9e000U, + 0x39a9e000U, 0x39a9e000U, 0x39a9e000U, 0x3b3f1c00U, 0x39a9e000U, 0x39a9e000U, + 0x39a9e000U, 0x3b3f1c00U, 0x39a9e000U, 0x3b3f1c00U, 0x39a9e000U, 0x39a9e000U, + 0x39a9e000U, 0xbb14a400U, 0x3b3f1c00U, 0x39a9e000U, 0x39cfa000U, 0x39cfa000U, + 0x39cfa000U, 0xbb35ac00U, 0x39cfa000U, 0x39cfa000U, 0x3b699400U, 0x39cfa000U, + 0x3b699400U, 0xbb35ac00U, 0x39cfa000U, 0x39cfa000U, 0x3b699400U, 0x39cfa000U, + 0xbb35ac00U, 0x39cfa000U, 0x39cfa000U, 0xbb35ac00U, 0x39cfa000U, 0xbb35ac00U, + 0x39cfa000U, 0x39cfa000U, 0x39cfa000U, 0xbb35ac00U, 0x39cfa000U, 0xbb35ac00U, + 0x39cfa000U, 0x39cfa000U, 0xbb35ac00U, 0x39cfa000U, 0x39cfa000U, 0x39cfa000U, + 0x39842000U, 0x39842000U, 0x3b14a400U, 0x39842000U, 0xbae73800U, 0x39842000U, + 0xbae73800U, 0xbae73800U, 0x39842000U, 0x39842000U, 0x39842000U, 0xbae73800U, + 0x3b14a400U, 0x39842000U, 0x39842000U, 0x39842000U, 0x3b14a400U, 0x39842000U, + 0x39842000U, 0xbae73800U, 0x39842000U, 0x3b14a400U, 0x39842000U, 0x39842000U, + 0x39842000U, 0x39842000U, 0xbae73800U, 0xbae73800U, 0x39842000U, 0x39842000U, + 0x3b14a400U, 0x39842000U, 0x3b48b500U, 0x39b26800U, 0x39b26800U, 0x39b26800U, + 0xbb1c1b00U, 0x39b26800U, 0xbb1c1b00U, 0x39b26800U, 0x39b26800U, 0x39b26800U, + 0x3b48b500U, 0x39b26800U, 0x3b48b500U, 0x39b26800U, 0x39b26800U, 0x39b26800U, + 0x39b26800U, 0x39b26800U, 0x39b26800U, 0x39b26800U, 0x3b48b500U, 0x3b48b500U, + 0x3b48b500U, 0x39b26800U, 0x39b26800U, 0x39b26800U, 0x3b48b500U, 0x39b26800U, + 0x39b26800U, 0x3b48b500U, 0x3b48b500U, 0x39b26800U, 0xbaff7200U, 0xb9631000U, + 0x3ac6ae00U, 0x3ac6ae00U, 0x3ac6ae00U, 0xb9631000U, 0xb9631000U, 0x3ac6ae00U, + 0x3ac6ae00U, 0xbaff7200U, 0xb9631000U, 0xb9631000U, 0xb9631000U, 0x3ac6ae00U, + 0xb9631000U, 0xb9631000U, 0xb9631000U, 0xbaff7200U, 0xb9631000U, 0xb9631000U, + 0xb9631000U, 0xbaff7200U, 0x3ac6ae00U, 0xb9631000U, 0xb9631000U, 0x3ac6ae00U, + 0xb9631000U, 0xb9631000U, 0xb9631000U, 0xb9631000U, 0xb9631000U, 0xbaff7200U, + 0xbb88d880U, 0xb9f34800U, 0xb9f34800U, 0xb9f34800U, 0xb9f34800U, 0xbb88d880U, + 0xb9f34800U, 0x3b54df00U, 0x3b54df00U, 0xb9f34800U, 0x3b54df00U, 0xb9f34800U, + 0xb9f34800U, 0xbb88d880U, 0x3b54df00U, 0xb9f34800U, 0xb9f34800U, 0xb9f34800U, + 0xb9f34800U, 0xb9f34800U, 0xb9f34800U, 0xb9f34800U, 0xb9f34800U, 0xb9f34800U, + 0xb9f34800U, 0xb9f34800U, 0xb9f34800U, 0xb9f34800U, 0x3b54df00U, 0xb9f34800U, + 0xb9f34800U, 0xb9f34800U, 0xb9631000U, 0xb9631000U, 0xbaff7200U, 0xbaff7200U, + 0xb9631000U, 0x3ac6ae00U, 0xb9631000U, 0xb9631000U, 0xb9631000U, 0xb9631000U, + 0x3ac6ae00U, 0x3ac6ae00U, 0xb9631000U, 0xb9631000U, 0xbaff7200U, 0xb9631000U, + 0xb9631000U, 0x3ac6ae00U, 0xb9631000U, 0xb9631000U, 0xb9631000U, 0xbaff7200U, + 0x3ac6ae00U, 0xb9631000U, 0xb9631000U, 0x3ac6ae00U, 0x3ac6ae00U, 0xb9631000U, + 0xb9631000U, 0xb9631000U, 0xb9631000U, 0x3ac6ae00U, 0xbaff7200U, 0xbaff7200U, + 0x3b243700U, 0x3991f800U, 0x3991f800U, 0x3991f800U, 0x3991f800U, 0x3991f800U, + 0xbaff7200U, 0x3991f800U, 0x3991f800U, 0x3991f800U, 0xbaff7200U, 0xbaff7200U, + 0x3991f800U, 0x3991f800U, 0x3991f800U, 0xbaff7200U, 0x3b243700U, 0xbaff7200U, + 0x3b243700U, 0x3991f800U, 0x3991f800U, 0x3991f800U, 0x3991f800U, 0xbaff7200U, + 0x3991f800U, 0x3b243700U, 0x3b243700U, 0xbaff7200U, 0x3991f800U, 0xbaff7200U, + 0xbb243700U, 0xb991f800U, 0xb991f800U, 0xb991f800U, 0xb991f800U, 0xb991f800U, + 0xbb243700U, 0xbb243700U, 0xb991f800U, 0xb991f800U, 0xbb243700U, 0xb991f800U, + 0xbb243700U, 0xb991f800U, 0xb991f800U, 0xb991f800U, 0x3aff7200U, 0x3aff7200U, + 0xb991f800U, 0xb991f800U, 0x3aff7200U, 0xb991f800U, 0xb991f800U, 0xb991f800U, + 0xb991f800U, 0x3aff7200U, 0xb991f800U, 0xb991f800U, 0xb991f800U, 0xbb243700U, + 0x3aff7200U, 0xb991f800U, 0x3b48b500U, 0x39b26800U, 0xbb1c1b00U, 0x39b26800U, + 0x39b26800U, 0x39b26800U, 0x39b26800U, 0x39b26800U, 0xbb1c1b00U, 0x3b48b500U, + 0x39b26800U, 0xbb1c1b00U, 0x39b26800U, 0x39b26800U, 0x3b48b500U, 0x3b48b500U, + 0x39b26800U, 0x39b26800U, 0x39b26800U, 0x39b26800U, 0xbb1c1b00U, 0x39b26800U, + 0x39b26800U, 0xbb1c1b00U, 0x39b26800U, 0x39b26800U, 0x39b26800U, 0x3b48b500U, + 0x3b48b500U, 0x3b48b500U, 0xbb1c1b00U, 0xbb1c1b00U, 0xbb243700U, 0x3aff7200U, + 0xb991f800U, 0xb991f800U, 0xb991f800U, 0xb991f800U, 0xbb243700U, 0xb991f800U, + 0x3aff7200U, 0xb991f800U, 0xb991f800U, 0xbb243700U, 0xbb243700U, 0xb991f800U, + 0x3aff7200U, 0xbb243700U, 0xb991f800U, 0xb991f800U, 0x3aff7200U, 0xb991f800U, + 0xb991f800U, 0x3aff7200U, 0x3aff7200U, 0xb991f800U, 0xbb243700U, 0xb991f800U, + 0xb991f800U, 0xb991f800U, 0xbb243700U, 0x3aff7200U, 0xb991f800U, 0xb991f800U, + 0xbb24fa80U, 0x3b541d80U, 0x39bc8c00U, 0xbb24fa80U, 0x39bc8c00U, 0xbb24fa80U, + 0x39bc8c00U, 0xbb24fa80U, 0x39bc8c00U, 0x39bc8c00U, 0x39bc8c00U, 0x39bc8c00U, + 0x39bc8c00U, 0xbb24fa80U, 0x39bc8c00U, 0x39bc8c00U, 0x3b541d80U, 0x3b541d80U, + 0x39bc8c00U, 0x3b541d80U, 0x39bc8c00U, 0x39bc8c00U, 0x39bc8c00U, 0x3b541d80U, + 0xbb24fa80U, 0x39bc8c00U, 0x3b541d80U, 0x39bc8c00U, 0x39bc8c00U, 0xbb24fa80U, + 0x39bc8c00U, 0x39bc8c00U, 0xbb06fb80U, 0x399a4400U, 0x399a4400U, 0x3b2d8c80U, + 0x3b2d8c80U, 0x399a4400U, 0x3b2d8c80U, 0x399a4400U, 0x399a4400U, 0xbb06fb80U, + 0x399a4400U, 0x399a4400U, 0x399a4400U, 0xbb06fb80U, 0xbb06fb80U, 0x399a4400U, + 0xbb06fb80U, 0x399a4400U, 0x3b2d8c80U, 0x399a4400U, 0x3b2d8c80U, 0x399a4400U, + 0x399a4400U, 0xbb06fb80U, 0x3b2d8c80U, 0x399a4400U, 0xbb06fb80U, 0x399a4400U, + 0x399a4400U, 0x3b2d8c80U, 0x399a4400U, 0x399a4400U, 0x39bc8c00U, 0x39bc8c00U, + 0x39bc8c00U, 0x39bc8c00U, 0x39bc8c00U, 0x39bc8c00U, 0xbb24fa80U, 0x3b541d80U, + 0xbb24fa80U, 0xbb24fa80U, 0x39bc8c00U, 0x39bc8c00U, 0x39bc8c00U, 0x39bc8c00U, + 0xbb24fa80U, 0x39bc8c00U, 0x39bc8c00U, 0x39bc8c00U, 0xbb24fa80U, 0x3b541d80U, + 0x39bc8c00U, 0x39bc8c00U, 0xbb24fa80U, 0x39bc8c00U, 0x39bc8c00U, 0x39bc8c00U, + 0x39bc8c00U, 0xbb24fa80U, 0x3b541d80U, 0x39bc8c00U, 0x39bc8c00U, 0x39bc8c00U, + 0xb96ff800U, 0xb96ff800U, 0xb96ff800U, 0x3ad1f900U, 0xbb06fb80U, 0xbb06fb80U, + 0xbb06fb80U, 0xbb06fb80U, 0xb96ff800U, 0xbb06fb80U, 0xb96ff800U, 0xb96ff800U, + 0xbb06fb80U, 0xb96ff800U, 0xb96ff800U, 0xb96ff800U, 0xb96ff800U, 0xb96ff800U, + 0xb96ff800U, 0x3ad1f900U, 0xbb06fb80U, 0xbb06fb80U, 0x3ad1f900U, 0x3ad1f900U, + 0x3ad1f900U, 0xb96ff800U, 0xb96ff800U, 0xbb06fb80U, 0xbb06fb80U, 0xb96ff800U, + 0x3ad1f900U, 0xbb06fb80U, 0x3b24fa80U, 0xb9bc8c00U, 0xb9bc8c00U, 0xb9bc8c00U, + 0x3b24fa80U, 0xb9bc8c00U, 0xb9bc8c00U, 0x3b24fa80U, 0xb9bc8c00U, 0xbb541d80U, + 0xb9bc8c00U, 0xb9bc8c00U, 0x3b24fa80U, 0xb9bc8c00U, 0x3b24fa80U, 0xb9bc8c00U, + 0xb9bc8c00U, 0xb9bc8c00U, 0xbb541d80U, 0xb9bc8c00U, 0xb9bc8c00U, 0xb9bc8c00U, + 0x3b24fa80U, 0xbb541d80U, 0x3b24fa80U, 0xb9bc8c00U, 0xb9bc8c00U, 0xb9bc8c00U, + 0xb9bc8c00U, 0xb9bc8c00U, 0xbb541d80U, 0xbb541d80U, 0xbb06fb80U, 0xbb06fb80U, + 0x399a4400U, 0x3b2d8c80U, 0x399a4400U, 0x399a4400U, 0x399a4400U, 0x399a4400U, + 0xbb06fb80U, 0x399a4400U, 0x399a4400U, 0x3b2d8c80U, 0x3b2d8c80U, 0x399a4400U, + 0x3b2d8c80U, 0x399a4400U, 0x399a4400U, 0x399a4400U, 0xbb06fb80U, 0x3b2d8c80U, + 0x399a4400U, 0xbb06fb80U, 0x399a4400U, 0x399a4400U, 0x399a4400U, 0xbb06fb80U, + 0xbb06fb80U, 0x399a4400U, 0xbb06fb80U, 0x3b2d8c80U, 0x399a4400U, 0xbb06fb80U, + 0xb9bc8c00U, 0xb9bc8c00U, 0xb9bc8c00U, 0xb9bc8c00U, 0xb9bc8c00U, 0xb9bc8c00U, + 0x3b24fa80U, 0x3b24fa80U, 0xbb541d80U, 0xb9bc8c00U, 0xb9bc8c00U, 0xb9bc8c00U, + 0xb9bc8c00U, 0xb9bc8c00U, 0xb9bc8c00U, 0xbb541d80U, 0xbb541d80U, 0xbb541d80U, + 0xbb541d80U, 0xb9bc8c00U, 0xbb541d80U, 0xb9bc8c00U, 0x3b24fa80U, 0xb9bc8c00U, + 0xb9bc8c00U, 0xb9bc8c00U, 0xb9bc8c00U, 0xb9bc8c00U, 0x3b24fa80U, 0xb9bc8c00U, + 0xb9bc8c00U, 0x3b24fa80U, 0x3b909fc0U, 0x3a008e00U, 0x3a008e00U, 0x3a008e00U, + 0x3a008e00U, 0x3a008e00U, 0x3a008e00U, 0x3a008e00U, 0x3a008e00U, 0x3a008e00U, + 0x3a008e00U, 0x3a008e00U, 0x3a008e00U, 0xbb60f880U, 0xbb60f880U, 0x3a008e00U, + 0x3b909fc0U, 0x3a008e00U, 0x3a008e00U, 0x3a008e00U, 0x3a008e00U, 0x3a008e00U, + 0xbb60f880U, 0x3a008e00U, 0x3a008e00U, 0x3b909fc0U, 0x3a008e00U, 0x3a008e00U, + 0x3a008e00U, 0xbb60f880U, 0x3a008e00U, 0x3a008e00U, +}; + + +// IQ3_XXS (ggml 18) has no committed checkpoint slice here, so its input is +// one legal SYNTHETIC block instead: an f16 d of 0.0125 followed by 96 +// payload bytes from a fixed LCG, which varies both the 4 grid-index bytes +// and the packed sign/scale word of every group. The EXPECTED values are +// still the oracle's own, from the same 237ad9b96 build. It is here because +// `DequantGgufRowToF32` omitted `case 18` for the same reason it omitted 19 +// and 66, and the three are fixed together in one shared branch. +inline constexpr uint8_t kIq3xxsGoldenBlocks[98] = { + 0x66, 0x22, 0x53, 0x89, 0x85, 0xed, 0x76, 0xe8, 0xd9, 0x88, 0x4a, 0x8b, + 0x02, 0xcf, 0xf5, 0x45, 0xd1, 0x95, 0x1c, 0x39, 0x63, 0xb6, 0x31, 0x3c, + 0x51, 0xd3, 0x80, 0xe8, 0x9d, 0xca, 0xf9, 0x7f, 0x57, 0xf8, 0xbb, 0x86, + 0x12, 0x2e, 0x65, 0x1e, 0x0a, 0x74, 0x1b, 0x21, 0x12, 0x80, 0x5a, 0x00, + 0x3f, 0xf4, 0x7c, 0x5f, 0x57, 0x5b, 0x0a, 0x6d, 0xf7, 0xe1, 0xc6, 0x05, + 0x86, 0xd6, 0x73, 0x86, 0xdd, 0xe1, 0x6a, 0x72, 0xb3, 0x04, 0xde, 0xc9, + 0xca, 0x53, 0xea, 0x23, 0xdb, 0x74, 0x60, 0x90, 0x42, 0xd5, 0x50, 0x30, + 0x69, 0xb0, 0x5a, 0x91, 0xf6, 0xc0, 0xb4, 0xcb, 0xb4, 0xc0, 0xfa, 0x5d, + 0x42, 0xa8, +}; + +inline constexpr uint32_t kIq3xxsGoldenBits[256] = { + 0x3c4cc000U, 0xbd7ff000U, 0x3db32800U, 0xbd199000U, 0x3db32800U, 0xbc4cc000U, + 0xbe465a00U, 0x3d7ff000U, 0x3e465a00U, 0x3d7ff000U, 0xbe0cc400U, 0x3d7ff000U, + 0x3e465a00U, 0xbc4cc000U, 0xbc4cc000U, 0xbe465a00U, 0xbd199000U, 0x3d199000U, + 0xbd7ff000U, 0xbd7ff000U, 0x3db32800U, 0x3db32800U, 0xbe265c00U, 0x3e265c00U, + 0xbde65800U, 0x3d7ff000U, 0xbe465a00U, 0x3e0cc400U, 0x3d199000U, 0xbc4cc000U, + 0x3e465a00U, 0xbd7ff000U, 0x3ed32600U, 0xbe0cc400U, 0xbf2ff500U, 0xbed32600U, + 0xbfc18d80U, 0x3f9e5c80U, 0xc0085de0U, 0xbf2ff500U, 0xbf9e5c80U, 0xbe0cc400U, + 0x3e0cc400U, 0x3e0cc400U, 0xbf2ff500U, 0x40085de0U, 0x3f2ff500U, 0xbfc18d80U, + 0xbf2ff500U, 0xbfc18d80U, 0x3f2ff500U, 0xc0085de0U, 0x3e0cc400U, 0xbe0cc400U, + 0x3ed32600U, 0x3ed32600U, 0x3f765700U, 0xbfc18d80U, 0xbf765700U, 0xbfc18d80U, + 0xbf2ff500U, 0x3e0cc400U, 0x3ed32600U, 0x3f765700U, 0x3f6ff100U, 0xbf6ff100U, + 0x3f6ff100U, 0xbe3ff400U, 0x3f0ff700U, 0xc01bf640U, 0xc039f460U, 0x3e3ff400U, + 0xbf0ff700U, 0xbf0ff700U, 0xbe3ff400U, 0x3f6ff100U, 0x4039f460U, 0x3fa7f580U, + 0xbf6ff100U, 0x3fd7f280U, 0x3f6ff100U, 0x3fa7f580U, 0xc003f7c0U, 0xbe3ff400U, + 0x3e3ff400U, 0xbf0ff700U, 0xbe3ff400U, 0x3f0ff700U, 0x3e3ff400U, 0xbe3ff400U, + 0xbfa7f580U, 0x3f0ff700U, 0x3fa7f580U, 0xbf6ff100U, 0x3fd7f280U, 0xc003f7c0U, + 0x3eacc200U, 0x40a75bf0U, 0x401729c0U, 0x3fd7f280U, 0x401729c0U, 0xc01729c0U, + 0xc08c5da0U, 0x408c5da0U, 0x406d8ac0U, 0x3fd7f280U, 0x3fd7f280U, 0x401729c0U, + 0x408c5da0U, 0xbeacc200U, 0x3f819180U, 0xc06d8ac0U, 0x3eacc200U, 0xc08c5da0U, + 0x401729c0U, 0xc0a75bf0U, 0x40425a40U, 0x3f819180U, 0x401729c0U, 0x3fd7f280U, + 0x3f819180U, 0xbeacc200U, 0x40425a40U, 0xbf819180U, 0x401729c0U, 0xc01729c0U, + 0x401729c0U, 0xc0a75bf0U, 0x404a59c0U, 0x40258f40U, 0x4000c4c0U, 0x40258f40U, + 0xbf5cbf00U, 0x40258f40U, 0xc04a59c0U, 0x3fb7f480U, 0x3e932a00U, 0x4000c4c0U, + 0x3f5cbf00U, 0x3e932a00U, 0x404a59c0U, 0xc08e90b0U, 0xc0258f40U, 0x3e932a00U, + 0x3e932a00U, 0x3fb7f480U, 0xbe932a00U, 0x3fb7f480U, 0x4000c4c0U, 0xc000c4c0U, + 0x3fb7f480U, 0x3e932a00U, 0xc000c4c0U, 0xc08e90b0U, 0x3e932a00U, 0x3e932a00U, + 0x3e932a00U, 0x3e932a00U, 0x3fb7f480U, 0x3fb7f480U, 0x3e9ff600U, 0xbfc7f380U, + 0x3fc7f380U, 0xbe9ff600U, 0xc09af650U, 0x405bf240U, 0xbfc7f380U, 0x3e9ff600U, + 0x3e9ff600U, 0xc00bf740U, 0x3f6ff100U, 0x3e9ff600U, 0x3e9ff600U, 0xc09af650U, + 0x400bf740U, 0x3fc7f380U, 0x3e9ff600U, 0xbfc7f380U, 0x405bf240U, 0xbf6ff100U, + 0xbe9ff600U, 0x3e9ff600U, 0xbe9ff600U, 0x3e9ff600U, 0xc00bf740U, 0xbfc7f380U, + 0xbe9ff600U, 0x3f6ff100U, 0x405bf240U, 0x4033f4c0U, 0x3fc7f380U, 0xc09af650U, + 0x405bf240U, 0x405bf240U, 0xbfc7f380U, 0x3fc7f380U, 0xbfc7f380U, 0xbfc7f380U, + 0x409af650U, 0xbf6ff100U, 0xbf6ff100U, 0xbe9ff600U, 0xc033f4c0U, 0x3f6ff100U, + 0xc033f4c0U, 0x3fc7f380U, 0x405bf240U, 0x3f6ff100U, 0xc00bf740U, 0xc09af650U, + 0x3e9ff600U, 0x3e9ff600U, 0xbe9ff600U, 0x3f6ff100U, 0xbf6ff100U, 0x3fc7f380U, + 0xc05bf240U, 0x3f6ff100U, 0xc00bf740U, 0x409af650U, 0x409af650U, 0x3f6ff100U, + 0x3fc7f380U, 0x4081f7e0U, 0x3fa7f580U, 0xbf498d00U, 0x3e865e00U, 0xc038c140U, + 0xc0822b10U, 0xbf498d00U, 0xbe865e00U, 0xbe865e00U, 0xbf498d00U, 0xc01729c0U, + 0x4038c140U, 0xbfa7f580U, 0xbfa7f580U, 0xbe865e00U, 0x4038c140U, 0xc038c140U, + 0xbe865e00U, 0x40822b10U, 0x3f498d00U, 0xbfa7f580U, 0x3f498d00U, 0x401729c0U, + 0x4038c140U, 0x3fa7f580U, 0x405a58c0U, 0xc01729c0U, 0x3e865e00U, 0x405a58c0U, + 0x40822b10U, 0x3f498d00U, 0xbfa7f580U, 0x405a58c0U, +}; + +} // namespace vllm_test diff --git a/tests/vt/test_ops_quant_dot.cpp b/tests/vt/test_ops_quant_dot.cpp index 73ee865fc..986ff2eb3 100644 --- a/tests/vt/test_ops_quant_dot.cpp +++ b/tests/vt/test_ops_quant_dot.cpp @@ -42,6 +42,7 @@ #include +#include "iq1_golden_vectors.h" // oracle-produced IQ1_S / IQ1_XXXS goldens #include "vt/cpu/cpu_threadpool.h" // Threadpool::SwapForTesting (via -I src) #include "vt/cpu/cpu_quant_iq_tables.h" // kIq1sGrid provenance check #include "vt/device.h" @@ -133,24 +134,39 @@ const WeightCase kWeightCases[] = { {vt::DType::kIQ3_XXS, 256, 98, 0, -1, -1, "iq3_xxs"}, {vt::DType::kIQ2_S, 256, 82, 0, -1, -1, "iq2_s"}, // IQ1_S (1.5625 bpw) is 96.92 % of the parameters of the - // Qwen3.8-2.4T-A95B UD-IQ1_S checkpoint ENG-EXPERT-STREAM targets. - // IQ1_S needs its own ceiling, measured not guessed. Its weights are - // ternary times a per-32 scale, so a super-block spans a wider dynamic - // range than a 4-6 bit codebook does, while Q8_K gives the activation ONE - // scale per 256 elements. The residual sits at ~5.2e-4 on the thinnest - // shape here (m=4, n=1 is FOUR dot products, so the statistic is noisy), - // against 5e-4. Set to 2e-3: ~4x the observed residual, and still an order - // of magnitude below the WEAKEST decode defect measured by mutation. - // Dropping the delta term reads 2.7e-2 (the smallest signal of the nine), - // the wrong delta sign bit 3.0e-1, ignoring the qh scale 9.4e-1, dropping - // the delta sign 5.7, and ignoring the 3 high grid-index bits 5.7e+1. So - // the ceiling sits in the empty band between noise and every defect. - {vt::DType::kIQ1_S, 256, 50, 0, -1, -1, "iq1_s", 2e-3}, - // IQ1_XXXS (1.1875 bpw) is 96.92 % of the Qwen3.8-2.4T-A95B UD-Q1_0 - // checkpoint. Type 66, from the pinned FORK oracle `llama-cpp-unsloth`. - // Same ceiling and the same reason as IQ1_S: ternary lanes times a per-32 - // scale against a Q8_K activation that gets one scale per 256. - {vt::DType::kIQ1_XXXS, 256, 38, 0, -1, -1, "iq1_xxxs", 2e-3}, + // Qwen3.8-2.4T-A95B UD-IQ1_S checkpoint ENG-EXPERT-STREAM targets, and + // IQ1_XXXS (1.1875 bpw) is 96.92 % of the UD-Q1_0 one. Both need a ceiling + // above upstream's 5e-4: their weights are ternary times a per-32 scale, so + // a super-block spans a wider dynamic range than a 4-6 bit codebook does, + // while Q8_K gives the activation ONE scale per 256 elements. + // + // 6e-4, set from the measurement rather than chosen for headroom. Every + // shape in this case, all 12 per type, re-measured 16 August 2026 with the + // ceiling forced to 1e-12 so doctest prints each captured value: + // + // iq1_s max 5.240e-4 at m=4 n=1, then 3.498e-4 at m=1 n=1 + // iq1_xxxs max 3.109e-4 at m=1 n=1, then 2.637e-5 at m=1 n=7 + // + // The n=1 column is where both the signal and the noise live: the NMSE + // denominator there is a single dot product, so it neither averages nor + // cancels. Every n=7 and n=16 shape sits below 3e-5. + // + // An earlier revision set this to 2e-3, described as ~4x the residual. That + // was a relaxation past the point where the statistic can discriminate. + // Doubling kIq1sDelta to 0.25 moves iq1_s to 6.967e-4, which 6e-4 fails and + // 2e-3 passes. Measure before widening a bound. + // + // Be precise about what this bound is worth, because the same mutation + // moves iq1_xxxs the WRONG WAY: 3.109e-4 unmutated against 1.420e-4 with the + // doubled delta, so NO ceiling catches that defect on this statistic. An + // NMSE against a dequant-f32 reference cannot seal a decode parameter at + // all, since both sides decode through the same `BlockToFloat`. What seals + // it is the pair of golden-vector cases below, which compare against the + // ORACLES bit for bit. This ceiling bounds quantization error, which is its + // own job, and is kept tight enough to stay a second signal where it can be + // one. + {vt::DType::kIQ1_S, 256, 50, 0, -1, -1, "iq1_s", 6e-4}, + {vt::DType::kIQ1_XXXS, 256, 38, 0, -1, -1, "iq1_xxxs", 6e-4}, {vt::DType::kMXFP4, 32, 17, -1, -1, 0, "mxfp4"}, }; @@ -665,6 +681,79 @@ TEST_CASE("kIq1xxxsGrid is the PINNED FORK table, not a look-alike") { CHECK(lanes[2] == 397); // +1 } +TEST_CASE("kIq1sDelta is upstream IQ1S_DELTA, not a value this tree chose") { + // `ggml/src/ggml-common.h:1121` at the pinned 237ad9b96 is + // `#define IQ1S_DELTA 0.125f`. The FORK reuses that same macro for IQ1_XXXS + // (`dequantize_row_iq1_xxxs` and `ggml_vec_dot_iq1_xxxs_q8_K` both spell + // IQ1S_DELTA), so one constant serves BOTH encodings here and one wrong value + // corrupts both at once. + // + // Sealed by value for exactly the reason the two grids are sealed by digest: + // `DequantIQ1_S`, `DequantIQ1_XXXS`, `VecDotIQ1_SQ8_K` and + // `VecDotIQ1_XXXSQ8_K` all read THIS definition, so every one of them agrees + // with every other whatever it holds. Consistency is not correctness. The + // grid seal stopped one table short of this constant, and a doubled delta + // survived the whole suite until the golden vectors below were added. + CHECK(vt::cpu::kIq1sDelta == 0.125F); +} + +// Compare a decode against the oracle-produced goldens, BIT for BIT. Both sides +// evaluate the same f32 expression `dl * (grid[j] + delta)`, so equality is +// exact and any difference means a decode PARAMETER diverged, not that rounding +// moved. Asserting per element (rather than on a digest or a mismatch count) +// makes the assertion total say how many values were actually examined. +void CheckAgainstOracle(vt::DType dtype, const uint8_t* blocks, + const uint32_t* golden_bits, size_t n) { + std::vector got(n); + vt::cpu::BlockToFloat(dtype)(blocks, got.data(), static_cast(n)); + for (size_t i = 0; i < n; ++i) { + CAPTURE(i); + uint32_t bits = 0; + std::memcpy(&bits, &got[i], sizeof(bits)); + CHECK(bits == golden_bits[i]); + } +} + +TEST_CASE("IQ1_S decodes REAL checkpoint bytes as the PINNED ORACLE does") { + // The first independent reference this encoding has. Everything else in this + // file decodes the weight with `vt::cpu::BlockToFloat`, the function under + // test, so it is independent only in the summation: a wrong scale field, a + // wrong delta magnitude or a flipped delta sign moves the reference and the + // kernel together and they still agree. + // + // Here the expected values come from ggml-org/llama.cpp @ 237ad9b96 running + // its OWN `ggml_get_type_traits(GGML_TYPE_IQ1_S)->to_float`, and the inputs + // are real `blk.0.ffn_gate_exps.weight` bytes from the UD-IQ1_S checkpoint. + // Provenance and the reproduction recipe are in iq1_golden_vectors.h. + // + // This seals the DEQUANT arm. The vec_dot arm is tied to it by the G3 + // "vec_dot matches f64 dequantize-then-dot" case above, so a defect injected + // into either one alone now fails somewhere, and a defect injected into both + // fails here. + CHECK(std::size(vllm_test::kIq1sGoldenBlocks) == 4 * 50); // 4 blocks + CHECK(std::size(vllm_test::kIq1sGoldenBits) == 4 * 256); + CheckAgainstOracle(vt::DType::kIQ1_S, vllm_test::kIq1sGoldenBlocks, + vllm_test::kIq1sGoldenBits, + std::size(vllm_test::kIq1sGoldenBits)); +} + +TEST_CASE("IQ1_XXXS decodes REAL checkpoint bytes as the PINNED FORK does") { + // Same argument as the IQ1_S case above. The oracle is the fork, + // unslothai/llama.cpp @ 36fe8e1cc, because no upstream llama.cpp defines ggml + // type 66 at all; the inputs are real UD-Q1_0 bytes. + // + // What this does and does not establish is the same as the spec's + // 1179648-weight run, of which these four blocks are the committed slice: it + // removes transcription error from OUR C++ and proves the block layout is + // read correctly. It does NOT make the fork gateable, because the fork's own + // decode is still unvalidated against anything but itself (#933). + CHECK(std::size(vllm_test::kIq1xxxsGoldenBlocks) == 4 * 38); // 4 blocks + CHECK(std::size(vllm_test::kIq1xxxsGoldenBits) == 4 * 256); + CheckAgainstOracle(vt::DType::kIQ1_XXXS, vllm_test::kIq1xxxsGoldenBlocks, + vllm_test::kIq1xxxsGoldenBits, + std::size(vllm_test::kIq1xxxsGoldenBits)); +} + TEST_CASE("G3 vec_dot is bit-exact run to run (fixed reduction order)") { for (const WeightCase& c : kWeightCases) { CAPTURE(std::string(c.name));