Skip to content

IQ1_S/IQ1_XXXS decode: every parameter but the grid is pinned only by self-consistency, and the loader's expansion path cannot decode types 18/19/66 #1023

Description

@localai-bot

Found by an independent review of the IQ1_S (ggml 19) / IQ1_XXXS (ggml 66) decode that landed in #946, and reproduced here before repair.

Every decode parameter except the grid was pinned only by self-consistency. The grids carry an FNV-1a digest and a lane census, and that seal works. It stops one table short. ReferenceDotF64 (tests/vt/test_ops_quant_dot.cpp:231-236) and the G3 MatmulBTQuant NMSE reference (:915) both decode the weight with vt::cpu::BlockToFloat, which is the function under test, so they are independent only in the summation. Three defects were injected, each applied and compiled, and the whole suite stayed green with an unchanged assertion count:

Mutation Site Baseline result
kIq1sDelta 0.125F to 0.25F (affects BOTH encodings) cpu_quant_iq_tables.h:422 UNCAUGHT
IQ1_S delta sign inverted in dequant AND vec_dot cpu_quant_dequant.cpp:389, cpu_quant_dot.cpp:633 UNCAUGHT
IQ1_S scale read from qh bits 13-15 instead of 12-14, both paths same two files UNCAUGHT

The NMSE ceiling was widened past the point where it discriminates. tests/vt/test_ops_quant_dot.cpp:148,153 set nmse_max = 2e-3 for both new types. Re-measured over all 12 shapes per type: unmutated iq1_s peaks at 5.240e-4 and iq1_xxxs at 3.109e-4. With the doubled delta iq1_s reads 6.967e-4, so 2e-3 passes the defect and 6e-4 fails it. On iq1_xxxs the same defect moves the statistic the WRONG WAY, 3.109e-4 down to 1.420e-4, so no ceiling catches it there at all: an NMSE against a dequant-f32 reference cannot seal a decode parameter, because both sides decode through the same BlockToFloat.

The loader's expansion path cannot decode the target checkpoint. gguf_dequant.cpp:107-116 lists no case 19 and no case 66, so DequantGgufRowToF32 throws unsupported ggml type although vt::cpu::BlockToFloat decodes both in one line. RouteGgufTensor (gguf_keep_quant.cpp:122) routes a tensor to kExpandBf16 whenever cpu_ref is set (the VT_CPU_REF oracle switch), keep-quant is off, K is ragged, or the role is not verbatim, so the reference lane refuses the model. ggml 18 (IQ3_XXS) has the same omission, pre-existing since the DeepSeek-V4 UD-IQ2_XXS port; it is the ffn_down encoding of that vehicle. All three are one shared switch branch and are fixed together here.

Three smaller ones, all in the same files: cpu_quant_dot.cpp:591 says the IQ1_S kernel reads bsums "and the others do not", while Q2_K, Q5_K and Q4_K already do; cpu_quant_iq_tables.h:1-5 calls the file a 1:1 mirror of llama.cpp @ 237ad9b96 and indexes its tables, but now also carries kIq1xxxsGrid from unslothai/llama.cpp @ 36fe8e1cc and indexes neither new table; and both new census cases in tests/vllm/test_gguf_keep_quant.cpp claim TOTAL coverage of 1702 tensor records while enumerating six of the seven encodings, summing to 864 (F32, 838 tensors, was missing).

Repair. Seal kIq1sDelta against upstream IQ1S_DELTA (ggml-common.h:1121 @ 237ad9b96), and add golden-vector fixtures for all three encodings whose EXPECTED values are produced by the oracles themselves rather than by this tree: ggml_get_type_traits(type)->to_float in a build of ggml-org/llama.cpp @ 237ad9b96 for types 18 and 19, and of unslothai/llama.cpp @ 36fe8e1cc for type 66. The IQ1 inputs are real blk.0.ffn_gate_exps.weight bytes from the two target checkpoints, so this also commits a slice of the 1179648-weight bit-identity run the spec records. Set the NMSE ceiling from the measurement. Complete the census so it sums to the declared record count.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions