Clarify GGUF quantized storage fidelity and BitShift fallback compute

[justinchuby]

Problem

A graph containing BitShift/BitwiseAnd followed by DequantizeLinear and floating-point MatMul can look as though a purportedly quantized model was converted to FP32. That interpretation conflates two separate properties:

1. Storage quantization — whether weights remain packed (for example, 4-bit codes plus scales/zero-points) in the saved ONNX package.
2. Compute quantization — whether the selected execution provider executes a native/fused quantized kernel or expands the weights to floating point for computation.

Mobius needs to report both properties explicitly and must not imply source-format fidelity merely because the target storage remains quantized.

What the BitShift graph means

com.microsoft::MatMulNBits is emitted as the native graph operator. Mobius also registers an inlineable standard-ONNX Function body for runtimes that do not implement that custom operator.

The Function body:

1. uses BitwiseAnd and BitShift to unpack low/high 4-bit nibbles from packed UINT8 weights;
2. casts the codes to UINT4;
3. applies block scales/zero-points through DequantizeLinear;
4. performs floating-point MatMul.

This is a portable compute fallback over quantized storage. Inlining the Function does not replace the packed initializer with a dense FP32 initializer. When the native MatMulNBits operator remains in the graph, the actual kernel is execution-provider dependent; CPU configurations may use either float dequantization or dynamically quantized INT8 dot products. We must not promise a particular compute kernel from the storage format alone.

Concrete Gemma4 evidence

Pinned artifact:

• Repository: unsloth/gemma-4-E2B-it-GGUF
• Revision: 0314792d7f1f7e229411f620751375812bb9faf2
• File: gemma-4-E2B-it-Q4_K_M.gguf
• Size: 3,106,738,272 bytes
• LFS SHA-256: 740185b21d22ceb83a11c3aa62ad5842ef32c70f6096d756bbee85a1e4ec34b8

Header-only tensor census (no tensor payload download):

Source dtype	Tensors	Source bytes	Current affine INT4/block-32 disposition
Q4_K	212	1,121,845,248	Lossy dequantize + requantize
Q6_K	34	215,470,080	Lossy 6-bit → 4-bit requantization
Q5_K	1	1,614,807,040	Lossy 5-bit → 4-bit requantization (per_layer_token_embd.weight)
F32	353	111,270,028	Preserved as float; not quantized
BF16	1	27,525,120	Preserved as float; not quantized

The resulting projections/embeddings remain packed INT4 in the target package, but this conversion is not source-faithful Q4_K_M and must not be labeled as though the original GGUF quantization was preserved.

Current fidelity classes

The classification must be based on the actual tensor qtype and selected graph/runtime path, never the filename or preset name.

Byte-preserved native blocks

When the selected text-only runtime path supports BlockQuantizedMatMul, these formats retain their serialized GGUF blocks:

• MXFP4
• IQ4_NL, IQ4_XS
• IQ3_S, IQ3_XXS
• IQ2_XXS, IQ2_XS, IQ2_S
• IQ1_S, IQ1_M

The same qtypes are not automatically faithful in paths that normalize every tensor to a common affine MatMulNBits layout (notably current multimodal normalization).

Numerically lossless affine repack

These supported formats can be rearranged into the corresponding affine target without changing their dequantized values:

• Q4_0
• Q4_1
• Q8_0

Q1_0 has multiple incompatible layouts and must be classified by the detected layout rather than by its shared type ID.

Lossy quantized-to-quantized conversion

These source formats are lossy when normalized to affine INT4/block-32:

• Q4_K (two-level super-block scales/minima cannot be represented exactly by the target layout)
• Q5_K / Q5_0 when reduced to 4-bit
• Q6_K when reduced to 4-bit
• mixed presets containing any tensor that does not exactly match the selected target layout
• native IQ/MXFP4 formats when forced through multimodal/common-affine normalization rather than the native block path

The output may still be storage-quantized, but it must be described as converted INT4, not as faithful preservation of the source preset.

Float conversion

--dequantize / keep_quantized=False intentionally creates floating-point weights. Such output must not carry a quantized-storage claim.

Required behavior

• Preflight every mapped GGUF tensor and classify it as:
  • byte-preserved native;
  • numerically lossless repack;
  • lossy quantized requantization;
  • dequantized float;
  • rejected.
• Emit one prominent warning before conversion when any quantized tensor will be lossy, including source qtypes, tensor counts, source bytes, and target format.
• State which qtypes in that artifact are lossless under the selected path.
• Persist a machine-readable quantization_report.json with:
  • source qtype census;
  • per-disposition counts/bytes;
  • target storage format;
  • source_faithful and storage_quantized booleans;
  • compute-path/capability description without claiming a specific provider kernel;
  • explicit float tensors.
• Never name a lossy normalized result Q4_K_M; describe it as, for example, INT4 affine block-32 converted from a mixed GGUF.
• Verify that Function inlining leaves packed initializers packed and does not create dense FP32 weight initializers.

This policy intentionally allows lossy quantized-to-quantized conversion, but only with truthful warning, metadata, and naming.