Skip to content

docs: align quantization and registry counts #575

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mudler merged 1 commit into from
Aug 13, 2026
Merged

docs: align quantization and registry counts #575

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/BUILD.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -270,9 +270,9 @@ of operations. Per-op detail is in the
|---|---|
| NVFP4 W4A4 / W4A16 | Both gate-model paths run on GB10, token-exact. FP4 tactics match vLLM; Marlin NVFP4 W4A16 grouped-MoE is the 35B expert path |
| compressed-tensors NVFP4A16 (W4A16), dense | Correctness-complete via the Marlin weight-only path; speed not yet measured |
| GGUF F32 / F16 / Q4_0 / Q8_0 / Q3_K / Q4_K / Q5_K / Q6_K | Supported. On CPU the six block encodings compute directly on the compressed blocks (`VT_GGUF_KEEP_QUANT=0` disables it). GPU builds still expand GGUF weights |
| GGUF F32 / F16 and block quantization | Supported. CPU keeps the supported Q, IQ, and MXFP4 blocks compressed through the matrix multiply. CUDA also keeps the supported Q8_K-activation formats compressed; other formats fall back to expansion or CPU compute. Set `VT_GGUF_KEEP_QUANT=0` to disable the direct path. See [STATUS.md](STATUS.md) for the exact format and backend coverage |
| FP8 (W8A8) | The 35B ModelOpt static per-tensor projection slice is implemented; generic FP8 modes and FP8 KV remain open |
| MXFP4 / MXFP8 | Planned |
| compressed-tensors MXFP4 (W4A16) | Qwen3 dense weights load and run through the Marlin path on CUDA. Qwen3-8B is correctness-gated and benchmarked against vLLM; c1 passes the speed floor, while c2-c8 remain below it. MXFP8 compute remains open |

## Environment variables

Expand Down
2 changes: 1 addition & 1 deletion docs/STATUS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ are `VOID`; no product behavior changed.

Supported-model registry guard (2026-08-06): the public per-architecture list in
[FEATURES](FEATURES.md) is CI-bound to the C++ registry by
`scripts/check-supported-models.py` (+ mutation test), so the 30
`scripts/check-supported-models.py` (+ mutation test), so the 37
`REGISTER_VLLM_MODEL` architectures and the FEATURES rows cannot drift.

GCC 12 production-library maintenance (2026-07-31): the two known `-Werror`
Expand Down
Loading