Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .agents/issue-index.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion .agents/quantization-matrix.md

Large diffs are not rendered by default.

429 changes: 399 additions & 30 deletions .agents/specs/qwen38-27b-quant-arms.md

Large diffs are not rendered by default.

5 changes: 4 additions & 1 deletion docs/FEATURES.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,9 @@ are our reading of their documented behavior, not measurements.
| GPTQ | ◐ CPU dequant | ✅ | ✅ | ☐ |
| MXFP4 compressed-tensors | ◐ W4A16 Marlin, mem 2.63x less. gate_up FUSION + decode-graph default-ON; #44 3/3, 32B 6/6. **`VT_MARLIN_DENSE` DEFAULT-ON** (`KERNEL-MARLIN-DENSE-EXEC`): dense marlin 48-CTA, byte-faithful, beats MoE (c8 0.969) | ✅ | ✅ | ☐ |
| Compressed-tensors `mixed-precision` (`config_groups`, ordered regex `targets`) | ◐ scheme read from the config, never from a dtype probe; an arm with no loader is REFUSED BY NAME. On `unsloth/Qwen3.8-27B-NVFP4` the W4A4 group loads, the FP8 W8A8 group and the `kv_cache_scheme` are refused (#821) | ✅ | ✅ | ☐ |
| ModelOpt `MIXED_PRECISION`: STATIC per-tensor FP8 + W4A16_NVFP4 | ☐ UNTRIED. `r0b0tlab/Qwen3.8-27B-NVFP4-MTP-sm121` carries 208 `input_scale`, `F32` scalar `weight_scale` and `"dynamic": false`, so the three `unsloth` blockers are absent (#1574) | ✅ | ✅ | ☐ |
| ModelOpt `MIXED_PRECISION` (`quantized_layers`, exact module names) | ◐ the declared `quant_algo` is CROSS-CHECKED against the shipped operand names at load; a disagreement, an unimplemented algo or an unknown operand family is REFUSED BY NAME. `r0b0tlab/...-MTP-sm121` loads (#821) | ✅ | ✅ | ☐ |
| ModelOpt `MIXED_PRECISION`: the cross-check moves no arm | ✅ `nvidia/Qwen3.6-27B-NVFP4` @`0893e160` is NOT refused, and one case says so: same format, wildcard exclusions, an `input_scale` on every NVFP4 module, a `kv_cache_scheme` shipping zero scales (#821) | ✅ | ✅ | ☐ |
| ModelOpt `MIXED_PRECISION`: STATIC per-tensor FP8 + W4A16_NVFP4 | ◐ LOADS, never RUN. `r0b0tlab/Qwen3.8-27B-NVFP4-MTP-sm121` carries 208 `input_scale`, `F32` scalar `weight_scale` and `"dynamic": false`, so the three `unsloth` blockers are absent (#1574, #821) | ✅ | ✅ | ☐ |
| fp8 weights, per-tensor scale | ✅ | ✅ | ✅ | ☐ |
| Block-wise (fine-grained 128x128) FP8, the `weight_scale_inv` layout | ◐ RUNS on CPU (#1189 M4/M6): 10 projections as 7 GEMMs, `gate_up`+QKV merged. CUDA sm120: N,K %128==0 only; 7 shapes MATCH CPU ref on GB10, no token gate (#1437) ([spec](../.agents/specs/vt-matmul-fp8-block-cuda.md)) | ✅ | ✅ | ☐ |
| Per-tensor FP8 W8A8 linear is a shared seam any model can bind | ✅ `models/dense_fp8_gemm.h` + `layers::Fp8W8A8LinearMethod` (#940), bound via `layers::MakeLinearMethod`. One definition, CUDA only ([spec](../.agents/specs/vt-fp8-shared-seam.md)) | ✅ `Fp8LinearMethod` | ✅ | ☐ |
Expand Down Expand Up @@ -367,6 +369,7 @@ CPU elementwise GEMM (f32/f16/bf16) runs AVX2 and AVX-512 tiers on x86 where the
| LTX-2.5 resolution | Off-grid sizes refused, naming the offending axis and a size you can actually pass; frames still round | `--width`/`--height` must divide 64 (two-stage) or 32 (one-stage), from the VAE factor times the phase downscale ([#919](https://github.com/mudler/vllm.cpp/issues/919)). `--frames` rounds to `8k + 1`. No size cap |
| LTX-2.5 arms nothing can request | Declared, not requestable | `int8-convrot` (ComfyUI-only), single-node multi-GPU, `BetaScheduler` (upstream selects no scheduler either). No flag or extra asks for these. `multishot` was RETIRED (absent upstream) and `kLoraFusion` too (now served) |
| Qwen3.8-27B quantized arms (Q4_K_M GGUF, its `clip` mmproj, the `unsloth` "NVFP4" = `mixed-precision`) | All three files ACCOUNTED against committed manifests (866 / 334 / 1968 names), none RUN: no token gate, Q4_K_M PENDING on #857 | BF16 gated (#915). `--mmproj` (ABI v22) loads a `clip` GGUF; a load refuses a file with a tensor no loader reads. "NVFP4" @`7d6f8d4d`: W4A4 loads, FP8 refused ([spec](../.agents/specs/qwen38-27b-quant-arms.md)) |
| Qwen3.8-27B, the SECOND NVFP4 artifact (`r0b0tlab/...-MTP-sm121`, a ModelOpt checkpoint) | 2001 names ACCOUNTED per scheme against four committed manifests; LOADS, never RUN: no token gate (#1632) | @`36f717a2`: 208 per-tensor STATIC FP8 + 193 W4A16_NVFP4 modules, both halves load. Not the `unsloth` format ([spec](../.agents/specs/qwen38-27b-quant-arms.md)) |
| Multi-GPU execution | Hardware-blocked | TP proven equal to tp=1 on CPU; no 2-GPU box to run it |
| LoRA end to end | CPU brick landed | Unwired standalone; not usable through the server |
| Multimodal over HTTP | Image request path wired; forward + codec pending | `ROAD-V1-MM` W1-W3 landed. Open: no mm-forward on `Request.mm_features`; no image codec. Video/audio/multi-image now **refuse** with HTTP 400 rather than drop ([#686](https://github.com/mudler/vllm.cpp/issues/686)) |
Expand Down
4 changes: 4 additions & 0 deletions docs/USAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -246,6 +246,10 @@ lists other published arms when they have not been used as a gated checkpoint.
| Qwen3.8-27B GGUF projector | `mmproj-BF16.gguf` | 931,146,432 bytes | `unsloth/Qwen3.8-27B-GGUF` @ `fe1e2a23d973adb629709749dc4f6756df66ef10` | `83ee4f4f205fa514161778c41df1ea14144faa0f713510893b63c2395f5c2d53` | BF16 `clip` projector loads and validates through `--mmproj` | No request path runs the loaded projector |
| Qwen3.8-27B mixed FP8 and NVFP4 | `model.safetensors` | 22,568,192,096 bytes | `unsloth/Qwen3.8-27B-NVFP4` @ `7d6f8d4d72f56b92b3cdbf22f156b90e1bab0108` | `c473512c70eace07e2256fe9fd76596ac03e3295bee7d54cfb72676416afcc05` | NVFP4 modules load | FP8 modules and quantized KV cache are refused |
| Qwen3.8-27B MTP drafter | `model_mtp.safetensors` | 849,400,392 bytes | `unsloth/Qwen3.8-27B-NVFP4` @ `7d6f8d4d72f56b92b3cdbf22f156b90e1bab0108` | n/a (non-quantized) | BF16 MTP artifact is present | MTP execution is owed |
| Qwen3.8-27B ModelOpt NVFP4 shard 1 of 4 | `model-00001-of-00004.safetensors` | 9,965,644,108 bytes | `r0b0tlab/Qwen3.8-27B-NVFP4-MTP-sm121` @ `36f717a22990e82c54c1d48ee77c491b87825680` | Locally computed hash is owed; the bytes are not mirrored here, and the four shards are verified semantically instead (header parse plus data end equal to the published size); #821 | 208 per-tensor static FP8 and 193 W4A16_NVFP4 modules load | The declared FP8 KV cache is unread; #1593 |
| Qwen3.8-27B ModelOpt NVFP4 shard 2 of 4 | `model-00002-of-00004.safetensors` | 9,985,743,924 bytes | `r0b0tlab/Qwen3.8-27B-NVFP4-MTP-sm121` @ `36f717a22990e82c54c1d48ee77c491b87825680` | Locally computed hash is owed; #821 | Same arms as shard 1 | The declared FP8 KV cache is unread; #1593 |
| Qwen3.8-27B ModelOpt NVFP4 shard 3 of 4 | `model-00003-of-00004.safetensors` | 1,120,886,516 bytes | `r0b0tlab/Qwen3.8-27B-NVFP4-MTP-sm121` @ `36f717a22990e82c54c1d48ee77c491b87825680` | Locally computed hash is owed; #821 | Same arms as shard 1 | The declared FP8 KV cache is unread; #1593 |
| Qwen3.8-27B ModelOpt MTP drafter | `model-00004-of-00004.safetensors` | 849,400,592 bytes | `r0b0tlab/Qwen3.8-27B-NVFP4-MTP-sm121` @ `36f717a22990e82c54c1d48ee77c491b87825680` | Locally computed hash is owed; #821 | Fifteen BF16 MTP tensors are present and unquantized | MTP execution is owed |
| Qwen3.8-2.4T-A95B | `UD-Q1_0` ten-file GGUF split | about 370 GiB | `unsloth/Qwen3.8-2.4T-A95B-GGUF` @ `567d3e6ac26c5474b18311e619c04350fb9a5556` | `b7770552b2ac24e7334c917bc92e90e218e87cfe29484db65e62e8ef2a60334d` (shard 1); `2765517f833c736338d3ab34354e1c10eb8d79e62325f998285b435e5cf03dcd` (shard 2) | CPU expert streaming from disk | CUDA refuses a checkpoint that exceeds device capacity |
<!-- checkpoint-registry:end -->

Expand Down
Loading
Loading