Qwen3.8-27B three-way at each engine's best: one checkpoint ours/vLLM/SGLang all serve, and three of the four recorded NVFP4 blockers are properties of the unsloth artifact rather than the format

[localai-bot]

A published third-party package serves Qwen3.8-27B on a single GB10 and states
its numbers with the exact flags that produced them:
r0b0tlab/qwen38-27b-nvfp4-sm121-vllm,
with a SGLang sibling at
r0b0tlab/qwen38-27b-nvfp4-sm121-sglang.
Both serve the SAME checkpoint,
r0b0tlab/Qwen3.8-27B-NVFP4-MTP-sm121
@ 36f717a22990e82c54c1d48ee77c491b87825680. Their published claims, same box,
r0b0bench core-subset, think-off:

Engine	Profile	c1	c2	c4	c6	c8
vLLM	DFlash2 K8	67.1	121.5	211.5	279.2	—
SGLang	DFlash2 K8	68.6	124.3	212.0	276.4	—
vLLM	MTP K3	27.8 dedicated	—	34.61	—	82.89
SGLang	EAGLE	28.38 dedicated	—	—	—	123.90
vLLM	AR (no spec)	11.35	—	—	—	—

We have no number on this subject at any of those settings. This issue owns
closing that, and the three-way measurement that follows.

The finding that resizes the work

QUANT-QWEN38-27B-NVFP4-ARM records four blockers against
unsloth/Qwen3.8-27B-NVFP4. Three of them are properties of THAT ARTIFACT and
not of the format, and the checkpoint the competitor numbers were actually
measured on does not carry any of the three. Read from the artifact on
2026-08-21 — model.safetensors.index.json, and shard 1's safetensors header by
range request — not from its model card:

Recorded blocker (unsloth)	r0b0tlab tree
the unconditional .input_scale read, against *.input_scale appearing ZERO times	208 input_scale tensors present, F32 scalar []. The unconditional read succeeds
a per-channel BF16 weight_scale that ReadF32Scalar refuses on BOTH count and dtype	weight_scale is F32 scalar [] on every FP8 module. Per-tensor static, which is what ReadF32Scalar wants
no representation for a DYNAMIC per-token activation scheme	config_groups.group_0 sets "dynamic": false on BOTH weights and input_activations. The dynamic scheme is not used
a scheme never read from the config at all	NOT discharged for this artifact. See the correction below

The NVFP4 half is also the easier one. unsloth ships nvfp4-pack-quantized
W4A4; this tree ships W4A16_NVFP4, group_size 16, weight-only, in the
standard two-level layout our tree already runs: weight U8 [17408, 2560],
weight_scale F8_E4M3 [17408, 320] (5120/16 = 320), weight_scale_2 F32 [].

Accounting, corrected 2026-08-21 — the first version of this issue said 594
weight_scale and omitted A_log, and those buckets did not sum to 2001. The
594 came from a SUBSTRING match that also caught weight_scale_2. Exact
suffixes, which do sum:

count	suffix
937	.weight
401	.weight_scale
208	.input_scale
193	.weight_scale_2
166	.bias
48	.A_log
48	.dt_bias
2001	total

The 193/208 split still matches the publisher's stated "193 W4A16_NVFP4 + 208
FP8". W4's accounting gate is pinned to the unsloth NAME SET (1953 + 15 = 1968)
and does not cover this one.

k_scale and v_scale appear ZERO times, while hf_quant_config.json sets
kv_cache_quant_algo: "FP8". So the fp8 KV scales are defaulted by the engine
rather than read, and the serve flag --kv-cache-dtype fp8 is load-bearing for
CORRECTNESS, not only memory: the publisher's canary is 19 x 23, which returns
437 with the flag and 417 without it.

What actually blocks us

#	Blocker	Owner	State
1	fp8 KV cache on CUDA — the store kernel, the paged-attention read dequant, and half-sized blocks in the real runner	KV-FP8 / QUANT-KV-FP8	W1 landed CPU-only. The CUDA half is owed and is on the critical path, because the target profile cannot be served correctly without it
2	the NVFP4 + static-FP8 tower resolving and RUNNING on this name set	QUANT-QWEN38-27B-NVFP4-ARM W5	W4 landed the resolver against a different name set. W5 is unclaimed
3	DFlash2's candidate selector and speculator	SPEC-DFLASH2 W3/W4/W6 (#1314)	W1, W2 landed. The draft is still refused BY NAME at RefuseDflash2CandidateSelector
4	MTP execution	QUANT-QWEN38-27B-NVFP4-ARM	owed (docs/USAGE.md: "BF16 MTP artifact is present; MTP execution is owed") — needed only for the mtp comparison row, not for AR or DFlash2

Blocker 1 and 2 together buy the AR row (11.35), which is the first cell we
can honestly contest. Blocker 3 buys the headline DFlash2 row.

Both competitor arms are now reachable

They were not. The lease's worker pod has no image path
(#1265), which is why the
SGLang oracle is gateable = no. The developer authorized rc hold plus ssh
to the leased host for this campaign on 2026-08-21, recorded in
.agents/developer-preferences.md. The lease stays the mutex, so the fleet
still reports the device busy; what the host shell adds is sudo docker run for
the published images and sudo nvidia-smi -lgc for clock pinning
(#1354 — every c1 pairing of
the 2026-08-19 series came back PAIRING_VERDICT=DISCARD on 12.9-26.4%
within-run SM-clock spread against a 5% ceiling, with no lever available inside
the pod).

Deliverable

One table, three engines, ONE checkpoint, each engine on its own best published
profile, every flag recorded, plus a reproduce recipe in docs/USAGE.md that a
reader can run. A cell we lose is published as a cell we lose.

Related: #979 scopes a four-way on this model and concludes no single
quantization is common to all four engines. That conclusion stands for
llama.cpp. It is now FALSIFIED for the other three: this checkpoint is common to
ours, vLLM and SGLang, and #979's matrix owes that correction.

Corrected 2026-08-21, before any work started

Blocker 3 in the table above was read at cae8ace0c and was already stale when
this issue was filed. Re-derived against 947e5f648:

SPEC-DFLASH2 is no longer a mechanism blocker. W3 (the candidate
selector), W4 (the path walk) and W5 (the GGUF drafter arm) all landed on
2026-08-20. RefuseDflash2CandidateSelector is RETIRED, both containers draft
end to end on a GPU, and the startup refusal is gone from both. What the row
still owes is W6's gate and the fact that no published DFlash2 artifact has
been LOADED yet — which is a thing this campaign does anyway, because its
subject is exactly such an artifact.

The corrected order:

#	Blocker	Owner	State at 947e5f648
1	fp8 KV on CUDA — the store kernel, the paged-attention read dequant, half-sized blocks and --kv-cache-dtype threading	KV-FP8 W2 and W3	the critical path. W1 landed CPU-only; W2, W3 and W4 all read later
2	the NVFP4 + static-FP8 tower resolving and RUNNING on this name set	QUANT-QWEN38-27B-NVFP4-ARM W5	unclaimed. W4 landed the resolver against the unsloth name set (1968), not this one (2001)
3	the DFlash2 GATE, not the mechanism	SPEC-DFLASH2 W6	mechanism DONE 2026-08-20
4	MTP execution	QUANT-QWEN38-27B-NVFP4-ARM	owed; needed only for the MTP comparison row

This moves the headline cell closer. The DFlash2 column stops being "port a
speculator" and becomes "gate a speculator that already runs", so the whole
campaign now hangs on fp8 KV.

Corrected 2026-08-21 (second correction): the format is ModelOpt, not compressed-tensors

The fourth blocker was recorded above as discharged by W4's compressed-tensors
resolver. That is wrong for THIS artifact, and the reason matters more than the
error. quantization_config declares:

quant_method : modelopt
quant_algo   : MIXED_PRECISION
ignore       : []            # empty, against unsloth's 303 entries

It carries BOTH a config_groups block (the compressed-tensors shape this issue
originally read) and a quantized_layers block naming 401 exact modules (the
ModelOpt shape). The DECLARED method is modelopt, so ct::Config stops at
quant_method != "compressed-tensors" and nothing in this tree reads this
config at all. Every routing decision then falls to a tensor-name probe, which
can be silently wrong in both directions.

So the work is not "extend W4's resolver to a second name set". It is the first
production wiring of the ModelOpt reader. That is what PR #1603 does.

[localai-bot]

First measured cells, 2026-08-21. Both competitor arms, one box, one instrument.

All three servers ran the SAME checkpoint, r0b0tlab/Qwen3.8-27B-NVFP4-MTP-sm121 @ 36f717a2, four shards sha256-verified against the publisher's final-sota-shards.sha256. All were launched by the publishers' OWN scripts, unmodified. All were read by ONE instrument — vllm bench serve from the published vLLM image, driven against each server in turn — because a cross-engine throughput claim measured by two different harnesses is not a comparison.

Every arm passed the publisher's 19 x 23 -> 437 canary (9/9 checks) BEFORE any timing leg. That gate is not ceremony here: a broken DFlash2 selector emits well-formed drafts, the verify is lossless, and only acceptance falls — so the tokens still look right while the speed claim is meaningless.

Dedicated c1, 512 -> 2048, --seed 0 --ignore-eos --temperature 0

Arm	output tok/s	TPOT	duration	mode
vLLM AR (no speculation)	9.71	102.74 ms	210.98 s	eager
vLLM DFlash2 K8	22.28	44.41 ms	91.91 s	eager
SGLang DFlash2 K8	66.87	14.72 ms	30.63 s	CUDA graphs

Failed requests: 0 on every leg. Clocks pinned, verified DURING the runs at 0.29% within-run spread (120 samples, 2080-2086 MHz), zero throttle samples, peak 60 C.

Why SGLang is 3x ahead, established rather than assumed

The 3.0x throughput gap and the 3.0x TPOT gap are the same fact. At ~15 ms/token, host dispatch dominates and CUDA graphs are worth exactly this kind of multiple. SGLang captures them (target_verify=15.83 s, draft_decode=14.18 s, bs 1-22). Every published vLLM profile ships --enforce-eager.

So I removed --enforce-eager from the vLLM DFlash2 profile, changing nothing else. It does not start.

File "vllm/model_executor/models/qwen3_dflash2.py", line 277, in forward
    for start in range(0, flat.shape[0], chunk):
torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints violated
  - You marked L['candidate_ids'].size()[0] as dynamic but your code specialized it to be a constant (56)
  - You marked L['hidden_states'].size()[0] as dynamic but your code specialized it to be a constant (56)
RuntimeError: Engine core initialization failed.

The DFlash2 selector's forward runs a host-side Python loop over a tensor dimension (for start in range(0, flat.shape[0], chunk)), which specialises a dimension torch.compile was told is dynamic. --enforce-eager is therefore not a conservative default in that package. It is mandatory, and the eager penalty is not a methodology artefact of ours to correct away.

That settles ## 4 D1 for this profile: eager IS vLLM's production configuration for DFlash2 on this hardware, because no other configuration exists. The 3x is real and attributable to a concrete architectural difference.

What this means for our own port, concretely

We do not have that loop. .agents/specs/dflash2-spec-decode.md records that vt::Dflash2PathWalk runs "in UPSTREAM'S OWN GRID, one program per request with the step loop INSIDE it", chosen because the host-side sequential walk cost 28% of the 27B draft step in DSpark (#436) before SampleSequentialDevice moved it. We already graph a speculative verify path for DSpark behind VT_SPEC_DECODE_GRAPH (+12.2%/+3.5% on the 35B cells).

So the thing blocking vLLM from a 3x on this profile is a thing this project deliberately designed out. Graphing our DFlash2 verify is a named lever with a measured size, and it is worth its own row. That is a hypothesis with an obvious test, not a claim.

One published claim does not reproduce, and one comparison is unresolved

AR c1. Published 11.35; measured 9.71 with their own tool, their image, their flags, on an unthrottled box. Three instruments agree: our probe pinned 9.59, our probe unpinned 9.37, their tool 9.71. Their suite header says it ran on "node2", a different physical Spark, so this is a statement about this box. Note also that the same run reports Peak output token throughput 11.00 and Total token throughput 12.20, both nearer 11.35 than the sustained figure, so their headline may be a peak or a total-token number.

The DFlash2 ladder is NOT resolved and must not be quoted as a win. Their ladder is "r0b0bench aggregate client throughput", a term they never define. Our ladder (1024 -> 256, their protocol):

c	output tok/s	total tok/s (in+out)	their published
1	18.92	95.50	67.1
2	28.74	145.11	121.5
4	50.79	256.42	211.5
6	59.10	298.37	279.2

If "aggregate" counts input+output we are above them at every point; if it counts output only we are below by 3.5-4.7x. Two readings, opposite verdicts, and picking the flattering one would be the whole failure mode this campaign exists to avoid. Recorded UNRESOLVED pending a definition of their metric.

Their published pair also deserves a question: they report vLLM 67.1 against SGLang 68.6 and call the ladder "within ~3%", while shipping eager for one engine and graphs for the other. On this box those two profiles are 3x apart.

Instrument notes worth keeping

Two vllm bench serve fields are junk on a speculative arm and are not quoted above: at c1 it reported Peak output token throughput 9.00 BELOW a mean of 22.28, and Mean ITL 3496 ms against Mean TPOT 44.41 ms. DFlash2 emits in bursts of up to K=8, and any metric derived from gaps between streamed chunks misreports. Duration, output throughput and TPOT are the trustworthy fields. This also weakens reading their AR "peak 11.00" as their headline.

Two client invocations failed with exit 0 and empty output before the SGLang cell was obtained: the vLLM CLI builds its arg parser via VllmConfig and dies with "Failed to infer device type" when the client container has no GPU visible, and the image's ENTRYPOINT is vllm serve, so docker run ... vllm bench serve parses as vllm serve vllm bench serve. In both cases the pipeline's exit status was the grep's, not the container's. Taking either at face value would have recorded "SGLang returns no result" — a claim about the engine that was entirely a claim about the invocation.