diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index b6cd9205d..010ca6846 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -21723,3 +21723,119 @@ OWED: re-run the pre-reimage protocol (single cold oracle invocation) on THIS box. If it reproduces ~0.97 here, cold JIT is confirmed as the inflator and every recorded ratio needs revising. If it reproduces ~0.83, the machine changed and the old numbers stand for the old box. +## Q38-27B-BF16 — Qwen3.8-27B bf16 online serving vs the pinned oracle, first numbers on this checkpoint (2026-08-15, `row/MODEL-Q38-27B-GATE`, base `origin/main`, GB10 sm_121a, #915) + +TOKEN AXIS FIRST, because a speed number on a checkpoint that has not been +adjudicated is not a result. `Qwen/Qwen3.8-27B` @`1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0`, +bf16, 55,586,114,863 bytes over 18 shards. Greedy, 7 prompts x 16 tokens vs +`0.23.1rc1.dev1511+g555967922` / FlashInfer `0.6.15.post1`: 4/7 prompts STRICT +16/16, three first-divergence positions, all EXACT fp32 TIES at 0.000 mnats with +our token at rank 3 / 2 / 2. `ALL_TIES_OR_IN_BAND` against `kNearTieMnats = 500`. +Detail and the instrument argument in +[`specs/qwen38-27b-bf16-gate.md`](specs/qwen38-27b-bf16-gate.md). + +BUILD AND RECIPE. Source `11a42dc4c46b9f6d78d9a43064a8b29880a45a46`, staged by +`git archive` (md5 `2801ab2e49079393ea01b00381d5e6a8` verified both ends), built +in `nvidia/cuda:13.0.1-devel-ubuntu24.04` with `--runtime nvidia`: + +``` +cmake -S /src -B /src/build -G Ninja -DCMAKE_BUILD_TYPE=Release \ + -DVLLM_CPP_CUDA=ON -DVLLM_CPP_CUDA_ARCHITECTURES=121a \ + -DVLLM_CPP_TRITON=ON -DVLLM_CPP_CUTLASS_FETCH=ON -DVLLM_CPP_SERVER=ON +``` + +Fast path ASSERTED from the configure log, not assumed: `CUTLASS found ... +enabling sm120a NVFP4 cutlass GEMM`, `CUDA feature fa2: ENABLED for [121a]`, +`fp4-mma` / `cutlass-nvfp4` / `cutlass-fp8` / `marlin-nvfp4` all ENABLED, 22 +`sm_121a` Triton AOT manifest lines, and `cuobjdump -lelf` on the shipped +library reporting `sm_121a`. `NINJA_EXIT=0`, zero `error:` hits, zero ENOSPC. +`vllm-server` md5 `bda95d34a7e2587c6e2195e365f77bc0`, `libvllm.so.0.0.3` md5 +`bacf61ad3090af11c8fa13e3eab955bd`, both relinked at 15:22:59Z (the script +deletes the artifacts up front, so their existence after the build is itself +evidence the link ran). + +THE DENOMINATOR IS vLLM'S PRODUCTION CONFIG. No `--enforce-eager` anywhere on +the vLLM arm, so CUDA graphs are ON. `--language-model-only` is passed to BOTH +arms so the denominator is not handicapped by a vision tower neither workload +uses (#414). Timed requests are issued only by the oracle's own +`vllm bench serve`, so the client is identical across arms. One `flock` for the +whole series, so the two arms are never interleaved with a neighbour's job. + +`--mamba-ssm-cache-dtype float32` on the vLLM arm is FIDELITY, NOT A HANDICAP, +and this was checked rather than inherited: `config.json` for this checkpoint +declares `text_config.mamba_ssm_dtype = float32`, which is what vLLM's own +Qwen3.5 verification hook copies into `mamba_ssm_cache_dtype`, and our C++ +boundary mirrors the same split (BF16 conv + FP32 SSM, +[`specs/gdn-semantics.md`](specs/gdn-semantics.md)). Passing it makes resolved +behaviour explicit instead of relying on the hook. + +WORKLOAD. `--dataset-name random --random-input-len 1024 --random-output-len 128 +--random-range-ratio 0 --request-rate inf --ignore-eos --temperature 0 --seed 0`, +`num-prompts = 6 x concurrency`, concurrency 1 / 4 / 8, 3 reps per arm, +interleaved ours/vllm so clock or thermal drift lands on both alike, page cache +dropped before every leg. Server config identical across arms: +`--max-num-seqs 32 --max-num-batched-tokens 8192 --max-model-len 2048 +--no-enable-prefix-caching`. + +RESULT, 3 paired reps, every leg retained, medians over reps. vLLM completed +every request in all nine of its legs. We did not, and that decides the shape of +this record: + +| Leg | c1 | c4 | c8 | +|---|---|---|---| +| ours completed | 5, 5, 5 of 6 | 24, 24, 24 of 24 | 36, 37, 36 of 48 | +| vLLM completed | 6, 6, 6 of 6 | 24, 24, 24 of 24 | 48, 48, 48 of 48 | +| output tok/s, ours / vLLM | 2.37 / 3.50 | 15.01 / 15.58 | 15.96 / 27.85 | +| median TPOT ms, ours / vLLM | 220.6 / 223.6 | 239.0 / 234.3 | 261.1 / 241.4 | +| output throughput ratio | WITHHELD | 0.963x | WITHHELD | +| median ITL ratio | 1.013x | 1.008x | 1.021x | +| median TTFT ratio | 0.733x | 0.881x | 1.268x | + +WHY TWO CELLS ARE WITHHELD RATHER THAN QUOTED. `output_throughput` is tokens +divided by the leg's wall duration, and that duration still contains the time a +FAILED request spent before dying. At c1 our leg reads 2.37 tok/s against vLLM's +3.50, which is 0.677x, while median TPOT in the SAME result file reads 220.6 ms +against 223.6 ms, which is 1.014x in our favour. One of those two numbers is +describing dropped requests, not speed. Quoting 0.677x would have recorded a 32% +throughput deficit that the per-token evidence beside it contradicts. Filed as +[#931](https://github.com/mudler/vllm.cpp/issues/931), which also records that +NO harness here asserted `failed == 0` before summarising a ratio. + +THE FAILURES ARE REPRODUCIBLE AND ASYMMETRIC, which is what makes them a defect +rather than noise: 1 of 6 at c1 in all three reps, 0 of 24 at c4 in all three, +and 12/11/12 of 48 at c8, against 0 failures in all nine vLLM legs on the identical +workload from the identical client. Our server logged NOTHING for them: the +captured log is 27 lines, all startup, with no error, no request line and no +rejection, sampled live during a failing leg by a read-only sidecar. So the +failures are silent server-side, which points at the HTTP or connection layer +rather than an application-level rejection, and the cause is not yet named. + +WHAT c4 SAYS, being the one cell where both arms completed everything: we are at +0.963x output throughput and 1.008x median ITL. Per-output-token we are at +parity, and the throughput deficit at c4 is small. That is the only speed claim +this row supports today. + +RESOURCE AXES. Cold start to first `/health` is 53 s against vLLM's 780 s +(14.7x, medians of 3), ours reproducible to the second (53/53/53; vLLM +786/780/771). Host memory +after warmup is 42.5 GiB against 110.1 GiB (2.59x), and that one carries a +caveat rather than a win: vLLM was run with `--gpu-memory-utilization 0.85`, +which pre-reserves the KV pool on a unified-memory box, so the figure is what +the CONFIGURED engine holds, not what the model needs. + +CONTENTION ACTUALLY OBSERVED. The box was shared throughout with other +campaigns. This series queued behind an LTX-2.5 121-frame render and then an +`oracle_run.sh goldens` job, ~44 minutes of waiting on `$HOME/gpu.lock`, and +took the lock for the whole series so no leg was interleaved with a neighbour. +Load average at each leg boundary was recorded before the leg ran: 2.84, 0.91, +0.34, 1.03. A k3s-managed `local-ai-worker` pod was present and restarting +throughout; it was NOT stopped, because no developer preference or task +authority covers managing another service on a shared host, so it is recorded as +observed contention rather than removed. + +CLOCKS. Pinned under the lock with a trap that always resets: `nvidia-smi -lgc +2190` accepted (`gpuClkMin 2190, gpuClkMax 2190`), observed flat 2184 MHz during +the legs with `clocks_event_reasons.active = 0x0`, one boot id +`03717c9d-63c8-4652-a8fe-a63d012c5718`, persistence mode Disabled. 2184 MHz is +the same clock the existing clock-controlled 27B and 35B grids were taken at, so +those are comparable to this one. diff --git a/.agents/issue-index.md b/.agents/issue-index.md index e3fad2048..425494772 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -239,6 +239,8 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#907](https://github.com/mudler/vllm.cpp/issues/907) | `BACKEND-CUDA-COMP-CORE` | Five tests are red on dgx.casa (GB10, sm_121a) at `main`, proven PRE-EXISTING by a control build at `10b8bbdaa`: `test_capi` (SIGSEGV in an ABI v8 custom-logits-processor case, plausibly [#547](https://github.com/mudler/vllm.cpp/issues/547) or [#844](https://github.com/mudler/vllm.cpp/issues/844)), `test_cuda_ops` 439/440, `test_linear_method` 83/85, `test_ops_gdn` 4899/4900 ([#614](https://github.com/mudler/vllm.cpp/issues/614)), `test_qwen3_5_gdn_spec_routing` 119/123. Three of the five had no issue at all, which is why this exists | bug | | [#912](https://github.com/mudler/vllm.cpp/issues/912) | `ENG-EXPERT-STREAM` | Stream routed experts from NVMe so a model larger than device memory runs. Target `Qwen/Qwen3.8-2.4T-A95B`, REGISTERED against `Qwen3_5MoeForCausalLM` and blocked only on capacity: 370 GiB at UD-Q1_0 against 128 GB of unified memory on GB10. The only one of the three offload rows that helps on a unified-memory host, because `ENG-WEIGHT-OFFLOAD` and `ENG-HYBRID-PLACEMENT` both move bytes inside one physical pool. Cheaper than the spec assumed: on the GGUF path the mmap'd file already IS the bank and the per-expert slicer landed 2026-07-22 ([#824](https://github.com/mudler/vllm.cpp/issues/824)) | feature | | [#911](https://github.com/mudler/vllm.cpp/issues/911) | — | A `file:line` anchor into a file the row is ITSELF editing is stale by default, and spec BODIES are checked by nothing: `check-agent-record.py`'s `MATRIX_PATHS` (`:521`, `:529-530`) covers the five matrices, `feature-matrix.md` and `specs/model-family-inventory.md`, leaving 4772 line-carrying citations across 315 `.agents/specs/*.md` unexamined (positive control: 2314 line-less `.cpp` mentions match the same shape). `ltx25-prompt-adaln.md` shipped EIGHT stale repo-local anchors across two repair commits, moved by its own `020381676` and by `98f8e046d` (#658), then SEVEN more that were correct at `00613767d` and wrong at the merge of `origin/main`, because `0785cfc4d` (#882) added 70 lines to `ltx2_video.cpp` and 306 to `test_ltx2_video.cpp` ahead of every one. The obvious checker is a TAUTOLOGY — reading the span out of the file it validates reports 27 of 27 fresh on the same tree where reading the spans against their CLAIMS finds seven stale. Remedy is already in use and unwritten: `path:NN @ ` for a historical claim, claim-sourced uniqueness re-derivation for a live one, re-run after the merge. Narrower than [#632](https://github.com/mudler/vllm.cpp/issues/632) on surface and sharper on mechanism. Listed under `## Owed` in [`ltx25-prompt-adaln.md`](specs/ltx25-prompt-adaln.md) | bug | +| [#915](https://github.com/mudler/vllm.cpp/issues/915) | `MODEL-MM-qwen3-5-qwen3-5-for-conditional-generation` | Qwen3.8-27B (bf16): the token gate and every speed axis are OWED — the checkpoint loads and runs, and has never been adjudicated or measured. [`porting-inventory.md`](porting-inventory.md) deviation 17 already recorded that `Qwen/Qwen3.8-27B`@`1d4bf0f2` "fits GB10 and loads on current `main` with no code change" and that "its own token-exact gate is OWED and unrun", but no issue named either axis. Loading is not a token and a token is not a throughput number. [#821](https://github.com/mudler/vllm.cpp/issues/821) owns the NVFP4 and Q4_K_M arms of the same checkpoint and covers neither. Spec: [`qwen38-27b-bf16-gate.md`](specs/qwen38-27b-bf16-gate.md) | verification | +| [#931](https://github.com/mudler/vllm.cpp/issues/931) | `MODEL-MM-qwen3-5-qwen3-5-for-conditional-generation` | Our OpenAI server fails requests under concurrency where vLLM does not, and the failures silently corrupt every throughput ratio taken against it. Found benchmarking Qwen3.8-27B bf16 ([#915](https://github.com/mudler/vllm.cpp/issues/915)) on GB10: identical client, identical workload, one interleaved series under one GPU lock, ours `failed` 1/6 at c1 and 12/48 at c8 against vLLM's 0/6. `output_throughput` divides tokens by a wall duration that still contains the dead request, so our c1 leg reads 0.675x while median TPOT in the SAME file reads 1.017x in our favour. No harness asserted `failed == 0`, and our server logged nothing — the captured log is 27 startup lines, taken at readiness rather than after the leg | bug | | [#810](https://github.com/mudler/vllm.cpp/issues/810) | `MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm` | NemotronH is not reachable through `include/vllm.h`: `GPUModelRunner::initialize_kv_cache` rebuilds the RECURRENT half of the allocation from `config_.linear_*` instead of the `MambaSpec` the model published, so every non-Qwen3.5 hybrid is refused by Qwen3.5's name at `runner.cpp:525`, and per-layer membership comes from `config_.layer_types[l] == "linear_attention"` rather than `KVCacheGroupSpec::layer_names`. Spec [`nemotron-h-abi-e2e.md`](specs/nemotron-h-abi-e2e.md); the attention half at `runner.cpp:539-607` is already spec-driven and is the model to mirror. Note the SAFETY constraint recorded there: neutering the check alone reaches a forward that ignores `attn_kv`/`gdn_state`/`num_reqs`, which is strictly more dangerous than the refusal | bug | | [#873](https://github.com/mudler/vllm.cpp/issues/873) | `GATE-CI-CONCURRENCY` | `main` went RED on six release/registration gates after the #865 `ci.yml` rewrite: `check-release-binary-contract.py` and `check-test-registration.py` credit a checker to CI only through an UNCONDITIONAL job, and #865 gave `agent-record` an `if:`; the closed-PR skip is re-expressed through `needs:` and the byte-exact Windows PR proof schema restored (spec [`ci-concurrency.md`](specs/ci-concurrency.md)) | bug | | [#874](https://github.com/mudler/vllm.cpp/issues/874) | — | `windows-msvc-cpu`/`windows-msvc-vulkan` still start on a CLOSED pull request: `check-release-workflow.py::validate_pr_ci` compares their whole job mapping for equality, so neither an `if:` clause nor a `needs:` guard can be added. Listed under `## Owed` in [`ci-concurrency.md`](specs/ci-concurrency.md) | bug | diff --git a/.agents/model-matrix.md b/.agents/model-matrix.md index 9d711eda0..db1444311 100644 --- a/.agents/model-matrix.md +++ b/.agents/model-matrix.md @@ -91,7 +91,7 @@ Engaged architectures (the 53 non-`INVENTORIED` rows): |---|---|---|---|---| | ✅ | `Qwen3ForCausalLM` | Qwen3 dense (0.6B/1.7B/4B/32B) | near-tie-robust token-exact 16/16 on 0.6B+4B vs vLLM 0.25.0; NVFP4A16 (W4A16) dense quant also gated; c1 every-axis speed parity, c8 decode residual; async-serving device token-ids mirror ported (`ROW-SERVE-ASYNC-DENSE-MIRROR`, #31 fix into the shared dense `EmbedInto`) — `test_qwen3_dense_async_serving` RED→GREEN; sibling scope CLOSED (#323): `60e71a0e` fixed the eager path; `DenseDecodeGraphForward` ran first and replayed against stale HOST ids, so it now declines while the mirror is live and falls back to the proven eager path. Async gate 7/7 across Qwen3-0.6B/4B + Llama/Mistral/InternLM2 | `MODEL-TEXT-qwen3-qwen3-for-causal-lm` | | ✅ | `Qwen3MoeForCausalLM` | Qwen3-Coder-30B-A3B (MoE) | STRICT token-exact 6/6 vs vLLM 0.25.0; 11/16 speed-grid cells at/above graphed vLLM, c1/c2 residual | `MODEL-TEXT-qwen3-moe-qwen3-moe-for-causal-lm` | -| ✅ | `Qwen3_5ForConditionalGeneration` | Qwen3.6-27B (text path) | text-gen STRICT token-exact 235/235 vs vLLM 0.25.0; mm INPUT pipeline (M0/M1) landed + processor-parity gate PASS; **M3-W0 landed** (vision-inclusive checkpoint `Qwen/Qwen3.6-27B` 51.7 GiB bf16 with 333 `visual.*` FOUND+fits+downloaded; 27B vision config resolved — depth 27/out 5120/**EMPTY deepstack**; MRoPE `[11,11,10]`/rot 64/theta 1e7; the bf16 GDN-hybrid loader ALREADY handles it). **M3-b LANDED 2026-07-25: image→text STRICT token-exact 32/32 vs vLLM 0.25.0** — Qwen3.6-27B image understanding works end-to-end (forked GDN-hybrid VL forward gated on mm input ⇒ text byte-identical; 27B/35B/Coder inertness re-passed 235/315/138). **M3d LANDED 2026-07-25: video→text STRICT token-exact 32/32 vs vLLM 0.25.0** — video works end-to-end too (`Qwen3_5VLGenerateGreedyVideo` reuses the M3c processor/windowed-tower/video-MRoPE on the GDN-hybrid backbone). **Qwen video modalities COMPLETE: image+video both work e2e** (audio N/A for Qwen). **VISION-FORWARD SPEED (2026-07-28, `CLAIM-MM-SPEED-QWEN-IMAGE`, multimodal-speed.md §16): the mm-forward tower BEATS vLLM** — per-image tower forward 142.3 ms (flash `AttentionDenseFlash`, hd-72) vs vLLM 0.25.0 ~250 ms eager encode = 0.57×; attribution-first nsys REFUTED a bigger lever (the t=784 vision attention is serial-latency-bound, flash only 1.04× over warp), STRICT 32/32 image/video HELD + goldens md5 unchanged. Row stays `PARTIAL` — vision-forward speed BEATS vLLM; **umbrella speed pending** on batched c2+/serving | `MODEL-MM-qwen3-5-qwen3-5-for-conditional-generation` | +| ✅ | `Qwen3_5ForConditionalGeneration` | Qwen3.6-27B (text path) | text-gen STRICT token-exact 235/235 vs vLLM 0.25.0; mm INPUT pipeline (M0/M1) landed + processor-parity gate PASS; **M3-W0 landed** (vision-inclusive checkpoint `Qwen/Qwen3.6-27B` 51.7 GiB bf16 with 333 `visual.*` FOUND+fits+downloaded; 27B vision config resolved — depth 27/out 5120/**EMPTY deepstack**; MRoPE `[11,11,10]`/rot 64/theta 1e7; the bf16 GDN-hybrid loader ALREADY handles it). **M3-b LANDED 2026-07-25: image→text STRICT token-exact 32/32 vs vLLM 0.25.0** — Qwen3.6-27B image understanding works end-to-end (forked GDN-hybrid VL forward gated on mm input ⇒ text byte-identical; 27B/35B/Coder inertness re-passed 235/315/138). **M3d LANDED 2026-07-25: video→text STRICT token-exact 32/32 vs vLLM 0.25.0** — video works end-to-end too (`Qwen3_5VLGenerateGreedyVideo` reuses the M3c processor/windowed-tower/video-MRoPE on the GDN-hybrid backbone). **Qwen video modalities COMPLETE: image+video both work e2e** (audio N/A for Qwen). **VISION-FORWARD SPEED (2026-07-28, `CLAIM-MM-SPEED-QWEN-IMAGE`, multimodal-speed.md §16): the mm-forward tower BEATS vLLM** — per-image tower forward 142.3 ms (flash `AttentionDenseFlash`, hd-72) vs vLLM 0.25.0 ~250 ms eager encode = 0.57×; attribution-first nsys REFUTED a bigger lever (the t=784 vision attention is serial-latency-bound, flash only 1.04× over warp), STRICT 32/32 image/video HELD + goldens md5 unchanged. Row stays `PARTIAL` — vision-forward speed BEATS vLLM; **umbrella speed pending** on batched c2+/serving. **SECOND CHECKPOINT TOKEN-GATED 2026-08-15 (`Qwen/Qwen3.8-27B`@`1d4bf0f2`, bf16, [#915](https://github.com/mudler/vllm.cpp/issues/915), [spec](specs/qwen38-27b-bf16-gate.md)): 4/7 prompts STRICT 16/16 vs the pinned oracle `555967922`, and all THREE first-divergence positions are EXACT fp32 TIES** — oracle-minus-ours and top-2 gap both **0.000 mnats**, our token at rank **3 / 2 / 2** in the oracle top-20, so `ALL_TIES_OR_IN_BAND` against `kNearTieMnats = 500`. Every one is the [#910](https://github.com/mudler/vllm.cpp/issues/910) tie-break signature and nothing else: vLLM's pick carries the LOWER token id (1814/11/16309) and ours the HIGHER (22960/13/27180) at a bit-identical logprob. Only the first divergence per prompt is adjudicable, so this is three numbers; a raw position count over the grid is NOT a quality score and is not recorded as one. Adjudicated twice on the pinned oracle's fp32 logprobs — a greedy re-decode and an independent TEACHER-FORCED probe that asserts the echoed prefix — because the earlier `transformers` bf16 CPU probe could not resolve below one bf16 ULP (every runner-up gap it printed was a multiple of 0.125) and so could not have reported anything but a tie. **SPEED on the same checkpoint, vs vLLM's PRODUCTION graphed config at the pin, clocks 2184 MHz: 1 of 3 concurrency cells established.** c4 is the only cell where both arms completed every request — **0.963x** output throughput, **1.008x** median ITL. c1 and c8 throughput are **WITHHELD, not quoted**: our server failed 1/6 in all three reps and 12/11/12 of 48 where vLLM failed none in nine legs ([#931](https://github.com/mudler/vllm.cpp/issues/931)), and `output_throughput` divides tokens by a duration still containing the dead request, so c1 reads 0.677x while median TPOT in the SAME file reads 1.014x in our favour. Cold start **53 s vs 780 s = 14.7x**; host memory after warmup **42.5 vs 110.1 GiB = 2.59x**, caveated because vLLM's is set by `--gpu-memory-utilization 0.85` pre-reserving KV | `MODEL-MM-qwen3-5-qwen3-5-for-conditional-generation` | | ✅ | `Qwen3_5MoeForConditionalGeneration` | Qwen3.6-35B-A3B (NVFP4 text path + PUBLISHED BF16 text path; vision implemented, NOT gated) | text-gen STRICT token-exact 315/315 vs vLLM 0.25.0 on the NVFP4 requant; **the PUBLISHED bf16 repo now generates too and is oracle-gated (2026-08-15, #740 + #864): greedy 7 prompts x 3 repeats x 16 tokens @`995ad96e` vs the pinned oracle, 6/7 prompts STRICT 16/16, the seventh one exact logit tie (`top2_gap_mnats = 0.0`) our on-device argmax breaks toward the higher id where `torch.argmax` takes the lower (#910) — PASS under the ratified near-tie doctrine; only the FIRST divergence per prompt is adjudicable, so the raw 108/112 position count is NOT a quality score. SACRED inertness 3/3 with real counts, goldens byte-identical (27B 235/235, 35B 315/315, Coder 138/138, 688 assertions). NO throughput, latency or memory number exists for this checkpoint**; mm INPUT pipeline (M0/M1) landed + processor-parity gate PASS; vision tower LOADED and the forked VL forward LANDED (#891, M2/M3), but the binding image/video token-exact gates vs the pinned oracle are OWED, so the row stays `PARTIAL` | `MODEL-MM-qwen3-5-qwen3-5-moe-for-conditional-generation` | | 🚧 | `Qwen3_5ForCausalLM` | Qwen3.5 text-only dense arm (no checkpoint fits this box) | **REGISTERED, NOT RUN-GATED (2026-08-12, issue #490).** Ahead-of-pin forward port of upstream PR vllm#50210 (`ad5d29db7`): additive registration against the EXISTING dense factory + one backbone weight-namespace decision per checkpoint (`model.` vs `model.language_model.`, mixed index REFUSED). Dispatch, flat-config and namespace tests green; 27B/35B/Coder inert (VL prefix is still the seam default, goldens md5 unchanged). **NO token-exact oracle run and NO speed number — the run gate is OWED**, and no text-only Qwen3.5 checkpoint fits GB10. Unlike the MoE arm, this loader DOES route BF16/FP8/NVFP4 per projection by tensor presence, so a flat bf16 checkpoint is expected to load | `MODEL-TEXT-qwen3-5-qwen3-5-for-causal-lm` | | 🚧 | `Qwen3_5MoeForCausalLM` | `Qwen/Qwen3.8-2.4T-A95B` (2.4T MoE — DOES NOT FIT this hardware) | **REGISTERED, NOT RUN-GATED (2026-08-12, issue #490).** Same ahead-of-pin port against the EXISTING MoE factory: 3.8 is the token-exact Qwen3.6-35B-A3B GDN-hybrid MoE backbone at larger scale, every knob config-driven, with the BACKBONE weight names identical modulo the backbone prefix. **CORRECTED 2026-08-12 — REGISTRATION IS NOT LOADABILITY.** This loader reads ONLY per-expert NVFP4 routed experts; the published `Qwen/Qwen3.8-2.4T-A95B` and `Qwen/Qwen3.6-35B-A3B` indices ship 3-D STACKED experts and ZERO `weight_scale`/`input_scale` (read live 2026-08-12), and our gated 35B row reads the REQUANTIZED `nvidia/Qwen3.6-35B-A3B-NVFP4` — so at that date **the bf16/stacked MoE routed-expert arm was NOT implemented and such a checkpoint was REFUSED by a message naming it**. **SUPERSEDED 2026-08-15 — THE LOADER HALF IS CLOSED, WITH A TOKEN.** #740 landed the 3-D stacked bf16 routed-expert reader and #864 the bf16 GDN tower, attention tower, shared expert and `lm_head`, all selected by tensor presence; both published indices now satisfy the load plan completely, and the binding token-exact greedy gate RAN on `Qwen/Qwen3.6-35B-A3B` bf16 @`995ad96e` vs the pinned oracle at 6/7 prompts STRICT 16/16 (the seventh an exact logit tie, #910), with SACRED 27B/35B/Coder inertness 3/3 and goldens byte-identical. **THE RUN GATE FOR THIS CLASS IS STILL OWED, and the reason is now purely hardware:** that gate ran through `Qwen3_5MoeForConditionalGeneration`, not through `Qwen3_5MoeForCausalLM`, whose only published checkpoint is 2.4T bf16 ≈ 4.8 TB / FP8 ≈ 2.4 TB against 128 GB unified. NO token and NO speed number exist for this class; its load plan resolves against the published index, which is name/shape/dtype resolution and not a token. Also owed: MTP and GGUF arms for 3.8 | `MODEL-TEXT-qwen3-5-qwen3-5-moe-for-causal-lm` | diff --git a/.agents/oracles/transformers.md b/.agents/oracles/transformers.md index b655db657..c280ae84a 100644 --- a/.agents/oracles/transformers.md +++ b/.agents/oracles/transformers.md @@ -20,6 +20,18 @@ placeholder expansion directly from it ([`../specs/audio-track.md`](../specs/audio-track.md) §A0), and the C++ STFT is gated against those goldens. +**It cannot adjudicate a near-tie, and one attempt is recorded here so the next +agent does not repeat it.** A CPU `transformers` probe of the three Qwen3.8-27B +divergences ([`../specs/qwen38-27b-bf16-gate.md`](../specs/qwen38-27b-bf16-gate.md), +[#915](https://github.com/mudler/vllm.cpp/issues/915)) reported all three as +exact ties — and every runner-up gap it printed was an exact multiple of +**0.125**, one bf16 ULP in that exponent range. An instrument quantized to one +ULP cannot resolve a gap below one ULP, so "tied in bf16" does not imply "tied +in the model" and agreement with it is not confirmation. vLLM computes logprobs +from **fp32** and separates pairs bf16 collapses, so a logit-margin verdict +comes from the pinned vLLM oracle; this one stands only as a secondary +cross-check, with that limitation part of the result. + ```oracle-pin id = transformers role = secondary diff --git a/.agents/specs/qwen38-27b-bf16-gate.md b/.agents/specs/qwen38-27b-bf16-gate.md new file mode 100644 index 000000000..fcb3d6f8d --- /dev/null +++ b/.agents/specs/qwen38-27b-bf16-gate.md @@ -0,0 +1,208 @@ +# Qwen3.8-27B (bf16): the token gate and the speed axes + +**Row:** `MODEL-MM-qwen3-5-qwen3-5-for-conditional-generation` +(`.agents/model-matrix.md`) — `Qwen/Qwen3.8-27B` declares +`Qwen3_5ForConditionalGeneration`, the architecture that row owns. +**Issue:** [#915](https://github.com/mudler/vllm.cpp/issues/915) +**Related:** [#821](https://github.com/mudler/vllm.cpp/issues/821) owns the +NVFP4 / Q4_K_M arms of the same checkpoint; [#910](https://github.com/mudler/vllm.cpp/issues/910) +owns the tie-break divergence this gate ran into three times. +**Lifecycle:** `PARTIAL` +**Owner:** unassigned + +## Scope + +`Qwen/Qwen3.8-27B` @ `1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0`, bf16, text +path. Two axes, in order: a greedy token gate against the pinned oracle, then — +only if that gate is clean — throughput, latency and memory against vLLM's +production configuration. + +Out of scope: the NVFP4 and Q4_K_M arms (#821), the vision path, any fix for +#910, and advancing the parity pin. + +## Why this row needs no port + +[`porting-inventory.md`](../porting-inventory.md) deviation 17 records the +finding this spec measures against: `Qwen/Qwen3.8-27B` "is the already-gated +Qwen3.6-27B shape retrained — `config.json` differs in exactly one key +(`transformers_version`) and the safetensors tensor-name set is identical (1199 +names, zero difference either direction)". So no loader, forward or registry +change is owed, and none was made. What was owed is evidence, and the same +entry says so: "its own token-exact gate is OWED and unrun". + +## Design of the adjudication + +Greedy, 7 prompts x 16 tokens, both arms on the same prompts and token counts. +The oracle capture is deterministic across 3 repeats (`deterministic: true`, +`multi_member_cells: 0`), so a strict per-prompt comparison is well-posed. + +**Only the first divergence per prompt is adjudicable.** After it the two arms +are conditioned on different prefixes, so any later position compares two +different conditionings rather than one disagreement. Three prompts diverge, so +there are three numbers — not thirty-four. A position count over the whole grid +is not a quality score and is not recorded as one. + +At each first divergence the two arms agree on every preceding token, so both +are conditioned on a BYTE-IDENTICAL prefix. Feeding that prefix to the pinned +oracle and reading its **fp32** next-token distribution measures exactly one +thing: how far apart the oracle itself holds its own choice and ours. + +## Risks + +- **A bf16 instrument cannot resolve a bf16 tie.** A `transformers` CPU probe + reported all three as exact ties, and every runner-up gap it printed was an + exact multiple of 0.125 — one bf16 ULP in that exponent range. It could not + have reported anything else, so it is a secondary cross-check with that + limitation stated, never the answer. vLLM computes logprobs from fp32 and can + separate pairs bf16 collapses. +- **A re-decode probe assumes the prefix it needs.** Reading the distribution + at step `d` of a fresh greedy decode is only valid if that decode reproduced + the captured prefix. Mitigated by a second, teacher-forced probe that feeds + the prefix as token IDs and asserts both the echoed prefix and that the + oracle's top-1 equals the captured token. +- **A stale binary measures a tree that does not exist.** Mitigated by + rebuilding at `origin/main` and re-running both axes on that binary. +- **A degraded build voids every number.** Mitigated by asserting the + configure-log fast-path lines (CUTLASS, FA2, Triton AOT `sm_121a`) and + aborting the build otherwise. + +## Tests + +This row changes no `src/`, `include/` or `tests/` file, so it ports no test. +`git diff origin/main..HEAD` over those three paths is empty and that emptiness +is the claim. What it adds is evidence, listed under Evidence required. + +## Gates + +- Token gate: every first-divergence position within `kNearTieMnats = 500` of + the oracle's teacher-forced argmax, on the pinned oracle, or the row fails. +- Speed: recorded only if the token gate is clean. vLLM's **production** + configuration is the denominator — never `--enforce-eager`. Same client + (`vllm bench serve`) drives both arms. +- Both under `$HOME/gpu.lock`, clocks pinned, contention recorded per leg. + +## Evidence required + +- Oracle identity asserted (`vllm.__version__`, `flashinfer`) with ABORT on + mismatch, plus a `Python.h` precondition so a missing header aborts loudly + rather than dying inside Triton's JIT. +- Per divergence: the oracle top-2 gap in millinats, our token's rank in the + oracle top-20, and a verdict against the band. +- The build recipe, revisions, checkpoint size, binary md5, boot id, SM clock + and the contention actually observed. + +## Stop conditions + +- If any divergence is out of band, STOP: report it as a real divergence, run + no benchmark on that checkpoint, and let no record imply it is + baseline-ready. Do not attempt a fix. +- If the box is not quiet at a leg boundary, wait or drop the leg. A number + taken under load is worse than no number. + +## Now + +`PARTIAL`. The token axis is closed and passing. The speed axis is closed only +at **c4** (0.963x throughput, 1.008x ITL); c1 and c8 throughput are withheld until +[#931](https://github.com/mudler/vllm.cpp/issues/931) closes, because our server +drops requests there and vLLM does not. Nothing else advances this row. + +## Outcome + +**Measured, token axis.** `Qwen/Qwen3.8-27B` @`1d4bf0f2`, bf16, 55,586,114,863 +bytes over 18 shards, against the pinned oracle `0.23.1rc1.dev1511+g555967922` +with FlashInfer `0.6.15.post1`, greedy, 7 prompts x 16 tokens. The oracle +capture is deterministic over 3 repeats (`multi_member_cells: 0`). **4/7 prompts +STRICT 16/16**, and the three first-divergence positions adjudicate as **exact +fp32 ties**: + +| Prompt | Pos | Oracle | Ours | Top-2 gap | Oracle - ours | Our rank | +|---|---:|---|---|---:|---:|---:| +| `Once upon a time,` | 2 | 1814 `" world"` | 22960 `" magical"` | 0.000 mnats | 0.000 mnats | 3 | +| `The largest planet in our solar system is` | 1 | 11 `","` | 13 `"."` | 0.000 mnats | 0.000 mnats | 2 | +| `import numpy as np` | 8 | 16309 `" matplotlib"` | 27180 `" scipy"` | 0.000 mnats | 0.000 mnats | 2 | + +The teacher-forced logprobs are `-1.5257947444915771`, `-0.74363690614700317` and +`-1.4836434125900269`; the greedy re-decode reads `-1.524564266204834`, +`-0.7444034814834595` and `-1.4876692295074463`. The two conditioning paths +differ in the last few thousandths of a nat, which is the point of reading it +twice — and the GAP is exactly `0.0` under both, on both members of each tied +pair. `INTEGRITY_OK=True`: for all three, the engine echoed the supplied prefix +back unchanged and its teacher-forced top-1 equals the captured greedy token, so +prefill and incremental decode do not disagree here. + +`ALL_TIES_OR_IN_BAND` against `kNearTieMnats = 500`. Every one is +[#910](https://github.com/mudler/vllm.cpp/issues/910) and nothing else: the +oracle's pick carries the LOWER token id and ours the HIGHER at a bit-identical +logprob. `Once upon a time,` is a THREE-way tie — `" world"`, `" land"` and +`" magical"` all sit at `-1.524564266204834`. + +**The shared prefix is proven, not assumed.** The adjudication is only valid if +both arms are conditioned on the same bytes. Two things establish it. Our +tokenizer reproduces the oracle's prompt ids **7/7 exactly**, checked directly +through `examples/tokenize` against `capture.json`'s `prompt_ids` — which the +token gate itself could not show, because our server's `echo` does not return +prompt tokens. And the generated ids agree up to the divergence by construction +of "first divergence". + +**Rejected as the answer, kept as a cross-check.** A `transformers` CPU probe +called all three exact ties, and its own output refutes it as evidence: every +runner-up gap it printed was a multiple of **0.125**, one bf16 ULP in that +exponent range. An instrument that cannot resolve below one ULP cannot report +anything but a tie, so agreement with it is not confirmation. It is recorded +under the existing [transformers](../oracles/transformers.md) pin with that +limitation attached. + +**Read twice, on the oracle's fp32 logprobs.** A greedy re-decode reads the +distribution at step `d` of the oracle's own deterministic decode. An +independent teacher-forced probe feeds the prefix as token IDs, then asserts +both that the engine echoed that prefix back and that the oracle's top-1 equals +the captured token — so a decode that had silently wandered before `d` cannot +pass as a valid conditioning. Both abort on an oracle-identity mismatch +(`vllm.__version__`, `flashinfer`) and on a missing `Python.h`, which otherwise +surfaces as an opaque failure inside Triton's JIT. + +**The result does not depend on a stale tree.** The first run used a binary +built at `4a183b731`, 13 commits behind `main`, and `qwen3_5.cpp`, +`qwen3_5_dense.cpp` and `qwen3_5_weights.cpp` had all changed in between — so it +bound nothing. Rebuilt at `11a42dc4c` with the fast path asserted (CUTLASS, FA2, +Triton AOT `sm_121a`, `sm_121a` baked into the library) and re-run: **the same +4/7, the same three positions, the same tokens**. The intervening changes are +empirically inert on this path rather than assumed to be. + +**Measured, speed axis, and one of three cells is all it supports.** Against +vLLM's production graphed configuration (no `--enforce-eager`, +`--language-model-only` on both arms, same `vllm bench serve` client, one +`flock`, clocks pinned to a flat 2184 MHz, one boot id), random 1024-in / +128-out, 6 prompts per concurrency unit, paired reps interleaved ours/vLLM: + +| Axis | c1 | c4 | c8 | +|---|---:|---:|---:| +| ours / vLLM completed | 5,5,5 / 6,6,6 of 6 | 24x3 / 24x3 of 24 | 36,37,36 / 48x3 of 48 | +| Output throughput | WITHHELD | **0.963x** | WITHHELD | +| Median ITL | 1.013x | 1.008x | 1.021x | +| Median TPOT | 1.014x | 0.980x | 0.925x | +| Median TTFT | 0.733x | 0.881x | 1.268x | + +Plus cold start **53 s vs 780 s = 14.7x** and host memory after warmup +**42.5 vs 110.1 GiB = 2.59x**, the latter caveated because vLLM's figure is set +by `--gpu-memory-utilization 0.85` pre-reserving KV rather than by the model. + +**Two cells are WITHHELD, and that is the finding.** Our server failed 1 of 6 +requests at c1 in all three reps and 12/11/12 of 48 at c8, where vLLM failed none +in nine legs on the identical workload from the identical client +([#931](https://github.com/mudler/vllm.cpp/issues/931)). `output_throughput` +divides tokens by a wall duration that still contains the dead request, so the +c1 cell reads 0.677x while median TPOT in the SAME file reads 1.014x in our +favour. Recording 0.677x would have published a 32% deficit contradicted by the +evidence beside it. No harness here asserted `failed == 0` before summarising a +ratio, which is the gap that let this become quotable in the first place. + +The failures are silent server-side: a read-only sidecar sampled our server's +log live during a failing leg and it is 27 lines, all startup, no error, no +request line, no rejection. So the cause is not named yet, and a controlled +reproduction recording HTTP status and exception class is the next step. + +**What was NOT established.** Nothing about the vision path on this checkpoint, +nothing about its NVFP4 or Q4_K_M arms (#821), no claim that #910 is fixed (this +is the second checkpoint to be costed by it), and no throughput number at c1 or +c8 until #931 closes. Concurrencies above 8 were not run. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index ee0435ae1..862cd38f8 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -52,6 +52,7 @@ The first series free of both, at the pin, graphed, and at a pinned clock is in | Qwen3.6-35B-A3B | NVFP4 `modelopt_mixed` | 0.25.0 ROLLBACK, SUPERSEDED | 2/18 | 3-rep grid 2026-08-05 @`1ea26427`: 0.93-1.03x, VOID as ratios (#520, #414). At the pin, clocks pinned: **0.995x c1 / 0.946x c4 TPOT**. ★ probe found a prod async batch-1 greedy DEGENERATION bug the mirror fixes | | DeepSeek-V2-Lite | bf16 MLA | 0.25.0 ROLLBACK, SUPERSEDED | 4/25 | Attributed miss, row stays `ACTIVE` | | Qwen3.5-4B | bf16 direct-load | 0.26.0.dev0 | **1.0283x tput, `PENDING`** | OPEN: TTFT/TPOT/E2E 1.085/1.017/1.029x, VRAM +118.7 MiB ([data](bench-evidence/qwen35-4b-sm120-main-20260807.md)) | +| Qwen3.8-27B | bf16 (@`1d4bf0f2`) | 0.26.0.dev0 at the pin, graphed, clocks 2184 MHz | **1 of 3 concurrency cells** | Token gate PASSES. Only c4 is like-for-like: tput **0.963x**, ITL **1.008x**. c1/c8 tput NOT ESTABLISHED: we failed 1/6 and ~11/48 requests, vLLM none ([#931](https://github.com/mudler/vllm.cpp/issues/931)) | ### GDN prefill kernels by GPU @@ -186,6 +187,23 @@ on the default. c16 re-checked on the default: 2312.9/2303.9/2294.4 (median **2303.9**), c32 2942.7 (no regression). Root cause + file:line in the benchmark record. The same P0 hit classic dense `Qwen3ForCausalLM` (quant-independent), fixed by `ROW-SERVE-ASYNC-DENSE-MIRROR` (see the MXFP4 Qwen3-8B row). The intake-drain lever likewise measured NEUTRAL (2026-08-06, `VT_INTAKE_DRAIN` A/B 3+3 reps): admitting during the forward wait collapses intake -91% but shifts it into queued, arrival-to-scheduled invariant, so the recorded INTAKE term is an attribution boundary over a GPU-bound prefill wait, not reducible; lever reverted, byte-exact `VT_LOOP_TRACE` probe kept. +### Qwen3.8-27B (bf16) by concurrency + +| Axis | c1 | c4 | c8 | +|---|---:|---:|---:| +| Requests completed, ours / vLLM | 5,5,5 / 6,6,6 of 6 | 24,24,24 / 24,24,24 of 24 | 36,37,36 / 48,48,48 of 48 | +| Output token throughput | NOT ESTABLISHED, we dropped requests ([#931](https://github.com/mudler/vllm.cpp/issues/931)) | **0.963x** | NOT ESTABLISHED, we dropped requests ([#931](https://github.com/mudler/vllm.cpp/issues/931)) | +| Total token throughput | NOT ESTABLISHED | **0.918x** | NOT ESTABLISHED | +| Median ITL, over completed only | 1.013x | **1.008x** | 1.021x | +| Median TPOT, over completed only | 1.014x | 0.980x | 0.925x | +| Median TTFT, over completed only | 0.733x | 0.881x | 1.268x | +| Median E2EL, over completed only | 1.003x | 0.974x | 0.983x | +| ours / vLLM output tok/s | 2.37 / 3.50 | 15.01 / 15.58 | 15.96 / 27.85 | +| ours / vLLM median TPOT ms | 220.6 / 223.6 | 239.0 / 234.3 | 261.1 / 241.4 | +| Cold start to first `/health` | **53 s vs 780 s = 14.7x**, medians of 3 (ours 53/53/53, vLLM 786/780/771) | same binary | same binary | +| Host memory after warmup | **42.5 vs 110.1 GiB = 2.59x**, but vLLM's is set by `--gpu-memory-utilization 0.85` pre-reserving KV, so it is what the configured engine holds, not what the model needs | | | +| Why only c4 counts | `output_throughput` divides tokens by a wall duration that still contains the dead request, so a cell where one arm dropped requests is withheld, not quoted | 3 paired reps, clocks 2184 MHz | token gate: 4/7 strict, 3 exact fp32 ties ([#915](https://github.com/mudler/vllm.cpp/issues/915)) | + ### DeepSeek-V2-Lite (MLA) Medians of 3 reps, 1,024 in / 128 out. The vLLM arm runs `--moe-backend triton`, diff --git a/docs/FEATURES.md b/docs/FEATURES.md index d28fd55dc..eb7ce1153 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -109,7 +109,7 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks. | Architecture | Tested checkpoint(s) | Correctness gate | Speed vs reference | |---|---|---|---| -| `Qwen3_5ForConditionalGeneration` | Qwen3.6-27B NVFP4 (`unsloth` @`890bdef7`, `nvidia` @`0893e160`); Qwen3.5-4B BF16 | 27B strict 235/235 text + 32/32 image/video; 4B cached 3/3 | `unsloth` 27B at/above vLLM, `nvidia` ModelOpt 0.85x; 4B throughput 1.021x. Loads BF16, FP8 and NVFP4 (CT + ModelOpt naming); a `modelopt_mixed` FP8 tower stays NATIVE (#164), GDN `in_proj_qkvz` merged. CUDA/CPU only | +| `Qwen3_5ForConditionalGeneration` | Qwen3.6-27B NVFP4 (`unsloth` @`890bdef7`, `nvidia` @`0893e160`); Qwen3.5-4B BF16; **Qwen3.8-27B BF16** @`1d4bf0f2` | 27B strict 235/235 text + 32/32 image/video; 4B cached 3/3; Qwen3.8-27B 4/7 strict, 3 exact fp32 ties in band (#915) | `unsloth` 27B at/above vLLM, ModelOpt 0.85x; 4B 1.021x; 3.8-27B c4 **0.963x**, c1/c8 withheld (#931). Loads BF16/FP8/NVFP4 (CT + ModelOpt); `modelopt_mixed` FP8 tower NATIVE (#164), GDN `in_proj_qkvz` merged. CUDA/CPU | | `Qwen3_5MoeForConditionalGeneration` | Qwen3.6-35B-A3B (NVFP4 text; published BF16 text + vision tower) | NVFP4 strict 315/315 vs vLLM 0.25.0; published BF16 6/7 prompts strict 16/16 vs the pin, 7th an exact tie (#910). Image/video IMPLEMENTED, NOT GATED (#891): the tower loads and runs, mm gate OWED | gate model: 0.93x to 1.03x grid; NO BF16 or mm speed claim | | `Qwen3_5ForCausalLM`, `Qwen3_5MoeForCausalLM` | none: no text-only Qwen3.5 checkpoint fits this hardware | **NO RUN GATE, OWED.** Gated on `test_qwen3_8_text_only.cpp`; NO token claim. Loader reads stacked BF16 experts (#740) plus BF16 towers, shared expert and `lm_head` (#864), so both published indices satisfy the load plan | not measured | | `Qwen3ForCausalLM` | Qwen3 dense 0.6B/1.7B/4B/32B, NVFP4A16 | near-tie strict 16/16 vs vLLM 0.25.0 | c1 every-axis parity, c8 decode residual | diff --git a/docs/STATUS.md b/docs/STATUS.md index 04e50ee10..9847c4462 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -787,10 +787,47 @@ checkpoint that fits GB10 appears; the MoE one when a fitting `Qwen3_5MoeForCausalLM` checkpoint does. `Qwen/Qwen3.8-2.4T-A95B` is ~4.8 TB bf16 (~2.4 TB FP8) against 128 GB of unified -memory and no smaller Qwen3.8 sibling exists, so it remains unrunnable here. Its -load plan resolves completely against the published index — name, shape and -dtype resolution, **not a token**. Still owed: the MTP and GGUF arms for 3.8. -This does not advance the parity pin. +memory, so it remains unrunnable here. Its load plan resolves completely against +the published index — name, shape and dtype resolution, **not a token**. Still +owed: the MTP and GGUF arms for 3.8. This does not advance the parity pin. + +**A smaller Qwen3.8 sibling DOES exist, and this page said otherwise until +2026-08-15.** `Qwen/Qwen3.8-27B` @`1d4bf0f2` (55.6 GB bf16, 18 shards) fits GB10 +and declares `Qwen3_5ForConditionalGeneration` — the already-gated Qwen3.6-27B +shape retrained, `config.json` differing in exactly one key. It closes neither +text-only run gate above, because it is not a `Qwen3_5[Moe]ForCausalLM`. + +**It is now token-gated ([#915](https://github.com/mudler/vllm.cpp/issues/915), +[spec](../.agents/specs/qwen38-27b-bf16-gate.md)).** Greedy, 7 prompts x 16 +tokens vs the pinned oracle on GB10: **4/7 prompts STRICT 16/16**, and all three +first-divergence positions are **EXACT fp32 ties** — top-2 gap and +oracle-minus-ours both **0.000 mnats**, our token at rank 3 / 2 / 2 in the +oracle top-20, so `ALL_TIES_OR_IN_BAND` against `kNearTieMnats = 500`. All three +are the [#910](https://github.com/mudler/vllm.cpp/issues/910) tie-break and +nothing else: vLLM takes the lower token id, we take the higher. + +Only the first divergence per prompt is adjudicable, so that is three numbers, +and a raw position count over the grid is not a quality score. The tie verdict +rests on the oracle's **fp32** logprobs, read twice — a greedy re-decode and a +teacher-forced probe that asserts the prefix it conditions on. A `transformers` +bf16 CPU probe agreed, but is recorded as secondary only: every runner-up gap it +printed was a multiple of 0.125, one bf16 ULP, so it could not have resolved a +real gap below that and could not have reported anything but a tie. + +**Speed: one of three concurrency cells is established, and the reason the other +two are not is a defect of ours.** Against vLLM's production graphed config on +GB10, clocks pinned at 2184 MHz, c4 is the only cell where both arms completed +every request: **0.963x** output throughput and **1.008x** median ITL. At c1 and +c8 our server failed 1 of 6 in all three reps and 12/11/12 of 48 where vLLM failed none, so +those throughput cells are **withheld, not quoted** +([#931](https://github.com/mudler/vllm.cpp/issues/931)) — the metric divides +tokens by a duration that still contains the dead request, which is why c1 reads +0.677x while median TPOT in the same file reads 1.014x in our favour. + +Resource axes on the same series: cold start to first `/health` **53 s vs +780 s = 14.7x**, and host memory after warmup **42.5 vs 110.1 GiB = 2.59x** — +the latter with the caveat that vLLM's figure is set by +`--gpu-memory-utilization 0.85` pre-reserving KV on a unified-memory box. Larger DeepSeek / GLM / MiniMax / Gemma-4 variants are recorded as **hardware-blocked** (they do not fit 119 GiB of unified memory on this box) or