diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 25929b944..31dc52c58 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -25101,3 +25101,132 @@ narrower f64 vector explains without anything further. SGLang-Omni is still `gateable = no`. Every reference axis in `docs/BENCHMARKS.md` stays `PENDING`. Everything above is an internal two-arm number on one named box. + +## ENG-EXPERT-STREAM-DEVICE W0e — the CPU arm is measured and the CUDA arm loads without generating (2026-08-18/19, `dgx:gpu0`, source `95883dcae`, #1124) + +One `rc hold` (`edb4b3d0-5d6e-422f-ade6-bff5339e3396`, 22:13:12Z to 00:58Z, +released by interrupting its client). GB10, sm_121a, driver 580.173.02, CUDA +13.0.88, 20 cores, 122,502 MiB RAM, 30,625 MiB swap. Checkpoint +`Qwen3.8-2.4T-A95B-UD-Q1_0` (369.97 GiB, 10 shards) from LOCAL NVMe at +`/home/mudler/ckpt/qwen3.8-q1_0`. Prompt fixed as token ids +`760,6511,314,9338,369`, greedy, 32 tokens, `--max-num-seqs 1`, +`VT_GGUF_PREFAULT=0 VT_MOE_EXPERT_STREAM=1 VT_MOE_EXPERT_STREAM_STATS_EVERY=1`. +Page cache dropped before every arm. Same binary both arms, built with CUTLASS +4.5.0 and Triton AOT: the configure log reports `fp4-mma`, `cutlass-nvfp4`, +`cutlass-fp8`, `marlin-nvfp4` and `fa2` all `ENABLED for [121a]`, plus +`FlashAttention-2 prefill/decode: ENABLED` and `Triton AOT W2: embedded trees +[sm_80;sm_86;sm_89;sm_90a;sm_100a;sm_121a]`. + +Harness: `benchmarks/expert_stream_device_w0e.cpp`, a thin C-ABI client. It +exists because one generation has to yield three things no shipped command gives +together: the generated token ids (`vllm_complete_tokens`), a per-step arrival +timestamp (a logits processor used as a pure observer, which edits nothing so +the argmax is unchanged), and the expert-stream counters at both ends of the run. + +### CPU arm, four runs, two slot counts + +| slots | arena | load | TTFT | steady s/token, steps 4-32 | generate | peak RSS | min avail | peak swap | +|---|---|---|---|---|---|---|---|---| +| 4000 | 9.28 GiB | 271.1 s | 85.90 s | median 11.22, min 9.62, max 12.51 | 502.3 s | 86.5 GiB | 16,840 MiB | not sampled | +| 4000 | 9.28 GiB | 255.7 s | 79.09 s | median **11.05**, min 9.43, max 13.25 | 460.7 s | 86.5 GiB | 16,347 MiB | 6,883 MiB | +| 8000 | 18.55 GiB | 261.6 s | 94.25 s | median 45.40, min 20.73, max 82.68 | 1643.2 s | 86.6 GiB | 6,985 MiB | not sampled | +| 8000 | 18.55 GiB | 266.5 s | 132.74 s | median 39.98, min 16.06, max 94.30 | 1581.6 s | 86.6 GiB | 6,941 MiB | 30,625 MiB (all) | + +The 8000-slot rows are a MEMORY result, not a cache result. A bigger cache came +out slower on every pairing of the medians: 3.62x same-rep on the pair carrying +the publishable figure (39.98 against 11.05), 4.05x pairing rep 1 with rep 1 +(45.40 against 11.22), and 3.56x to 4.11x over all four pairings. It also came +out much less steady, stated as the max/min ratio of the steady window: 3.99x in +rep 1 (20.73 to 82.68 s) and 5.87x in rep 2 (16.06 to 94.30 s), against 1.30x +and 1.40x at 4000 slots. + +The cause is NOT that the arena does not fit. 18.55 GiB of arena beside 62 GiB +of dense weights is 80.55 GiB on a 119.63 GiB box, and the columns agree that +nothing overflowed: peak RSS moves 86.5 -> 86.6 GiB for a 9.27 GiB arena delta. +The columns that DO move are `min avail`, 16,347 -> 6,941 MiB, a 9,406 MiB fall +that is about the arena delta, and peak swap, 6,883 -> 30,625 MiB, all of it. +The best-supported reading of that pattern is page-cache displacement: the +borrowed 370 GiB expert mapping is served out of whatever memory is free, the +arena takes that memory, and the reclaim pressure it creates pushes anonymous +pages to swap. That is a reading of these columns and not a proven mechanism. +This run sampled no page-cache size and no major-fault counter, so it cannot +separate displacement from plain reclaim pressure, and a run that wants to +settle it has to sample both. + +The operational conclusion does not depend on which of the two it is. Both reps +of each slot count reproduce each other, so this is the box and not a fluke, +more slots is not a free knob, and the publishable figure is the 4000-slot one. + +Per-token deltas, 4000-slot rep 2, seconds: 79.09, 51.14, 12.52, then 11.69, +11.51, 11.35, 11.59, 13.25, 11.58, 9.48, 12.16, 11.22, 11.42, 11.51, 11.05, +10.70, 10.69, 11.93, 11.12, 11.26, 10.99, 9.85, 10.39, 9.58, 11.35, 10.17, +11.03, 10.31, 9.43, 10.25, 10.38, 10.64. Step 1 is prefill; steps 2 and 3 are +still filling a cold cache. + +G0-LIVE, gated on the DIFFERENCE and never the total. At 4000 slots the +after-prefill snapshot reads `steps=1 hits=0 misses=10074 evictions=0 fills=4000 +bytes=9961472000 exhausted=6074` and the final one `steps=32 hits=37096 +misses=58538 evictions=48464 fills=52464 bytes=130654666752 exhausted=6074`, so +the decode-phase delta is 0 over 31 decode steps. At 8000 slots the same pair +reads 2074 and 2074, delta 0. Counters are byte-identical between the reps of +each slot count. + +All four runs produced the same 32 ids: +`11751,13,11751,369,264,3177,7172,303,279,17631,919,314,9338,11,383,279,181474,10629,13,1049,369,279,7526,3177,303,9338,321,369,3750,364,1141,25438`, +which detokenize to " Paris. Paris is a city located in the northern part of +France, on the Seine River. It is the largest city in France and is known for +its iconic", `finish_reason=length`. + +### CUDA arm, seven attempts, zero decode steps + +| slots | arena | prompt tokens | load | RSS after load | steps | peak system used | peak swap | +|---|---|---|---|---|---|---|---| +| 8000 | 18.55 GiB | 5 | 267.2 s | 61.20 GiB | 0 | 100,215 MiB | not sampled | +| 4000 | 9.28 GiB | 5 | 271.6 s | 61.20 GiB | 0 | 94,737 MiB | not sampled | +| 4000 | 9.28 GiB | 5 | 255.3 s | 61.20 GiB | 0 | 120,351 MiB | 30,569 MiB | +| 3500 | 8.12 GiB | 1 | 262.8 s | 61.20 GiB | 0 | 120,296 MiB | 28,138 MiB | +| 3500 | 8.12 GiB | 1 | 265.3 s | 61.20 GiB | 0 | 120,347 MiB | 30,461 MiB | +| 3500 | 8.12 GiB | 1 | 272.3 s | 61.20 GiB | 0 | 120,306 MiB | 30,172 MiB | +| 64 | 0.15 GiB | 1 | 268.9 s | 61.20 GiB | 0 | 118,257 MiB | 30,211 MiB | + +The LOAD is new and it works: this checkpoint used to refuse on `--device cuda` +(#1123) and now loads, and the `[expert-stream] ON` banner prints on the device +arm, which is the first production evidence that W0b's CUDA leg is reached and +answers true on real hardware. + +The last row is the diagnosis. A 0.15 GiB arena dies where an 18.55 GiB one +does, 124x smaller and no further along, so the slot arena is not the cost. The +in-place tower fallback is not the cost either: a 1-token prompt has a protected +set of 93 x 3 x 10 = 2,790 slices, fits 3500 slots with no fallback taken, and +behaves exactly like the 5-token prompt whose 13,950-slice set fits no arena +here. + +Growth is anonymous, not file-backed, so the GPU is not pinning the mapping's +pages through its address translation. That was the first hypothesis and it is +refuted. Sampling the container process every 5 s: `RssAnon` 8.1, 13.9, 25.9, +38.3, 49.4, 61.4 GB through the load while `RssFile` stays at 0.1-0.2 GB, then +`VmSwap` 0, 5.4, 11.4, 16.5, 27.1, 31.0 GB inside the forward. Host `RssAnon` +plus `VmSwap` reaches about 65 GB while the system reports ~119 GiB used, and +the ~42 GiB difference is device memory this unified part does not charge to the +process RSS. The non-expert weights are therefore resident twice, once as the +host-side `OwnedTensor` and once as the `ResidentWeight` device staging copy. +About 50 GiB of the 61.20 is the bf16 expansion the GDN V-head reorder forces on +`attn_qkv` and `ssm_out`, already measured in +[expert-streaming.md](specs/expert-streaming.md). Filed as +[#1299](https://github.com/mudler/vllm.cpp/issues/1299). + +Each CUDA run was stopped by a guard that kills the container when MemAvailable +plus SwapFree falls under a floor, because an out-of-memory kill on GB10 takes +the machine down rather than the process. The guard is not what makes them fail: +the last rows were climbing at roughly 10 GB of swap per minute with under +600 MiB of swap left. The guard's own first version counted MemAvailable ALONE +and killed two runs while 29 GiB of swap sat untouched, which is worth recording +because it reported a box about to die when the kernel had a whole tier left. + +Contention: the lease excluded every other `rc` job. Two things it did not +exclude are recorded rather than assumed away. An orphaned `VLLM::EngineCore` +from an earlier session held 3.32 GiB of host RSS throughout; it is not this +row's process and it was left alone. And the FIRST arm started seconds after the +previous holder's four-hour render released the device, with the one-minute load +average still at 17.5, which is one reason the first 8000-slot run is the +noisiest of the four. Every later arm started from a quiet box. diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 29698feda..1d2b52865 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -460,3 +460,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1375](https://github.com/mudler/vllm.cpp/issues/1375) | `MODEL-DIFFUSION-LTX25` | First end-to-end per-forward cost for the FULL 21.004 B LTX-2.5 DiT on GB10, measured on run `20260819T150230Z` with binary `0a43a750` built from [`7b9e207b1`](https://github.com/mudler/vllm.cpp/commit/7b9e207b1) (#1252). At 1024x576/25f (2304 latent tokens) the governor resolved **7 forward starts from the GPU busy/idle edge counter** and measured `per_forward ~162.0 s` with `first_dit = 481.5 s`, so the recipe's fixed 60 forwards (30 steps x 2 CFG legs, `ltx2_pipeline.cpp:521-529`) project **10 803 s against the rung's 7 153 s budget** and the rung was refused rather than run to the wall. The same lease then COMPLETED 768x448/25f (1344 tokens) in 2990 s, so the ceiling is geometry against lease length, not a defect. TWO instrument facts belong with the number, because both have already caused a wrong reading: `gpu_edges=0` means the GPU never went idle long enough to sample an edge (SATURATED), not that no work ran — this rung sampled 85% of 3191 samples above 50% utilisation; and `eu-stack` resolves no frames in the rc worker container, so phase attribution came from the cpu%/rss signature rather than from symbols. Owned by the LTX-2.5 row; spec [`ltx-2-5.md`](specs/ltx-2-5.md) | measurement | | [#1387](https://github.com/mudler/vllm.cpp/issues/1387) | `ENG-EXPERT-STREAM-DEVICE` | `docs/FEATURES.md`'s routed-expert-streaming row still read "CPU keep-quant towers only" after W0c made the seam take the slot arm on `is_cpu()` OR `host_memory_is_device_addressable()` and W0d made the load-time fit refusal drop those towers from its bound. `AGENTS.md` routes a feature-surface change to that page, and the change that moved the surface did not write it. `scripts/check-doc-checkpoint.py` said so, on commit `939755f99` of `row/ENG-EXPERT-STREAM-DEVICE-W0`: a measurement was appended to `.agents/benchmark-record.md` with no `docs/FEATURES.md` edit beside it. The PAGE is FIXED IN FLOW while repairing the fresh review of [#1377](https://github.com/mudler/vllm.cpp/pull/1377): the row now names both the device arm ([#1124](https://github.com/mudler/vllm.cpp/issues/1124)) and the residency condition ([#1378](https://github.com/mudler/vllm.cpp/issues/1378)), and the observability detail it displaced to stay inside the 220-character cell budget is stated at `docs/USAGE.md:4598-4620`. The GATE is NOT fixed and needs a decision: the checker walks a range one COMMIT at a time, so once a commit is published on a branch that may not be force-pushed, no later commit can make it green, and `scripts/agent-preflight.sh` keeps reporting `doc-checkpoint range` red on this branch until it merges. The squashed commit that lands on `main` carries both paths and passes. Changing the walk is checker semantics and needs its own row, spec and red-first evidence per `AGENTS.md` "Changing the rules or a checker", so it is not folded in here. Spec [`expert-stream-device-slots.md`](specs/expert-stream-device-slots.md) | bug | | [#1386](https://github.com/mudler/vllm.cpp/issues/1386) | — | `tools/bench/gpu_clock_state.py`'s `QUERY_FIELDS` collects nine fields and **none of them is thermal and none is electrical**, so the driver's own `SwThermalSlowdown` label can never be checked against a die reading on any window this helper has ever recorded. The measured consequence is that the nine windows of 2026-08-19 cannot distinguish a load transition from a thermal excursion. The concrete evidence is `clock-c1-r1.samples.json` in `/mnt/nas_share/rc/q38bf16/out/bench-20260819T035148Z/`: ours c1 r1 dips five times on the same period at the same `utilization.gpu = 96` — 48.83 s / 2177 MHz, 80.60 s / 2320 MHz, 109.28 s / 2210 MHz, 137.98 s / 2359 MHz, 166.07 s / 2268 MHz — and **two of those five carry `0x0000000000000000`**, no throttle bit at all (2210 and 2359), while three carry `0x20`. The 2210 MHz unlabelled dip is deeper than two of the three labelled ones, so the driver labels comparable excursions inconsistently and the bit alone cannot decide it. What would settle it: add `temperature.gpu` and `power.draw` to `QUERY_FIELDS`. That changes the clock-record schema, so it owes its own row and spec. Split out of [#1354](https://github.com/mudler/vllm.cpp/issues/1354) and owed under `## Owed` in [lease-clock-pinning.md](specs/lease-clock-pinning.md) | gap | +| [#1299](https://github.com/mudler/vllm.cpp/issues/1299) | `ENG-EXPERT-STREAM-DEVICE` | `Qwen3.8-2.4T-A95B UD-Q1_0` (369.97 GiB) now LOADS on `--device cuda` on a 119.631 GiB GB10 — `ENG-EXPERT-STREAM-DEVICE` W0 works, the slot store builds and the #1123 refusal correctly does not fire — and then exhausts the machine inside the FIRST forward, producing zero decode steps. Measured on `dgx:gpu0` inside one `rc hold` at source `95883dcae`, production stack (CUTLASS 4.5.0 found, `cutlass-fp8`/`cutlass-nvfp4`/`fa2` ENABLED for `[121a]`, Triton AOT `sm_121a`). Seven attempts, all identical: load 255-272 s, RSS after load 61.20 GiB, 0 decode steps, peak system `used` 118-120 GiB with 28-31 GiB of swap consumed. The slot arena is NOT the cost, and that is measured rather than argued: a **64-slot 0.15 GiB** arena dies exactly where an 8000-slot 18.55 GiB one does, and a 1-token prompt (protected set 2,790 slices, which fits 3500 slots with no in-place tower fallback at all) behaves exactly like a 5-token one. The growth is ANONYMOUS, not file-backed, so the GPU is not pinning the mapping's pages: `RssAnon` climbs 8.1 -> 61.4 GB through the load while `RssFile` stays at 0.1-0.2 GB, then `VmSwap` climbs 0 -> 31.0 GB inside the forward. Host `RssAnon` + `VmSwap` reaches ~65 GB while system `used` sits at ~119 GB, and the ~42 GiB difference is device memory the process RSS does not account for on this unified part — so the non-expert weights are resident TWICE, once as the host-side `OwnedTensor` and once as the `ResidentWeight` device staging copy. `.agents/specs/expert-streaming.md` measured the host half already: the GDN V-head reorder makes `attn_qkv` and `ssm_out` `kTransformedWeight`, which expands them from ~5.5 bits to bf16, about 50 GiB of the 61.20. The CPU arm pays it once and serves (32 tokens, steady 11.05 s/token at 4000 slots, decode-phase `exhausted` delta 0); the CUDA arm pays it twice and cannot. NOT #1123, whose refusal fired at load and no longer applies, and NOT a defect in W0b/W0c/W0d, which do what their spec says. This is the device-side half of the sentence `expert-streaming.md` already carries: "Whoever takes this needs BOTH: the streaming lane for the ~330 GiB of experts, and a transformed-weight path that does not expand." Owed under `## Owed` in [`expert-stream-device-slots.md`](specs/expert-stream-device-slots.md) | bug | diff --git a/.agents/specs/expert-stream-device-slots.md b/.agents/specs/expert-stream-device-slots.md index 226b041e7..7431f7f8b 100644 --- a/.agents/specs/expert-stream-device-slots.md +++ b/.agents/specs/expert-stream-device-slots.md @@ -10,11 +10,23 @@ platform may read it. ## Now -`ACTIVE`. **W0a HAS RUN on `dgx:gpu0` and answered `PAGEABLE_OK`. W0b, W0c and -W0d are implemented and unit-gated on the CPU tier. W0e is still QUEUED behind -the lease, so no decode number exists yet and none is claimed.** +`ACTIVE`. **W0a through W0e have all run. W0 does what it was built to do — +`--device cuda` now LOADS this checkpoint instead of refusing — and the CUDA arm +still produces no token, because the first forward exhausts the box for a reason +that is not the expert lane +([#1299](https://github.com/mudler/vllm.cpp/issues/1299)). The CPU arm is +measured, reproduced, and replaces the VOID #912 F1 figure.** See `## Evidence`. -What that means precisely, because "W0 landed" would overstate it: +The three gates, reported one result each: + +| Gate | Result | +|---|---| +| **G0-CORRECT** | **NO CUDA SIDE.** The CUDA arm emits zero tokens, so there is nothing to compare. The CPU side is byte-identical across four runs and two slot counts (32 ids, listed in `## Evidence`), which is the strongest half of the comparison that this hardware allows today. | +| **G0-LIVE** | **PASS on CPU, NOT REACHED on CUDA.** CPU: `steps=32`, `forced=0`, decode-phase `exhausted` delta **0** at both 4000 and 8000 slots. CUDA: the store BUILDS and prints its banner, and no step boundary is ever reached, so there is no snapshot pair to difference. | +| **G0-SPEED** | **CPU only, no ratio.** Steady decode **11.05 s/token**, which is rep 2's median over 29 samples (min 9.43, max 13.25) at 4000 slots; rep 1's median is 11.22, so the two reps agree within 1.5%. No CUDA number exists, so no ratio is reported and none may be inferred. | + +What that means precisely, because both "W0 landed" and "W0 failed" would +misstate it: * **W0b — the predicate.** `Platform::host_memory_is_device_addressable()`, base false, CUDA from `cudaDevAttrPageableMemoryAccess AND @@ -120,14 +132,18 @@ What that means precisely, because "W0 landed" would overstate it: `ConcurrentManagedAccess` — are not carried here, because the verdict turns on the two that are and inventing the other two would be worse than omitting them. -* **W0e — the measurement.** Not run. G0-CORRECT, G0-LIVE and G0-SPEED are all - `PENDING` on the lease. - -So `--device cuda` on `Qwen3.8-2.4T-A95B UD-Q1_0` is no longer refused by the -predicate on this box: the probe has measured `PageableMemoryAccess == 1`, and -the predicate was probed rather than assumed precisely so that this sentence -could change on evidence. What has NOT run is the load itself. The developer's -target remains that GPU figure, and W0e is what produces it. +* **W0e — the measurement.** RAN, on one `rc hold` on `dgx:gpu0`. It produced a + reproduced CPU figure, a CUDA load that works, and a CUDA arm that generates + nothing. `## Evidence` has all of it. + +Today `--device cuda` on `Qwen3.8-2.4T-A95B UD-Q1_0` **loads**, which is what +W0 was for and is new. It then dies in the first forward, and the cause is the +DENSE half of the model rather than the expert lane: the non-expert weights are +resident twice on a unified part, and a 0.15 GiB slot arena fails in exactly the +place an 18.55 GiB one does. That is +[#1299](https://github.com/mudler/vllm.cpp/issues/1299), listed under `## Owed`. +The developer's target remains a GPU figure, and this row cannot produce one +until #1299 moves. ## Scope @@ -380,6 +396,214 @@ gate that stays green without it measured a class, not a capability. **G-DISCRETE (owed, cannot run here).** See `## Owed`. +## Evidence + +W0a and W0e, `dgx:gpu0` (GB10, sm_121a, driver 580.173.02, CUDA 13.0.88, 20 +cores, 122,502 MiB RAM, 30,625 MiB swap), one `rc hold` +`edb4b3d0-5d6e-422f-ade6-bff5339e3396`, 2026-08-18T22:13:12Z to +2026-08-19T00:58Z, released by interrupting its client. Source `95883dcae`, +the head of PR [#1242](https://github.com/mudler/vllm.cpp/pull/1242). + +### The build, because a degraded one would have voided every number + +`cmake -DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_CUDA=ON +-DVLLM_CPP_CUDA_ARCHITECTURES=121a -DVLLM_CPP_CUTLASS_DIR=/cutlass +-DVLLM_CPP_TRITON=ON`, CUTLASS 4.5.0 staged host-side. The configure log +reports `fp4-mma`, `cutlass-nvfp4`, `cutlass-fp8`, `marlin-nvfp4` and `fa2` all +`ENABLED for [121a]`, `CUTLASS found at /cutlass; enabling sm120a NVFP4 cutlass +GEMM`, `FlashAttention-2 prefill/decode: ENABLED for arch(es) [121a]`, and +`Triton AOT W2: embedded trees [sm_80;sm_86;sm_89;sm_90a;sm_100a;sm_121a]`. +Both arms are the SAME binary, and every arm ran with the page cache dropped +first (`echo 3 > /proc/sys/vm/drop_caches`, ~90.9 GB available after each drop). + +### The harness, and why it is not `vllm-cli` + +`benchmarks/expert_stream_device_w0e.cpp`, a thin client of `include/vllm.h` +linked against the packaged shared library, which exports the C ABI and nothing +else. **The project builds it**, as the `expert-stream-device-w0e` target. It +was first written unwired, beside `marlin_moe_standalone.cpp`, on the reading +that a gate instrument is not a shipped capability. Wiring it found that the +file did not compile under the project's own flags at all — three backslash +continuations inside `//` comments, which `-Werror=comment` rejects — so the +recorded recipe was the only thing that had ever built it. An instrument +nothing compiles rots against the very ABI it measures, and a measurement whose +harness no longer builds cannot be reproduced, so the target is the correct +shape even though the file is not a capability. It needed to exist because the gate wants three things from ONE +generation, and no shipped command produces all three: the generated TOKEN IDS +(`vllm_complete_tokens`, ABI v13, which writes them into a caller buffer), a +PER-STEP arrival timestamp (a custom logits processor, invoked once per decode +step), and the expert-stream counters at both ends of the run +(`VT_MOE_EXPERT_STREAM_STATS_EVERY=1`, so the periodic line prints at step 1 and +every step after; at the default 16 a 32-token run prints two lines and a +5-token one prints none, which is indistinguishable from a dead lane). + +The logits processor is a PURE OBSERVER: it records and returns without touching +`logits`, so the argmax the sampler takes is the one it would have taken without +it. Its own `token_ids` view is recorded but NOT used for the gate, because +`tests/capi/test_capi.cpp` states that under the async scheduler that view can +lag the emitted tokens. + +The prompt is fixed as TOKEN IDS rather than as text, `760,6511,314,9338,369` +("The capital of France is", produced by `build/examples/tokenize` from the +checkpoint's own GGUF metadata), so both arms are fed byte-identical input and +no tokenizer step sits between the two. + +### The CPU arm: measured, reproduced, and it replaces the VOID #912 F1 figure + +`--device cpu`, `VT_GGUF_PREFAULT=0 VT_MOE_EXPERT_STREAM=1`, `--max-num-seqs 1`, +greedy, 32 tokens. Four runs, two slot counts, two reps each: + +| slots | arena | load | TTFT (step 1) | steady decode s/token, steps 4-32 | total generate | peak RSS | min avail | peak swap | +|---|---|---|---|---|---|---|---|---| +| 4000 | 9.28 GiB | 271.1 s | 85.90 s | median **11.22**, min 9.62, max 12.51 | 502.3 s | 86.5 GiB | 16,840 MiB | not sampled | +| 4000 | 9.28 GiB | 255.7 s | 79.09 s | median **11.05**, min 9.43, max 13.25 | 460.7 s | 86.5 GiB | 16,347 MiB | 6,883 MiB | +| 8000 | 18.55 GiB | 261.6 s | 94.25 s | median 45.40, min 20.73, max 82.68 | 1643.2 s | 86.6 GiB | 6,985 MiB | not sampled | +| 8000 | 18.55 GiB | 266.5 s | 132.74 s | median 39.98, min 16.06, max 94.30 | 1581.6 s | 86.6 GiB | 6,941 MiB | **30,625 MiB (all of it)** | + +**Read the 8000-slot rows as a memory result, not a cache result.** A bigger +cache came out slower, and the four available median pairings span 3.56x to +4.11x: 3.62x same-rep on the pair that produced the publishable figure (39.98 +against 11.05) and 4.05x pairing rep 1 with rep 1 (45.40 against 11.22). It also +came out far less steady: the steady window's max/min ratio is 3.99x in rep 1 +(20.73 to 82.68 s) and 5.87x in rep 2 (16.06 to 94.30 s), against 1.30x and +1.40x at 4000 slots. Whichever pairing is quoted, the direction is wrong. + +**The cause is NOT that the arena does not fit, and the arithmetic says so.** +18.55 GiB of arena beside this model's 62 GiB of dense weights is 80.55 GiB on a +119.63 GiB box, which fits with room to spare, and the columns agree: peak RSS +moves 86.5 -> 86.6 GiB for a 9.27 GiB arena delta. The two columns that DO move +are `min avail`, 16,347 -> 6,941 MiB, a 9,406 MiB fall that is about the arena +delta, and peak swap, 6,883 -> 30,625 MiB, which is all of it. The +best-supported reading of that pattern is **page-cache displacement**: the +borrowed 370 GiB expert mapping is served out of whatever memory is free, the +arena takes that memory, and the reclaim pressure it creates pushes anonymous +pages to swap. That is a reading of the columns and not a proven mechanism, and +this run cannot separate it from plain reclaim pressure, because it sampled no +page-cache size and no major-fault counter. The operational conclusion does not +depend on which of the two it is, and it is unchanged: **more slots is not a +free knob.** + +Both 8000-slot runs reproduce each other, so this is the box's behaviour and not +a fluke. **The publishable CPU figure is the 4000-slot one, 11.05 s/token**, +which is rep 2's median; rep 1's is 11.22, so the two reproduce within 1.5%. It +is the first live-cache streaming-ON decode number this project has; +`docs/BENCHMARKS.md:8` recorded streaming-ON as VOID (#912 F1, the step clock +dead from token 3) with a re-measure owed. + +Steps 1-3 are excluded from the steady figure and reported separately because +they are not steady state: step 1 is prefill, and steps 2-3 are still filling a +cold cache (85.90, 80.38, 16.06 s, then 11.7 and below). + +**G0-LIVE, gated on the DIFFERENCE and never the total.** At 4000 slots the +after-prefill snapshot is `steps=1 hits=0 misses=10074 evictions=0 fills=4000 +bytes=9961472000 exhausted=6074` and the final one is `steps=32 hits=37096 +misses=58538 evictions=48464 fills=52464 bytes=130654666752 exhausted=6074`, so +the decode-phase `exhausted` delta is **0** over 31 decode steps. At 8000 slots +the same pair reads 2074 and 2074, delta **0**. Both reps of each slot count +produce byte-identical counters, so the lane is deterministic. `steps=32 > 0`. +`forced` is 0 by construction and is deliberately absent from the stderr line: +its only setter is `detail::ExpertStreamSetForceFallback`, which +`src/vllm/model_executor/models/qwen3_5_internal.h:488` records as having no +production caller, and `qwen3_5.cpp:5521` (`++forced_` in +`Qwen35ExpertStream::Slice`) counts it separately from `exhausted` for exactly +that reason, in a comment that says so. + +**The tokens.** All four CPU runs, across both slot counts, produced the same 32 +ids: + +``` +11751,13,11751,369,264,3177,7172,303,279,17631,919,314,9338,11,383,279, +181474,10629,13,1049,369,279,7526,3177,303,9338,321,369,3750,364,1141,25438 +``` + +which detokenize to " Paris. Paris is a city located in the northern part of +France, on the Seine River. It is the largest city in France and is known for +its iconic", `finish_reason=length`, `completion_tokens=32`. + +### The CUDA arm: it loads, and it does not generate + +**The load is the new thing and it works.** `--device cuda` on this checkpoint +used to refuse ([#1123](https://github.com/mudler/vllm.cpp/issues/1123)); W0d's +conditional bound removes that refusal when the lane serves the towers, and the +load now completes in 255-272 s with 61.20 GiB resident. The lane then engages: +the `[expert-stream] ON slots=... resident=... GiB` banner prints on the device +arm, which is W0c's whole point. + +That banner is also the first production evidence that **W0b's CUDA leg is +reached and answers true on real hardware**. Neither the banner nor the +successful load is reachable unless `host_memory_is_device_addressable()` +returned true from the CUDA platform, so the leg no longer rests on the fake +platform in `test_expert_stream_device_slot` alone. The negative — a mutation +that makes it answer false — is still owed and is in `## Owed`. + +**Then the first forward exhausts the machine. Seven attempts, zero decode +steps, every one of them.** + +| slots | arena | prompt tokens | load | RSS after load | decode steps | peak system `used` | peak swap used | +|---|---|---|---|---|---|---|---| +| 8000 | 18.55 GiB | 5 | 267.2 s | 61.20 GiB | 0 | 100,215 MiB | not sampled | +| 4000 | 9.28 GiB | 5 | 271.6 s | 61.20 GiB | 0 | 94,737 MiB | not sampled | +| 4000 | 9.28 GiB | 5 | 255.3 s | 61.20 GiB | 0 | 120,351 MiB | 30,569 MiB | +| 3500 | 8.12 GiB | 1 | 262.8 s | 61.20 GiB | 0 | 120,296 MiB | 28,138 MiB | +| 3500 | 8.12 GiB | 1 | 265.3 s | 61.20 GiB | 0 | 120,347 MiB | 30,461 MiB | +| 3500 | 8.12 GiB | 1 | 272.3 s | 61.20 GiB | 0 | 120,306 MiB | 30,172 MiB | +| **64** | **0.15 GiB** | 1 | 268.9 s | 61.20 GiB | 0 | 118,257 MiB | 30,211 MiB | + +Each run was stopped by a guard that kills the container when MemAvailable plus +SwapFree falls under a floor, because an out-of-memory kill on GB10 takes the +whole machine down rather than the process. The guard is not what makes these +runs fail: the last two rows above were still climbing at roughly 10 GB of swap +per minute with under 600 MiB of swap left, so the kernel was seconds from the +kill the guard exists to prevent. + +**The expert lane is not the cost, and that is measured rather than argued.** +The 64-slot row is the control: a **0.15 GiB** arena dies in the same place an +**18.55 GiB** one does, 124x smaller and no further along. The in-place tower +fallback is not the cost either: a 1-token prompt has a protected set of +`93 x 3 x 10 = 2,790` slices, which fits 3500 slots with no fallback taken at +all, and it behaves exactly like the 5-token prompt whose 13,950-slice set +cannot fit any arena on this box. + +**The growth is ANONYMOUS, not file-backed**, so the GPU is not pinning the +mapping's pages through its address translation — which was the first +hypothesis, and it is wrong. Sampling the container process every 5 s through +the load and into the forward: + +``` +RssAnon: 8.1 -> 13.9 -> 25.9 -> 38.3 -> 49.4 -> 61.4 GB (through the load) +RssFile: 0.1-0.2 GB throughout, 4.6 GB at the very end +VmSwap: 0 -> 5.4 -> 11.4 -> 16.5 -> 27.1 -> 31.0 GB (inside the forward) +``` + +Host `RssAnon` plus `VmSwap` reaches about 65 GB while the system reports +~119 GiB `used`, and the ~42 GiB difference is device memory that this unified +part does not charge to the process RSS. **So the non-expert weights are +resident twice** — once as the host-side `OwnedTensor` and once as the +`ResidentWeight` device staging copy — and on a box where device memory IS host +memory, that doubling is what runs it out. + +[`expert-streaming.md`](expert-streaming.md) already measured the host half of +this: the GDN V-head reorder makes `attn_qkv` and `ssm_out` +`kTransformedWeight`, which expands them from about 5.5 bits to bf16, roughly +**50 GiB** of the 61.20. The CPU arm pays that once and serves. The CUDA arm +pays it twice and cannot. That spec's own sentence — "Whoever takes this needs +BOTH: the streaming lane for the ~330 GiB of experts, and a transformed-weight +path that does not expand" — is exactly this result, and W0 delivered the first +half. Filed as [#1299](https://github.com/mudler/vllm.cpp/issues/1299) and +listed under `## Owed`. + +### What was running beside the measurement + +The lease excluded every other `rc` job for its duration. Two things were not +excluded and are recorded rather than assumed away. An orphaned +`VLLM::EngineCore` process from an earlier session held 3.32 GiB of host RSS for +the entire window; it is not this row's process and it was left alone. And the +FIRST arm started at 22:13, seconds after the previous lease holder's four-hour +render released the device, with the one-minute load average still at 17.5 — +which is one of the two reasons the first 8000-slot run is the noisiest of the +four, the other being the swap it drove. Every later arm started from a quiet +box. + ## Dependencies | Dependency | Shape | @@ -524,7 +748,9 @@ re-derived here. | **`scripts/check-doc-checkpoint.py` stays RED on this branch for commit `939755f99` and cannot be made green here** ([#1387](https://github.com/mudler/vllm.cpp/issues/1387)). That commit appended a measurement to `.agents/benchmark-record.md` without writing `docs/FEATURES.md`, whose streaming row then said "CPU keep-quant towers only" after W0c had made a host-readable staging device take the slot arm. | The PAGE is repaired here. The GATE is not, and cannot be: the checker walks a range one COMMIT at a time, and `main` may never be force-pushed, so no later commit can make a published one green. The squashed commit that lands carries both paths and passes. Changing the walk is checker semantics and needs its own row, spec and red-first evidence. | | **The CUDA registrar's own probe assembly is still unmutated.** `src/vllm/platforms/cuda.cpp`'s `Registrar` reads `cudaDevAttrPageableMemoryAccess` and `cudaDevAttrIntegrated`, defaults each to 0 on a query failure, and hands the pair to `HostMemoryIsDeviceAddressableFromAttrs`. That call and those defaults compile only in a CUDA build, so nothing on the CPU tier can mutate them. | The RULE they feed is no longer part of this debt: #1378 extracted it and gated it over all four attribute pairs in `test_platform`, and both term-deletion mutations are RED there. What remains is narrower and honest -- the probe calls, the failure defaults, and the registration itself -- and it needs the same `dgx:gpu0` lease as W0e. Named here rather than folded into the W0b bullet, which used to claim more than it had. | | **G-DISCRETE: validate W1/W2 on a discrete NVIDIA GPU.** The measurement: on a device with VRAM V and `host_memory_is_device_addressable() == false`, load a GGUF whose `*_exps` towers exceed V, with the lane on, and gate (i) token-exactness against the CPU arm on the same checkpoint, (ii) decode-phase `exhausted` delta 0, (iii) peak device allocation <= non-expert remainder + arena. | No discrete NVIDIA GPU is reachable from this project. `dgx:gpu0` is a GB10 where device memory IS host memory, so a device store there exercises the plumbing and not the thing W1 exists for. Recorded rather than implied, because a gate nobody can run is not a gate. | +| **A mutation of W0b's CUDA leg.** `CudaPlatform::host_memory_is_device_addressable` compiles only in a CUDA build, so no CPU-tier gate can invert it. The bullet in `## Now` promised this line and the table did not carry it, which is fixed here. | **Half discharged by W0e and stated as half.** The lane engaged on a real `--device cuda` run — the `[expert-stream] ON` banner printed and the #1123 refusal did not fire — and neither happens unless the probed predicate returned true on the actual CUDA platform, so the leg is now proven REACHED and proven to answer true on a GB10. What is still owed is the negative: a mutation that makes it answer false and shows a gate go red. That needs a CUDA build with a test target, and W0e built with `-DVLLM_CPP_BUILD_TESTS=OFF` because the lease was for the measurement. | | **A zero-copy device filler (GPUDirect Storage / `cuFile`).** | W1 ships the staging bounce by choice, for the reasons in its design note. The measurement that would justify replacing it — a device-arm decode where the H2D leg is a measurable fraction of fill time — does not exist until W1 has run somewhere. | -| **The CPU arm's streaming decode figure is still VOID.** `docs/BENCHMARKS.md:8` records it as VOID (#912 F1) with a re-measure owed. | Owned by `ENG-EXPERT-STREAM` and arranged separately by the operator. It is the DENOMINATOR for G0-SPEED, not a precondition for G0-CORRECT or G0-LIVE. | +| **The CUDA arm loads and then exhausts the box in its first forward, so this row still has no GPU number.** [#1299](https://github.com/mudler/vllm.cpp/issues/1299). The non-expert weights are resident twice on a unified part, once as the host-side `OwnedTensor` and once as the `ResidentWeight` device staging copy, and about 50 GiB of that is the bf16 expansion the GDN V-head reorder forces on `attn_qkv` and `ssm_out`. | Not fixable inside this row's scope, and measured rather than inferred: a 0.15 GiB arena fails where an 18.55 GiB one does, and the growth is `RssAnon` while `RssFile` stays flat. The fix is a transformed-weight path that does not expand, or a staging path that releases the host copy — either is its own row with its own spec. W1 and W2 are unaffected: they are about WHERE a slice lives, and this is about the dense remainder beside it. | +| ~~**The CPU arm's streaming decode figure is still VOID.**~~ **CLOSED by W0e**, 2026-08-18: streaming-ON decode on a live cache is **11.05 s/token** steady at 4000 slots, rep 2's median with rep 1 at 11.22, and the decode-phase `exhausted` delta is 0 in the same run. See `## Evidence`. | Kept as a line rather than deleted because `docs/BENCHMARKS.md:8` still carries the parent row's VOID (#912 F1) text for `ENG-EXPERT-STREAM`, which owns that row's own re-measure. This row measured its own denominator and is no longer waiting on one. | | **`.agents/specs/expert-streaming.md`'s `## Owed` entry for #1124 still names no owning row ID.** | Not edited here on purpose; PRs #1200 and #1216 both edit that file. One-line follow-up once both land. | | **W1 may land UNREACHED if it is split from W2.** | The recommendation is one pull request. If a split is chosen, the commit body and the PR body must name what is unreached and name W2 as the owning wiring, per `## Nothing lands dead`. | diff --git a/.agents/specs/prompt-token-divergence.md b/.agents/specs/prompt-token-divergence.md new file mode 100644 index 000000000..121aca0fa --- /dev/null +++ b/.agents/specs/prompt-token-divergence.md @@ -0,0 +1,314 @@ +# SPEC-PROMPT-TOKEN-DIVERGENCE — the served prompt-token count, and what it is not + +Issue: [#1355](https://github.com/mudler/vllm.cpp/issues/1355) +Row: `SPEC-PROMPT-TOKEN-DIVERGENCE` +State: `READY` (diagnosis committed; the defect is not yet located) + +**The ID collides with a prefix.** `SPEC-` is this tree's SPECULATIVE-DECODING +row prefix (`SPEC-MTP-K-GT-1`, `SPEC-DSPARK-BLOCK-SIZE-GUARD`, `SPEC-DFLASH2`), +so this ID reads as a spec-decode row and is not one. It is recorded here rather +than renamed, because the ID is already committed, indexed and referenced. + +## Now + +The benchmark campaign of 19 August 2026 measured our server at **5,942** prompt +tokens where the pinned vLLM oracle measured **6,144** for a corpus both arms +took from byte-identical invocations of one `vllm bench serve` client. This spec +records what that divergence **is not**, with reproductions, and names the one +probe that can still locate it. It changes no product code. + +## Scope + +In scope: attributing the count. Out of scope: the speed verdict itself, and +with it every ours-over-vLLM ratio. What this spec corrects is OUR OWN arm's +numerator, on our own arm only (`## Consequence for the campaign's numbers`). + +## Upstream anchors (pin `5559679229`, `/home/mudler/_git/vllm`) + +- `vllm/benchmarks/datasets/datasets.py:557` `RandomDataset` — a prompt is a + contiguous slice of vocabulary ids, DECODED to text and then re-encoded. +- `vllm/benchmarks/datasets/datasets.py:495-549` `gen_prompt_decode_to_target_len` + — the decode/re-encode loop that pads or truncates until + `len(tokenizer.encode(prompt, add_special_tokens=False)) == target`, retrying + ten times and reporting the residual as `token_mismatch`. +- `vllm/benchmarks/datasets/utils.py:41` `get_sampling_params`. +- `vllm/benchmarks/serve.py:74-144` `_align_prompts_to_server_tokenizer` — posts + the FIRST prompt to the server's `/tokenize`, and on a disagreement re-tokenizes + every prompt through the server and truncates it to `prompt_len` ids. +- `vllm/benchmarks/lib/endpoint_request_func.py:245-247` — `output.prompt_len` + starts at the dataset's value and is OVERWRITTEN by `usage.prompt_tokens` from a + usage-only SSE frame. +- `vllm/benchmarks/serve.py:606,734,1222` — `total_input`, hence + `total_token_throughput`, and `input_lens`. + +**vLLM reporting exactly 1024 is not the surprising half.** The dataset +calibrates each prompt string until the client's own tokenizer encodes it to +exactly `--random-input-len` tokens, so a server whose tokenizer agrees with the +client's reports 1024 by construction. Regenerated locally at the pin: 48/48 +prompts, `token_mismatch == 0`. + +## What was measured + +Corpus regenerated from `--seed 0`, `--random-input-len 1024`, +`--random-range-ratio 0`, tokenizer `/mnt/nas_share/rc/ckpt/qwen3.8-27b-hf` +(`tokenizer.json` sha256 `0997f410c57a1f4e53b09e4be8f4a172d90edd9564368fb0847030937229b9f3`, +byte-identical to the `ckpt-sha256.txt` manifest the campaign wrote from the +STAGED copy the server read). + +1. **The corpus is the campaign's corpus.** State the measurement precisely, + because the shortfall did NOT reproduce and cannot reproduce here: our + tokenizer encodes all 48 regenerated prompts to 1024, so they come short + nowhere on x86-64. What agrees is the INDEX SET. The 19 regenerated prompts + that carry 74-150 combining marks are exactly the 19 indices the campaign's + `bench-20260819T035148Z/c8-r1.json` reports short — 0, 1, 9, 10, 11, 13, 14, + 15, 16, 19, 21, 22, 25, 28, 29, 30, 33, 45, 47. Prompts 0-5 of the c8 leg are + byte-identical to the c1 leg because `numpy` short-circuits a zero-width + `integers()` draw and consumes no state, so both legs share one offset + stream. A 48-bit index agreement between an independently drawn corpus and + the served counts is not a coincidence: these are the campaign's prompts. +2. **Our tokenizer sources are byte-exact ON X86-64.** + `vllm::tok::Tokenizer::FromHfJson` + `Encode` at this revision returns the + SAME IDS, not merely the same count, as HF `tokenizers` for all 48 prompts. The tokenizer sources are unchanged + between the campaign's build `1dac4f9a7` and `origin/main` + (`git diff 1dac4f9a7 origin/main -- src/vllm/tokenizer include/vllm/tokenizer` + is empty), so the comparison is against the code that ran. +3. **The transport PRIMITIVES are not it.** The same result over a real socket + through `third_party/httplib` + `nlohmann::json::parse` + + `EncodeWithSpecialTokens`, with the body sent three ways: `ensure_ascii` + escapes (what `aiohttp` sends), raw UTF-8, and `Transfer-Encoding: chunked`. + 48/48 at 1024. **This was a standalone harness and NOT the production request + path.** It never ran `from_json(const nlohmann::json&, CompletionRequest&)`, + which is where the `prompt` string is actually extracted + (`protocol.cpp:280-281`), and it never ran `serving_completion.cpp`. That + segment is x86-testable and is UNTESTED, and it sits inside the gap this + elimination would otherwise be read as closing. +4. **Not a build-flag or memory defect.** `-O0`, `-O3`, `-O3 -funsigned-char` + (the aarch64 `char` signedness, which is the obvious portability suspect) and + `-fsanitize=address,undefined` all return 1024 on all 48 with no diagnostic. +5. **Not any pre-tokenizer or BPE option TRIED.** Every wrong `SplitPattern` + (`kQwen2Classic`, `kLlama3`, `kTekken`, `kGpt4o`, `kGpt2`) and every + misclassification of `\p{M}` produces MORE tokens (1117-1306), never fewer. + **That direction is a property of the cases tried, not a law**, and this spec + claims only the former. A COARSER split PERMITS merges a finer split forbids, + so it can lower a count rather than raise it — that is the mechanism behind + `\p{N}` against Qwen's `\p{N}{1,3}` — and two measured cases already sit + below 1024: `ignore_merges=true` gives 1022, and the fresh review measured a + Split pre-tokenizer with `behavior: "Removed"` giving 0. The reason every + pattern tried went UP is narrower and sufficient: the merge table was learned + under one canonical split, so a deviation from it costs tokens. Mechanisms + that can lower a count are therefore NOT excluded here, and three are named + in `## Diagnosis`. +6. **Not the client's alignment pass, in EITHER direction.** `_fix_one` + (`serve.py:132-138`) returns the request UNCHANGED when + `len(tokens) <= req.prompt_len`, and truncates only above it. No prompt in + this corpus exceeds 1024 server ids — the anomaly is 915, which is below — + so the bytes on the wire are the same whether the pass ran or not, and + whether or not the warning that says it ran was truncated + (`## Consequence for the campaign's numbers`). Simulated against our + tokenizer under every wrong pattern it likewise always predicts 1024, never + 915. +7. **Not Unicode normalisation.** The checkpoint declares `normalizer: NFC` and + `tokenizer.cpp:388-391` records the deviation that we accept it and do not + apply it. Every one of the 48 prompts is ALREADY NFC, so the deviation is + inert here. (It is still owed: a client that sends non-NFC text gets a + different tokenization from HF. The tokenizer parity goldens feed it no + non-NFC text either, so nothing in this tree exercises the deviation at all. + Not this bug.) + +The divergence correlates perfectly with combining marks: all 19 prompts at the +campaign's short indices carry 74-150 characters in `Mn`/`Mc`, and all 29 at its +exact indices carry 0-2. Equivalently +they are the prompts drawn from the multilingual region of the vocabulary +(bytes-per-character 1.4-2.2), against pure-ASCII (1.0) and pure-CJK (3.0) +neighbours that are exact. No transformation tried — deleting or replacing any +category, page or codepoint range, prefix truncation, NFC/NFD/NFKC — lands on the +served numbers. + +## Diagnosis + +**This is not a harness artifact, not a corpus property, and not a defect in the +tokenizer SOURCES AS EXERCISED ON X86-64.** All three were candidate +explanations and all three are falsified above. That qualifier is load-bearing: +what item 2 and item 4 establish is that these sources produce the reference ids +on x86-64 at four optimisation settings and under ASan and UBSan. The server +binary that reported the anomaly is aarch64, and the probe below exists to catch +a code-generation difference in exactly that class. What remains is that the +server binary (`bin/vllm-server`, sha256 `7d0c3cafb224…`, aarch64, built from +`1dac4f9a7`) reported a count that the identical source, the identical +`tokenizer.json` and the identical prompt bytes do not produce on x86-64. + +**That sha256 is the run's, and the conflicting value beside the artifacts is +already adjudicated.** `NOTES.txt` in the evidence directory records +`ab0b9a1e6144…`, which is the binary of the PREVIOUS day's run +(`out/bench-20260818T213222Z/job.log:17,22`). Three artifacts of the run this +spec diagnoses give `7d0c3cafb224…` instead: `sha256sum` over +`/mnt/nas_share/rc/q38bf16/bin/vllm-server`, `out/RESULT.txt:5`, and +`out/bench-20260819T035148Z/job.log:17,18`, which asserted `WANT_BIN_SHA256` +equal to `GOT_BIN_SHA256` at launch. +`.agents/benchmark-record.md:24510-24514` settled this exact conflict in favour +of the executing artifact, so it is not re-opened here. The hash is not +load-bearing for the diagnosis — both builds carry +`WANT_SHA=1dac4f9a7…` and the identical `SRC_SHA256=c74c45d1…`, two +non-reproducible builds of one tree, and the argument rests on the source and on +aarch64 — but it is load-bearing for reproducing the run. + +**The conclusion rests on the data path, not on an enumeration.** No list of +things that can go wrong inside a tokenizer is needed, and none is offered, +because a missed entry would not weaken this: + +- `usage.prompt_tokens` is `response.prompt_token_ids.size()` + (`serving_completion.cpp:78`, the streaming path `vllm bench serve` drives; + `res.prompt_token_ids.size()` at `:283` is its non-streaming sibling); +- that vector is `RequestOutput::prompt_token_ids` + (`output_processor.cpp:284`), copied from the request state the engine + recorded at admission (`output_processor.cpp:174`); +- and that state carries the ids `InputProcessor` produced by calling + `tokenizer_.EncodeWithSpecialTokens(prompt)` (`input_processor.cpp:259-260`). + +The reported number IS the id vector the model consumed. 915 ids where the +client calibrated the prompt to 1024 therefore means the model consumed a +different prompt, WHATEVER the mechanism, and that holds against a mechanism +nobody has listed. **The model saw a different prompt.** If the mechanism is +confirmed, it reaches every token-exact gate driven over HTTP with non-ASCII +prompts, not only benchmarking. + +**Three mechanisms that nothing above excludes**, recorded so that the search is +not read as narrower than it is: + +- **A merge-table or vocabulary ORDER difference.** A hash order or a + comparator that resolves differently on aarch64 can change which merges rank + first and so LOWER a count with no text lost at all. This is the aarch64 + hypothesis itself, and the x86-64 id-for-id agreement in item 2 cannot see it. +- **A UTF-8 decode truncation inside `Encode`**, which loses text rather than + re-ranking merges. +- **The `/v1/completions` request-parse segment**, which item 3 did not + exercise: `CompletionRequest`'s `from_json` (`protocol.cpp:280-281`) and + `serving_completion.cpp`. + +## The one probe left + +Everything reproducible without hardware has been reproduced. The remaining +question needs the box, and needs no GPU beyond a server that is already up: + +1. On `dgx:gpu0` inside a lease, against a running `vllm-server` on this + checkpoint, POST prompt 0 of the regenerated corpus to `/tokenize` + (`add_special_tokens: false`) and to `/v1/completions`. + + **There is ONE tokenizer instance and five borrows of it**, so this probe + does not compare two tokenizer states and must not be read as doing so. + `server_main.cpp:1229` takes + `const vllm::tok::Tokenizer& tokenizer = loaded->tokenizer()` + (`model_loader.h:381`, returning `LoadedEngine::tokenizer_` at + `model_loader.h:528`); `api_server.cpp:1229` passes that object's ADDRESS + into the non-owning `ApiServer::tokenizer_` (`api_server.h:340`); and + `model_loader.cpp:1528` constructs `input_processor_(tokenizer_, …)`, whose + member is a REFERENCE (`input_processor.h:152`) to the same object. + `model_loader.cpp:1529` and `:1508` borrow it a fourth and fifth time, by + ADDRESS for `output_processor_(&tokenizer_)` and by REFERENCE for + `MakeNativeBackendFactory(tokenizer_, …)`, whose parameter is declared + `const tok::Tokenizer&` and which the header requires the object to outlive + (`backend_native.h:155-158`). The one owning instance is + move-constructed at `model_loader.cpp:1429`, and the count is of THIS path: + the C ABI takes a sixth borrow at `vllm_c.cpp:1395`, off the HTTP path this + probe drives. What the two endpoints do NOT share is the HANDLER around it: + body parse, string extraction, and `Encode` against + `EncodeWithSpecialTokens`. + + Read the three outcomes as follows: + + - **They disagree** — the difference is in the two HANDLERS, not between two + tokenizer states. + - **They agree at 915** — this does NOT isolate the tokenizer. Both handlers + sit behind the same httplib and the same nlohmann, so a common parse or + transport truncation and an ARM tokenizer defect are indistinguishable at + this point. + - **They agree at 1024** — `## Stop conditions`. +2. **The discriminating test is `examples/tokenize` built natively on aarch64** + and run over the same prompt bytes, with no server, no HTTP and no JSON in + the path. It is the step that separates an ARM code-generation difference + from everything on the server path, and on an agreement at 915 it is the next + step rather than a fallback. +3. Whichever way it falls, the prompt that reproduces it is the smallest failing + test, and it enters through the HTTP entry point rather than by constructing a + tokenizer by hand. + +## Owed + +- **The aarch64 lane runs no tokenizer gate.** `.github/workflows/ci.yml:1086` + builds exactly four targets on `ubuntu-24.04-arm` — `test_cpu_isa_arm`, + `test_ops_matmul_elem`, `test_ops_quant_dot`, `test_ops_quant_repack`. The + tokenizer parity goldens never execute on ARM, so an ARM-only tokenizer defect + is unguarded by construction. The golden corpus itself is not the gap: it + carries 28 combining marks and passes on x86. +- The NFC deviation at `src/vllm/tokenizer/tokenizer.cpp:388-391` is accepted + rather than applied. Neither the golden corpus nor any other gate feeds it + non-NFC text, so the deviation is unexercised rather than passing. + +## Consequence for the campaign's numbers + +`total_token_throughput = (total_input + total_output) / duration` +(`serve.py:734`) takes `total_input` from the server's own report. Our arm's +`total_input` is short by the missing prompt tokens, so OUR OWN published figure +is deflated against the corpus the client actually built, and the axis is not +comparable between the two arms at all. `.agents/benchmark-record.md:24201` +already annotates the oracle's cell "do not set this beside ours" for that +reason. + +This spec corrects OUR numerator and stops there. Both columns are our arm: + +| our leg | as published | with input = 1024 x N, the corpus the client built | correction | +|---|---|---|---| +| c1, mean of 3 | 38.4819 | 39.6404 | +3.01 % | +| c8, mean of 3 | 195.9628 | 203.6222 | +3.91 % | + +**No ours-over-vLLM ratio is derived here, and none may be derived from these +numbers.** `tools/bench/gpu_clock_state.py compare` returned +`PAIRING_VERDICT=DISCARD` on all three c1 pairings; `docs/BENCHMARKS.md:216` +records the c1 ratio as OWED for that reason; and +`.agents/benchmark-record.md:24229` states that no ours-over-vLLM ratio is +derived from those absolutes, here or anywhere else. A number the clock gate +discarded is not a number, and a corrected numerator does not turn one into a +number. Nor would such a ratio be independent evidence if the gate ever allowed +it: with the inputs imputed equal on both arms and the outputs equal at 768, a +corpus-matched total-token ratio reduces algebraically to the ratio of the two +durations, which is exactly what `output_throughput` already divides. The speed +verdict is out of scope (`## Scope`). + +`output_throughput` is biased upward because the shortfall is real work our +server did not do: 3.29 % fewer prompt tokens at c1, 4.23 % at c8. Weighted by +the prefill share of wall time the bias is **0.10-0.15 % at c1** and **about +0.6 % at c8** — measured run-to-run CV over the three reps is **0.048 %** (c1) +and **0.251 %** (c8), so the bias exceeds the noise on both legs and is not +absorbed by it. (The campaign published **0.039 %** and **0.205 %**. That is the +SAME measurement in the population rather than the sample form, not an +independent check of it: the ratio is exactly the Bessel factor sqrt(3/2) = +1.2247 on three reps, and 0.039 % x 1.2247 = 0.048 %, 0.205 % x 1.2247 = +0.251 %.) + +One correction to the campaign's record: the absence of a +`WARNING: tokenizer mismatch` line is NOT evidence that the alignment pass found +the tokenizers in agreement. `bench.sh:237` pipes the client through `tail -40`, +and both that warning and `WARNING: /tokenize unavailable` are printed before the +result block, so both are cut. The saved logs cannot say which happened. Nothing +about the served counts turns on which it was, because `_fix_one` is inert at +915 either way (item 6). + +## Reproduction + +No GPU, no lease, no download. `python3` with `numpy` + `transformers`, and a +C++20 compiler. + +1. Mirror `RandomDataset` at the pin (seed 0, `input_len` 1024, `range_ratio` 0, + `prefix_len` 0) against the checkpoint's tokenizer; dump the prompts. +2. Compile `src/vllm/tokenizer/{tokenizer,bpe,pretokenizer,unicode_data}.cpp` + with a stub for `vllm::GgufFile::FindKv` (needed only to link `FromGguf`) and + encode each prompt. +3. Compare ids against HF and counts against + `bench-20260819T035148Z/c8-r1.json`'s `input_lens`. + +## Stop conditions + +Stop and report `NEEDS_DECISION` if the probe shows `/tokenize` and +`/v1/completions` agreeing at 1024 on the box, because that would mean the +campaign's `usage` frames did not come from the tokenizer at all and the +attribution above is wrong. diff --git a/benchmarks/expert_stream_device_w0e.cpp b/benchmarks/expert_stream_device_w0e.cpp new file mode 100644 index 000000000..88b45bb3d --- /dev/null +++ b/benchmarks/expert_stream_device_w0e.cpp @@ -0,0 +1,286 @@ +// w0e_gen — ENG-EXPERT-STREAM-DEVICE W0e measurement harness (issue #1124). +// +// A THIN CLIENT of the public C ABI (include/vllm.h) only: vllm_engine_load + +// vllm_complete, with a custom logits processor used as a PURE OBSERVER. The +// processor edits nothing, so the argmax the sampler takes is byte-identical to +// a run without it; it exists because it is the only public surface that hands +// back the request's GENERATED TOKEN IDS, which is what G0-CORRECT gates on, +// and because it is invoked once per decode step, which is what gives the +// per-token arrival deltas G0-SPEED needs. +// +// The AUTHORITATIVE id list comes from vllm_complete_tokens (ABI v13), which +// fills a caller buffer with the generated ids. The processor's own token_ids +// view is recorded too, but it is NOT trusted for the gate: the capi suite +// records that under the async scheduler that view can LAG the emitted tokens, +// because the bookkeeping is fed back by update_from_output. Pass --prompt-ids +// to take that path; without it the harness falls back to vllm_complete, whose +// only id evidence is the processor's lagging view. +// +// It builds as the `expert-stream-device-w0e` target, and links `vllm::shared` +// rather than `vllm::vllm`, so it exercises the packaged C ABI and nothing else +// and cannot reach an internal header even by accident. It was originally left +// out of CMake beside marlin_moe_standalone.cpp, on the reading that a +// gate instrument is not a shipped capability. That is the wrong trade for THIS +// file: an instrument nothing compiles rots silently against the very ABI it +// measures, and the numbers in `.agents/benchmark-record.md` cannot be +// reproduced from a file that no longer builds. So the project builds it. +// +// cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release +// -DVLLM_CPP_CUDA=ON -DVLLM_CPP_CUDA_ARCHITECTURES=121a +// -DVLLM_CPP_CUTLASS_DIR=$HOME/cutlass -DVLLM_CPP_TRITON=ON +// cmake --build build --target expert-stream-device-w0e -j 6 +// +// One arm, with the lane on and its statistics line at every step so that the +// after-prefill snapshot and the final one are both in the log. (Written on one +// line: a backslash continuation inside a `//` comment is what -Werror=comment +// rejects, and this file is compiled with the project's flags now.) +// +// VT_GGUF_PREFAULT=0 VT_MOE_EXPERT_STREAM=1 VT_MOE_EXPERT_STREAM_SLOTS=8000 +// VT_MOE_EXPERT_STREAM_STATS_EVERY=1 ./build/examples/expert-stream-device-w0e +// --model .gguf --device cuda --max-tokens 32 --max-num-seqs 1 +// --prompt-ids 760,6511,314,9338,369 +// +// `VT_GGUF_PREFAULT=0` is load-bearing for a model larger than memory, and +// STATS_EVERY=1 is load-bearing for a short run: at the default 16 a healthy +// 32-token run prints no periodic line at all, which looks exactly like a dead +// lane. +#include "vllm.h" + +#include +#include +#include +#include +#include +#include +#include + +namespace { + +struct Obs { + std::vector t; // epoch seconds at each LP call + std::vector> ids; // ids seen at each LP call +}; + +double Now() { + return std::chrono::duration( + std::chrono::system_clock::now().time_since_epoch()) + .count(); +} + +// PURE OBSERVER: records and returns. It must not touch `logits`. +void ObserveLp(const int32_t* token_ids, int32_t n, float* logits, + int32_t vocab, void* ud) { + (void)logits; + (void)vocab; + Obs* o = static_cast(ud); + o->t.push_back(Now()); + o->ids.emplace_back(token_ids, token_ids + (n > 0 ? n : 0)); + std::fprintf(stderr, "[w0e] lp_call=%zu t=%.6f n_ids=%d\n", o->t.size(), + o->t.back(), n); + std::fflush(stderr); +} + +long ProcKb(const char* key) { + std::FILE* f = std::fopen("/proc/self/status", "r"); + if (f == nullptr) return -1; + char line[512]; + long v = -1; + while (std::fgets(line, sizeof(line), f) != nullptr) { + if (std::strncmp(line, key, std::strlen(key)) == 0) { + v = std::atol(line + std::strlen(key)); + break; + } + } + std::fclose(f); + return v; +} + +void PrintMem(const char* tag) { + std::printf("W0E_MEM %s VmRSS_kB=%ld VmHWM_kB=%ld\n", tag, ProcKb("VmRSS:"), + ProcKb("VmHWM:")); + std::fflush(stdout); +} + +const char* NextArg(int argc, char** argv, int& i) { + if (i + 1 >= argc) { + std::fprintf(stderr, "w0e_gen: missing value after %s\n", argv[i]); + std::exit(2); + } + return argv[++i]; +} + +} // namespace + +int main(int argc, char** argv) { + std::string model, prompt = "The capital of France is", tag = "run"; + std::vector prompt_ids; + int32_t device = 0; // 0 auto, 1 cpu, 2 cuda + int max_tokens = 33; + long long kv_bytes = 0; + int max_num_seqs = 1; + + for (int i = 1; i < argc; ++i) { + const std::string f = argv[i]; + if (f == "--model") { + model = NextArg(argc, argv, i); + } else if (f == "--prompt") { + prompt = NextArg(argc, argv, i); + } else if (f == "--tag") { + tag = NextArg(argc, argv, i); + } else if (f == "--prompt-ids") { + const std::string s = NextArg(argc, argv, i); + size_t p = 0; + while (p < s.size()) { + size_t q = s.find_first_of(", ", p); + if (q == std::string::npos) q = s.size(); + if (q > p) prompt_ids.push_back(std::atoi(s.substr(p, q - p).c_str())); + p = q + 1; + } + } else if (f == "--max-tokens") { + max_tokens = std::atoi(NextArg(argc, argv, i)); + } else if (f == "--kv-cache-memory") { + kv_bytes = std::atoll(NextArg(argc, argv, i)); + } else if (f == "--max-num-seqs") { + max_num_seqs = std::atoi(NextArg(argc, argv, i)); + } else if (f == "--device") { + const std::string d = NextArg(argc, argv, i); + if (d == "auto") { + device = 0; + } else if (d == "cpu") { + device = 1; + } else if (d == "cuda") { + device = 2; + } else { + std::fprintf(stderr, "w0e_gen: bad --device %s\n", d.c_str()); + return 2; + } + } else { + std::fprintf(stderr, "w0e_gen: unknown flag %s\n", f.c_str()); + return 2; + } + } + if (model.empty()) { + std::fprintf(stderr, "w0e_gen: --model required\n"); + return 2; + } + + std::printf("W0E_TAG=%s\nW0E_DEVICE=%d\nW0E_ABI=%d\nW0E_MAX_TOKENS=%d\n", + tag.c_str(), static_cast(device), + static_cast(vllm_abi_version()), max_tokens); + std::printf("W0E_PROMPT=%s\n", prompt.c_str()); + std::fflush(stdout); + PrintMem("start"); + + vllm_model_params mp = vllm_model_params_default(); + mp.model_path = model.c_str(); + mp.device = device; + mp.max_num_seqs = max_num_seqs; + // 0.0 is the ABI's "unset" spelling; vllm-cli does the same so a plain run + // does not look like an explicit ask (#1165). + mp.gpu_memory_utilization = 0.0; + if (kv_bytes > 0) mp.kv_cache_memory_bytes = kv_bytes; + + const double t_load0 = Now(); + std::printf("W0E_LOAD_START=%.6f\n", t_load0); + std::fflush(stdout); + vllm_engine* eng = nullptr; + vllm_status st = vllm_engine_load(&mp, &eng); + const double t_load1 = Now(); + if (st != VLLM_OK) { + std::printf("W0E_LOAD_STATUS=%d\nW0E_LOAD_ERROR=%s\n", static_cast(st), + vllm_last_error()); + std::printf("W0E_LOAD_SECS=%.3f\nW0E_RESULT=LOAD_FAILED\n", + t_load1 - t_load0); + std::fflush(stdout); + return 1; + } + std::printf("W0E_LOAD_SECS=%.3f\nW0E_LOAD_END=%.6f\n", t_load1 - t_load0, + t_load1); + std::fflush(stdout); + PrintMem("after_load"); + + Obs obs; + vllm_sampling_params sp = vllm_sampling_params_default(); + sp.temperature = 0.0f; // greedy + sp.max_tokens = max_tokens; + sp.logits_processor = &ObserveLp; + sp.logits_processor_user_data = &obs; + + vllm_completion out{}; + std::vector out_ids(256); + int32_t n_out_ids = 0; + const bool tokens_path = !prompt_ids.empty(); + std::printf("W0E_ENTRY=%s\n", tokens_path ? "vllm_complete_tokens" : "vllm_complete"); + if (tokens_path) { + std::printf("W0E_PROMPT_IDS_N=%zu\nW0E_PROMPT_IDS=", prompt_ids.size()); + for (size_t i = 0; i < prompt_ids.size(); ++i) { + std::printf("%s%d", i == 0 ? "" : ",", prompt_ids[i]); + } + std::printf("\n"); + } + std::fflush(stdout); + const double t_gen0 = Now(); + if (tokens_path) { + st = vllm_complete_tokens(eng, prompt_ids.data(), + static_cast(prompt_ids.size()), &sp, + out_ids.data(), + static_cast(out_ids.size()), &n_out_ids, + &out); + } else { + st = vllm_complete(eng, prompt.c_str(), &sp, &out); + } + const double t_gen1 = Now(); + std::printf("W0E_GEN_STATUS=%d\nW0E_GEN_SECS=%.3f\n", static_cast(st), + t_gen1 - t_gen0); + if (st != VLLM_OK) { + std::printf("W0E_GEN_ERROR=%s\nW0E_RESULT=GEN_FAILED\n", vllm_last_error()); + std::fflush(stdout); + vllm_engine_free(eng); + return 1; + } + std::printf("W0E_PROMPT_TOKENS=%d\nW0E_COMPLETION_TOKENS=%d\nW0E_FINISH=%s\n", + out.prompt_tokens, out.completion_tokens, + out.finish_reason != nullptr ? out.finish_reason : "(none)"); + std::printf("W0E_TEXT_BEGIN\n%s\nW0E_TEXT_END\n", + out.text != nullptr ? out.text : ""); + + // Per-step arrival: LP call k fires when step k's logits are ready, i.e. at + // the moment token k becomes available. k == 1 is the end of prefill (TTFT). + std::printf("W0E_LP_CALLS=%zu\n", obs.t.size()); + for (size_t k = 0; k < obs.t.size(); ++k) { + const double prev = (k == 0) ? t_gen0 : obs.t[k - 1]; + std::printf("W0E_STEP k=%zu t=%.6f dt=%.6f n_ids=%zu\n", k + 1, obs.t[k], + obs.t[k] - prev, obs.ids[k].size()); + } + // AUTHORITATIVE: the ids vllm_complete_tokens wrote. + if (tokens_path) { + std::printf("W0E_OUT_IDS_N=%d\nW0E_OUT_IDS=", n_out_ids); + for (int32_t i = 0; i < n_out_ids; ++i) { + std::printf("%s%d", i == 0 ? "" : ",", out_ids[static_cast(i)]); + } + std::printf("\n"); + } + // ADVISORY: the longest id prefix the processor's view showed. Recorded so a + // reader can see the lag rather than infer it; never the gate's evidence. + if (!obs.ids.empty()) { + size_t best = 0; + for (size_t k = 0; k < obs.ids.size(); ++k) { + if (obs.ids[k].size() >= obs.ids[best].size()) best = k; + } + const std::vector& ids = obs.ids[best]; + std::printf("W0E_LP_IDS_N=%zu\nW0E_LP_IDS=", ids.size()); + for (size_t i = 0; i < ids.size(); ++i) { + std::printf("%s%d", i == 0 ? "" : ",", ids[i]); + } + std::printf("\n"); + } + PrintMem("after_gen"); + std::printf("W0E_RESULT=OK\n"); + std::fflush(stdout); + + vllm_completion_free(&out); + vllm_engine_free(eng); + PrintMem("after_free"); + std::fflush(stdout); + return 0; +} diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 333863f9b..60aefb83a 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -6,7 +6,7 @@ |---|---|---|---| | **Weight load (`ENG-LOAD-DIRECT-UPLOAD`, #150)** | Qwen3.6-27B bf16 (50.098 GiB), GB10 Vulkan, same binary both arms, interleaved under one GPU lock | **Load phase 1.54x warm** (19.27 -> 12.48 s), **1.61x cold** (52.62 -> 32.75 s); bytes moved **100.196 -> 81.260 GiB**. Every ON leg beat every OFF leg | byte-identical; 6/6 token-exact | | **Expert streaming vehicle (`ENG-EXPERT-STREAM`, [#912](https://github.com/mudler/vllm.cpp/issues/912))** | Qwen3.8-2.4T-A95B `UD-Q1_0` (370 GiB) on ONE GB10, 119 GiB, c1 | **Loads and generates** (streaming OFF): resident **62 GiB**, decode **66.7 s/tok**. Experts BORROW the mmap; ~6.7 GB/token as 4 KiB faults, **~50x** off NVMe. Streaming-ON decode **VOID** (#912 F1); re-measure owed | correct answer; no oracle runs this | -| **Device expert slices (`ENG-EXPERT-STREAM-DEVICE`, [#1124](https://github.com/mudler/vllm.cpp/issues/1124))** | Qwen3.8-2.4T-A95B `UD-Q1_0`, `--device cuda` vs `cpu`, one GB10, greedy, 32 tokens | Decode **PENDING**: W0e queued, no floor. W0a **MEASURED** (`dgx:gpu0`, `PAGEABLE_OK`): pageable=1, integrated=1, a kernel read+wrote a 2,490,368 B host-vector slot, 2.06-2.28x. [record](../.agents/benchmark-record.md) | correct answer; no oracle runs this | +| **Device expert slices (`ENG-EXPERT-STREAM-DEVICE`, [#1124](https://github.com/mudler/vllm.cpp/issues/1124))** | Qwen3.8-2.4T-A95B `UD-Q1_0` 370 GiB, `--device cuda` vs `cpu`, one GB10, greedy 32 tokens, **4000 expert-stream slots**, 2026-08-18, [record](../.agents/benchmark-record.md) | CPU **MEASURED**, replaces #912 F1's VOID: **11.05 s/token** steady, rep 2's median, rep 1 11.22 (1.5%), live cache. 8000 slots is 3.6-4.1x slower. CUDA: loads, dies in forward 1, 0 steps (#1299). W0a `PAGEABLE_OK` | correct answer; no oracle runs this | | **Structured state record (active)** | v1 scalar + relational + Git-history contracts | No benchmark. At `776c56f1`: 157 imports = 3,231,342 exact bytes; append preserved all 156 wrappers/rows. 95 tests: validator/core 44 (checker 20 + core 24), NOW 18, migration 22, cutover 11. New raw-row mutation guard. | n/a | | **Binary release (ACTIVE; Windows pre-alpha pending)** | v0.0.2 shipped eight primary archive/checksum/provenance triplets + two indexes (26 assets) from source SHA `7020de93652ca920424a10ac5255b34810dd2f24`, run `31466516224` | Windows W14-W16 implemented. **PENDING:** native hosted gates, merged-SHA ten-tuple dry run, matching-hardware evidence, v0.0.3-pre.1 publication, 32-asset audit | W12 optional/non-primary | | **Container images (ACTIVE; arm64 cuda verified on GB10 + Orin 2026-08-11)** | `ENG-RELEASE-CONTAINERS` ([spec](../.agents/specs/container-images.md)) | cpu amd64 783 MB; cuda arm64 **1.71 GB**. GB10 `sm_121a`: `/health`+`/version`+SIGTERM on `--gpus all`. Orin `sm_87` (Tegra): Qwen3-0.6B **generates**, GPU **GR3D 95-97%** | n/a | diff --git a/docs/STATUS.md b/docs/STATUS.md index 2c42b4439..d9987ec55 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -164,7 +164,7 @@ token-for-token correctness against the pinned oracle. | LoRA / multi-LoRA adapters | In progress (W1+W2 CPU runtime), not yet usable end-to-end | Highest-demand missing feature. W1+W2 landed the punica CPU brick (shrink/expand, `-1`-slot skip) plus packed adapters and the wrapped layer family (merged qkv/gate_up slices, TP slicing, embedding and logits LoRA). CPU-gated vs double references, RED-first: `test_punica_cpu` 8/8 + `test_lora_layers` 16/16 (4,647 assertions); the fully-sharded (S-LoRA) apply REFUSES at tp_size>1 instead of returning a partial delta. W3-W7 (mapping, adapter load, manager, endpoints, GPU kernels + model gate) are in `.agents/specs/lora-adapter.md`. No model can be served with an adapter yet | | Safetensors loading | Supported | Both gate models plus every registered dense/MoE family | | GGUF loading (F32/F16/BF16/Q4_0/Q8_0/Q2_K/Q3_K/Q4_K/Q5_K/Q6_K/IQ2_XXS/IQ3_XXS/IQ2_S/MXFP4/NVFP4) | Supported; compute-in-quant (keep-quant) on CPU AND now CUDA for the six K-block encodings PLUS Q2_K/IQ2_XXS/IQ3_XXS (DeepSeek-V4 W8, 2026-07-29 - the FIRST CUDA keep-quant GGUF k-quant GEMM `KERNEL-QUANT-CIQ-GEMM-CUDA`, MMVQ-style dequant-in-kernel, GB10-gated 92401/92401 vs the CPU oracle, so a CUDA runner keeps blocks compressed and dots them on the GPU instead of the ARM cores); **NVFP4 now COMPUTES IN FP4 on CUDA for the dense-MLP and full-attention projections (2026-07-29, `CLAIM-GGUF-NVFP4-COMPUTE`), no longer materialize-only** | Weights in six block encodings stay compressed from file to matmul on CPU (no BF16 expansion). NVFP4 (ggml type 40) DEQUANTIZES, including the per-tensor (per-expert) `.scale` sidecar the container keeps outside the blocks; gated BIT-EXACT against the compressed-tensors NVFP4 path on real Qwen3.6-27B bytes from both containers. **It no longer expands to bf16 on CUDA:** an NVFP4 matmul/expert weight is REPACKED at load into the same (`weight_packed [N,K/2]`, `weight_scale [N,K/16]`) operand pair the compressed-tensors path produces - a pure byte permutation, gated BYTE-IDENTICAL against that container - and the existing `vt::MatmulNvfp4*` kernels run on it, so no new kernel exists and no numerics are re-derived. Covers the dense MLP + full-attention q/k/v/o and the MoE shared/routed experts; the GDN `in_proj_*` family and `ssm_out` still expand (the V-head reorder rewrites their layout) and a CPU build still expands everything - the documented `part` subset. **MEASURED GB10 (2026-07-29), same-binary A/B, 2 reps/arm:** peak RSS **50.8 -> 25.7 GiB**, load-and-generate **1:58 -> 0:41**; the 256 moved projections cost 35 840 MiB expanded vs 10 080 fp4-resident (3.56x). **The safetensors-sibling divergence CLOSES:** the fp4 arm is token-IDENTICAL over 24 greedy tokens where the same binary's bf16 arm diverges at index 4. REPORTED, not gated: the containers are not the same model - the GGUF NVFP4-quantizes 192 GDN `in_proj` tensors the safetensors keeps BF16 (mean rel. weight error ~0.18) and their activation global scales differ, so identity is not guaranteed and a cross-container throughput arm is invalid. SACRED gates unmoved: `test_qwen27_paged_engine` 235/235, `test_qwen36_paged_engine` 315/315. **The MoE (35B) stacked-expert arm is HARDWARE-GATED too (2026-07-29)**: the real 35B A3B NVFP4 GGUF loads and generates through the fp4 path, its 120 routed-expert stacks x 256 experts repack to the modelopt safetensors' operands with ZERO differing bytes over 840 sampled (tensor, expert) slabs, and all 840 per-expert `.scale[e]` are bit-identical to that expert's `weight_scale_2` - the scale INDEXING, mutation-proved against a `scales[0]`-for-all and an expert-0-slab-for-all mutant. Same-binary A/B: peak RSS 68.5 -> 22.7 GiB (3.01x), load-and-generate 1:51.9 -> 0:28.8, tokens IDENTICAL (the 35B routed experts run the W4A16 grouped GEMM in both arms). Recorded OPEN: this case's 24-token greedy stream is NOT run-to-run stable (1 of 3 `use_a16` and 1 of 4 reference runs differed), so the binding results are the weight-level byte identity and the residency audit, not token-exactness; `test_qwen36_paged_engine` is token-exact at ITS engine params, so the instability belongs to this case's configuration and attributing it is owed work. It also FIXED a latent defect the MoE arm made reachable: the two fp4 fused MoE blocks issued the router GEMM assuming the safetensors `[K,N]` gate layout and threw `matmul: inner dims mismatch` on the GGUF's `[N,K]`; `MoeRouterLogits` now branches on `nk` (inert for safetensors, SACRED unmoved). **Q2_K (id 10) + IQ2_XXS (id 16) DEQUANTIZE (2026-07-29, `CLAIM-DSV4-GGUF-LOADER`):** the ~2-bit types the single-Spark `DeepSeek-V4-Flash-GGUF UD-IQ2_XXS`/`UD-Q2_K_XL` vehicles use, ported 1:1 from llama.cpp `ggml-quants.c` (`iq2xxs_grid` codebook + signs; Q2_K nibble sub-scale/min), unit-gated on hand-derived bytes (`test_gguf_dequant` 15/15). Dequant-only (no vec_dot -> expand-bf16). A V4-GGUF model cannot RUN yet: the name map (tensor-manifest-blocked) + the V4 forward (W3-W8) remain. **Multi-shard split GGUF READING (2026-08-03, `CLAIM-GGUF-SPLIT-SHARDS`):** `GgufFile::Open` now transparently stitches llama.cpp `gguf-split` shards (`...-00001-of-00003.gguf`) — every shard mmap'd, tensor tables merged, KV metadata from shard `00001`, sibling mappings kept alive by the primary so keep-quant mmap-borrows stay valid across shards (`OwnsSpan` is shard-aware); `VT_GGUF_NO_SPLIT=1` opts out; unit-gated (`test_gguf` split-merge / no-split / count-mismatch cases, 33/33 local). This unblocks the real 3-shard `unsloth/DeepSeek-V4-Flash-0731 UD-IQ2_M` (~91 GiB), whose layout is the NATIVE `deepseek4` arch — per-block `ffn_gate_tid2eid` hash tables (hash layers 0/1/2) + `hc_*` MHC + DSA compressor/indexer are all PRESENT (name-map 1328/1328), `vocab_size` derives from `token_embd` — NOT a standard llama.cpp conversion, so no loader-layout change is owed. It now loads THROUGH 1324/1328 tensors; the sole remaining gap is 4 routed-expert slabs quantized with IQ2_S (id 22, ×2) + MXFP4 (id 39, ×2) — encodings we have GGUF block traits for but no keep-quant vec_dot, so they hit the expand→dequant path which lacks them. Expanding those 4 expert tensors to bf16 would add ~17 GiB (~106 GiB total → OOM-reboot risk), so the memory-safe fix is an IQ2_S+MXFP4 keep-quant kernel (CPU dequant dispatch + `iq2s_grid` + CUDA `DotSuperblock`), spec'd as the next brick **IQ2_S (id 22) + MXFP4 (id 39) DEQUANTIZE + KEEP-QUANT on CPU (2026-08-03, `CLAIM-DSV4-UDIQ2M-QUANT`, off-GPU):** the extra per-tensor "dynamic" encodings the `unsloth/DeepSeek-V4-Flash-GGUF UD-IQ2_M` checkpoint mixes into its last routed-expert slabs (IQ2_S `ffn_gate/up` dotting Q8_K, MXFP4 `ffn_down` dotting Q8_0) — ported 1:1 from llama.cpp `ggml-quants.c` @ 237ad9b96 (`iq2s_grid` 1024-entry codebook + DIRECT sign bytes; MXFP4 `kvalues_mxfp4` + `e8m0_to_fp32_half` micro-scaling, distinct from the compressed-tensors `E8M0ToF32` NVFP4 path). CPU dequant + keep-quant `vec_dot`, unit-gated on hand-derived golden bytes (`test_gguf_dequant` 17/17), an INDEPENDENT f64 dequant-then-dot + GEMM NMSE (`test_ops_quant_dot` 19/19), and keep-quant routing (`test_gguf_keep_quant` 37/37) — all CPU-green, so UD-IQ2_M's four previously-`unsupported ggml type 22/39` slabs now load COMPRESSED (no ~17 GiB bf16 expansion that OOM-reboots the box). CUDA: the IQ2_S device `DotSuperblock` is wired into the Q8_K grouped-MoE GEMM and now **CUDA-BUILT + LINKED on GB10 (sm_121a, CUDA 13.0, `-Werror`, 2026-08-03 integration)** — it compiles clean and the merged binary links; MXFP4's device dot (`DotMXFP4`) is written but NOT wired (Q8_0-activation needs a separate 32-block GEMM) so it is marked `[[maybe_unused]]` to keep the ready math without tripping nvcc #177-D, and on GPU MXFP4 CPU-fallbacks like Q4_0/Q8_0. The V4-GGUF forward + a real UD-IQ2_M GPU load/coherence run are owed | -| Larger-than-pool GGUF on an accelerator (`ENG-EXPERT-STREAM-DEVICE`, [#1124](https://github.com/mudler/vllm.cpp/issues/1124)) | W0a measured `PAGEABLE_OK` on `dgx:gpu0`; W0b-W0d implemented and unit-gated; W0e still queued, so no decode number exists and none is claimed | A 369.96 GiB GGUF refuses on `--device cuda` against a 119.631 GiB pool (#1123). W0 serves `*_exps` slices from the HOST slot store, keyed on a PROBED `host_memory_is_device_addressable()` and keep-quant towers (#1378) | +| Larger-than-pool GGUF on an accelerator (`ENG-EXPERT-STREAM-DEVICE`, [#1124](https://github.com/mudler/vllm.cpp/issues/1124)) | W0a-W0e all RUN. It now **loads** on `--device cuda` and does **not** generate: forward 1 exhausts the box, 0 steps ([#1299](https://github.com/mudler/vllm.cpp/issues/1299)). CPU serves, 11.05 s/token at 4000 slots | W0 removed the #1123 refusal: `*_exps` from the HOST slot store on a PROBED `host_memory_is_device_addressable()`, keep-quant/keep-f16 only (#1378). The DENSE half blocks it now: 0.15 GiB dies where 18.55 GiB does | | AWQ / GPTQ quantization | W0 spike + W1 CPU INT4 dequant primitive; not yet loadable end to end | INT4 unpack+dequant-to-bf16 for BOTH community formats, mirroring vLLM 1:1 (AWQ reverse-order `awq_triton.py`; GPTQ `qdq_4.cuh` with zero_offset v1/v2 + act-order g_idx). Unit-gated RED-first (hand-computed known bytes + double-precision roundtrip). NOT wired to a loader, no GPU Marlin compute, no model run yet: config recognizer (W2), Marlin GPU GEMM riding the vendored NVFP4 Marlin (W4), CPU e2e (W3), GPTQ 8/2/3-bit (W5) and MoE (W6) are named next bricks. See [.agents/specs/awq-gptq-quant.md](../.agents/specs/awq-gptq-quant.md) | | MXFP4 (compressed-tensors `mxfp4-pack-quantized`) | Compute PROVEN (#38); GQA-swap ON (#49); decode-graph+gate_up FUSION default-ON. `VT_MARLIN_DENSE` DEFAULT-ON (`KERNEL-MARLIN-DENSE-EXEC`): dense marlin 48-CTA byte-faithful (32B 0.000, 263/263), binding beats #51 every axis (c1 1.020, c8 0.969, mem 2.63x). **`QUANT-CT-MXFP4-FINAL-STACK` TERMINAL — both last levers exhausted: num_splits cap `VT_FA2_NSPLITS_CAP` gated-OFF (c1-only, self-corrects@c8; 32B strict char-identical); glue folds via `vt::FusedChain`; `FLASH-AUDIT` #68: c8 flash gap +12.5us/call is occupancy/L2-bound; `-use_fast_math` TRIED, REGRESSES flash (168.8→189.8), rejected. c1 1.020x PASS, c2-c8 0.962-0.969.** state.md | Shared with DeepSeek-V4-Flash + Kimi-K3 MXFP4 paths. CPU E8M0 dequant 5/5·1142. GPU W4A4 + MoE-expert e2e later | | CPU backend vs llama.cpp | 20-core Arm at floor, **denominator SUPERSEDED** (fork `237ad9b96`, owed a re-take vs stock `b10451`, #1003); RPi5/A76 below floor `GATING`; **x86_64 open on every axis** | Pi: AAPCS64 beats SDOT 3.66-5.08%; llama.cpp 2.17x pf / 1.53x dec faster (0.461x/0.653x); RSS -24.2% vs stock `b9892`; 64-tok byte-exact. BF16 GEMM open. x86_64 first measured 2026-08-11 (#433): peak RSS 1.0022x = hairline OPEN GAP (6.33 MB against us), throughput pending a quiet host, `G5` load-discipline gate failing, quant path portable-tier only (CIQ `G5`) | @@ -1045,6 +1045,17 @@ throughput does not. Output throughput is biased UP if the prompts were truly truncated, by roughly 0.13% at c1 and 0.4-0.6% at c8 — larger than the 0.039% and 0.205% CVs published beside them, so the bias is not inside the stated precision. +**That caveat now carries a row.** `SPEC-PROMPT-TOKEN-DIVERGENCE` is READY +([#1355](https://github.com/mudler/vllm.cpp/issues/1355), +[spec](../.agents/specs/prompt-token-divergence.md)): diagnosis committed, defect +not located, no product code changed. Off the box our tokenizer sources return +HuggingFace's IDS, not merely its counts, for all 48 regenerated prompts on +x86-64 at four optimisation settings, under ASan and UBSan, and over a real +socket. The count is not a reporting artifact: `usage.prompt_tokens` is the id +vector the engine consumed, so a short one means the model saw a different +prompt. What is left needs the ARM box, because the server binary was aarch64 and +that CI lane builds four kernel targets and runs no tokenizer gate at all. + **Our half of the debt is discharged. Neither cell is a ratio, and each is blocked for its own reason.** At c1 vLLM also completed every request (**4.2835 tok/s**, CV 0.033%) and both absolutes stand, but diff --git a/docs/USAGE.md b/docs/USAGE.md index 99289d35f..d0388517d 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -4408,7 +4408,7 @@ save. ```sh VT_MOE_EXPERT_STREAM=1 \ -VT_MOE_EXPERT_STREAM_SLOTS=8000 \ +VT_MOE_EXPERT_STREAM_SLOTS=4000 \ ./build/examples/vllm-cli --model /models/Qwen3.8-2.4T-A95B-UD-Q1_0-00001-of-00008.gguf \ --prompt "The capital of France is" --max-tokens 16 ``` @@ -4467,13 +4467,23 @@ Four limits, stated plainly rather than left to be discovered. file, against the residency this process resolved, and a file that mixes a kept tower with a staged one keeps the whole bound as well (issue [#1378](https://github.com/mudler/vllm.cpp/issues/1378)). -* **No speed claim is attached.** The decode measurement on the one machine that - answers capable has not run at the time of writing; `docs/BENCHMARKS.md` - carries it as `PENDING` and records what HAS been measured there, which is the - device probe itself. Device access to host-resident weights on that part has a - recorded penalty, and this lane reads ~6.95 GB of expert bytes per token that - way, so a CUDA arm slower than the CPU arm is a real possible outcome. Read - the benchmarks file before assuming the GPU is the faster arm here. +* **The load now succeeds and the generation does not, so there is still no + speed claim.** The measurement ran on the one machine that answers true + (GB10, 2026-08-18) and it split: `--device cuda` loads this checkpoint in + 255-272 s, which it could not do before, and then exhausts the machine inside + its first forward without emitting a token + ([#1299](https://github.com/mudler/vllm.cpp/issues/1299)). The slot arena is + measurably not the cause — a 64-slot 0.15 GiB arena fails exactly where an + 8000-slot 18.55 GiB one does — so raising or lowering + `VT_MOE_EXPERT_STREAM_SLOTS` will not get you a token. **Use `--device cpu` + for this checkpoint today.** That arm serves it at a steady **11.05 s/token + at 4000 slots**, which is the count both recipes in this section set and the + only count that figure holds for. The same binary at 8000 slots measured a + 39.98-45.40 s/token median over two runs, and the second of them consumed all + 30,625 MiB of the box's swap, so **more slots is not a free knob here**: the + extra 9.27 GiB of arena takes the free memory the borrowed 370 GiB expert + mapping is served out of. Read `docs/BENCHMARKS.md` before assuming the GPU is + the faster arm here. ### The same thing as config, and which one wins @@ -4487,7 +4497,7 @@ below that, where weights stay borrowed out of the file mapping. ```sh ./build/examples/vllm-server --model /models/Qwen3.8-2.4T-A95B-UD-Q1_0-00001-of-00008.gguf \ --offload-config '{"vllm_cpp":{"mmap":{"enabled":true,"prefault":false}, - "expert_stream":{"enabled":true,"slots":8000}}}' + "expert_stream":{"enabled":true,"slots":4000}}}' ``` | Key | Environment equivalent | Default | diff --git a/examples/CMakeLists.txt b/examples/CMakeLists.txt index 1e57bf5e4..8b376377a 100644 --- a/examples/CMakeLists.txt +++ b/examples/CMakeLists.txt @@ -361,6 +361,21 @@ target_link_libraries(vocoder-conv-ab PRIVATE vllm::vllm) target_include_directories(vocoder-conv-ab PRIVATE ${CMAKE_SOURCE_DIR}/src) vllm_cpp_set_warnings(vocoder-conv-ab) +# expert-stream-device-w0e: the W0e measurement harness for row +# ENG-EXPERT-STREAM-DEVICE (#1124). It exists because one generation has to +# yield three things together that no shipped command produces at once: the +# generated token ids, a per-step arrival timestamp, and the expert-stream +# counters at both ends of the run — and a live gate needs the DECODE-phase +# delta of those counters, not the total, because prefill exhaustion here is +# structural. Unlike the A/B harnesses above it is a pure client of +# `include/vllm.h` and links `vllm::shared`, so it stays inside the ABI-client +# rule and needs no `src/` include path. Not gated on CUDA: the CPU arm is the +# baseline the CUDA arm is compared against. +add_executable(expert-stream-device-w0e + ${CMAKE_SOURCE_DIR}/benchmarks/expert_stream_device_w0e.cpp) +target_link_libraries(expert-stream-device-w0e PRIVATE vllm::shared) +vllm_cpp_set_warnings(expert-stream-device-w0e) + # vulkan-gemm-ab: the VK-C tactic A/B — cooperative matrix vs the portable scalar # GEMM, SAME BINARY with VT_VULKAN_COOPMAT as the only variable. Built only in a # Vulkan build, because it links the Vulkan context directly to report which