measure(ENG-EXPERT-STREAM-DEVICE): W0e ran, the CPU arm has a live-cache number and the CUDA arm loads without generating (#1124, #1299) - #1414
Merged
Conversation
…che number and the CUDA arm loads without generating (#1124, #1299) W0e is the measurement W0b-W0d existed to make possible. It ran on one `rc hold` on `dgx:gpu0` against the real 369.97 GiB `Qwen3.8-2.4T-A95B UD-Q1_0`, and it split. The CPU arm is now MEASURED and replaces the VOID #912 F1 figure: 11.05 s/token steady, two reproductions, live cache, 4000 slots. The 8000-slot arm is 3.6x SLOWER, because an 18.55 GiB arena cannot sit beside 62 GiB of dense weights on a 119.631 GiB part without swapping, so more slots is not a free knob. The CUDA arm LOADS, which it could not do before this row, and then generates nothing: it exhausts the box inside its first forward, 0 steps, 7 attempts. The slot arena is measurably not the cause, because a 64-slot 0.15 GiB arena fails in exactly the place an 8000-slot 18.55 GiB one does. That is #1299, and it is the DENSE half of the model rather than the expert lane. No speed claim is attached to the CUDA arm and `docs/BENCHMARKS.md` says so. The harness earns its own paragraph, because it did not build. It landed in no CMakeLists, so nothing in the project compiled the instrument that produced this row's only numbers. Wiring it as the `expert-stream-device-w0e` target showed the file does not compile under the project's flags at all: three backslash continuations inside `//` comments, which `-Werror=comment` rejects. The recorded manual `g++ -O2` recipe carries no `-Werror`, so that recipe was the only thing that had ever built it, and it would have rotted against the very C ABI it measures with nobody noticing. It now builds, links `vllm::shared` so it stays a pure `include/vllm.h` client, and `ldd` shows `libvllm.so.0` and nothing internal. This branch is rebuilt on `main` rather than merged into it. #1377 landed as a squash, which destroyed this branch's merge base, and merging produced thirteen conflicts including product files this change does not touch. Reapplying the eight-file delta onto `main` leaves three record conflicts, each resolved by key: the benchmark log unions two disjoint sections, and the two keyed table rows take this row's newer state while keeping the keep-quant/keep-f16 mechanism #1378 corrected. Verified afterwards that every one of main's #1378 repairs survives in the spec and that nothing of main's is missing beyond the W0e status text this measurement supersedes. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
… 11.05 figure was measured at (#1124, #1299) A fresh review returned FAIL on `f51c8708`. This repairs its six findings and changes no product code. The one an operator would have paid for is the recipe. `docs/USAGE.md` set `VT_MOE_EXPERT_STREAM_SLOTS=8000` in the copy-paste block and `"slots":8000` in the `--offload-config` example, and sixty-six lines below promised 11.05 s/token. That figure is the 4000-slot arm; this row's own record measures the 8000-slot arm at a 39.98-45.40 s/token median with all 30,625 MiB of swap consumed, on a box where an out-of-memory kill takes the machine down rather than the process. Both recipes now set 4000, and every place that publishes 11.05 names 4000 beside it, so the number carries its own denominator. The causal claim beside those numbers was wrong and the columns say so. "An 18.55 GiB arena does not fit beside 62 GiB of dense weights on a 119.63 GiB box" does not close: 18.55 plus 62 is 80.55. Peak RSS moves 86.5 to 86.6 GiB for a 9.27 GiB arena delta, while `min avail` falls 16,347 to 6,941 MiB, about the arena delta, and swap rises 6,883 to 30,625 MiB. The record and the spec now read that as page-cache displacement, and say plainly that it is the best-supported reading of these columns rather than a proven mechanism, because the run sampled no page-cache size and no major-fault counter. The operational conclusion is unchanged: more slots is not a free knob. Three smaller repairs. The spec's `forced` anchor pointed at `qwen3_5.cpp:5389`, which is a W4 header comment; the counter is `++forced_` at `qwen3_5.cpp:5521` inside `Qwen35ExpertStream::Slice`, whose own comment states the reason the spec attributes to it. "11.05 s/token, 2 reps" is now stated as rep 2's median with rep 1 at 11.22; "3.6x slower" as the 3.56x-4.11x span of the four available pairings, with 4.05x for rep 1 against rep 1; and "4x more variable" as the max/min ratio it always was. Finally the #1299 row in `.agents/issue-index.md` moves from line 400 to the end of the file, because "append a row at the end" is the premise the `merge=union` driver is safe under. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
This was referenced Aug 19, 2026
localai-bot
added a commit
that referenced
this pull request
Aug 20, 2026
… twice, and `--device cuda` now decodes a 369.97 GiB checkpoint on a 119.631 GiB GB10 (#1299) (#1427) Replaces #1326. That pull request is correct and reviewed, and its base is gone: it was stacked on `row/ENG-EXPERT-STREAM-DEVICE-W0` at `95883dcae`, and #1377 squash-landed the same W0 work onto `main` as `5f4eb356e`. A squash destroys the merge base, so #1326 now presents product-code conflicts that are two unrelated histories describing one change rather than a disagreement. This branch is that same change rebuilt on `main`, with every one of `main`'s later repairs kept. ## What the defect was With W0's lane on, `Qwen3.8-2.4T-A95B UD-Q1_0` (369.97 GiB) LOADED on `--device cuda` on a 119.631 GiB GB10 and then exhausted the machine inside its first forward: zero decode steps, seven attempts, every one identical (#1299). The measurements name it rather than a reading of the code. A **0.15 GiB** slot arena died exactly where an 18.55 GiB one did, so the arena is not the cost. A 1-token prompt, whose protected set fits with no in-place fallback at all, behaved identically to a 5-token one, so prefill protection is not it. Growth was ANONYMOUS (`RssAnon` 8.1 to 61.4 GB) while file-backed stayed flat, so nothing was pinning the mapping. Host anon plus swap reached ~65 GB against system `used` ~119 GB, and the ~42 GB difference is device memory that unified memory does not charge to RSS. So every non-expert weight was resident twice: once as the host `OwnedTensor`, once as `ResidentWeight`'s device staging copy. On a part where device memory IS host memory, the second copy buys nothing and costs everything. ## The change `ResidentWeight` now takes the same branch W0c gave `KqExpertSlice`, on the same probed predicate: where `Platform::host_memory_is_device_addressable()`, it returns a tensor over `w.bytes.data()` instead of `Alloc` + `Copy` into `w.d_dev`. A DISCRETE device answers false, falls through, and is byte-identical to before, asserted by its own case rather than by inspection. Safety is by alignment, not by surveying kernels. The staging branch is a verbatim byte copy returning the same dtype, shape and dropped marker set, so the only thing a consumer can notice about the substitution is the pointer's alignment. `kDeviceAliasAlignment` is 256 because cuBLASLt's `CUBLASLT_MATMUL_PREF_MIN_ALIGNMENT_A_BYTES` defaults to 256 and this tree never sets it, which dominates every explicit pointer gate in the CUDA kernels -- at least seven across four files, the strictest asking 32. A plain `std::vector<uint8_t>` gives 16 and no more, so `MakeHostBytesDeviceAliasable` re-homes an OWNED misaligned buffer once into an aligned block: one memcpy that REPLACES the host-to-device copy it removes. A misaligned BORROW declines and stages instead, because copying a clean file-backed GGUF mapping into anonymous memory would create the residency this change exists to remove. Three fresh-review findings and one defect found in flow ride with it, each with its own red-first case: `MoeBlockBf16Cuda` released host mirrors the aliased device pointer table still pointed at, so the release now asks `HostMirrorIsRedundant`; the dense release guarded on `d_dev || d_dev_f32`, and `d_dev_f32` is an f32 UPCAST that can never stand in for the raw bytes; the `no host bytes` refusal fired ABOVE the device-copy memo and turned an always-served case into a throw; an already-aligned direct-upload borrow re-ran issue #150's `madvise(MADV_DONTNEED)` on every forward step; and #1320's i8mm-repack tripwire was missing on both arms of `ResidentWeight`. ## The result, and the gate that does not pass It decodes. **32/32 steps, peak RSS 97.75 GiB, decode-phase `exhausted` delta 0** (G0-LIVE PASS). G0-CORRECT FAILS: the CUDA ids diverge from the CPU arm at step 7 on a near-tie, where the CPU arm's own runner-up is exactly the token CUDA emitted, 1.4% behind, and one step later the margin is 0.1%. G0-SPEED is therefore VOID by this row's own stop condition and **no rate is claimed anywhere**. ## Can the alias move a logit? Measured on the target silicon, and no The probe compares what a consumer can actually see. `rc` job `7c7a05e9-be87-48f4-94ae-1bbe0340f063` on `dgx:gpu0` -- `NVIDIA GB10 sm_121`, driver 580.173.02, cuBLASLt 130101, the predicate re-derived in the job's own output as `pageableMemoryAccess=1 integrated=1` -- ran six checkpoint shapes crossed with both cuBLASLt formulations the dense path issues, 12 measurements, `PROBE_EXIT=0`, `PROBE_FAILURES=0`. A repeated heuristic call is identical 12/12; the tree's unset preference equals the documented 256 default 12/12; weakening the promise to 16 moves nothing 12/12; and `cublasLtMatmul` output is bit-exact between a `cudaMalloc` operand and a 256-aligned host block 12/12, `differing=0`. At least five distinct algorithm configurations appear across the twelve and they differ from the earlier `thor:gpu0` leg's, so the heuristic was re-resolved rather than replayed and the instrument discriminates. The structural reason needs no lease: `cublasLtMatmulAlgoGetHeuristic` takes no operand pointers, so alignment reaches it only through a preference this tree never sets. **So the alias does not cause the step-7 divergence.** Excluding one cause is not identifying another. What DOES cause it is unmeasured, and it is carried under `## Owed` with its next traceable step named: a two-arm dump of the step-7 forward that names the first differing tensor. ## What `main` already had, and what this rebuild kept Every repair #1377 and #1378 landed survives, unmodified. None of the files carrying them is in this diff: - `GgufExpertTowersReachSlotLane`, the fifth lane term, in `gguf_device_fit.{h,cpp}` and read by the loader. - `PeekRoute` promoted into `include/vllm/model_executor/model_loader/gguf_keep_quant.h`. - `HostMemoryIsDeviceAddressableFromAttrs` extracted into `src/vllm/platforms/platform.cpp` and gated over all four attribute pairs. Their gates are green with W0f in the tree, which is the live evidence rather than an inspection: `test_gguf_device_fit` 17 cases / 130 assertions, `test_gguf_device_fit_reach` 14 / 66, `test_platform` 14 / 114. Two more of `main`'s repairs are inside files this branch does touch, and both are kept: the `HostAddressable` RAII guard in `test_expert_stream_device_slot.cpp` (the one conflicting case now uses it, including for its discrete arm, instead of writing the flag directly), and `docs/USAGE.md`'s limit list, which grows from four limits to six rather than back to two, keeping the model-family bullet and the keep-quant OR keep-f16 residency bullet. #1414 then landed the W0e measurement while this branch was gating. Its facts are carried, not overwritten: the CPU arm's **11.05 s/token at 4000 slots** on a live cache is the standing figure and the 4000-slot count is what both recipes set, and the 8000-slot slowdown keeps `main`'s attribution -- the extra 9.27 GiB of arena takes the free memory the borrowed 370 GiB mapping is served out of, which is the page cache rather than an arena that fails to fit. ## Records, and one union-merge duplicate `docs/STATUS.md`, `docs/BENCHMARKS.md` and `docs/FEATURES.md` take one scoped row edit each; every other key is byte-for-byte unchanged. `.agents/benchmark-record.md` is a pure append of 208 lines with 0 deletions, and it now carries TWO sections rather than one merged one, because #1414's run and this one are different trees: `95883dcae` (loads, generates nothing) and `9c783a8be` (32/32 steps). The spec's `## Evidence` mirrors that split, and each side says which figures are its own and which must not be mixed. `.agents/issue-index.md` appends one row (#1320). It appended two, and the second was the union-merge duplicate `AGENTS.md` warns about: #1414 appended a `#1299` row on main and this branch appended its own, and `merge=union` kept both without a conflict. `scripts/check-agent-record.py` caught it. Main's row is kept byte-for-byte and this branch's is dropped, which is the only resolution the append-only rule allows. Two record edits are consequences of this change rather than part of it, named so a reviewer does not have to work out why they are in the diff. Adding 198 lines to `qwen3_5_weights.h` moved `struct Fp8Weight` from line 342 to line 540, which `scripts/check-agent-record.py` reported as a stale-anchor regression (33 against a baseline of 32); the three citations of it are repointed, in `.agents/quantization-matrix.md` (both the text and the `#L` fragment, which is the one the checker reads) and in two specs. And appending a measurement to `.agents/benchmark-record.md` obliges `docs/FEATURES.md` under `scripts/check-doc-checkpoint.py`, so the routed-expert-streaming row now says the staging device decodes and that its token gate fails. That is #1387's shape, caught before the commit was published rather than after. ## The fresh review returned FAIL, and this is the repair Five findings, four of them in the record rather than in the product code, which survived eleven of twelve mutations and is unchanged apart from one comment. **The alignment constant had no gate (MEDIUM).** `kDeviceAliasAlignment` was 256 and every assertion about it in the tree is written `% vllm::kDeviceAliasAlignment == 0`, which is a tautology in the constant. The review lowered it to 16 and the whole suite reported `SUCCESS` at exit 0 -- the promise this change's entire safety argument rests on could be deleted with no gate saying so. That argument is that cuBLASLt is PROMISED 256: `CUBLASLT_MATMUL_PREF_MIN_ALIGNMENT_A_BYTES` defaults to 256 and this tree never overrides it, with `grep -rn MIN_ALIGNMENT src/vt/` empty against a positive control on `CUBLASLT_MATMUL_PREF_MAX_WORKSPACE_BYTES` in the same file. A case in `test_resident_weight_host_addressable.cpp` now pins the literal. RED-FIRST, and re-taken on the pushed head `ac5e1ad8f`. Before the case, the constant at 16 left every one of the eight suites GREEN at exit 0 with identical counts -- the reviewer's mutation 5 reproduced exactly. With the case, the same mutation gives compile rc 0, `git diff --stat` 1 file / 1 insertion / 1 deletion, `test cases: 13 | 12 passed | 1 failed`, `Status: FAILURE!`, exit 1. Restored byte-for-byte by sha256 (`7f1d10b8...`), rebuilt, 13 cases / 72 assertions / `SUCCESS!` / exit 0. The reviewer's mutation to 24 still reds with its original `std::bad_alloc` and SIGABRT intact beside the new failure -- exit 134, 5 cases failed while the `assertions:` line read 1 failed of 16, which is why the `Status:` line is what was read. The pin masks nothing, and the existing `%` assertions stay because they check the buffers. **The header stated two things that were false (MEDIUM).** Its rationale block said this row's CUDA-versus-CPU token divergence IS the two arms' GEMM arithmetic, which the spec explicitly forbids asserting, and said the GB10 leg is owed when the spec and `.agents/benchmark-record.md` both record it as RAN. The block now says what the probe established -- the alias is EXCLUDED as the cause, on the target silicon -- and says plainly that the cause is not identified. Three public projections had carried that same error outward, so the correction does not stop at the header. `docs/BENCHMARKS.md` and `docs/STATUS.md` both read "G0-CORRECT FAIL on a near-tie ... not the alias"; both now say the cause is NOT identified, in fewer characters than before. `docs/USAGE.md` said it hardest, in a user-facing document: "the divergence is a measured near-tie rather than a disagreement about the model" and "the two arms run genuinely different GEMM kernels". What is measured there is the MARGIN, and that is what it now says. **The disowned ratio is deleted rather than restated (LOW).** It was correctly labelled VOID and not claimed, and no rate reaches any public document. But digits survive a copy-paste and a disclaimer does not, and this repository has watched a disowned number become a quoted one. Both medians stay in the table, so anyone entitled to the quotient can divide; what is gone is the pre-computed string. `docs/FEATURES.md` gets back "not throughput", which stops a reader treating the lane as a speed feature and which survives in `docs/USAGE.md`. **One anchor went stale inside this pull request.** The rewritten rationale block adds 23 net lines above `struct Fp8Weight`, and `check-agent-record.py` caught `qwen3_5_weights.h:540` going stale, 33 against a baseline of 32. The anchor is repaired, never the baseline: the struct moved 540 -> 563, and `sed -n '540,553p'` before and `sed -n '563,576p'` after hash identically (`9f809a38...`), so the shift maps the cited range byte for byte rather than being re-derived by eye. Two prose citations the checker does not scan are corrected with it. ANCHOR-ROT is back to 38. ## NOT REACHED, and named here as `## Nothing lands dead` requires No CI gate reaches the alias branch through a production entry point. `test_expert_stream_wiring` enters `Qwen3_5Model::Forward` and the reachability mutation reds it, but it runs on the CPU device, where `ResidentWeight` returns at the `is_cpu()` early return about ninety lines above the alias branch. In CI the branch is reached only through `detail::StageWeightForTest`, a test-only seam. This is deliberate rather than an oversight. The branch is selected by `needs_weight_staging() && host_memory_is_device_addressable()`, and no CPU tier can register a platform that answers both -- exactly one machine this project can reach does. The device evidence is real and is the stronger of the two: the W0e run entered the branch **43,501 times** through `Qwen3_5Model::Forward` on `dgx:gpu0`. It is simply not repeatable in CI, and closing it needs either a GPU CI lane on a probed-capable part or a production entry point a fake staging platform can drive end to end. - What is unreached: the `MakeHostBytesDeviceAliasable` alias branch of `ResidentWeight`, on any CI runner. - Row that owns the wiring: `ENG-EXPERT-STREAM-DEVICE`. - Issue that tracks it: #1299 - Listed under `## Owed` in `.agents/specs/expert-stream-device-slots.md`. The disclosure was in the spec and in neither the pull request body nor any commit body. Because this repository sets `squash_merge_commit_message = PR_BODY`, this body IS the landed commit message, so a disclosure absent from it is a disclosure that does not land. ## `origin/main` merged, and the two keyed records resolved BY KEY GitHub reported this pull request CONFLICTING while `git merge-tree` read clean locally, which is the forge ignoring the `merge=union` driver on the append-only `.agents/issue-index.md`. Materialising the merge here clears it. `origin/main` moved TWICE during this repair, so there are two merges: `63d87805c` (#1399) and then `7f9c6802e` (#1436, #1428), which landed while the gate was running. Both are clean, and both had their keyed records verified by key rather than taken on the driver's word. `.agents/issue-index.md` is the exact union at each step, ending at 454 rows with every one of this branch's 452 and every one of main's 453 present, no extra row and no duplicate; main's #1396 and #1411 rows arrive and this branch's #1320 row stays. Eight files were co-edited across the two merges -- `include/vllm/model_executor/models/qwen3_5_weights.h`, `src/vllm/model_executor/models/qwen3_5_dense_weights.cpp`, `.agents/specs/qwen38-27b-quant-arms.md`, `tests/CMakeLists.txt` and the four public pages -- and each line main added was checked line by line to be present in the merged file: all are, and all five of this branch's edits in those pages survive. The one line either merge drops relative to main is this branch's own W0f change at `9c30dfe7d`. `struct Fp8Weight` is still at `qwen3_5_weights.h:563`, so the repaired anchor is still correct after both. `check-agent-record.py` was re-run AFTER each merge, not before, because a shared summary counter byte-identical on both sides merges without a conflict while both row changes apply. Conflict markers were checked with `git grep` under a positive control, because four record checkers return 0 on a file that contains them (#1417). ## Gate Re-taken on the pushed head `f423df992`, from a FRESH build directory rather than incrementally, because the second merge brought main's source in and an incremental build masks `-Werror`. Clean-configure plus full build, `CMAKE_RC=0` and `BUILD_RC=0` read from the unpiped commands, 0 warnings, and `grep -c 'No space left on device'` = 0 on every log with 61 GB free -- stated because this box has hit 100% during a run of this row and an ENOSPC presents as a code verdict rather than as infrastructure. Eight focused suites, each with a non-zero case count and an explicit `Status: SUCCESS!`: | Suite | cases | assertions | exit | |---|---|---|---| | `test_expert_stream_device_slot` | 5 | 45 | 0 | | `test_resident_weight_host_addressable` | 13 | 72 | 0 | | `test_load_direct_upload` | 16 | 203 | 0 | | `test_expert_stream_wiring` | 4 | 882 | 0 | | `test_gguf_device_fit` | 17 | 130 | 0 | | `test_gguf_device_fit_reach` | 14 | 66 | 0 | | `test_platform` | 14 | 114 | 0 | | `test_gguf_keep_quant` | 39 | 6093 | 0 | `check-agent-record.py`, `check-public-doc-tables.py`, `check-issue-index-append-only.py` and `check-doc-checkpoint.py` all exit 0, run from this worktree's own `scripts/` copy because a checker resolves its root from its own path. A full `ctest` on the pre-merge head was 561 of 564, with three failures none of which is this change: `test_qwen3_5_decode_graph_seam` SEGFAULTs on `main` itself (#1403), `test_nemotron_h_paged_forward` throws `No valid attention backend for device type 0`, and `test_qwen3_5_moe_vision` failed under parallel load with `safetensors: empty file` on a fixed `/tmp` path while the disk was full, then passed 7 of 7 serially. `test_cpu_x86_llamacpp_floor` is not registered in this tree, so there is nothing to re-run for it. Issue: #1299 Row issue: #1124 Tripwire filed and covered in flow: #1320 Spec: `.agents/specs/expert-stream-device-slots.md` #1124, #1299 and #1320 all stay OPEN: the row is not finished, the correctness gate does not pass, and #1320's loader-policy half belongs to `QUANT-GGUF-KEEPQ-LOADER`. Do not merge this before a fresh review, and note that CI cannot give a verdict on this repository right now -- every completed workflow run is `cancelled` (#1285), so treat it as REMOTE_UNVERIFIED and never as green. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
localai-bot
added a commit
that referenced
this pull request
Aug 20, 2026
…#1211) The capability landed across #912, #1119, #1132, #1377, #1414 and #1427, and nothing a user reads said how to reach it. AGENTS.md binds the checkpoint pin to docs/USAGE.md, and revision 567d3e6ac26c5474b18311e619c04350fb9a5556 lived only in the agent record: USAGE.md named this checkpoint three times and pinned it zero times, so nobody outside .agents/ could reproduce a single number the project publishes about it. This lands the recipe as a fifth per-model section beside Muse Glimmer 30B, Nemotron-3.5-Lightning-30B, MiniMax-H3 and MiniMax-Music3. It LINKS the "Streaming routed experts from disk" section rather than restating it, so the schema, the precedence rule, the #1135 reach limits and the per-device limits each keep one home. No docs/examples/ directory. `examples/` in this tree is runnable C++ CMake targets and `docs/` is one topic per file listed in the README index, so a new markdown genre would have had exactly one member and would have duplicated facts that already have owners. ## The section was drafted on 18 August, and three changes have landed since The draft is reconciled against them rather than merged as written. The decode subsection quoted a warm 667.0 s time to first token and 44.2 s/token. Neither figure names its expert-stream state, and no record in the tree carries either one outside a single commit message and a single index row, so both are deleted. What replaces them is measured: the CPU arm runs at 11.05 s/token at 4000 expert-stream slots, with 8000 slots at 39.98 to 45.40 s/token and streaming off at 66.7 s/token. The streaming-off row carries no ratio against the other two, because it was taken on a different source tree on a different date. The recipe served with expert streaming off, which is not the arm any current figure was measured on. It now turns streaming on at 4000 slots, matching both recipes in the mechanism section. The subsection headed "The expert slot cache is off, and it has not yet earned being on" is gone. Its premise was the VOID steady-decode figure of #912 F1, and W0e replaced that VOID with a live-cache measurement. Its second premise, that the fill copies from a pointer into the mmap and so still takes the page fault it exists to avoid, is retracted as unestablished in the same record. What a reader still needs from it, that the default is off and that 8000 slots is worse than 4000, is already carried by the section this one links. No `--device cuda` speed number appears anywhere. That arm decodes the checkpoint as of W0f, its token gate against the CPU arm fails at token 7 on a measured near-tie, and docs/BENCHMARKS.md therefore carries G0-SPEED as VOID. The claim that `--offload-config` does not reach vllm-cli is deleted. It reaches vllm-cli, the server's generate and chat path, the server's pooling and embedding path, and the C ABI. The transcription-only path is the one that refuses the flag, and the mechanism section already says so. ## The shard count is settled against the artifact Two copy-paste commands named -00001-of-00008.gguf. A GGUF split writes the total into every member's name, so that is a different file name, and an operator gets a file-not-found after a 370 GiB download. The report is #1420. Settled from the mirrored copy, rather than from a document and rather than from the HuggingFace tree API, whose lfs.oid is fabricated for a gated repo: - shard 1's own metadata declares split.count = 10 and split.tensors.count = 1702, and it carries zero tensors; - shard 1's sha256 recomputes to b7770552b2ac24e7334c917bc92e90e218e87cfe29484db65e62e8ef2a60334d, equal to the download manifest, which also gives shard 2 as 2765517f833c736338d3ab34354e1c10eb8d79e62325f998285b435e5cf03dcd; - the ten files sum to exactly 397,256,393,248 bytes, that is 369.97 GiB. Both commands now name -00001-of-00010.gguf. Issue #1420 stays open for the operator, because this pull request repairs one document and that issue also asks whether any other document repeats the file name. The per-token arithmetic is re-derived from the same metadata rather than transcribed. block_count 93, expert_count 512 and expert_used_count 10 give 93 x 10 x 3 = 2790 slices at 2,490,368 bytes, which is 6.95 GB (6.47 GiB) per token, so a ~5 GB/s NVMe puts the floor at 1.39 s/token, about 0.72 tok/s, and not the 1.29 s and 0.77 tok/s a GiB-for-GB substitution produces. Closes #1194. ## The fresh review returned FAIL, and this is the repair The factual core was found sound: every number traced to a record, no unsourced figure, no CUDA speed claim, and no asserted cause for the step-7 divergence. Five findings sat on top of it. None of them moves a verified fact. **The issue was linked in two of the three required places.** AGENTS.md binds #1194 to the index row, the owning row's spec and this body, and only the index row carried it. No checker catches that, because check-agent-record.py is satisfied by the row naming an owning row ID. .agents/specs/expert-streaming.md now carries a section recording what landed, why the pin is settled against the artifact rather than against a document, and which binary produced the figures. A git grep for the issue link now returns the spec and the index, run beside a control that proves the pattern reports nothing for a number that is absent. **The section claimed to repeat nothing and repeated five facts.** "Nothing here repeats them" was false for the 4000-slot decode figure, the 8000-slot figures, the per-token expert bytes, the --device cpu guidance and the VOID speed axis. The copies agree today. The W0e re-measure this row owes would leave a corrected copy beside a stale one that still reads true, which is the shape that put an 8000-slot figure under a 4000-slot heading on the first revision of this change. Every copy the reader does not need is gone, and the opener now names the three facts the recipe repeats on purpose and says why a recipe cannot omit them: which device to use, the expert bytes a token reads, and the two streaming decode figures. The decode figures name their one record, ENG-EXPERT-STREAM-DEVICE W0e in .agents/benchmark-record.md, so a re-measure has one place to look. **The published recipe is not the binary that was measured.** Every figure came from benchmarks/expert_stream_device_w0e.cpp, a purpose-built C ABI client that reports the token ids, a per-step timestamp and the expert-stream counters together. The section publishes a vllm-server command. That difference is now disclosed beside the prompt and environment differences the section already disclosed, and it is a disclosure rather than a caveat: at seconds per token the server's HTTP and SSE framing sits far below the run-to-run spread the section prints. **A stated bound was false.** Rep 1's 11.22 against 11.05 is 1.5385%, so "agrees within 1.5%" is a bound the numbers do not meet. It now reads 1.54% above, recomputed rather than transcribed. **The gate evidence was pinned to a SHA that would never land.** The branch was behind origin/main, which made the trailer and commit-style gates SKIP while reporting nothing about this tree, and the recorded evidence named a commit one behind the head. main is merged, and both trailer gates now EXECUTE. ## Gates From this branch's own worktree, at caffae2, with origin/main at d0eff4f and verified an ancestor of that head. Every rc was captured unpiped. - scripts/agent-preflight.sh: rc 0, zero gates failed and zero gates skipped. commit-trailers and commit-style both report ok rather than SKIP. - check-agent-record.py: rc 0, ENGINE=168 MODEL=377 QUANT=84 KERNEL=52 BACKEND=85 ANCHOR-ROT=38. - check-public-doc-tables.py: rc 0. - check-issue-index-append-only.py: rc 0. - check-doc-checkpoint.py: rc 0. - check-commit-trailers.py --range origin/main..HEAD: rc 0. - check-commit-style.py --range origin/main..HEAD: rc 0. - All four record checkers resolve ROOT from their own __file__ to this worktree, asserted rather than assumed. - git grep for conflict markers over the whole tree: no match, run beside a positive control that proves the pattern matches. The merge of origin/main touched one keyed record. Both incoming commits append to .agents/issue-index.md and neither edits an existing row, so the merge applied four appends and changed nothing else. UNOWNED_HIGH_WATER in scripts/check-agent-record.py is byte identical on both sides and was not double-applied. No build was run and none is needed. No src/, include/, tests/ or CMake file is touched by this branch. ## What this does not settle The #1194 index row was rewritten in place on this branch, which the append-only rule permits only because the row has never been on main. Reviewers should read it as new rather than as an edit. docs/BENCHMARKS.md line 8 and docs/STATUS.md still describe the streaming-ON decode figure as VOID per #912 F1, while BENCHMARKS.md line 9 records 11.05 s/token as replacing that VOID. That contradiction is #1442, it sits in a record this pull request does not own, and it stays with the ENG-EXPERT-STREAM row. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
localai-bot
added a commit
that referenced
this pull request
Aug 20, 2026
…ge conflict marker (#1450) Four record gates return rc 0 on a `docs/STATUS.md` carrying a spliced merge conflict. Each measures its own budget, and none asks whether a merge tool wrote into the file at all, so a keyed row that is half one branch and half another satisfies every budget it is measured against. An earlier revision of #1414's branch carried exactly that file, the full record gate set ran, and it reported clean; a person caught it by reading `git diff --stat`. This adds one tree-scoped checker, in one place rather than four, and wires it where the pre-push record gates already run. ## The rule, and why the separator is conditional A line that starts with seven `<` and a space opens a hunk, and a line that starts with seven `>` and a space closes one. Either fails, anywhere in a tracked text file. A line of exactly seven `=` fails only when a start marker opened a hunk above it. The separator stays conditional because a bare row of `=` is legal markdown: it is the setext heading underline and it is a horizontal rule. Two independent narrowings keep the gate off ordinary text, and both were measured rather than assumed. **Exactly seven:** five shipped files already carry a rule of eight or more `=` (three logs under `docs/bench-evidence/gdn-replayssm-w0-20260818/` and two tokenizer corpora under `tests/parity/goldens/`), so a `^=+` rule would have been red on arrival. **Adjacency:** even a line of exactly seven is silent unless a real marker opened a hunk. The diff3 `|||||||` marker is deliberately absent. A diff3 conflict still carries the start and end markers, so nothing escapes, and a line of seven `|` is a plausible empty row in a repository whose records are wide markdown tables. ## No allowlist, and no per-change registration Every pattern is built by character repetition rather than written as a literal, so the checker and its suite carry no marker at column 0 and need no exemption from the gate they implement. The suite builds its fixtures in temporary git repositories. Nothing in the repository has to register itself with this gate, and no file accumulates a list that every change must append to. AGENTS.md § *Records* forbids that surface: if N concurrent pull requests edit file F, F is a lock. A document that quotes a conflict marker on purpose indents it. That remedy is printed in the failure message, and it is per instance and in the text, not a registry entry. ## RED-BEFORE The defect was re-derived rather than quoted, on a detached scratch worktree at `b537a5344`. Five lines were spliced into the `docs/STATUS.md` capability table: a start marker, a duplicated keyed row, a separator, the same row again, and an end marker. The mutation asserted its own application against the file's sha256 before and after, and `git diff --stat` read `docs/STATUS.md | 5 +++++`. ``` === check-public-doc-tables rc=0 === OK: docs/BENCHMARKS.md and docs/FEATURES.md are human-readable keyed tables, docs/STATUS.md is inside its size ratchet, and .agents/benchmark-record.md carries the append-only record. === check-agent-record rc=0 === agent record OK: ENGINE=168 MODEL=377 QUANT=84 KERNEL=52 BACKEND=85 ANCHOR-ROT=38 === check-doc-checkpoint --base b537a53 --head HEAD rc=0 === OK: public documents match the claims this change makes. === check-issue-index-append-only --base b537a53 --head HEAD rc=0 === OK: issue index append-only ``` The two range-scoped checkers ran over a detached scratch **commit**, not over the working tree. A working-tree mutation of a commit-reading checker returns 0 because the checker never reads the mutated bytes, and that 0 looks exactly like a gate that cannot detect the defect. The suite was written before the checker existed and was seen failing: `Ran 17 tests`, `FAILED (failures=17)`, rc 1. ## GREEN-AFTER Same tree, same splice, the new gate: ``` root: /dev/shm/red-1417 docs/STATUS.md:129: conflict start marker: <<<<<<< HEAD docs/STATUS.md:131: conflict separator inside the hunk opened at line 129: ======= docs/STATUS.md:133: conflict end marker: >>>>>>> origin/main conflict markers: 3 findings in 1 file; examined 3728 tracked text files (1077 binary, 1 symlink, 0 absent skipped, 4806 tracked paths) Resolve the merge before committing. A document that quotes a conflict marker on purpose must indent it, so that it does not start at column 0. rc=1 ``` On the clean tree of this branch: ``` root: /home/mudler/_git/vllm.cpp/.claude/worktrees/fix-1417 conflict markers: 0 findings in 0 files; examined 3733 tracked text files (1077 binary, 1 symlink, 0 absent skipped, 4811 tracked paths) rc=0 ``` **3739 tracked text files examined**, out of 4817 tracked paths: 1077 binary, 1 symlink and 0 absent were skipped, and each class is counted and printed rather than dropped in silence. (It read 3733 of 4811 before the last merge of `main`; both figures are the same run on different trees.) A gate that cannot say how many things it examined has not reported, so an examined count of zero exits 2 instead of printing a clean verdict, and a file the run could not READ exits 2 as well rather than being called clean. The report also names the resolved root, because a checker resolves its root from its own path and a run from a linked worktree can otherwise print OK about the shared checkout. The clean-tree case does not assert a floor. It asserts EQUALITY with git's own tracked text set, derived at read time from `git grep -I --name-only -e ''` plus the tracked files that have no lines at all. Both sides are re-derived on every run, so nothing is stored and nothing drifts; the two agreed byte for byte. The suite: `Ran 21 tests`, `OK`, `cases run: 21`, rc 0. The count is asserted by the runner itself, so a suite that ran zero cases exits 2 rather than printing OK. Cost: 0.29 s of CPU for the whole tree, against 0.9 s before the scan was moved off decoding 142 MB of text to bytes. Wall time on this box ranged 1.17 s to 4.60 s under load from two sibling sessions, which is why the spec's stop condition bounds CPU and not wall. ## The gate is armed, proven by mutation Each mutation was applied in a scratch worktree under `/dev/shm`, never in the reviewed tree. Each asserted that its bytes actually changed, printed the new sha256, compiled the result before running anything, and restored the file byte-identically afterwards. | Mutation | Suite | |---|---| | M1 the detection core deleted (`scan_text` returns nothing) | **RED**, 8 failures | | M2 the separator adjacency guard `open_at is not None` dropped | **RED**, 1 failure | | M3 the examined count deleted from the report | **RED**, 5 failures | | M5 the index dedupe removed (`return names`) | **RED**, 1 failure | | M6 an unreadable file filed as a finding again | **RED**, 1 failure | | M7 the file count split on the first colon again | **RED**, 1 failure | | M4 the whole-buffer reject deleted | GREEN, and expected | Each mutation reds the case that owns it. M1 to M3 were re-run after the refactor, so no earlier commit's mutation proof was silently disarmed by a later one. **M2 is the finding this pull request is most worth reading for.** On the first run it stayed GREEN: the adjacency guard was in the source and no case could tell whether it was there. A file with no marker leaves the scan early, so the guard never decided anything in any fixture, and a new gate carrying a branch no mutation can falsify has the same defect it was added to fix. `test_a_separator_after_the_hunk_closes_is_not_named` was written to close it. It gives a file a closed hunk and then a legal setext underline below it; the exit code is 1 either way, so the case asserts the report, and it carries its own positive control (three named lines) before it claims a fourth is absent. M4 is green on purpose and is recorded as such. The whole-buffer reject is a pure optimization that decides nothing the per-line pass would decide differently, and its docstring says so, so a later reader cannot mistake it for the semantic guard. ## The four existing checkers are unchanged Byte-identical to `origin/main`, verified by sha256 rather than by eye: ``` scripts/check-public-doc-tables.py same=YES scripts/check-agent-record.py same=YES scripts/check-doc-checkpoint.py same=YES scripts/check-issue-index-append-only.py same=YES ``` All four still pass on this tree, tree-scoped and range-scoped, at rc 0. This adds a gate; it alters no existing gate's semantics. ## Repairs from the fresh review The review returned PASS with two LOW findings. Both are repaired here, because both bear on the principle this change elevates. **An unmerged index triple-counted.** `git ls-files` emits stages 1, 2 and 3 for a conflicted path, so during a live merge the checker read one file three times and reported `9 findings in 1 file; examined 3 tracked text files`. The verdict stayed 1, so nothing was ever missed, but the count was wrong in exactly the state the gate exists for, and no fixture built an unmerged index. `tracked_paths` now deduplicates with `dict.fromkeys`, which keeps git's order. `test_an_unmerged_index_is_counted_once` makes `git merge` genuinely fail and asserts the three stage entries before it asserts anything else, so the dedupe is falsifiable rather than merely present. **The clean-tree floor was 27% of the real count.** `> 1000` against a real 3733 would have stayed green over a scan that collapsed to markdown alone. It is now equality with git's own text set, as described above. Three accuracy repairs came with them. An unreadable file was filed as a finding, so an `OSError` exited 1 and printed "Resolve the merge before committing"; it now takes its own list and exits 2, because a file this run could not read is a file it cannot call clean. The summary file count split each report line on its first colon and merged two tracked paths sharing a prefix. And the creation-mutation comments in `check-pr-size.py` claimed the disabled stub fails *every* case that runs the checker: measured, it fails **16 of 21**, and the five survivors assert only that an ordinary document exits 0, or never run the checker at all. The mutation is still detected; the sentence was wrong, and it is corrected in both places. The spec's `## Risks` now records the unmerged index as **not seen** when the spec was written, rather than as a trade-off that was weighed and dropped. ## Where it runs `CHECKERS` in `scripts/agent-preflight.sh`, deliberately first, so a reader who sees it fail knows to stop reading the verdicts below it; `SUITES` for the mutation suite; and a step in the `agent-record` job of `.github/workflows/ci.yml`, so a pull request is gated even when preflight was never run. The workflow was re-parsed with a loader that rejects duplicate mapping keys: 17 jobs, one registered step. PyYAML accepts workflows GitHub rejects, and a duplicate key yields zero jobs. ## One in-flow repair, tracked separately as #1448 Registering a new checker requires an entry in `CREATION_MUTATIONS`, so this change necessarily edits `scripts/check-pr-size.py`, and a change to that file must prove its own HEAD checker/test pair passes. It could not: `test_every_tracked_and_current_change_path_is_classified` sweeps every tracked path, and ten files of `docs/bench-evidence/gdn-replayssm-w0-20260818/` have been unclassified since 2026-08-18, because `BENCH_EVIDENCE` matches exactly one path segment. That suite is RED on `origin/main`, verified in a detached worktree at `9ecaf1bb3`, and it is in no preflight `SUITES` entry, so preflight is green over it. It surfaces only as `ERROR: HEAD checker/test pair failed for 'scripts/check-pr-size.py'`, which reads as a defect in the change under review. Repaired here, per AGENTS.md § *Every change starts from an issue*: a `BENCH_EVIDENCE_RUN` pattern that names the surface rather than widening a rule, restricted to the extensions the directory carries and deliberately excluding `.md` and `.json`, which already classify as `public_document` and would be silently reclassified because the `evidence` arm is tested first. Classification of all 4811 tracked paths was captured before and after: exactly those ten move, and no other path changes class. Issue #1448 tracks it and is not closed by this body's keyword, so it can be closed by hand after review confirms the repair. Fourth instance of that class after #856, #668 and #989. ## Gates All re-run after the last merge of `origin/main` `0adeb8b0e`, so the base is an ancestor of HEAD and the trailer gates EXECUTE rather than skip. - `python3 scripts/check-conflict-markers.py` — rc 0, 3739 files examined. - `python3 tests/scripts/test_check_conflict_markers.py` — rc 0, 21 cases, `OK`. - `python3 scripts/check-agent-record.py` — rc 0, re-run after the merge. - `python3 scripts/check-public-doc-tables.py` — rc 0. - `python3 scripts/check-doc-checkpoint.py --base origin/main --head HEAD` — rc 0. - `python3 scripts/check-issue-index-append-only.py --base origin/main --head HEAD` — rc 0. - `python3 scripts/check-pr-size.py --base origin/main --head HEAD` — rc 0, `OK: every explicit path class is within its review budget.` - `python3 scripts/check-commit-trailers.py --range origin/main..HEAD` — rc 0 over 7 commits, EXECUTED rather than skipped. - `python3 scripts/check-commit-style.py --range origin/main..HEAD` — rc 0, same range. One accounted-for red, and it is not this change's: `test_cpu_x86_llamacpp_floor` reports `NO_QUIET_WINDOW` exit 4 where the case expects 2. That is #618's load dependence, measured on this box at load average 122-129 and later 157-225, driven by sibling agent sessions. This branch touches no benchmark harness, and the same suite re-run on the same sources at load average 2.5 reported `Ran 10 tests`, `OK`, rc 0. The full `scripts/agent-preflight.sh` on this head, after the merge: **85 gates `ok`, ZERO skipped**, and rc 1 for that one suite alone. `commit-trailers` and `commit-style` both read `ok` rather than `SKIP`, which is the point of merging `main` first: a skipped gate reports nothing about the tree, and the harness refuses the green banner over one. `windows-msvc-cpu` and `windows-msvc-vulkan` are red on every pull request (#584, #968) with no `main` baseline. This diff carries no C++. Closes #1417 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces #1301, which cannot be merged: #1377 landed as a squash and destroyed that branch's merge base, so merging
maininto it produced thirteen conflicts including product files this change does not touch. This branch is the same eight-file delta rebuilt onmain.W0e is the measurement W0b-W0d existed to make possible. It ran on one
rc holdondgx:gpu0against the real 369.97 GiBQwen3.8-2.4T-A95B UD-Q1_0,and it split.
The CPU arm is now MEASURED and replaces the VOID #912 F1 figure: 11.05
s/token steady at 4000 slots, which is rep 2's median over 29 samples. Rep 1's
median is 11.22, so the two reproduce within 1.5%. Live cache, decode-phase
exhausteddelta 0 over 31 decode steps in the same run.The 8000-slot arm is slower on every pairing of the two medians, 3.56x to 4.11x
across the four available pairings and 4.05x pairing rep 1 with rep 1, and it is
much less steady: the steady window's max/min ratio is 3.99x and 5.87x against
1.30x and 1.40x at 4000 slots. The cause is NOT that the arena fails to fit.
18.55 GiB of arena beside 62 GiB of dense weights is 80.55 GiB on a 119.63 GiB
box, and peak RSS barely moves for the 9.27 GiB arena delta, 86.5 to 86.6 GiB.
What moves is
min avail, 16,347 to 6,941 MiB, which is about the arena delta,and peak swap, 6,883 to 30,625 MiB, all of it. The best-supported reading of
that pattern is page-cache displacement: the borrowed 370 GiB expert mapping is
served out of whatever memory is free, the arena takes that memory, and the
reclaim pressure it creates pushes anonymous pages to swap. That is a reading of
the columns rather than a proven mechanism, because this run sampled no
page-cache size and no major-fault counter. The operational conclusion holds
either way, and it is the one to act on: more slots is not a free knob, and the
publishable figure is the 4000-slot one.
The CUDA arm LOADS, which it could not do before this row, and then generates
nothing: it exhausts the box inside its first forward, 0 steps, 7 attempts. The
slot arena is measurably not the cause, because a 64-slot 0.15 GiB arena fails
in exactly the place an 8000-slot 18.55 GiB one does. That is #1299, and it is
the DENSE half of the model rather than the expert lane. No speed claim is
attached to the CUDA arm and
docs/BENCHMARKS.mdsays so.The harness earns its own paragraph, because it did not build. It landed in no
CMakeLists, so nothing in the project compiled the instrument that produced
this row's only numbers. Wiring it as the
expert-stream-device-w0etargetshowed the file does not compile under the project's flags at all: three
backslash continuations inside
//comments, which-Werror=commentrejects.The recorded manual
g++ -O2recipe carries no-Werror, so that recipe wasthe only thing that had ever built it, and it would have rotted against the
very C ABI it measures with nobody noticing. It now builds, links
vllm::sharedso it stays a pureinclude/vllm.hclient, andlddshowslibvllm.so.0and nothing internal.A fresh review returned FAIL on the first head, and the second commit repairs
its six findings. The copy-paste recipe and the
--offload-configexample indocs/USAGE.mdset 4000 slots rather than 8000, so an operator who copies themgets the arm the published number was measured on instead of the 39.98-45.40
s/token one; every place that publishes 11.05 now names 4000 beside it; the
"does not fit" story above is restated as what the columns support; the spec's
forcedanchor points atqwen3_5.cpp:5521rather than a W4 header comment 132lines earlier; and the #1299 row in
.agents/issue-index.mdis appended at theend of that append-only file rather than inserted in its middle.
This branch is rebuilt on
mainrather than merged into it. #1377 landed as asquash, which destroyed this branch's merge base, and merging produced thirteen
conflicts including product files this change does not touch. Reapplying the
eight-file delta onto
mainleaves three record conflicts, each resolved bykey: the benchmark log unions two disjoint sections, and the two keyed table
rows take this row's newer state while keeping the keep-quant/keep-f16
mechanism #1378 corrected. Verified afterwards that every one of main's #1378
repairs survives in the spec and that nothing of main's is missing beyond the
W0e status text this measurement supersedes.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]