diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 9a0af0356..06b0bd728 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -479,6 +479,7 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1439](https://github.com/mudler/vllm.cpp/issues/1439) | `LTX25-DEVICE-RESIDENCY` | `ltx2 video: a render through the ABI emits a phase table that SUMS to wall` asserts `CHECK(leaves >= 0.95 * wall)` (`tests/vllm/multimodal/test_ltx2_video.cpp:2854`) and is **RED on `origin/main`**, not on the branch that found it. MEASURED on one x86 box, one build directory, `CMAKE_BUILD_TYPE` empty as CI has it, with this lane's four files reverted so the binary IS main at `89261c955`: six in-suite runs read 94.32%, 95.20%, 93.74%, 94.20%, 94.69% and one `-tc` run 94.19% - **five of six red**, 93 cases / 3321 assertions / 1 failed, exit 1. The W0-live merge reads 93.82%, 93.68%, 94.34%, 94.62%, 94.39% in-suite (96 cases / 3555 assertions / 1 failed) and 94.12% with `VLLM_RENDER_PROGRESS=0`, so neither the new emitter nor its call site is the term; one `-tc` run passed at 95.40%. It is NOT box contention, and the run that disproves that is main's single green: it had `wall=0.579684s`, more than double every other run, because the box was LOADED - a slower render passes, since the un-named residue grows more slowly than the wall it is divided by. The residue is 4.80% to 6.32% of `wall` across all twelve runs (0.0128 s to 0.0278 s against a `wall` of 0.220 s to 0.580 s), so a 95% floor sits INSIDE the measurement's own range at the 64x64 / 9-frame FIXTURE scale and the case decides by coin flip, mostly red. The tolerance was argued for the 21.004 B render, where the same residue would be a far smaller fraction. NOT FIXED IN FLOW, deliberately: naming the un-named time, or bounding `unaccounted_seconds` beside the ratio so the assertion says the same thing at both scales, is a change to a gate's semantics and needs its own row, spec and red-first evidence per `AGENTS.md` "Changing the rules or a checker". Found while merging `origin/main` into `row/LTX25-RESIDENCY-W0-LIVE` after [#1419](https://github.com/mudler/vllm.cpp/pull/1419) was auto-closed by its base branch being deleted. Listed under `## Owed` in [`ltx25-device-residency.md`](specs/ltx25-device-residency.md) | bug | | [#1435](https://github.com/mudler/vllm.cpp/issues/1435) | `VT-MATMUL-FP8-BLOCK-CUDA` | **A CUDA build without CUTLASS headers segfaults on the block-wise FP8 path instead of refusing by name, and configure reports the feature ENABLED anyway.** Measured 2026-08-20 on `dgx:gpu0` (GB10, cc 12.1) in an `rc` lease at `63d87805c`. `src/vt/cuda/cuda_matmul_fp8_block_cutlass.cu` is the only TU that registers `kMatmulFp8BlockScaled` for `kCUDA` and it reaches `target_sources(vllm PRIVATE ... ${_FP8_CUTLASS_SOURCES})` only through TWO nested conditions in `CMakeLists.txt` -- the outer `if(VLLM_CPP_CUTLASS)` (one occurrence in the file) and, inside it, the `if(VT_CUTLASS_FP8_ARCHS)` that SETS `_FP8_CUTLASS_SOURCES` -- so with no CUTLASS headers the op is unregistered, a device tensor reaches the portable HOST kernel, and the process SIGSEGVs while the reference tier prints `correct but slow`. Anchored by SYMBOL rather than by line on purpose, and the reason is measured on this very row: the guard sat at `CMakeLists.txt:1695@63d87805c` -- the only line number in this row, sha-qualified so it stays resolvable -- and by the time this row was written that number was already stale, then rotted AGAIN by four lines when `origin/main` was merged into the row's own branch mid-review. Two rots inside one pull request. A bare head-relative number here would be permanent, because this index is append-only and no checker scans it, and it would resolve to plausible unrelated code rather than to nothing: at `63d87805c` that line was the guard, and two trees later it was an NCCL include directory. Resolve the anchors above by grepping the symbols; do not trust any number quoted for them. Reachable on a default build: `VLLM_CPP_CUTLASS_FETCH` defaults OFF (the single `option(VLLM_CPP_CUTLASS_FETCH ...)` line, `OFF` at its default argument) and CUTLASS is not a submodule. This contradicts the design note in `cuda_matmul_fp8_block_cutlass.cu`, which states an unsupported build `keeps refusing by name -- which is the honest answer and not the #960/#844 fall-through`: the refusal IS real on the model path (`RefuseUnrunnableQwen3_5DenseFp8Block`, plus the two `VT_CHECK`s) but a caller entering through the op registry, which is what the ported upstream test does, bypasses all three. Second, smaller defect in the same run: `vt_cuda_report_feature` printed `CUDA feature cutlass-fp8: ENABLED for [121a]` for a build whose `build.ninja` contains ZERO references to either FP8 CUTLASS TU, because the report runs on the arch intersection alone, before CUTLASS-header detection. `cmake/CudaArchFeatures.cmake` opens by naming this exact class as what the feature table exists to prevent. NOT fixed in flow: both are code changes on a CUDA path needing their own row, spec and hardware re-gate. Owned by row `VT-MATMUL-FP8-BLOCK-CUDA`, under `## Owed` in [`vt-matmul-fp8-block-cuda.md`](specs/vt-matmul-fp8-block-cuda.md) | bug | | [#1437](https://github.com/mudler/vllm.cpp/issues/1437) | `VT-MATMUL-FP8-BLOCK-CUDA` | **The mainloop-scaled CUTLASS block-wise FP8 GEMM throws `cutlass Invalid status` on upstream's own ported case, on the arch it targets.** FIRST on-hardware execution of #1189 M5 (`489a9a4c0`), measured 2026-08-20 on `dgx:gpu0` (GB10, driver 580.173.02, compute capability 12.1) in an `rc` lease at `63d87805c`, CUDA 13.0 Release, `-DVLLM_CPP_CUDA_ARCHITECTURES=121a -DVLLM_CPP_CUTLASS_FETCH=ON`. G2 (upstream's `test_w8a8_block_fp8_cutlass_matmul` ported whole: M=32, N=576, K=7168, block [128,128], bf16 out) and G7 both THROW from `gemm_op.can_implement(args)`, so CUTLASS rejects the configuration before any launch -- not a numerical disagreement and not a launch failure. G6, G8 and G9 PASS. That this was an EXECUTION and not a skip rests on the throw TEXT, not on a count: `vt cuda: matmul_fp8_block_scaled: cutlass Invalid status` is assembled by the `VT_CUTLASS_CHECK` macro in `src/vt/cuda/cuda_matmul_fp8_block_cutlass.cu`, and the message prefix `matmul_fp8_block_scaled: cutlass ` occurs in NO other translation unit (the per-tensor sibling `cuda_matmul_fp8_cutlass.cu` defines a same-named macro but writes a different message), in a TU compiled only when CUTLASS is found, so no host fallback can produce that string; the run also logged ZERO `[vt reference-tier]` lines, and artifact provenance was confirmed by `cuobjdump --list-elf` reporting `cuda_matmul_fp8_block_cutlass.cu.1.sm_121a.cubin`. Suite: 5 cases, 3 passed, 2 failed, **34 assertions, 0 failed** -- recorded, but attributed rather than leaned on, because **27** of the 34 are G6, which has no `HasCuda()` guard and prints identically on a GPU-less box (8 grid entries x 3 `CHECK` + 3 tile-config `CHECK`s); the device-only remainder is G8's 2 and G9's 5, and a pure skip of this file prints 27. NO shape has had its output compared with the CPU reference: G2 and G7 are the only two cases that make that comparison and BOTH threw before their first assertion -- G7 aborted at `Grid()[0]`, the same M=32 N=576 K=7168 case, so its other seven shapes, including the second ragged-N entry M=8 N=576 K=1024, were never attempted -- and G8 compares a device f32 out against a device bf16 out, which is the kernel against ITSELF. Hypothesis recorded in the issue and explicitly NOT isolated: N=576 is 4*128+64, a ragged final scale block, which the test header records as upstream's reason for choosing it; G8 passing makes the failure shape-dependent rather than universal. This moves the row from `unmeasured` to `measured and failing`, a worse position than the spec previously described. NO speed claim: the lease took no clock control and recorded no contention. Owned by row `VT-MATMUL-FP8-BLOCK-CUDA`, under `## Owed` in [`vt-matmul-fp8-block-cuda.md`](specs/vt-matmul-fp8-block-cuda.md) | measurement | +| [#1442](https://github.com/mudler/vllm.cpp/issues/1442) | `ENG-EXPERT-STREAM-DEVICE` | `docs/BENCHMARKS.md:8` (row `ENG-EXPERT-STREAM`, [#912](https://github.com/mudler/vllm.cpp/issues/912)) read "Streaming-ON decode **VOID** (#912 F1); re-measure owed" while `docs/BENCHMARKS.md:9`, ONE LINE DOWN, recorded **11.05 s/token at 4000 slots**, live cache, decode-phase `exhausted` delta 0, explicitly "replacing #912 F1's VOID". `docs/STATUS.md:129` repeated the VOID claim. The re-measure was NOT owed: it landed in [#1414](https://github.com/mudler/vllm.cpp/pull/1414) (`c805ccbb3`) and is recorded at `.agents/benchmark-record.md`, section `ENG-EXPERT-STREAM-DEVICE W0e` -- 11.05 is rep 2's median over steps 4-32 and rep 1's 11.22 sits 1.54% above it. NO GATE CATCHES IT, and that is the per-row record shape working as designed: the two rows are keyed on different IDs, so `check-public-doc-tables.py` sees two well-formed rows and `check-agent-record.py` sees two individually consistent lifecycle states, and no checker here compares a claim in one keyed row against a claim in another. A superseding measurement in row B therefore leaves row A's statement standing until somebody edits row A. FIXED IN FLOW, record text only, no behavior change, and stated PER FILE because the two cells never said the same thing. `docs/BENCHMARKS.md:8` carried no cause at all; it keeps its `**VOID** (#912 F1)` and now names `ENG-EXPERT-STREAM-DEVICE` where it said "the next row", because a positional pointer inside a 253-row table breaks the moment a row is inserted above it and no checker validates one. `docs/STATUS.md:129` never said a re-measure was owed, and its replacement is not adjacent either: that figure is at `docs/STATUS.md:167`, 38 rows further down the SAME table, which runs 126-190 unbroken. It keeps the cause verbatim (the step clock had no caller, so the cache died in token 3) and gains the same named pointer. NEITHER cell repeats 11.05 s/token, because a fact stated twice is the shape that lets two records drift apart. The same contradiction one level down is closed in the same change: `.agents/specs/expert-streaming.md`'s `## Owed` still told a reader the re-measure was owed and blocked on an unreachable `dgx.casa`, so a fixed public cell routed them to a stale row record, and that entry is now struck as CLOSED by W0e with the void's cause kept as the reason the replacement was designed to gate `exhausted` on a DECODE-PHASE DELTA. The 4 KiB demand-fault mechanism displaced from the 220-character `BENCHMARKS.md` cell to make room is stated in full in `.agents/benchmark-record.md` under "Why it is slow, and by how much" (~6.7 GB of expert bytes per token served as 4 KiB faults at ~100 MB/s against an NVMe that sustains ~5 GB/s). NO CUDA NUMBER MOVED and none may: W0f (`e67b2a4ba`) makes `--device cuda` decode 32/32 steps at peak RSS 97.75 GiB, G0-CORRECT FAILS on a step-7 near-tie, and this row's own stop condition therefore makes G0-SPEED VOID. Owned by row `ENG-EXPERT-STREAM-DEVICE`, whose #1414 landing owed the lifecycle write to the row it superseded; spec [`expert-stream-device-slots.md`](specs/expert-stream-device-slots.md) | bug | | [#1417](https://github.com/mudler/vllm.cpp/issues/1417) | `GATE-CONFLICT-MARKERS` | **Four record gates return rc 0 on a document carrying literal conflict markers.** Re-derived on a detached scratch worktree at `b537a5344`: five lines spliced into the `docs/STATUS.md` capability table (start marker, a duplicated keyed row, separator, the same row again, end marker) leave `scripts/check-public-doc-tables.py`, `scripts/check-agent-record.py`, `scripts/check-doc-checkpoint.py` and `scripts/check-issue-index-append-only.py` all at rc 0 with their normal OK messages. The two range-scoped checkers were run over a scratch COMMIT, because a working-tree mutation of a commit-reading checker returns 0 without ever reading the mutated bytes. Closed by one tree-scoped checker, `scripts/check-conflict-markers.py`, wired into `scripts/agent-preflight.sh` and the `agent-record` CI job: it refuses a line that starts with seven `<` or seven `>` and a space, and a line of exactly seven `=` only when a start marker opened a hunk above it. The separator stays conditional because a bare row of `=` is a legal setext heading underline, and five shipped files already carry lines of eight or more `=`. No allowlist: the checker builds its patterns from character repetition, so it and its suite carry no marker at column 0. Spec [gate-conflict-markers.md](specs/gate-conflict-markers.md) | bug | | [#1448](https://github.com/mudler/vllm.cpp/issues/1448) | `GATE-CONFLICT-MARKERS` | **`scripts/check-pr-size.py`'s `classify_path` has no entry for a per-run `docs/bench-evidence//` directory, so its own suite is red on `main`.** `BENCH_EVIDENCE` matches exactly one path segment, and the ten files of `docs/bench-evidence/gdn-replayssm-w0-20260818/` (landed 2026-08-18) match none of it. Measured at `9ecaf1bb3`: sweeping every tracked path through `classify_path` leaves exactly 10 unclassified, and `test_every_tracked_and_current_change_path_is_classified` is RED in a detached worktree at that SHA. That suite is in no preflight `SUITES` entry, so preflight is green over it; it surfaces only through the checker-evidence contract, where it reads as `ERROR: HEAD checker/test pair failed for 'scripts/check-pr-size.py'` and looks like a defect in the change under review. Registering any NEW checker requires editing `CREATION_MUTATIONS`, so every future checker was blocked behind it. Fourth instance after [#856](https://github.com/mudler/vllm.cpp/issues/856), [#668](https://github.com/mudler/vllm.cpp/issues/668) and [#989](https://github.com/mudler/vllm.cpp/issues/989). FIXED in flow by a `BENCH_EVIDENCE_RUN` pattern restricted to the extensions the directory carries, excluding `.md` and `.json` because the `evidence` arm is tested before `public_document` and would silently reclassify `docs/bench-evidence/mxfp4-qwen/*`. Verified to move exactly those 10 paths and no others | bug | | [#1447](https://github.com/mudler/vllm.cpp/issues/1447) | `ENG-EXPERT-STREAM` | **`docs/USAGE.md` said EVERY Qwen3.8 decode figure came from the W0e C ABI harness; the 66.7 s/token streaming-off row of 16 August 2026 came from `vllm-server`, the same binary the section tells the reader to run.** Introduced by #1211 and fixed in the same flow: the sentence is scoped to the W0e and W0f runs and the exception is named, in `docs/USAGE.md` and in the spec paragraph that mirrors it. Provenance read at the source, `.agents/specs/expert-streaming.md:837` (server entry point) and `:905`. | record | diff --git a/.agents/specs/expert-stream-device-slots.md b/.agents/specs/expert-stream-device-slots.md index ce0055770..340ef8988 100644 --- a/.agents/specs/expert-stream-device-slots.md +++ b/.agents/specs/expert-stream-device-slots.md @@ -272,6 +272,34 @@ one.** G0-CORRECT fails on the step-7 near-tie, so G0-SPEED is VOID by this row' own stop condition. A decode number exists in `../benchmark-record.md` for the record; it is not a result and no ratio may be inferred from it. +**The public pages now agree with this row** +([#1442](https://github.com/mudler/vllm.cpp/issues/1442)). Both carried the +`ENG-EXPERT-STREAM` row's VOID after W0e replaced it, and they carried it +differently, so the repair is stated per file rather than as one claim about +two. + +* `docs/BENCHMARKS.md:8` read "Streaming-ON decode **VOID** (#912 F1); + re-measure owed". It named no cause, and line 9, the row directly under it, + already recorded the replacement. It now reads "Streaming-ON decode **VOID** + (#912 F1); re-measured LIVE by `ENG-EXPERT-STREAM-DEVICE`", which names the + row instead of its position, because a positional pointer inside a 253-row + table breaks the moment a row is inserted above it and no checker validates + one. +* `docs/STATUS.md:129` read "Streaming lands but its decode figure is VOID: the + step clock had no caller, so the cache died in token 3". It said nothing about + a re-measure being owed, and its replacement is not adjacent: that figure sits + at `docs/STATUS.md:167`, 38 rows further down the SAME table, which spans + lines 126 to 190 unbroken. It keeps the cause verbatim and gains the same + named pointer. + +Neither cell repeats 11.05 s/token. `docs/BENCHMARKS.md:9` and +`docs/STATUS.md:167` already carry it, and one number stated in two keyed rows +is what lets the two drift apart. W0e's landing owed that write to the row it +superseded and never made it. No gate can catch the class: the rows are keyed on +different IDs, so `check-public-doc-tables.py` sees two well-formed rows and +`check-agent-record.py` sees two individually consistent lifecycle states, and +no checker here compares a claim in one keyed row against a claim in another. + ## Scope **In scope.** @@ -964,7 +992,7 @@ re-derived here. | Owed | Why it is open | |---|---| | **`kQwen3MoeFactory.streams_routed_experts = true` is a CORRECT declaration that nothing READS today.** The flag's only reader is the loader's lane block, which is on the GGUF path, and `kGgufArchArms` (`model_loader.cpp`) maps no `general.architecture` onto `Qwen3MoeForCausalLM` (Qwen3-Coder), so no GGUF load can resolve to that factory. | It is set anyway because it is TRUE: `qwen3_moe.cpp` composes the same `RunMoeBlock` the Qwen3.5 MoE forward does, which is why it holds an `EndStepGuard` at all, so its experts do reach `KqExpertSlice`. Declaring it false to make every setting reachable would put a false statement in the registry, and the safe-direction default would then hide it. Named here per `## Nothing lands dead` rather than left for the next reader to find: `ENG-EXPERT-STREAM-DEVICE` owns the wiring under [#1124](https://github.com/mudler/vllm.cpp/issues/1124), and the flag becomes read the moment a `qwen3moe` GGUF arch arm exists. The `Qwen3_5Moe*` setting beside it IS read, and its gate is now EVIDENCED rather than asserted: mutation M-A3 (`kQwen3_5MoeFactory.streams_routed_experts = false`, `qwen3_5_moe.cpp`) was listed as NOT RUN in #1377's pull request body, was then run by that pull request's fresh review, and was re-run during the #1378 repair with the result recorded -- compile status 0, `git diff --stat` 1 file / 1 insertion / 1 deletion, `test_gguf_device_fit_reach` 14 cases with 2 failed and 66 assertions with 6 failed, exit status 1, tree restored byte-identical by sha256. **The laguna half of the same claim is VACUOUS and is not evidence for anything.** Mutation M-A3c (`kLagunaFactory.streams_routed_experts = true`) is GREEN, and correctly so: `laguna` has no entry in `kGgufArchArms` (`model_loader.cpp`), so a Laguna GGUF is refused as an unsupported architecture before the fit check runs and no setting on that factory can reach the lane. Nothing is owed to make it gateable -- manufacturing a gate for an unreachable flag would be worse than saying this -- and `DeepseekV4ForCausalLM`, which DOES have an arch arm, is the case that carries the architecture term's weight. | -| **`scripts/check-doc-checkpoint.py` stays RED on this branch for commit `939755f99` and cannot be made green here** ([#1387](https://github.com/mudler/vllm.cpp/issues/1387)). That commit appended a measurement to `.agents/benchmark-record.md` without writing `docs/FEATURES.md`, whose streaming row then said "CPU keep-quant towers only" after W0c had made a host-readable staging device take the slot arm. | The PAGE is repaired here. The GATE is not, and cannot be: the checker walks a range one COMMIT at a time, and `main` may never be force-pushed, so no later commit can make a published one green. The squashed commit that lands carries both paths and passes. Changing the walk is checker semantics and needs its own row, spec and red-first evidence. | +| ~~**`scripts/check-doc-checkpoint.py` stays RED on this branch for commit `939755f99` and cannot be made green here** ([#1387](https://github.com/mudler/vllm.cpp/issues/1387)). That commit appended a measurement to `.agents/benchmark-record.md` without writing `docs/FEATURES.md`, whose streaming row then said "CPU keep-quant towers only" after W0c had made a host-readable staging device take the slot arm.~~ **CLOSED 2026-08-20** ([#1442](https://github.com/mudler/vllm.cpp/issues/1442)'s flow). The page half landed twice over: `5f4eb356e` (#1377) wrote the row and `e67b2a4ba` (#1427) refined it for W0f, so `docs/FEATURES.md:64` now names the staging device and both accepted residencies, keep-quant and keep-f16. Measured rather than asserted: `git grep 'CPU keep-quant towers only' origin/main -- docs/FEATURES.md` is rc 1 with no output, and the same grep at `5f4eb356e^` is rc 0 with one hit, which is the positive control that makes the empty result absence rather than a wrong pattern. | Kept as a line rather than deleted, because the SURVIVING half is a different question with a different owner and deleting the entry would lose the pointer to it. Whether a per-commit record gate should be satisfiable after its commit is published is [#573](https://github.com/mudler/vllm.cpp/issues/573), owned by `ENG-RECORD-CONFLICT-SURFACES`, and it is open. Changing the walk is checker semantics and needs its own row, spec and red-first evidence, so it was not folded into this row. Nothing in this branch touched `scripts/`. | | **The CUDA registrar's own probe assembly is still unmutated.** `src/vllm/platforms/cuda.cpp`'s `Registrar` reads `cudaDevAttrPageableMemoryAccess` and `cudaDevAttrIntegrated`, defaults each to 0 on a query failure, and hands the pair to `HostMemoryIsDeviceAddressableFromAttrs`. That call and those defaults compile only in a CUDA build, so nothing on the CPU tier can mutate them. | The RULE they feed is no longer part of this debt: #1378 extracted it and gated it over all four attribute pairs in `test_platform`, and both term-deletion mutations are RED there. What remains is narrower and honest -- the probe calls, the failure defaults, and the registration itself -- and it needs the same `dgx:gpu0` lease as W0e. Named here rather than folded into the W0b bullet, which used to claim more than it had. | | **G-DISCRETE: validate W1/W2 on a discrete NVIDIA GPU.** The measurement: on a device with VRAM V and `host_memory_is_device_addressable() == false`, load a GGUF whose `*_exps` towers exceed V, with the lane on, and gate (i) token-exactness against the CPU arm on the same checkpoint, (ii) decode-phase `exhausted` delta 0, (iii) peak device allocation <= non-expert remainder + arena. | No discrete NVIDIA GPU is reachable from this project. `dgx:gpu0` is a GB10 where device memory IS host memory, so a device store there exercises the plumbing and not the thing W1 exists for. Recorded rather than implied, because a gate nobody can run is not a gate. | | **A mutation of W0b's CUDA leg.** `CudaPlatform::host_memory_is_device_addressable` compiles only in a CUDA build, so no CPU-tier gate can invert it. The bullet in `## Now` promised this line and the table did not carry it, which is fixed here. | **Half discharged by W0e and stated as half.** The lane engaged on a real `--device cuda` run — the `[expert-stream] ON` banner printed and the #1123 refusal did not fire — and neither happens unless the probed predicate returned true on the actual CUDA platform, so the leg is now proven REACHED and proven to answer true on a GB10. What is still owed is the negative: a mutation that makes it answer false and shows a gate go red. That needs a CUDA build with a test target, and W0e built with `-DVLLM_CPP_BUILD_TESTS=OFF` because the lease was for the measurement. | diff --git a/.agents/specs/expert-streaming.md b/.agents/specs/expert-streaming.md index be6761410..8c48eb28f 100644 --- a/.agents/specs/expert-streaming.md +++ b/.agents/specs/expert-streaming.md @@ -1626,7 +1626,7 @@ Carried debt for this row. Each item names why it is not closed here. | Owed | Why it is open | |---|---| -| **Re-measure decode on a LIVE cache.** The `docs/BENCHMARKS.md` decode figure for this row was taken with the step clock dead from token 3 onward and is void. | Needs `dgx.casa` and the 370 GiB checkpoint. The box was unreachable for this repair (`No route to host`), and this host has no CUDA device and cannot hold the model. | +| ~~**Re-measure decode on a LIVE cache.** The `docs/BENCHMARKS.md` decode figure for this row was taken with the step clock dead from token 3 onward and is void.~~ **CLOSED by `ENG-EXPERT-STREAM-DEVICE` W0e**, 2026-08-18 (`c805ccbb3`, [#1414](https://github.com/mudler/vllm.cpp/pull/1414)): streaming-ON decode on a live cache is **11.05 s/token** steady at 4000 slots, rep 2's median over steps 4 to 32 with rep 1 at 11.22, and the decode-phase `exhausted` delta is 0 in the same run. Recorded in [`../benchmark-record.md`](../benchmark-record.md) under `ENG-EXPERT-STREAM-DEVICE W0e`. | Kept as a line rather than deleted, because the reason the figure was void is what shaped the replacement. `Qwen35ExpertStream::EndStep()` had no caller, so `protected_this_step` never cleared, `Acquire` returned -1 once the cache filled and every slice from partway through token 3 came from the mmap, which IS the baseline. That is why W0e gates `exhausted` on its DECODE-PHASE DELTA rather than on a total, and why it reports two reps. The host blocker this entry named is spent: the measurement ran on `dgx:gpu0` inside one `rc` lease. The TTFT row of the same 16 August run was never void and is unchanged. | | **The `pread` path has never run on the model.** `EnsureFile` now has a CPU-local gate that drives it through the production seam from a temp file and proves the `file_offset + offset` composition (#1091 finding 4), so it is no longer UNREACHED. It is still unmeasured on a real checkpoint. | Same host. Three earlier attempts were OOM-killed at 48.6 GiB anon beside another session's 32.6 GiB job. | | **A run that REQUESTS streaming and never builds a store prints no statistics line.** The `[expert-stream] ON ...` banner is absent in that case too, so no-banner means "nothing reached the lane" and banner-without-line means "the process died"; the docs state all four shapes. | A teardown hook that could report it is not reachable from any test on a CPU-only host, because `Reserve` and `Get` sit in one call chain and a device platform is what separates them. Landing it would have been an untestable branch added to fix an untestable-branch problem. Needs `dgx.casa` (see #1091). There is NO such hook in the tree: `~Qwen35ExpertStream` is the only production path to the final line, and it prints it directly rather than through `ExpertStreamFlushStats`, which has no production caller at all. The header now says both, rather than describing the hook that was rejected (#1106). | | **Three of the four step guards land UNREACHED.** `Qwen3_5MTPModel::Forward`, `Qwen3_5Model::ForwardDense` and `Qwen3_5ReplayLayer` are parity-only entry points with no caller outside `tests/`, so their `Qwen35ExpertStreamStep` guard is reached by no production path. Only `Qwen3_5MTPModel::ForwardPaged` is (`runner.cpp:2183` -> `spec_decode/mtp/speculator.cpp:107,262`), and even that caller is "UNREACHABLE unless a speculator is configured" (`runner.cpp:2120`) — so a DEFAULT-configuration run reaches none of the four, which is a weaker statement than "one of four is reached" and is recorded here rather than rounded up. Owning row `ENG-EXPERT-STREAM`; tracked as [#1108](https://github.com/mudler/vllm.cpp/issues/1108). | Nothing is deleted, because the guards are correct where they sit and cost nothing, and the alternative — add the guard later, together with the caller — is precisely how this row lost its step boundary in the first place. It closes when one of those entry points gains a production caller, or when they are retired as parity references. Neither is scheduled and neither should be forced by the record. | diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 7f9b0bc2e..d93506aab 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -5,7 +5,7 @@ | Reference | Workload | Headline | Tokens | |---|---|---|---| | **Weight load (`ENG-LOAD-DIRECT-UPLOAD`, #150)** | Qwen3.6-27B bf16 (50.098 GiB), GB10 Vulkan, same binary both arms, interleaved under one GPU lock | **Load phase 1.54x warm** (19.27 -> 12.48 s), **1.61x cold** (52.62 -> 32.75 s); bytes moved **100.196 -> 81.260 GiB**. Every ON leg beat every OFF leg | byte-identical; 6/6 token-exact | -| **Expert streaming vehicle (`ENG-EXPERT-STREAM`, [#912](https://github.com/mudler/vllm.cpp/issues/912))** | Qwen3.8-2.4T-A95B `UD-Q1_0` (370 GiB) on ONE GB10, 119 GiB, c1 | **Loads and generates** (streaming OFF): resident **62 GiB**, decode **66.7 s/tok**. Experts BORROW the mmap; ~6.7 GB/token as 4 KiB faults, **~50x** off NVMe. Streaming-ON decode **VOID** (#912 F1); re-measure owed | correct answer; no oracle runs this | +| **Expert streaming vehicle (`ENG-EXPERT-STREAM`, [#912](https://github.com/mudler/vllm.cpp/issues/912))** | Qwen3.8-2.4T-A95B `UD-Q1_0` (370 GiB) on ONE GB10, 119 GiB, c1 | **Loads and generates**, streaming OFF: resident 62 GiB, decode 66.7 s/tok. Experts BORROW the mmap; ~6.7 GB/token, ~50x off NVMe. Streaming-ON decode **VOID** (#912 F1); re-measured LIVE by `ENG-EXPERT-STREAM-DEVICE` | correct answer; no oracle runs this | | **Device expert slices (`ENG-EXPERT-STREAM-DEVICE`, [#1124](https://github.com/mudler/vllm.cpp/issues/1124), [#1299](https://github.com/mudler/vllm.cpp/issues/1299))** | Qwen3.8-2.4T-A95B `UD-Q1_0` 370 GiB on `cuda` vs `cpu`, one GB10, greedy 32 tokens, **4000 expert-stream slots** (8000 is 3.6-4.1x slower), [record](../.agents/benchmark-record.md) | CPU **11.05 s/token** steady, replacing #912 F1's VOID. CUDA **G0-LIVE PASS**: 32/32 steps, `exhausted` delta 0, peak RSS 97.75 GiB. **G0-CORRECT FAIL**, cause NOT identified (not the alias: GB10). **G0-SPEED VOID** | correct; no oracle | | **Structured state record (active)** | v1 scalar + relational + Git-history contracts | No benchmark. At `776c56f1`: 157 imports = 3,231,342 exact bytes; append preserved all 156 wrappers/rows. 95 tests: validator/core 44 (checker 20 + core 24), NOW 18, migration 22, cutover 11. New raw-row mutation guard. | n/a | | **Binary release (ACTIVE; Windows pre-alpha pending)** | v0.0.2 shipped eight primary archive/checksum/provenance triplets + two indexes (26 assets) from source SHA `7020de93652ca920424a10ac5255b34810dd2f24`, run `31466516224` | Windows W14-W16 implemented. **PENDING:** native hosted gates, merged-SHA ten-tuple dry run, matching-hardware evidence, v0.0.3-pre.1 publication, 32-asset audit | W12 optional/non-primary | diff --git a/docs/STATUS.md b/docs/STATUS.md index ab8e2e04d..fec4edfa0 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -126,7 +126,7 @@ token-for-token correctness against the pinned oracle. | Capability | State | Notes | |---|---|---| | Qwen3.6-27B (NVFP4) text generation | Correctness-complete; speed is CHECKPOINT-dependent | Token-exact GB10 on both. `unsloth` @`890bdef7` beats vLLM every c (1.007-1.045x), 115/124; `nvidia` @`0893e160` **flat 0.937-0.956 c1-c32** (#349; 0.838 void) | -| Qwen3.8-2.4T-A95B (`UD-Q1_0`, 370 GiB) | **Loads and generates on ONE 119 GiB GB10**; speed is the gap | Resident 62 GiB; 66.7 s/tok, streaming OFF. Streaming lands but its decode figure is VOID: the step clock had no caller, so the cache died in token 3 ([#912](https://github.com/mudler/vllm.cpp/issues/912)) | +| Qwen3.8-2.4T-A95B (`UD-Q1_0`, 370 GiB) | **Loads and generates on ONE 119 GiB GB10**; speed is the gap | Resident 62 GiB; 66.7 s/tok, streaming OFF. Streaming-ON decode was VOID: the step clock had no caller, so the cache died in token 3 (#912). Re-measured LIVE in the `ENG-EXPERT-STREAM-DEVICE` row | | Qwen3.6-35B-A3B (NVFP4, GDN MoE) | Correctness-complete; **canonical 0.918-0.972x c1-c32** @`348c265d` (first c16/c32), STALE as of 2026-08-12 (+136 src commits then, incl. a +2.05% c8 lever); regrid owed | Token-exact SYNC+ASYNC; `VT_ASYNC_DEVICE_MIRROR` ON fixes async batch-1 token-0 degeneration. Router warp kernel (#378) device-gated 315/315 both arms, kernel 1.363x, step-level not separable | | Qwen3.6-35B-A3B (published BF16, GDN MoE) — TEXT | Correctness-gated 2026-08-15 (#740, #864, both `DONE`); **no throughput, latency or memory number exists for this checkpoint, and none is claimed** | Greedy vs the pinned oracle @`995ad96e`: **6/7 prompts STRICT 16/16**; the 7th is one exact logit tie (`0.0 mnats`) our argmax breaks toward the higher id (#910). SACRED 3/3, goldens byte-identical | | Qwen3.6-35B-A3B (BF16) — IMAGE / VIDEO | Implemented, **NOT gated** (#891); row stays `PARTIAL` | The 333 `model.visual.*` tensors load and the tower computes (`[1,28,28]`→`[196,2048]`, finite, absmax 2.08) on sm_110 FALLBACK attention. The token-exact mm gate vs the oracle is OWED |