diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md index 39f6bd964..d20b0eb92 100644 --- a/.agents/engine-matrix.md +++ b/.agents/engine-matrix.md @@ -39,7 +39,7 @@ forensics: roadmap_v1.md and the parity ledger. | Area | Rows | `ANCHOR-BACKFILL` | `PARTIAL` | `SPIKE` | `READY` | `ACTIVE` | `GATING` | `DONE` | `INVENTORIED` | |---|---:|---:|---:|---:|---:|---:|---:|---:|---:| | Engine and scheduling | 30 | 6 | 3 | 1 | 3 | 8 | 2 | 2 | 5 | -| KV cache and memory | 24 | 7 | 3 | 2 | 3 | 3 | 2 | 0 | 4 | +| KV cache and memory | 25 | 7 | 3 | 2 | 4 | 3 | 2 | 0 | 4 | | Parallelism | 6 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 5 | | Sampling and generation | 15 | 4 | 2 | 0 | 0 | 4 | 0 | 1 | 4 | | Structured output and tools | 7 | 0 | 4 | 0 | 0 | 2 | 0 | 0 | 1 | @@ -48,7 +48,7 @@ forensics: roadmap_v1.md and the parity ledger. | LoRA and adapters | 2 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 1 | | Long context and attention | 10 | 5 | 0 | 0 | 1 | 0 | 1 | 0 | 3 | | Loading, tokenizer, config | 11 | 3 | 3 | 0 | 1 | 1 | 1 | 1 | 1 | -| **Total** | **165** | **35** | **17** | **4** | **13** | **38** | **8** | **11** | **38** | +| **Total** | **166** | **35** | **17** | **4** | **14** | **38** | **8** | **11** | **38** | ## Engine core and scheduling @@ -118,6 +118,7 @@ lifecycle are unchanged. | `ENG-HYBRID-PLACEMENT` | Hybrid device placement: per-tensor-group device assignment resolved at model build, delivering routed-MoE expert COMPUTE on the CPU backend while attention/dense/router/norms stay on GPU (surpass-track — vLLM has the CPU MoE kernels but selects them platform-wide via `current_platform.is_cpu()`, so hybrid placement is ABSENT in pin). Moves compute toward the weights, the inverse of `ENG-WEIGHT-OFFLOAD`/`ENG-EXPERT-STREAM`; composes with `BACKEND-DISTRIBUTED-TP` sharding on an orthogonal axis and refuses where they conflict. Issue [#149](https://github.com/mudler/vllm.cpp/issues/149) (CPU-MoE half only) | T2 | absent in-pin (placement): `vllm/model_executor/layers/fused_moe/oracle/mxfp4.py:533`, `oracle/fp8.py:129`, `oracle/int8.py:53`, `oracle/int_wna16.py:111`, `oracle/unquantized.py:97,202`, `oracle/w4a8_int8.py:40` (all `current_platform.is_cpu()`); kernels present but platform-gated: `vllm/model_executor/layers/fused_moe/experts/cpu_moe.py`, `cpu_fused_moe.py:398,430`, `vllm/_custom_ops.py:3790,3803`; secondary oracle llama.cpp @ `237ad9b96` (`gateable = yes`): `common/arg.cpp:2451-2478` (`-ot`/`-cmoe`/`-ncmoe`), `common/common.h:1046-1054` (`LLM_FFN_EXPS_REGEX`), `src/llama-model-loader.cpp:1158-1160`, `src/llama-model.cpp:1032`, `include/llama.h:530`, auto-fit `common/fit.h:24` + `common/fit.cpp:457,485` (TP conflict refused at `:181`) | - | - | [hybrid-placement.md](specs/hybrid-placement.md) | `READY` | - | | `ENG-RESIDENCY-CONFIG` | The host-RAM→DISK weight-residency tier as a CONFIG surface: a vllm.cpp-original `vllm_cpp` extension key inside the existing `--offload-config` document (`{"vllm_cpp":{"mmap":{"enabled","prefault"},"expert_stream":{"enabled","slots","slot_bytes"}}}`), reaching the loader through `EngineParams::weight_residency` and installed in `LoadedEngine::FromModelDir` ahead of all weight I/O. Closes the asymmetry the two rows above created: `ENG-WEIGHT-OFFLOAD`'s device→host tier is configurable and expert-AWARE (`cpu_offload_params:["experts"]`), while the tier that actually makes `Qwen3.8-2.4T-A95B UD-Q1_0` (370 GiB) serve on a 119 GB box was environment-only. The MIRRORED structs are untouched — upstream has no disk tier, so `include/vllm/config/offload.h` stays a byte-faithful transcription and the extension is vllm.cpp-original by construction. Precedence is **env var > JSON config > built-in default**, deliberately: the `VT_*` variables exist so a benchmark arm is switchable without restarting the server, and an A/B in flight depends on it. `VT_MOE_EXPERT_STREAM_STATS_EVERY` stays environment-only by decision (it changes only a diagnostic cadence — the instrument, not the configuration). Also fixes [#1109](https://github.com/mudler/vllm.cpp/issues/1109) in flow. Issue [#1110](https://github.com/mudler/vllm.cpp/issues/1110) | T1 | ABSENT in-pin, and not a mirror gap: `vllm/config/offload.py:12` is `Literal["auto","uva","prefetch"]` with no disk arm, `vllm/model_executor/offloader/uva.py:21` is CPU-blanket UVA and `offloader/prefetch.py:557-560` is cpu-only — nothing upstream reads a weight off a file at inference time. Surface SHAPE mirrored from the flag next door: `vllm/entrypoints/openai/cli_args.py` JSON-object config arguments, parsed and REFUSED at startup | `include/vllm/config/weight_residency.h`, `src/vllm/config/weight_residency.cpp`; field `include/vllm/entrypoints/model_loader.h`; install `src/vllm/entrypoints/model_loader.cpp` (`LoadedEngine::FromModelDir`); flag `src/vllm/entrypoints/openai/server_main.cpp`; ABI `src/capi/vllm_c.cpp`; resolve sites `src/vllm/model_executor/model_loader/gguf_keep_quant.cpp` (`VT_GGUF_MMAP`), `src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp` (`VT_GGUF_PREFAULT`), `src/vllm/model_executor/models/qwen3_5.cpp` (`VT_MOE_EXPERT_STREAM`, `_SLOTS`, `_SLOT_BYTES`) | [`tests/vllm/config/test_weight_residency_config.cpp`](../tests/vllm/config/test_weight_residency_config.cpp) (parse, unknown-key REFUSAL, the mirrored config coming out byte-identical from the same document, precedence in both directions, the latch); [`tests/vllm/entrypoints/test_weight_residency_reach.cpp`](../tests/vllm/entrypoints/test_weight_residency_reach.cpp) (`FromModelDir` and the C ABI install it); [`tests/vllm/entrypoints/openai/test_serve_residency_config.cpp`](../tests/vllm/entrypoints/openai/test_serve_residency_config.cpp) (the REAL `VllmServerMain` on a real argv — the suite the reachability mutation deletes the install call site under) | [weight-residency-config.md](specs/weight-residency-config.md) | `ACTIVE` | `CLAIM-ENG-RESIDENCY-CONFIG` | | `KV-SGLANG-RADIX-CACHE` | **SGLang RadixAttention behavior-parity scope (fuse-or-flag). VERDICT: already FUSED into our APC.** SGLang's radix TREE of KV prefixes (token/page-granular trie, longest-prefix match with mid-node split, LRU-over-tree eviction, `lock_ref` protection, `extra_key` tenant isolation) is functionally equivalent to our block-hash APC (per-block chain hash, block pool, block-LRU, ref-count, `extra_keys`). The ONLY behavioral delta is sharing granularity (token/page vs `block_size`), bounded to one block at the divergence point, already parametrized by our `block_size`, and output-neutral (APC-ON==APC-OFF token-exact, `KV-PREFIX-CACHE` W3). A second token-granular trie would be a redundant incompatible abstraction against MIRROR-vLLM ⇒ **NO distinct path**. The flag `--enable-radix-attention` is an ALIAS for the APC toggle (`enable_prefix_caching`); implementation = CLI alias + a C-ABI `enable_prefix_caching` tri-state field (currently absent from `vllm_model_params`). SGLang pin v0.5.15 `f63458b`. | T2 | SGLang `mem_cache/radix_cache.py:280` (`RadixCache`), `:217` (`TreeNode`), `:355`/`:648` (`match_prefix`/helper), `:563` (`evict` LRU heap), `:592`/`:607` (lock_ref), `:136`/`:191` (page-aligned match); `mem_cache/registry.py:83-94` (ChunkCache fallback); `server_args.py:755` (`--disable-radix-cache`), `:739` (eviction policy) | our APC (== the fused equivalent): `src/vllm/v1/core/kv_cache_utils.cpp:259,291`; `src/vllm/v1/core/kv_cache_coordinator.cpp:260`; `src/vllm/v1/core/kv_cache_manager.cpp:124`; toggle `examples/server/main.cpp:185`, `include/vllm/entrypoints/model_loader.h:76`; **alias landed (RW1, `CLAIM-SGLANG-IMPL`):** `--enable-radix-attention`/`--disable-radix-attention` server aliases for the same `enable_prefix_caching` tri-state `examples/server/main.cpp`; C-ABI `int32_t vllm_model_params.enable_prefix_caching` (0/1/2) ABI v6→v7 `include/vllm.h`, mapped in `src/capi/vllm_c.cpp` | `KV-PREFIX-CACHE` cache-ON proof reused (`tests/parity/test_qwen3_apc_e2e.cpp:235`, hits 0.807, token-exact); ABI round-trip `tests/capi/test_capi.cpp` "enable_prefix_caching tri-state" 9/9 (default 0, valid 0/1/2, out-of-range→INVALID_ARGUMENT); alias adds server-help + C-ABI contract only, NO engine change (RadixAttention is fused into APC) | [sglang-radixattention.md](specs/sglang-radixattention.md) | `ANCHOR-BACKFILL` | `CLAIM-SGLANG-IMPL` | +| `ENG-EXPERT-STREAM-DEVICE` | The DESTINATION half of expert streaming: where a streamed expert slice lives and which platform may read it, so a larger-than-pool GGUF serves on `--device cuda` instead of refusing at load. `ENG-EXPERT-STREAM` beside it owns the MECHANISM (cache, streamer, `pread` filler, host store) and is unchanged by this row. Three waves. W0 is the critical path and the only one that produces a GPU number on hardware this project owns: a probed `host_memory_is_device_addressable()` predicate lets a unified/integrated platform read slices out of the existing host slot store, while a discrete CUDA device keeps falling through to `KqResidentSlice` and keeps the #1123 refusal. W0 is FOUR edits rather than one guard, because the slot arm itself calls `ResidentWeight` (which stages the whole 1.1875 GiB tower on a staging platform, the exact allocation #1123 died on) and because the load-time fit refusal fires before any forward exists to take the slot arm. W1 adds `DeviceExpertSlotStore` plus the `CommitSlot` fill contract without which a device slot cannot be filled at all (`SlotForWrite` is handed straight to `::pread`); W2 adds the virtual `SlotForRead` that makes W1 reachable, since the read today is the CONCRETE `HostExpertSlotStore::Slot()`. Surpass-track: inference-time disk expert paging is ABSENT in pinned vLLM and no secondary oracle implements it, so the correctness reference is our own CPU arm and the gate is token-exactness against it. Sized: 2790 slices per token at 2,490,368 B is 6.95 GB per token, against 335.62 GiB of `*_exps` towers and a 119.631 GiB device pool. Issue [#1124](https://github.com/mudler/vllm.cpp/issues/1124) | T2 | absent in-pin: `vllm/model_executor/offloader/uva.py:21` (CPU-blanket UVA over whole parameters), `vllm/model_executor/offloader/prefetch.py:557-560` (cpu-only); no secondary oracle either, since llama.cpp `-ot`/`-ncmoe` (`common/arg.cpp:2451-2478` @ `237ad9b96`) moves expert COMPUTE and not slots. Mirrored platform seam this row extends: `vllm/platforms/interface.py:914` + `vllm/platforms/cuda.py:675` `is_integrated_gpu` | - | - | [expert-stream-device-slots.md](specs/expert-stream-device-slots.md) | `READY` | - | ## Parallelism and scale-out @@ -223,7 +224,7 @@ claims it. | `ENG-RELEASE-WINDOWS` | Native Windows x86_64 pre-alpha release extension: one adaptive MSVC/UCRT CPU bundle with AVX2 executed in CI and one Vulkan preview bundle, both deterministic ZIPs and authenticated by the existing release handoff | T0 | vLLM has no Windows release path; runtime behavior remains pinned to vLLM `555967922`. Platform substrate reference: llama.cpp `src/llama-mmap.cpp:520-590` @ `237ad9b961f009ae19ac29dbce4cd0c1251f94b3`; Win32 API is the OS authority | W14 Win32 portability/MSVC CPU, W15 deterministic ZIP/PE packaging + Vulkan, and W16 ten-tuple prerelease workflow/version/docs implemented for one PR | Linux portability/release mutation gates are local evidence only. Native `windows-2022` MSVC `/W4 /WX`, extracted runtime/ISA smokes, merged-SHA ten-tuple dry run, `v0.0.3-pre.1` publication, attestations, and exact 32-asset audit remain pending; no Windows ZIP exists yet | [windows-binary-release.md](specs/windows-binary-release.md); [#117](https://github.com/mudler/vllm.cpp/issues/117) | `ACTIVE` | `CLAIM-ENG-RELEASE-WINDOWS` | | `ENG-RELEASE-CONTAINERS` | Published OCI container images on GHCR, built by GitHub Actions: the same staged server bundle as `ENG-RELEASE-BINARIES`, shipped from one package `ghcr.io/mudler/vllm.cpp` with the lane in the tag — `:-cuda` / `-vulkan` / `-cpu`, the moving `:latest-cuda` / `:latest-vulkan` / `:latest-cpu`, and a bare `:latest` aliasing the cpu lane, with `ENTRYPOINT vllm-server`. Lanes `cuda` (one fat image covering every supported SM), `vulkan`, `cpu` (adaptive baseline); `rocm` blocked-preview, tracking its binary channel. Version tags are immutable; every `latest-` moves. Each lane is a `linux/amd64` + `linux/arm64` multi-arch manifest built on native runners — aarch64 is first-class here because GB10 (sm_121a), Thor (sm_110) and Orin (sm_87) are all arm64. The image contains the bundle and nothing else: no weights, no Python, no PyTorch, no compiler, no build tree. BOUNDARY: the GPU driver and container runtime stay on the host and are never bundled; Metal and MLX are NOT-CONTAINERIZABLE (no macOS container runtime and no Metal passthrough exists) and remain static-binary-only lanes, recorded as a permanent boundary rather than pending work. No image, workflow, registry package or pull is claimed to exist. | T0 | release image lanes `.buildkite/release-pipeline.yaml:34-170` and the published-image dependency boundary `docker/Dockerfile.cpu:262-290` @ `555967922` | `docker/Dockerfile` (cpu/vulkan/cuda targets calling the release scripts); `docker/healthcheck.sh`; `release/container-matrix.json`; `scripts/check-container-matrix.py`; `scripts/check-container-workflow.py`; `scripts/validate-container-image.py`; `scripts/container_tags.py`; `.github/workflows/containers.yml`; SIGTERM handler `src/vllm/entrypoints/openai/server_main.cpp` (`SignalShutdown`, all three `listen()` sites); the pre-existing `docker/Dockerfile.arm64` is an unrelated CPU bench cross-check | issues `#170`, `#312`, `#394`; `tests/scripts/test_check_container_matrix.py` 31/31; `test_check_container_workflow.py` 29/29; `test_check_cuda_fat_gencode.py` 7+4 subtests. **GB10 2026-08-11 (`promaxgb10-4ad8`, `sm_121a`, CUDA 13.3): arm64 cuda image 1.71 GB, 673/673 objects, ten-SM gencode audit PASS, and a REAL GPU boot -- `/health` 200, `/version` 200, in-container healthcheck, clean SIGTERM, `--gpus all`, host driver 580.159.03 injected.** cpu amd64 783 MB gated locally; cpu+vulkan amd64 green on hosted CI **arm64 cuda lane RUNTIME-VERIFIED on GB10 2026-08-11** -- the first accelerator-hardware evidence for any lane. Four defects were removed to get there, each found by building rather than reading: the CUDA 12.9 base could not compile `sm_110`, the BuildKit cache mount outlived its toolchain (both #366), Marlin gencode had drifted from the feature table and failed the audit on 14 correctly-compiled TUs (#394, blocking BOTH cuda tuples project-wide), and the validator could only ever produce build evidence because its boot smoke never passed `--gpus`. **NOT established: nothing is published to GHCR; amd64 cuda is unbuilt; the published arm64 image is SBSA (`targets/sbsa-linux`), so Tegra -- Thor `sm_110`, Orin `sm_87` -- is untested and NOT covered** **ORIN (Tegra) 2026-08-11: the SBSA image RUNS on Jetson AGX Orin `sm_87` (L4T R36.4.3, Docker 27.5.1) -- Qwen3-0.6B (rev `c1899de2`) loads and GENERATES via `/v1/completions`, tegrastats GR3D 95-97% during decode vs 14-15% idle.** Tegra needs `--runtime nvidia --gpus all`: `--gpus` alone is refused by the hook and `--runtime` alone mounts no driver | [container-images.md](specs/container-images.md); issues [#170](https://github.com/mudler/vllm.cpp/issues/170), [#312](https://github.com/mudler/vllm.cpp/issues/312), [#394](https://github.com/mudler/vllm.cpp/issues/394) | `ACTIVE` | `CLAIM-ENG-RELEASE-CONTAINERS-W1-W7` | | `ENG-DOCS-SITE` | Publish the 11 `docs/*.md` as a browsable GitHub Pages site at `https://mudler.github.io/vllm.cpp/` WITHOUT a second copy of the prose. A Hugo site at `website/` mounts `../docs` READ-ONLY and derives everything else from what is already in the files: each page title from the file's first `# H1`, the sidebar order from `website/data/nav.yaml`, and links through a Goldmark render hook (internal `.md` → site URL; the 139 `../.agents/**` and `../AGENTS.md` escapes → GitHub blob URLs, since the protocol tree is deliberately NOT published). **No file under `docs/` is modified, moved, renamed, or given front matter**, so `check-doc-checkpoint.py` and every protocol path reference keep working and there is no second surface that can drift — the whole point of the row. Custom lean layouts, NO theme and NO submodule: off-the-shelf docs themes read titles, weights and menus out of front matter this design deliberately does not have, so each would need its title partial, menu and link hook overridden anyway, and hugo-book additionally floors at Hugo 0.158 against the 0.146.3 pin CI and the local toolchain share. Hard prerequisite inside the repo: `classify_path` in `scripts/check-pr-size.py` FAILS CLOSED on `website/**` (verified: raises `ValueError: unclassified repository path`), so the classifier must learn the path or the PR cannot pass the project's own size gate. Hard prerequisite outside it: GitHub Pages must be enabled with the source set to GitHub Actions — the workflow is inert otherwise. A marketing landing page is explicitly OUT of scope (`README.md` stays the front door), as is any restructuring of `docs/`; the custom domain is parked behind the pending vLLM trademark question | T1 | NO vLLM analogue — upstream's docs are a separate mkdocs site and nothing in this row mirrors upstream *behavior*, so it carries no parity obligation. The STRUCTURAL reference is LocalAI's `.github/workflows/gh-pages.yml` (two Hugo sites merged into one Pages artifact), reduced to the docs half | read-only mount `website/hugo.toml:29`; title-from-H1 `website/layouts/partials/title.html:10`; link rewriting `website/layouts/_default/_markup/render-link.html:27`; guard `scripts/check-site.py:70`; deploy `.github/workflows/gh-pages.yml` | `tests/scripts/test_check_site.py:51,56,66,80,89,97` (6 mutation cases: clean tree, H1 stripped, doc absent from nav, nav entry with no file, duplicated entry, missing nav file); build evidence 14 pages with `docs/bench-evidence` + `docs/superpowers` absent from `public/` and no `href` ending in `.md`; 48 protocol links rewritten in `docs/status/`. NO published page is claimed: GitHub Pages is not yet enabled on the repository, which is the recorded stop condition holding this row at `GATING` | [gh-pages-docs-site.md](specs/gh-pages-docs-site.md); issue [#224](https://github.com/mudler/vllm.cpp/issues/224) | `READY` | `CLAIM-ENG-DOCS-SITE` | -| `ENG-RECORD-ANCHOR-RATCHET` | **The record's `path:line` citations were range-checked and never reported.** `check-agent-record.py` parsed BOTH forms: markdown links, and bare `` `file.cpp:123` `` through `RAW_LOCAL_ANCHOR_RE` since `ee511ca8a`. On a missing file or an out-of-range line `local_line_anchors` runs `continue`, so the bad anchor never reaches the caller, and `is_code_anchor` then answers with **any**, so one good sibling covers the rest. There was no symbol test and no report, and **32 of the 38** offenders are IN RANGE, so range-checking could not have found them. Measured at `8daa67b39`: **832 of 867** in-scope citations (**96.0%**) were already parsed and range-checked, and the **35** new to parsing sit under `.agents/`, `docs/` and `website/`; `EVIDENCED_STATES` omits `ACTIVE`/`READY` entirely and is deliberately NOT widened, because requiring an anchor there raises 85 errors across 53 rows. Even the fraction it saw was only range-checked, never checked to CONTAIN the symbol named beside it — every stale anchor found in the 2026-08-13/14 campaign was in range. LANDED as a device-leakage-shaped ratchet over a recorded baseline, never a bulk cleanup: the backlog is fixed by whoever next touches each row | T1 | none — this is our own record surface; the discipline mirrors AGENTS.md §Records ("cite the `file:line` you ported from") | parser + classifier + ratchet in `check-agent-record.py`: `BARE_CITATION_RE` `scripts/check-agent-record.py:1215` (the bare form), `cell_citations` `scripts/check-agent-record.py:1269` (both forms, with the adjacent-symbol rule), `classify_citation` `scripts/check-agent-record.py:1326` (OK / STALE / BROKEN), `RECORD_ANCHOR_STATES` `scripts/check-agent-record.py:1213` (gap 3: `ACTIVE` and `READY` join the count), `check_record_anchors` `scripts/check-agent-record.py:1449` (the two-way gate); budget in `scripts/record-anchor-baseline.json` | `RecordAnchorRatchet` `tests/scripts/test_agent_record.py:1397` — 10 cases, RED-first, including `test_one_good_link_does_not_cover_a_rotted_bare_citation` `tests/scripts/test_agent_record.py:1465`, the `any()` shape the rot hid in. Five mutants red it: report-only, `EVIDENCED_STATES` restored, links-only, first-citation-only, range-only. Measured baseline **38** (32 STALE + 6 BROKEN); gate wired in `scripts/agent-preflight.sh` and the `agent-record` CI job (`--report`) | [record-anchor-ratchet.md](specs/record-anchor-ratchet.md) | `ACTIVE` | `CLAIM-ENG-RECORD-ANCHOR-RATCHET` | +| `ENG-RECORD-ANCHOR-RATCHET` | **The record's `path:line` citations were range-checked and never reported.** `check-agent-record.py` parsed BOTH forms: markdown links, and bare `` `file.cpp:123` `` through `RAW_LOCAL_ANCHOR_RE` since `ee511ca8a`. On a missing file or an out-of-range line `local_line_anchors` runs `continue`, so the bad anchor never reaches the caller, and `is_code_anchor` then answers with **any**, so one good sibling covers the rest. There was no symbol test and no report, and **32 of the 38** offenders are IN RANGE, so range-checking could not have found them. Measured at `8daa67b39`: **832 of 867** in-scope citations (**96.0%**) were already parsed and range-checked, and the **35** new to parsing sit under `.agents/`, `docs/` and `website/`; `EVIDENCED_STATES` omits `ACTIVE`/`READY` entirely and is deliberately NOT widened, because requiring an anchor there raises 85 errors across 53 rows. Even the fraction it saw was only range-checked, never checked to CONTAIN the symbol named beside it — every stale anchor found in the 2026-08-13/14 campaign was in range. LANDED as a device-leakage-shaped ratchet over a recorded baseline, never a bulk cleanup: the backlog is fixed by whoever next touches each row | T1 | none — this is our own record surface; the discipline mirrors AGENTS.md §Records ("cite the `file:line` you ported from") | parser + classifier + ratchet in `check-agent-record.py`: `BARE_CITATION_RE` `scripts/check-agent-record.py:1227` (the bare form), `cell_citations` `scripts/check-agent-record.py:1281` (both forms, with the adjacent-symbol rule), `classify_citation` `scripts/check-agent-record.py:1338` (OK / STALE / BROKEN), `RECORD_ANCHOR_STATES` `scripts/check-agent-record.py:1225` (gap 3: `ACTIVE` and `READY` join the count), `check_record_anchors` `scripts/check-agent-record.py:1461` (the two-way gate); budget in `scripts/record-anchor-baseline.json` | `RecordAnchorRatchet` `tests/scripts/test_agent_record.py:1397` — 10 cases, RED-first, including `test_one_good_link_does_not_cover_a_rotted_bare_citation` `tests/scripts/test_agent_record.py:1465`, the `any()` shape the rot hid in. Five mutants red it: report-only, `EVIDENCED_STATES` restored, links-only, first-citation-only, range-only. Measured baseline **38** (32 STALE + 6 BROKEN); gate wired in `scripts/agent-preflight.sh` and the `agent-record` CI job (`--report`) | [record-anchor-ratchet.md](specs/record-anchor-ratchet.md) | `ACTIVE` | `CLAIM-ENG-RECORD-ANCHOR-RATCHET` | | `ENG-RECORD-CONFLICT-SURFACES` | Retire the shared record surfaces that make concurrent PRs conflict by construction. MEASURED at `origin/main` `d928e2c3` with `git merge-tree --write-tree` over every open PR: **16 of 29 conflict (55%), and 13 of the 16 conflict in bookkeeping files ONLY**, with no product code involved — `.agents/coordination.md` in 8, `.agents/NOW.md` in 5, `.agents/roadmap_v1.md` in 4, `scripts/check-public-doc-tables.py` in 4, `docs/STATUS.md` in 4, and any `src/`/`tests/` path in just 3. Three defects, each of which GUARANTEES rather than risks a collision. (1) `.agents/NOW.md` is a fixed-size shared buffer at EXACTLY 6000/6000 chars (`check-now-current.py:31`), so adding a row requires evicting another and every PR is a read-modify-write of one global — and the conflict is the LUCKY outcome, since a clean three-way merge would apply both evictions and both additions, silently dropping live rows and blowing the very budget the checker defends. (2) `STATUS_RATCHET = {"chars": 243245}` (`check-public-doc-tables.py:557`) is a hardcoded byte count of a DIFFERENT file that may only fall, so a PR owing `docs/STATUS.md` one lifecycle line must delete unrelated prose from another row to pay for it and edit the checker too; the checker's own comment at `:331` already records the failure (*"a ratchet pinned to the byte turns every concurrently merged row's one-line status edit into a spurious failure"*) and answered it with slack instead of removing the coupling. (3) `.agents/coordination.md`'s active-claims table is insert-at-one-anchor: the six ROCm GDN PRs (#334 #336 #341 #343 #345 #348) are ONE author's sequential stack that conflicts on nothing else, each appending a ~1,500-char row — the PR description, transcribed into a file every other claim also writes. It also contradicts the protocol it serves: `AGENTS.md` holds that *"History is git"* and *"There is no state log"*, yet both claims tables ARE state logs duplicating `gh pr list`, `row/` branch names and issue state; the argument that refuses a waiver registry applies unchanged to a claims registry. Precedent twice over — `policy.csv` retired in `0f3e44ee`, per-class line budgets retired 2026-08-10 because the gate fired on ordinary work. The exonerated surfaces share ONE property, one writer per file: `.agents/specs/.md` (one file per row, **zero conflicts** in the sample), the `*-matrix.md` inventories, and the append-only `.agents/benchmark-record.md`. SCOPE: remove `STATUS_RATCHET` and the doc-gating global counters while KEEPING the per-cell/per-paragraph caps (local, so they couple nothing); remove the active-claims table and derive claims from open PRs and branch names; drop `NOW.md`'s byte budget; order the roadmap's keyed tables by ID so distinct keys stop colliding at one anchor; and record the invariant — **no surface that every PR must write** — in `AGENTS.md`. No product source, kernel or gate semantic moves | T0 | NO vLLM analogue — this is local protocol machinery, so the mirror rule does not apply and no upstream `file:line` exists to port from. Governed instead by `AGENTS.md` §"Changing the rules or a checker", which requires a spec, a red-before test or mutation, and green-after evidence | - | - (spec-before-code: the red-before suites are named in the spec's Tests section — `tests/scripts/test_check_public_doc_tables.py`, `tests/scripts/test_check_now_current.py`, a mutation case per removed rule proving the obligation survives in the retained caps and `check-doc-checkpoint.py`, and a `git merge-tree` merge-shape regression that must be RED before the `NOW.md`/roadmap work and GREEN after) | [retire-shared-record-surfaces.md](specs/retire-shared-record-surfaces.md); issue [#364](https://github.com/mudler/vllm.cpp/issues/364) | `READY` | `CLAIM-ENG-RECORD-CONFLICT-SURFACES` | | `ENG-TRAILER-MERGE-ARTIFACTS` | The trailer gate rejects CORRECT commits because of paragraph placement, and that is why `main` is red on `agent-record`. `check-commit-trailers.py` reads trailers via `git interpret-trailers --parse`, which treats ONLY the final paragraph as the block; GitHub appends `Co-authored-by:` as a SEPARATE trailing paragraph on a squash merge, so a complete correct block becomes invisible and the gate reports it missing. MEASURED: piping `git show -s --format=%B dbd0d51c` into `git interpret-trailers --parse` prints nothing but the co-author line, and 13 of the last 30 commits on `main` fail the check -- unnoticed only because those runs were cancelled (#274), which HID the defect rather than causing it. FIX: fuse consecutive trailing TRAILER-SHAPED paragraphs before parsing. Nothing is relaxed -- the block must still exist, the marker must still sit above it, each declaration must still appear exactly once, and an AI co-author is still forbidden; the block is merely FOUND where the merge tool left it. A prose paragraph still terminates it. REJECTED IN FLIGHT and recorded because it is the more instructive half: a first attempt also collapsed identical duplicate trailers to fix the multi-commit-squash shape, which relaxes the uniqueness rule an existing test already pins. Rewriting that assertion to suit the change is what AGENTS.md forbids, and the distinction is real -- a doubled block is genuinely malformed and fixable at source, whereas the co-author case is a correct commit defeated by the parser. Reverted in full. SCOPE LIMIT, stated rather than implied: this fixes ONE of five observed shapes. `f64f2b71` (bot co-author) is a REAL violation the parse had been hiding and now correctly fails; `87308dea` (GitHub's `---------` separator), `b8293c88` (squash doubled the block) and `b580452d` (merge button, no trailers) stay red by design. Closing those is a merge-method change, not a checker change | T0 | NO vLLM analogue -- local protocol machinery, so the mirror rule does not apply and there is no upstream `file:line` to port from. Governed by `AGENTS.md` §"Changing the rules or a checker" | `scripts/check-commit-trailers.py:60` (`join_trailing_trailer_paragraphs`, `_is_trailer_paragraph`, and the fused `parsed_trailers`) | `tests/scripts/test_check_commit_trailers.py:1` 21 cases -- the RED-BEFORE appended-co-author case plus four GUARDS that keep the fusion bounded (doubled block still fails, contradictory declarations still fail, a no-trailer merge message still fails, prose after the block still fails), all four green before and after; closure [parity-ledger.md#L941](parity-ledger.md#L941) | [trailer-merge-artifacts.md](specs/trailer-merge-artifacts.md); issue [#406](https://github.com/mudler/vllm.cpp/issues/406) | `DONE` | `157080c8` | | `ENG-FORGE-COAUTHOR` | The forbidden-AI-trailer rule was catching ATTRIBUTION rather than an authorship claim, which is why bot-opened PRs red `main` on merge. GitHub composes the squash message itself and appends the account that opened the PR — `Co-authored-by: localai-org-maint-bot <...@users.noreply.github.com>` — and most PRs here are opened by a bot, so nearly every squash trips the AI-identity check. Real instance `f64f2b71`, invisible until #406 repaired the parse, which is why it reads as a new failure and is not one. The rule exists so an AI cannot claim it WROTE the code, and that stays; GitHub is recording who pressed the button, and the AI-involvement claim is already carried separately by `AI-Assisted` and `Assisted-by` in the same block. FIX: accept a `Co-authored-by` at a GitHub account noreply address even when the name matches an AI identity token, keyed on the FORGE'S OWN DOMAIN rather than the name so the exemption cannot be borrowed. A hand-written `Co-authored-by: Claude ` still fails; `Signed-off-by` is excluded from the exemption entirely, because a sign-off is a legal assertion about provenance rather than attribution. `AGENTS.md` records the same distinction in the same change so prose and checker cannot drift | T0 | NO vLLM analogue -- local protocol machinery, so the mirror rule does not apply and there is no upstream `file:line` to port from. Governed by `AGENTS.md` §"Changing the rules or a checker" | `scripts/check-commit-trailers.py:38` (`FORGE_ACCOUNT_EMAIL` and the forbidden-trailer skip) | `tests/scripts/test_check_commit_trailers.py:1` 25 cases -- the RED-BEFORE forge-bot case plus THREE guards that matter more than the relaxation because this LOOSENS a rule: a hand-written AI co-author still fails, `Signed-off-by` at the same noreply address still fails, and a human co-author still passes; all three green before and after. Real commit `f64f2b71` re-verified per commit | [forge-coauthor-attribution.md](specs/forge-coauthor-attribution.md); issue [#418](https://github.com/mudler/vllm.cpp/issues/418) | `ACTIVE` | `CLAIM-ENG-FORGE-COAUTHOR` | diff --git a/.agents/specs/expert-stream-device-slots.md b/.agents/specs/expert-stream-device-slots.md new file mode 100644 index 000000000..ab90156f1 --- /dev/null +++ b/.agents/specs/expert-stream-device-slots.md @@ -0,0 +1,384 @@ +# `ENG-EXPERT-STREAM-DEVICE` — a device-reachable destination for streamed expert slices + +Issue [#1124](https://github.com/mudler/vllm.cpp/issues/1124). Owning row +`ENG-EXPERT-STREAM-DEVICE` ([engine-matrix.md](../engine-matrix.md), KV cache and +memory). Parent row `ENG-EXPERT-STREAM`, spec +[expert-streaming.md](expert-streaming.md), which owns the streaming MECHANISM — +the cache, the streamer, the `pread` filler and the host store — and lists this +capability under its `## Owed`. This row owns only WHERE a slice lives and WHICH +platform may read it. + +## Now + +`READY`, spec committed, no implementation. W0 is the critical path: today +`--device cuda` on `Qwen3.8-2.4T-A95B UD-Q1_0` refuses at load by design +(`76f2a6d84`, issue #1123), so **there is no GPU number for this checkpoint to +take at all**, and nothing downstream of W0 produces one on the hardware this +project owns. The developer's target is that GPU figure. + +## Scope + +**In scope.** + +* **W0 — the integrated/unified path.** Let a platform whose host memory is + device-addressable read expert slices out of the existing host slot store, so + `--device cuda` serves this checkpoint on a GB10 instead of refusing. Keyed on + a PROBED property, never on a device name, an architecture string or a + compute capability. +* **W1 — the device slot store.** A second production `ExpertSlotStore` whose + slots are device allocations, with the fill contract that makes filling one + possible at all. +* **W2 — the device-capable read.** A virtual read on `ExpertSlotStore`, so the + seam `Qwen35ExpertStream::Slice` reads through stops being the concrete + `HostExpertSlotStore`. + +**Out of scope, and owed elsewhere.** Asynchronous prefetch and miss/compute +overlap (`ENG-EXPERT-STREAM` W6, conditional on a measurement). Composing +streaming with the grouped keep-quant MoE path. A Windows filler. A zero-copy +device DMA filler (GPUDirect Storage / `cuFile`) — see the W1 design note, which +records why a staging bounce is chosen instead and what would justify replacing +it. Families other than Qwen3.5 MoE. The KV/activation term of the load-time fit +bound, which `KV-WARMUP-PROFILE` owns. + +**Verdict on issue #1124's piece 3.** #1124 lists "the filler is `pread`-into-host" +as a fourth independent piece. It is not independent and it is not a wave: a +device store cannot be filled AT ALL through today's contract, because +`ExpertSlotStore::SlotForWrite` returns a pointer that `ExpertStreamer::EnsureFile` +hands straight to `::pread`, and `vt::Backend::DeviceMemoryIsHostAddressable` +is false for CUDA, so that pointer is not host-writable. W1 therefore carries a +fill-contract change or W1 cannot land. Planning it as a third wave after W1 +would plan a wave that deadlocks its predecessor. It is folded into W1 below. + +## Upstream chain + +**There is no oracle for this row, and that is a measured statement rather than +a search that came up empty.** Pinned vLLM `555967922` has inference-time expert +paging nowhere: `vllm/model_executor/offloader/uva.py:21` is a CPU-blanket UVA +offloader over whole parameters, and `vllm/model_executor/offloader/prefetch.py:557-560` +is cpu-only. The secondary-oracle table does not rescue it either — llama.cpp's +`-ot` / `-ncmoe` (`common/arg.cpp:2451-2478` @ `237ad9b96`) moves expert COMPUTE +to the host, which is `ENG-HYBRID-PLACEMENT`'s inverse design and not a slot +store. `docs/BENCHMARKS.md` already records the parent row as "correct answer; +no oracle runs this". + +So the reference for correctness is **our own CPU arm on the same checkpoint and +the same binary**, which is the shape `ENG-EXPERT-STREAM` already gates on, and +every gate below is stated against it. Nothing here is mirrored from vLLM, and +nothing here may claim to be. + +The one upstream surface this row DOES mirror is the platform seam it extends: +`vllm/platforms/interface.py:914` + `vllm/platforms/cuda.py:675` +`is_integrated_gpu`, already ported as `platforms::Platform::is_integrated_gpu` +and recorded in that header as "unwired today". W0 is what wires that family of +predicates, and it adds one sibling to it rather than a bespoke test at the call +site — the same discipline `needs_weight_staging` and `supports_fa2_attention` +were hoisted under (accelerator-seam audit §12.3). + +## Our baseline + +Every number below is re-verified against the tree at `fe24a3029` and against +the census recorded in [expert-streaming.md](expert-streaming.md) (both GGUF +tensor tables at revision `567d3e6ac26c5474b18311e619c04350fb9a5556`, 1702 +records parsed against 1702 declared). + +| Quantity | Value | +|---|---| +| one IQ1_XXXS expert tower `[E*N,K]`, E=512 | 1,275,068,416 B = 1.1875 GiB | +| all 279 `*_exps` towers | 360,374,599,680 B = **335.62 GiB** | +| whole tensor table (the fit bound's scope) | 397,245,341,184 B = 369.96 GiB | +| non-expert remainder | 36,870,741,504 B = 34.34 GiB | +| ... minus the MTP / `nextn` block a default load never stages | 27,930,252,800 B = **26.01 GiB** | +| one expert slice = one slot | 2,490,368 B = 2.375 MiB | +| slices per decoded token (93 layers x 10 experts x 3 matrices) | 2790, i.e. 6,948,126,720 B = **6.95 GB (6.47 GiB) per token** | +| device pool, `cudaMemGetInfo` total on `dgx:gpu0` | 128,452,956,160 B = 119.631 GiB, exactly `/proc/meminfo MemTotal` x 1024 | + +### The four pieces, re-derived at `fe24a3029` + +1. **No device store.** `include/vllm/model_executor/host_expert_slot_store.h::HostExpertSlotStore` + is the only production `ExpertSlotStore`; the only other subclass is a test + double. `include/vllm/model_executor/expert_streamer.h:8-9,30-31` says "the + production destination is a contiguous device-side slot array" and "production + writes to device memory". **Both sentences are false today**, and W1 makes them + true rather than adding a second claim beside them. +2. **No device-capable read.** `Qwen35ExpertStream` holds + `std::unique_ptr store_` (`qwen3_5.cpp:5621`) and reads + `store_->Slot(r.slot)` at `:5381` and `:5437` — the CONCRETE class. There is no + `SlotForRead()` on the interface, so the seam cannot be swapped even once a + device store exists. +3. **The filler is `pread`-into-host.** `ExpertStreamer::EnsureFile` takes + `store_.SlotForWrite(acq.slot)` and passes it to `::pread` + (`src/vllm/model_executor/expert_streamer.cpp:76-94`). Not host-writable on a + device allocation; see the scope verdict above. +4. **The consumer is CPU-gated.** `qwen3_5.cpp::KqExpertSlice` takes the slot arm + only under `GetPlatform(d.q.device.type).is_cpu()` (`:5714`); every other + platform falls through to `KqResidentSlice` (`:5218`) and stages the whole + tower. + +### Two facts #1124 does not name, and W0 fails without both + +**(a) The slot arm itself stages the tower.** Inside `KqExpertSlice` the slot +branch calls `ResidentWeight(d, w)` — solely to inherit dtype, device and repack +markers — and then overwrites `.data` with the slot pointer. On CPU +`ResidentWeight` aliases and costs nothing. On a staging platform it takes the +upload branch, `void* p = d.b.Alloc(nb)` at `qwen3_5.cpp:1095` with +`nb = w.bytes.size()`, which is **the whole 1.1875 GiB stacked tower** — and +memoizes it in `w.d_dev`, so the very first streamed slice allocates what +streaming exists to avoid. That is the identical allocation #1123 died on +(48 towers, partway through layer 16 of 93). Lifting the `is_cpu()` guard alone +therefore reproduces #1123 rather than fixing it. W0 must build the slot tensor +WITHOUT `ResidentWeight`, and must refuse loudly if any `*_exps` tower reaches +`ResidentWeight` while streaming is on. + +**(b) The load-time refusal fires first.** `CheckDeviceWeightFit` runs in +`src/vllm/entrypoints/model_loader.cpp` before the tokenizer and before any +weight I/O, gated on `target.needs_weight_staging()`, which is `true` on CUDA +including GB10 — the header at `include/vllm/platforms/interface.h:248-258` says +so explicitly and says why (`is_unified_memory()` / `is_integrated_gpu()` are the +WRONG predicate for staging, because GB10 is physically unified and the CUDA path +still binds distinct device pointers). The bound sums the WHOLE tensor table, +including all 335.62 GiB of `*_exps`, so it refuses this checkpoint before any +forward exists to take the slot arm. **W0 must make the refusal conditional on +the streaming lane serving those towers, or W0 produces no run.** This is the +single reason "lift one guard" is not one guard, and it is scoped here rather +than discovered during implementation. + +The arithmetic that makes the conditional refusal EXACT rather than a fudge +factor: with the lane on, the staged set is the non-expert remainder plus the +slot arena, both known at load. Default load, 8000 slots: +26.01 GiB + 8000 x 2,490,368 B (18.55 GiB) = **44.56 GiB against a 119.631 GiB +pool**. No headroom fraction is invented, and the over-count direction the +existing bound documents (issue #1136) is unchanged. + +### The CUDA kernel exists + +`src/vt/cuda/cuda_quant_dot.cu` carries `WType::kIQ1_XXXS` through +`LaunchGemm`, `LaunchGroupedGemm` and `LaunchGroupedFusedSwiGLU` +(`:1874,1967,2182`), transcribed from the pinned `llama-cpp-unsloth` fork +oracle. So the encoding holding 96.92 % of this checkpoint has a CUDA GEMM, and +W0 is not blocked on a missing kernel. This is stated because it is the first +thing that would have killed the wave. + +### The prefill exhaustion fact any W0 harness must handle + +The last two-arm run recorded `exhausted=7813` on an 8000-slot cache, identical +at steps 2, 3 and 4 — all of it in prefill. That is not a defect and no slot +count fixes it. One step is one forward, prefill processes the whole prompt in +one forward, and slices acquired within a step are protected from eviction, so +the peak protected set for a T-token prompt is + + 93 layers x 3 matrices x min(512, 10*T) experts + +which saturates at 142,848 slices = **331 GiB** for any T >= 52 — the whole +model. A decode step touches ~2790 and exhausts nothing. A gate that reads TOTAL +`exhausted` therefore reports a red for a healthy lane. See `## Gates` G0-LIVE +for the two admissible harness shapes and which one is primary. + +## Port map + +Nothing is ported; there is no upstream. This is the local change map. + +| Wave | Surface | Change | +|---|---|---| +| W0a | scratch probe, no tree change | print `cudaDevAttrPageableMemoryAccess`, `cudaDevAttrPageableMemoryAccessUsesHostPageTables`, `cudaDevAttrIntegrated`, `cudaDevAttrConcurrentManagedAccess` on `dgx:gpu0` | +| W0b | `include/vllm/platforms/interface.h`, `src/vllm/platforms/cuda.cpp`, `src/vllm/platforms/rocm.cpp` | new `virtual bool host_memory_is_device_addressable() const { return false; }` beside `is_integrated_gpu`; CUDA overrides from a probe taken once at registration next to the existing `cudaDevAttrIntegrated` probe; ROCm overrides from the `pageable_memory_access` capability it ALREADY probes (`rocm_backend.hip:96-103`) | +| W0c | `src/vllm/model_executor/models/qwen3_5.cpp` | `KqExpertSlice` takes the slot arm under `is_cpu()` OR `host_memory_is_device_addressable()`; the slot branch builds its tensor without `ResidentWeight`; a named `VT_CHECK` in `ResidentWeight` refuses a streamed `*_exps` tower reaching device staging | +| W0d | `include/vllm/model_executor/model_loader/gguf_device_fit.h`, `src/.../gguf_device_fit.cpp`, `src/vllm/entrypoints/model_loader.cpp` | the fit bound gains an explicit "these tensors are served by the slot lane, and the arena costs this instead" input; the loader passes it when the resolved config says streaming is on and the platform can read host slots | +| W1 | `include/vllm/model_executor/expert_streamer.h`, new `include/vllm/model_executor/device_expert_slot_store.h` | `CommitSlot(int32_t, size_t)` on `ExpertSlotStore` (no-op on the host store); `DeviceExpertSlotStore` allocating slots through `vt::Backend::Alloc` with one pinned host staging slot, `SlotForWrite` returning staging and `CommitSlot` doing the H2D; correct the two false sentences in `expert_streamer.h` | +| W2 | `expert_streamer.h`, `host_expert_slot_store.h`, `device_expert_slot_store.h`, `qwen3_5.cpp` | `virtual uint8_t* SlotForRead(int32_t)`; `Qwen35ExpertStream::store_` becomes `std::unique_ptr`; `:5381` and `:5437` read through the virtual; the store is selected from the platform | + +## Tests to port + +**None.** There is no upstream implementation, so there is no upstream test +suite, no fixture, no tolerance and no revision anchor to preserve. Recording +that explicitly is the requirement; inventing a "ported" label for locally +written tests would falsify the porting inventory. Scratch-written tests are +recorded as such in `porting-inventory.md` §9. + +The tests this row WRITES, red-first, with the defect each is proven against by +mutation: + +| Test | Proves | Mutation that must red it | +|---|---|---| +| `tests/vllm/platforms/test_platform.cpp` (extend) | the new predicate defaults false and the CUDA/ROCm assembly threads the probed value | flip the default to true; drop the assignment | +| `tests/vllm/model_executor/test_expert_slot_store.cpp` (new) | a device-flavoured store filled via `EnsureFile` yields byte-identical slot content to the host store | delete `CommitSlot`'s copy; return staging from `SlotForRead` | +| `tests/vllm/model_executor/test_gguf_device_fit.cpp` (extend) | with the lane on, the bound excludes `*_exps` and adds the arena; with it off, the bound is byte-identical to today | make the exclusion unconditional | +| `tests/vllm/entrypoints/test_gguf_device_fit_reach.cpp` (extend) | the loader reaches the conditional refusal from the production entry point | delete the production call site | +| a `qwen3_5` slot-arm unit gate | the slot branch never calls `ResidentWeight`, and a streamed tower reaching device staging throws by name | remove the `VT_CHECK`; restore the `ResidentWeight` call | + +## Gates + +Correctness first, always. No speed number from this row is admissible until +G0-CORRECT and G0-LIVE both pass in the SAME run that produced it — that is the +whole content of #912 F1, where a published decode figure measured a dead cache. + +**G0-CORRECT (W0, blocking).** `Qwen3.8-2.4T-A95B UD-Q1_0`, `--device cuda`, +streaming ON, greedy, one fixed prompt, 32 decoded tokens. The token IDs must be +**byte-identical** to `--device cpu`, streaming ON, same binary, same checkpoint, +same prompt. "It ran" is not a result and neither is prose that looks coherent. +A mismatch stops the wave and no speed number is reported. + +**G0-LIVE (W0, blocking).** From the same process: + +* `steps > 0` — the step clock advanced. `steps == 0` with a printed decode + figure is exactly the #912 F1 shape. +* `forced == 0` — no slice was served by the forced-fallback switch. +* **decode-phase `exhausted` delta == 0.** Not the total. The primary harness + snapshots `detail::ExpertStreamSnapshot()` immediately after the prefill step + and again at the end, and gates the DIFFERENCE. That is the shape a real + benchmark needs, because it works at any prompt length, and prefill exhaustion + is structural (see `## Our baseline`). +* Secondary control only, not the benchmark: a prompt of T <= 4 tokens with + >= 11,160 slots makes prefill itself fit — 11,160 x 2,490,368 B = 25.88 GiB of + arena plus 26.01 GiB of weights = 51.89 GiB, inside the 119.631 GiB pool — so + TOTAL `exhausted` is legitimately 0. It is a useful cross-check on the + snapshot arithmetic and it is not a workload, so it never replaces the diff. + +**G0-SPEED (W0, reporting).** Decode s/token, CUDA arm against the CPU arm, +same box, same `rc` lease, same prompt, same token count, three interleaved +reps, min and median both recorded, plus resident bytes and the fill/hit/byte +counters. **No floor is set and none may be invented.** This row's first number +is a measurement of an unknown, and a CUDA arm SLOWER than the CPU arm is a +real, publishable result that closes the unified shortcut — recorded in +`docs/BENCHMARKS.md` as a measured negative, not as a failure to be tuned away. + +**G1 (W1).** `DeviceExpertSlotStore` driven through `ExpertStreamer::EnsureFile` +produces byte-identical slot contents to `HostExpertSlotStore` on the same +input, on a CPU `vt::Backend`, red-first and mutation-proven per the table above. + +**G2 (W2, reachability).** Per `## Nothing lands dead`: delete the production +selection of the device store in a scratch copy and rerun the focused gate. A +gate that stays green without it measured a class, not a capability. + +**G-DISCRETE (owed, cannot run here).** See `## Owed`. + +## Dependencies + +| Dependency | Shape | +|---|---| +| **CPU decode re-measure on a live cache** | The DENOMINATOR for G0-SPEED, not a gate that blocks starting. `docs/BENCHMARKS.md:8` records streaming-ON decode as VOID (#912 F1) with a re-measure owed, and the operator is arranging it separately. G0-CORRECT and G0-LIVE need no denominator and can run first. **If the box frees up for only one run, take the CUDA arm** — it is the one that does not exist at all today, it produces the correctness verdict as well as a number, and the CPU denominator can follow. | +| `dgx:gpu0` | The only GB10. Leased through `rc`, never bare ssh; the hold sits behind a trap that releases on every exit path. Currently held by another agent's Nemotron gate, with a second agent on thor, so W0's measurement QUEUES. W0a/W0b/W0c/W0d are all buildable and unit-gatable without it. | +| The 370 GiB checkpoint | Already staged for the parent row; recipe in [expert-streaming.md](expert-streaming.md) "Loading a 370 GiB split GGUF". | +| PRs #1200 and #1216 | Both edit `.agents/specs/expert-streaming.md`. This row does not, deliberately — see `## Risks/decisions`. | +| #1126 (`Backend::DeviceMemoryInfo` has no CUDA override) | NOT a blocker. W0 and W1 read the budget from `residency_policy().device_memory_total_bytes`, which CUDA already probes at registration; they never touch the live free/total seam #1126 owns. | +| PR #1203 | Introduces `path::Symbol` citations and `scripts/check-symbol-anchors.py`. This spec uses symbol anchors where a symbol exists and lines only where the citation is to a comment or a specific statement. | + +## Work breakdown + +Each wave states its own gate and its own stop condition. A stop condition is +where the wave ENDS, not where it degrades quietly into the next one. + +### W0 — the integrated/unified path (critical path for the GPU number) + +* **W0a — the probe.** ~30 lines, no tree change, inside an `rc` lease: print + `cudaDevAttrPageableMemoryAccess`, `...UsesHostPageTables`, `cudaDevAttrIntegrated` + and `cudaDevAttrConcurrentManagedAccess` on `dgx:gpu0`. This is the cheapest + decisive experiment in the row and it runs first, because the whole W0 + mechanism rests on a GB10 CUDA kernel being able to dereference a pointer from + the host slot arena, which is plain `std::vector` storage. + **Stop condition:** if `PageableMemoryAccess == 0`, W0b as designed is wrong. + The alternative is a `cudaHostAlloc`/`cudaHostRegister` arena, which works on + ANY CUDA device including discrete but changes the store's allocator and its + ownership story. That is a different design and it comes back as + `NEEDS_DECISION` rather than being substituted silently. +* **W0b — the predicate.** `host_memory_is_device_addressable()`, base false, + CUDA from the W0a attribute, ROCm from its already-probed + `pageable_memory_access`. **Why not an existing predicate:** `is_cpu()` is what + is being lifted; `needs_weight_staging()` is true on CUDA everywhere and would + gate nothing; `is_unified_memory()` answers the opposite question (backend.h:58-66 + records that CUDA on GB10 reports unified while a `cudaMalloc` pointer is still + not host-dereferenceable); and `is_integrated_gpu()` alone is insufficient + because `rocm.cpp:52-58` already documents integrated-without-pageable-access as + a real device class. A discrete CUDA device answers false, keeps falling through + to `KqResidentSlice`, and therefore keeps hitting the #1123 refusal — which is + correct, and which is what W1/W2 exist to remove. + **Gate:** `tests/vllm/platforms/test_platform.cpp`, mutation-proven. + **Stop condition:** none; this wave is small and self-contained. +* **W0c — the slot arm without the tower.** Lift the guard, build the slot + tensor directly instead of through `ResidentWeight`, and add the named refusal + for a streamed `*_exps` tower reaching device staging. + **Gate:** the `qwen3_5` slot-arm unit gate above. + **Stop condition:** if the marker set `ResidentWeight` carries cannot be + reproduced without staging (a repack marker that only the upload path can + set), stop — that is a different change and it is not this wave. +* **W0d — the conditional refusal.** Teach the fit bound the streamed-tensor + exclusion and the arena, and pass it from the loader. + **Gate:** `test_gguf_device_fit` for the arithmetic and + `test_gguf_device_fit_reach` for the production reach; the lane-off bound must + be byte-identical to today. + **Stop condition:** if the exclusion cannot be expressed without the bound + taking a general per-tensor staging POLICY (the shape #1136 explicitly refuses + to invent), stop and return `NEEDS_DECISION`. The lane's tensor set is + `*_exps` and is knowable; a general policy input is not. +* **W0e — the measurement.** G0-CORRECT, G0-LIVE, G0-SPEED, on one lease. + **Stop condition:** a token mismatch, `steps == 0`, or a non-zero decode-phase + `exhausted` delta stops the wave and voids the number. + +### W1 — the device slot store (with its fill contract) + +`DeviceExpertSlotStore` allocates its slot array through `vt::Backend::Alloc`. +It cannot be filled by `pread` (scope verdict above), so W1 adds +`CommitSlot(int32_t slot, size_t bytes)` to `ExpertSlotStore`: `SlotForWrite` +returns a host-writable staging buffer, `pread` fills it as it does today, and +`CommitSlot` performs the single contiguous H2D. `HostExpertSlotStore::CommitSlot` +is a no-op, so the host path stays byte-identical and keeps its direct +`pread`-into-slot. `ExpertStreamer` calls `CommitSlot` only on the success path; +the existing `catch` that invalidates the cache entry already covers the +partial-fill case the parent spec documents at length. + +**Design answer: staging bounce, not a bespoke device filler.** A true zero-copy +filler (GPUDirect Storage / `cuFile`, or an `O_DIRECT` DMA into a device BAR +mapping) moves fewer bytes, needs a driver capability probe, a mount-level +check, an aligned-I/O path and a fallback for every case that fails those. The +bounce costs one extra host-to-device copy of 2.375 MiB per miss on top of a +disk read of the same size, lands in one wave, and keeps the zero-copy filler +genuinely optional rather than load-bearing. It is chosen for that reason and +not because it is faster; the measurement that would justify replacing it is a +device-arm decode where the H2D leg is a measurable fraction of fill time, and +that measurement does not exist yet. Recorded in `## Owed`. + +**Gate:** G1. **Reachability:** W1 alone lands UNREACHED — nothing selects the +store and nothing can read it, because the read is still the concrete +`HostExpertSlotStore::Slot()`. Per `## Nothing lands dead` that is admissible +only if the commit body, the PR body and this spec's `## Owed` all name it and +name W2 as the owning wiring. **The cheaper and more honest shape is to land W1 +and W2 as one pull request**, and that is the recommendation here; splitting +them is a scheduling choice that costs an explicitly-declared unreached slice. + +**Stop condition:** if `CommitSlot` cannot be added without changing every +existing `ExpertSlotStore` caller's contract in a way that alters host-path +behaviour, stop — the host path must stay byte-identical, and a change that +cannot preserve that is a different design. + +### W2 — the device-capable read + +`virtual uint8_t* SlotForRead(int32_t)` on `ExpertSlotStore`; +`Qwen35ExpertStream::store_` becomes `std::unique_ptr`; the two +`store_->Slot(...)` reads go through the virtual; the concrete store is selected +from the platform. **Gate:** G2, the call-site-deletion mutation. +**Stop condition:** if the returned pointer's device-ness cannot be expressed in +the `vt::Tensor` the GEMM binds without a second change to the tensor +construction, stop and re-scope — that is W0c's territory and it must not be +re-derived here. + +## Risks/decisions + +| Risk / decision | Call | +|---|---| +| **The GB10 ATS penalty could erase W0's win entirely.** Device access to host-resident weights on GB10 is recorded as carrying a real penalty, and this lane reads 6.95 GB per token that way. | Accepted as the thing being measured, not assumed away. G0-SPEED is what settles it, and the settling measurement is named: the CUDA arm's decode s/token against the CPU arm's, same box, same lease, three interleaved reps. A CUDA arm at or above the CPU arm's time is a genuine negative result — it closes the unified shortcut for this box, leaves W1/W2 standing for the discrete case, and is recorded in `docs/BENCHMARKS.md` as measured. It is a few hours, not a campaign, and that asymmetry is why W0 runs first. | +| W0 is FOUR edits, not one guard. | Stated rather than discovered. The two forcing facts are in `## Our baseline` (a): the slot arm's own `ResidentWeight` call stages the tower, and (b): the load-time refusal fires before any forward. Neither is in #1124's four pieces. W0d touches the fit bound, which is the one place this wave reaches into another row's surface; it is additive and exact (an explicit tensor exclusion plus the arena), it does not touch the KV/activation term `KV-WARMUP-PROFILE` owns, and if it cannot stay that way the W0d stop condition returns `NEEDS_DECISION`. | +| W1/W2 can be built here but only VALIDATED on hardware nobody here has. | Recorded as owed with its exact measurement rather than dressed as a gate. This host has no discrete NVIDIA GPU, and on `dgx:gpu0` device memory IS host memory, so a device store there proves the plumbing and not the capability. See `## Owed`. | +| A device store is the wrong shape if the answer is "put the slots in pinned host memory". | Open, and W0a's result informs it. A `cudaHostAlloc` arena is device-readable on discrete GPUs too, over PCIe, which is almost certainly too slow to serve 6.95 GB/token — but it is the fallback if `PageableMemoryAccess == 0` on GB10, and it is written down here so it is not re-invented as a surprise mid-wave. | +| This row does not edit `.agents/specs/expert-streaming.md`. | Deliberate. PRs #1200 and #1216 both edit that file today, and its `## Owed` entry for #1124 remains TRUE as written — it names the capability and the issue, and it does not name a row ID that this row's existence falsifies. Cross-linking is one-way, from here to there. Re-pointing that entry at this row is a one-line follow-up once both PRs land, and it is not worth a conflict now. | +| `ENGINE_ROWS` in `scripts/check-agent-record.py` is a shared counter, exactly the "measurement of one file stored in another" coupling AGENTS.md warns about. | Bumped 162 -> 163 for a real new row, with its justification paragraph, per the constant's own comment history. Checked against every open PR: none bumps it (#851 carries a stale `156` as diff context, #361 does not touch it), so this addition takes the lock cleanly. | +| Splitting the device capability out of `ENG-EXPERT-STREAM` rather than adding a W7-W9 to it. | The parent row's spec is 1600 lines and has two open PRs editing it. A separate row gives this capability an independent lifecycle state and a per-row spec surface, which is the shape AGENTS.md prefers (one file per row, read with a glob). The parent keeps the MECHANISM; this row owns the DESTINATION. | +| Streaming and the grouped keep-quant MoE path remain mutually exclusive. | Unchanged by this row, and unchanged by W0. `VT_MOE_EXPERT_STREAM=1` still disables grouping and says so once on stderr. Making them compose needs a slot-aware grouped GEMM and is its own row. | + +## Owed + +| Owed | Why it is open | +|---|---| +| **G-DISCRETE: validate W1/W2 on a discrete NVIDIA GPU.** The measurement: on a device with VRAM V and `host_memory_is_device_addressable() == false`, load a GGUF whose `*_exps` towers exceed V, with the lane on, and gate (i) token-exactness against the CPU arm on the same checkpoint, (ii) decode-phase `exhausted` delta 0, (iii) peak device allocation <= non-expert remainder + arena. | No discrete NVIDIA GPU is reachable from this project. `dgx:gpu0` is a GB10 where device memory IS host memory, so a device store there exercises the plumbing and not the thing W1 exists for. Recorded rather than implied, because a gate nobody can run is not a gate. | +| **A zero-copy device filler (GPUDirect Storage / `cuFile`).** | W1 ships the staging bounce by choice, for the reasons in its design note. The measurement that would justify replacing it — a device-arm decode where the H2D leg is a measurable fraction of fill time — does not exist until W1 has run somewhere. | +| **The CPU arm's streaming decode figure is still VOID.** `docs/BENCHMARKS.md:8` records it as VOID (#912 F1) with a re-measure owed. | Owned by `ENG-EXPERT-STREAM` and arranged separately by the operator. It is the DENOMINATOR for G0-SPEED, not a precondition for G0-CORRECT or G0-LIVE. | +| **`.agents/specs/expert-streaming.md`'s `## Owed` entry for #1124 still names no owning row ID.** | Not edited here on purpose; PRs #1200 and #1216 both edit that file. One-line follow-up once both land. | +| **W1 may land UNREACHED if it is split from W2.** | The recommendation is one pull request. If a split is chosen, the commit body and the PR body must name what is unreached and name W2 as the owning wiring, per `## Nothing lands dead`. | diff --git a/scripts/check-agent-record.py b/scripts/check-agent-record.py index 9303710e5..5b099c447 100644 --- a/scripts/check-agent-record.py +++ b/scripts/check-agent-record.py @@ -588,8 +588,20 @@ # head `19c93519`, base `9842d701`); the parity pin `555967922` does not carry the # architecture at all, so this row does not advance the pin. `READY`, spec # `specs/dflash2-spec-decode.md`, issue #1314. +# 163 since 2026-08-18: +`ENG-EXPERT-STREAM-DEVICE` (the DESTINATION half of expert +# streaming -- where a streamed slice lives and which platform may read it). Genuinely +# new and not expressible by `ENG-EXPERT-STREAM` beside it: that row owns the streaming +# MECHANISM -- the slot cache, the streamer, the `pread` filler and the host store -- +# and all of it landed and runs on `--device cpu`, while this row changes only the +# destination those bytes are written to and the predicate that decides who may read +# them. The two have different hardware requirements and different lifecycle states: +# the parent is `READY` with a live CPU lane, and this one cannot reach its own +# discrete-GPU gate on any box the project owns. Surpass-track, no oracle: inference-time +# disk expert paging is absent in pinned vLLM (`offloader/uva.py:21`, +# `offloader/prefetch.py:557-560`) and no secondary oracle implements it either. +# `READY`, spec `specs/expert-stream-device-slots.md`, issue #1124. # Bumped for a real new row, never to make a failing state transition pass. -ENGINE_ROWS = 165 +ENGINE_ROWS = 166 ENGINE_SUMMARY_SECTIONS = ( ("Engine and scheduling", "Engine core and scheduling"),