fix(ENG-CUDAGRAPH-BREAK): three registrations never read the async mirror's identifiers, on the graph path and the eager path alike (#1305)

.agents/issue-index.md

@@ -458,6 +458,7 @@ rather than merged. `scripts/check-agent-record.py` gates both.
 | [#1376](https://github.com/mudler/vllm.cpp/issues/1376) | `ENG-CUDAGRAPH-BREAK` | `main` was red on `tests/scripts/test_check_gate_commands.py`, measured at `601b576c6` in a detached worktree of `origin/main`: 8 failures of 44 tests, every one a comparison between the computed runnable population and `RUNNABLE_BASELINE`. `ENG-CUDAGRAPH-BREAK` was in the first and absent from the second. Cause: W5 of that row ([#1361](https://github.com/mudler/vllm.cpp/issues/1361)) filled its spec's `## Gates` section with runnable evidence, including a named test binary with its case and assertion counts and an exit status, which is exactly what moves a row into the runnable population. The ratchet's own error text instructs a re-pin in the SAME change, and the re-pin was not made. This is the growth case the ratchet exists to force a decision about, not a defect in that row's work. **It landed with no remote verdict**: the continuous integration lane that would have caught it independently has not executed for this repository since roughly 07:43Z on 19 August 2026, with runs queueing and none starting while GitHub reports Actions operational. FIXED IN FLOW while merging `origin/main` into `row/ENG-HF-MODEL-DOWNLOAD` for [#1280](https://github.com/mudler/vllm.cpp/issues/1280), because the fix is small and clear and a red `main` blocks every other row's gate. The entry is added with a justifying comment in the form the neighbouring entries use, no checker semantics change, and no test is weakened. After the re-pin the suite reports 45 tests OK and the audit reads 39 runnable of 119 gated rows | bug |
 | [#1378](https://github.com/mudler/vllm.cpp/issues/1378) | `ENG-EXPERT-STREAM-DEVICE` | The expert-stream lane fired without asking which RESIDENCY ROUTE the towers would take, which deleted the #1123 refusal on every non-keep-quant arm. The lane condition at `src/vllm/entrypoints/model_loader.cpp` had four terms and all four are properties of the DEVICE and the ARCHITECTURE: the platform stages, the platform can read host slots, the factory declares `streams_routed_experts`, streaming is requested. None of them is a property of the FILE under the PROCESS's policy. `KqExpertSlice`, the seam that serves a slice out of the host slot store, is reached only from the `expert_*_kq` arm, and `LoadExpertsOrNvfp4` (`qwen3_5_gguf_weights.cpp`) routes the same `blk.<n>.ffn_{gate,up,down}_exps.weight` tensors to `kNvfp4Fp4` and to `kExpandBf16` as well, both of which reach `MoeBlockFusedMarlinCuda`/`MoeBlockFusedCuda`/`MoeBlockBf16Cuda` and stage every tower through `ResidentWeight`. So on a GB10 with `--device cuda`, a larger-than-pool `qwen35moe` GGUF and `VT_MOE_EXPERT_STREAM=1`, EITHER `VT_GGUF_KEEP_QUANT=0` (a documented, supported opt-out) OR an NVFP4 GGUF turned the lane on, dropped every `*_exps.weight` from the device bound, charged an arena nothing allocates, and put back the 26-minute load and the `cudaMalloc: out of memory` first forward that #1123 exists to prevent. The W0c `expert_streamed` tripwire cannot see it either, because a tower on those arms is never claimed. The UNSAFE direction, unlike the two over-counts the bound documents. FOUND by the fresh review of [#1377](https://github.com/mudler/vllm.cpp/pull/1377) and PROVEN by mutation, not argued: adding `VT_GGUF_KEEP_QUANT=0` to the reach suite's fixture left all 12 cases green. FIXED IN FLOW in the repair of that same pull request: a fifth term, `GgufExpertTowersReachSlotLane`, asks the model loader's own `PeekRoute` about this file under this process's policy and accepts only the residencies that reach the kq arm; it is all-or-nothing in the safe direction, so one tower that would be staged keeps the WHOLE bound. Red first: two new reach cases (`VT_GGUF_KEEP_QUANT=0`, and an NVFP4 tower) go from LET THROUGH to REFUSED at the lane-off bound, 2 of 14 cases and 6 of 66 assertions failing before the term exists and 14/14 with it. Spec [`expert-stream-device-slots.md`](specs/expert-stream-device-slots.md) | bug |
 | [#1375](https://github.com/mudler/vllm.cpp/issues/1375) | `MODEL-DIFFUSION-LTX25` | First end-to-end per-forward cost for the FULL 21.004 B LTX-2.5 DiT on GB10, measured on run `20260819T150230Z` with binary `0a43a750` built from [`7b9e207b1`](https://github.com/mudler/vllm.cpp/commit/7b9e207b1) (#1252). At 1024x576/25f (2304 latent tokens) the governor resolved **7 forward starts from the GPU busy/idle edge counter** and measured `per_forward ~162.0 s` with `first_dit = 481.5 s`, so the recipe's fixed 60 forwards (30 steps x 2 CFG legs, `ltx2_pipeline.cpp:521-529`) project **10 803 s against the rung's 7 153 s budget** and the rung was refused rather than run to the wall. The same lease then COMPLETED 768x448/25f (1344 tokens) in 2990 s, so the ceiling is geometry against lease length, not a defect. TWO instrument facts belong with the number, because both have already caused a wrong reading: `gpu_edges=0` means the GPU never went idle long enough to sample an edge (SATURATED), not that no work ran — this rung sampled 85% of 3191 samples above 50% utilisation; and `eu-stack` resolves no frames in the rc worker container, so phase attribution came from the cpu%/rss signature rather than from symbols. Owned by the LTX-2.5 row; spec [`ltx-2-5.md`](specs/ltx-2-5.md) | measurement |
+| [#1390](https://github.com/mudler/vllm.cpp/issues/1390) | `ENG-CUDAGRAPH-BREAK` | `test_qwen3_5_decode_graph_seam` SIGSEGVs on `main` and every assertion passes. Measured at `5f68e60df`, which is `origin/main` exactly, CPU Release x86_64: 8 cases, 7 passed, 1 failed, an assertion line reading 135 of 135 passed and 0 failed, exit 139, with `W6: two spec shapes of EQUAL S and different q get two graphs` reporting `CRASHED: SIGSEGV`. The number a reader greps says 135/135, so only the exit status and the `CRASHED` line carry the verdict. ORDER-DEPENDENT: `-tc="W6*"` alone passes 9/9 exit 0, so the crash needs state an earlier case in the same process left behind — a doctest binary runs every case in one process, and a leaked pool block, a leaked backend or platform registration, or a driver slot captured under one shape and re-entered under another are all live candidates. `gdb -batch -ex run -ex bt` puts the fault inside `vt::cpu::PagedAttentionKernel` on a `vt::cpu::Threadpool` worker, which is what a block table, slot mapping or sequence length that does not describe the handed KV cache looks like. The crashing case is the one [#1374](https://github.com/mudler/vllm.cpp/issues/1374) added for the `(S, q, spec)` ring key. Found while landing [#1305](https://github.com/mudler/vllm.cpp/issues/1305) and NOT caused by it: reverse-applying that branch's whole source change and rebuilding leaves the same exit 139, and that change touches `qwen3_moe.cpp`, `deepseek_v2.cpp` and three registry translation units, none of which this binary's crashing case executes. NOT fixed in flow, because `AGENTS.md` routes a surprising fix to the normal row, spec and fresh-review path: it is a segmentation fault in another stage's newly landed code, its mechanism is an unlocated cross-case state leak, and `src/vllm/model_executor/models/qwen3_5.cpp` is under concurrent edit for [#1380](https://github.com/mudler/vllm.cpp/issues/1380). Owner: row `ENG-CUDAGRAPH-BREAK`, under `## Owed` in [eng-cudagraph-break.md](specs/eng-cudagraph-break.md) | bug |
 | [#1405](https://github.com/mudler/vllm.cpp/issues/1405) | `ENG-CUDAGRAPH-BREAK` | `scripts/check-gate-commands.py` exists to answer whether a gated row's `## Gates` names a command that CAN FAIL, and it drops `true`, `echo ok` and anything containing a pipe for that reason. It does not drop `git diff`. Measured at `96ed8346f`: `runnable_commands` over `.agents/specs/eng-cudagraph-break.md` returns exactly `['git diff']`, and that one always-zero command is the whole reason `ENG-CUDAGRAPH-BREAK` sits in `RUNNABLE_BASELINE` (`:414`). Its `## Gates` names no test binary, least of all `tests/test_qwen3_5_decode_graph_seam`, whose W6 case exited 139 on `main` ([#1390](https://github.com/mudler/vllm.cpp/issues/1390), [#1394](https://github.com/mudler/vllm.cpp/issues/1394)). The EXECUTION gates were surveyed at the same revision and are sound — `ctest` fails on exit 139 (`tests/CMakeLists.txt:30`, `.github/workflows/ci.yml:1013`), `scripts/mutation-harness.py:99-113` reads the exit code and `Status: FAILURE!` and says the summary line is not the authority, `tools/bench/gdn_packed_component.py:1893-1902` requires `Status: SUCCESS!` beside its pinned totals, and `scripts/music3-vocoder-conv-ab.sh:88` recovers `${PIPESTATUS[0]}` past its grep — so NO execution gate reads a crashed run as green, and the exposure is the record layer plus [#1376](https://github.com/mudler/vllm.cpp/issues/1376). Sharper than it reads: a SIGSEGV TRUNCATES the assertion count (135 crashed vs 138 complete on one unchanged file), so a gate pinning a total reports a crash as count drift. NOT fixed in flow: adding `git diff` to `_CANNOT_FAIL` moves rows out of the runnable population, reds `--check` and the exact `RUNNABLE_BASELINE` pin in `tests/scripts/test_check_gate_commands.py`, and `AGENTS.md` routes a semantic checker change to its own row, spec and fresh review | bug |
 | [#1406](https://github.com/mudler/vllm.cpp/issues/1406) | `ENG-CUDAGRAPH-BREAK` | The block-table bound [#1394](https://github.com/mudler/vllm.cpp/issues/1394) adds is CPU-ONLY. Six backends register `OpId::kPagedAttention` (`cpu_paged_attn.cpp:271`, `cuda_paged_attn.cu:2890`, `rocm_ops.hip:148`, `vulkan_ops.cpp:1604`, `metal_ops.mm:1119`, `tenstorrent_ops.cpp:3188`) and the byte-identical `block_table[r * bt_row + (j / block_size) * bt_col]` appears unguarded at `cuda_paged_attn.cu:230,385,521,676,837` and `rocm_paged_attn.hip:226,424,557,765`, with `j` bounded only by `seq_lens[r]`. The obvious repair — one bound at the shared seam `vt::PagedAttention` — is NOT correct as written, and this was measured rather than reasoned: the seam cannot dereference a device `seq_lens`, and its only host-readable quantity `PagedAttentionArgs::max_seq_len` is documented at `include/vt/ops.h:806-812` as a value where **an upper bound is safe**, so a refusal keyed on it rejects a call the contract permits. `row/ENG-CUDAGRAPH-SEAM-SIGSEGV` carried exactly that arm and WITHDREW it for this reason. Three candidate fixes are named in the issue; the smallest that reaches all six backends narrows `max_seq_len` to exact, which the production caller already passes (`v1/attention/backend.cpp:303`, `:57-58`) and which owes a caller audit and a spec | gap |
 | [#1394](https://github.com/mudler/vllm.cpp/issues/1394) | `ENG-CUDAGRAPH-BREAK` | The CPU paged attention reads `btab[r * bt_row + (j / block_size) * bt_col]` for every `j < seq_lens[r]` without checking that the block table has that many columns, so a caller with a short table gets an out-of-bounds read, a plausible block index out of it, and attention over the WRONG page — silently. `tests/vllm/models/test_qwen3_5_decode_graph_seam.cpp`'s `SpecAttnMeta` supplies one: hardcoded `block_table_num_cols = 1` against shape C's `seq_lens = 24` at `block_size = 16`. Found while fixing [#1380](https://github.com/mudler/vllm.cpp/issues/1380), whose `DevicePool` change moved the bytes after the table and turned the same read into a SIGSEGV on `thor:gpu0` (`gdb` at `src/vt/cpu/cpu_paged_attn.cpp:224` under `FullAttnBlockPaged`). PRE-EXISTING: the case passes at `origin/main` only because the read landed on bytes that decoded to an in-range index. FIXED IN FLOW in both halves — the kernel refuses a short table with one compare per request outside the token loop, and the helper sizes its table for the sequence length it declares. Owned by row `ENG-CUDAGRAPH-BREAK` | bug |