fix(BACKEND-ATTN-REGISTRY): CPU_ATTN was a name in a list, so a head size FLASH_ATTN declines left the CPU with no backend (#1371)

.agents/issue-index.md

@@ -458,6 +458,8 @@ rather than merged. `scripts/check-agent-record.py` gates both.
 | [#1376](https://github.com/mudler/vllm.cpp/issues/1376) | `ENG-CUDAGRAPH-BREAK` | `main` was red on `tests/scripts/test_check_gate_commands.py`, measured at `601b576c6` in a detached worktree of `origin/main`: 8 failures of 44 tests, every one a comparison between the computed runnable population and `RUNNABLE_BASELINE`. `ENG-CUDAGRAPH-BREAK` was in the first and absent from the second. Cause: W5 of that row ([#1361](https://github.com/mudler/vllm.cpp/issues/1361)) filled its spec's `## Gates` section with runnable evidence, including a named test binary with its case and assertion counts and an exit status, which is exactly what moves a row into the runnable population. The ratchet's own error text instructs a re-pin in the SAME change, and the re-pin was not made. This is the growth case the ratchet exists to force a decision about, not a defect in that row's work. **It landed with no remote verdict**: the continuous integration lane that would have caught it independently has not executed for this repository since roughly 07:43Z on 19 August 2026, with runs queueing and none starting while GitHub reports Actions operational. FIXED IN FLOW while merging `origin/main` into `row/ENG-HF-MODEL-DOWNLOAD` for [#1280](https://github.com/mudler/vllm.cpp/issues/1280), because the fix is small and clear and a red `main` blocks every other row's gate. The entry is added with a justifying comment in the form the neighbouring entries use, no checker semantics change, and no test is weakened. After the re-pin the suite reports 45 tests OK and the audit reads 39 runnable of 119 gated rows | bug |
 | [#1378](https://github.com/mudler/vllm.cpp/issues/1378) | `ENG-EXPERT-STREAM-DEVICE` | The expert-stream lane fired without asking which RESIDENCY ROUTE the towers would take, which deleted the #1123 refusal on every non-keep-quant arm. The lane condition at `src/vllm/entrypoints/model_loader.cpp` had four terms and all four are properties of the DEVICE and the ARCHITECTURE: the platform stages, the platform can read host slots, the factory declares `streams_routed_experts`, streaming is requested. None of them is a property of the FILE under the PROCESS's policy. `KqExpertSlice`, the seam that serves a slice out of the host slot store, is reached only from the `expert_*_kq` arm, and `LoadExpertsOrNvfp4` (`qwen3_5_gguf_weights.cpp`) routes the same `blk.<n>.ffn_{gate,up,down}_exps.weight` tensors to `kNvfp4Fp4` and to `kExpandBf16` as well, both of which reach `MoeBlockFusedMarlinCuda`/`MoeBlockFusedCuda`/`MoeBlockBf16Cuda` and stage every tower through `ResidentWeight`. So on a GB10 with `--device cuda`, a larger-than-pool `qwen35moe` GGUF and `VT_MOE_EXPERT_STREAM=1`, EITHER `VT_GGUF_KEEP_QUANT=0` (a documented, supported opt-out) OR an NVFP4 GGUF turned the lane on, dropped every `*_exps.weight` from the device bound, charged an arena nothing allocates, and put back the 26-minute load and the `cudaMalloc: out of memory` first forward that #1123 exists to prevent. The W0c `expert_streamed` tripwire cannot see it either, because a tower on those arms is never claimed. The UNSAFE direction, unlike the two over-counts the bound documents. FOUND by the fresh review of [#1377](https://github.com/mudler/vllm.cpp/pull/1377) and PROVEN by mutation, not argued: adding `VT_GGUF_KEEP_QUANT=0` to the reach suite's fixture left all 12 cases green. FIXED IN FLOW in the repair of that same pull request: a fifth term, `GgufExpertTowersReachSlotLane`, asks the model loader's own `PeekRoute` about this file under this process's policy and accepts only the residencies that reach the kq arm; it is all-or-nothing in the safe direction, so one tower that would be staged keeps the WHOLE bound. Red first: two new reach cases (`VT_GGUF_KEEP_QUANT=0`, and an NVFP4 tower) go from LET THROUGH to REFUSED at the lane-off bound, 2 of 14 cases and 6 of 66 assertions failing before the term exists and 14/14 with it. Spec [`expert-stream-device-slots.md`](specs/expert-stream-device-slots.md) | bug |
 | [#1375](https://github.com/mudler/vllm.cpp/issues/1375) | `MODEL-DIFFUSION-LTX25` | First end-to-end per-forward cost for the FULL 21.004 B LTX-2.5 DiT on GB10, measured on run `20260819T150230Z` with binary `0a43a750` built from [`7b9e207b1`](https://github.com/mudler/vllm.cpp/commit/7b9e207b1) (#1252). At 1024x576/25f (2304 latent tokens) the governor resolved **7 forward starts from the GPU busy/idle edge counter** and measured `per_forward ~162.0 s` with `first_dit = 481.5 s`, so the recipe's fixed 60 forwards (30 steps x 2 CFG legs, `ltx2_pipeline.cpp:521-529`) project **10 803 s against the rung's 7 153 s budget** and the rung was refused rather than run to the wall. The same lease then COMPLETED 768x448/25f (1344 tokens) in 2990 s, so the ceiling is geometry against lease length, not a defect. TWO instrument facts belong with the number, because both have already caused a wrong reading: `gpu_edges=0` means the GPU never went idle long enough to sample an edge (SATURATED), not that no work ran — this rung sampled 85% of 3191 samples above 50% utilisation; and `eu-stack` resolves no frames in the rc worker container, so phase attribution came from the cpu%/rss signature rather than from symbols. Owned by the LTX-2.5 row; spec [`ltx-2-5.md`](specs/ltx-2-5.md) | measurement |
+| [#1371](https://github.com/mudler/vllm.cpp/issues/1371) | `BACKEND-ATTN-REGISTRY` | CPU had NO attention backend for a head size FLASH_ATTN declines, and the summary line said `0 failed` while it happened. `CpuPlatform::get_attn_backend_priority` returned `{CPU_ATTN, FLASH_ATTN}` where `CPU_ATTN` was a name in a list and nothing else — never registered, so every CPU selection fell through to `FLASH_ATTN`, recorded as behavior-preserving because our CPU paged-attention kernel reads that backend's NHD layout. It was behavior-preserving only while `FLASH_ATTN` accepted everything CPU asked. [#1332](https://github.com/mudler/vllm.cpp/issues/1332) M1 (`369ea7fd4`) gave it `flash_attn.py:170-178`'s `head_size % 8 == 0` rule, and a CPU request with `head_size` 6 then matched no registered backend at all: `SelectAttentionBackendName` throws out of `GPUModelRunner::initialize_kv_cache` with `No valid attention backend for device type 0 from {FLASH_ATTN: [head_size not supported]}`. Attributed by clean before/after on one row's own two shas across two boxes and two architectures (`68a0ff378` on thor sm_110, 13/13 passed; `e35c14d52` on dgx sm_121a, 11 of 13 thrown), so neither box- nor arch-specific. **The reporting shape is the second half of this issue**: assertions collapse `3272 -> 18` while doctest still prints `0 failed`, because a THROWN case runs no assertions — a gate grepping `assertions:` reads eighteen assertions, eighteen passed, zero failed, and calls it clean. Fixed by registering `CPU_ATTN` (`src/vllm/v1/attention/backends/cpu_attn.cpp`), which is the answer upstream `cpu.py:75-87` gives for every CPU request; the FA2 rule is untouched, because it is right about the FA2 kernel and was never about ours | bug |
+| [#1389](https://github.com/mudler/vllm.cpp/issues/1389) | `BACKEND-ATTN-REGISTRY` | Metal, Vulkan and Tenstorrent inherit FlashAttention-2's head-size rule from a backend class they share by NAME, so a head size their own kernels run has no backend. Each registers `FlashAttentionBackend` for `FLASH_ATTN` (`src/vllm/v1/attention/backend.cpp:395-417`) and each returns `{"FLASH_ATTN"}` as its entire non-MLA priority list; the documented precondition for those rows is a shared NHD KV layout, not a shared kernel. Since [#1332](https://github.com/mudler/vllm.cpp/issues/1332) M1 that class also carries `supports_head_size` = `head_size % 8 == 0 && head_size <= 256` (`include/vllm/v1/attention/backend.h:394-397`, `flash_attn.py:170-178` @ pin `5559679229`), a true statement about a CUDA kernel none of those three devices runs. Same defect [#1371](https://github.com/mudler/vllm.cpp/issues/1371) reported for CPU, and the refusal names FlashAttention on boards that have never run FlashAttention. `kROCM` is NOT affected — `RocmAttentionBackend` declares no head-size constraint. CPU is fixable by mirroring upstream's own CPU answer; these three have no upstream equivalent, so the fix must AUTHOR each device's declared capabilities and needs its own spec. Found while fixing #1371 and not fixed in that flow: three platforms, no local runnable gate, and an authored capability surface is what `AGENTS.md` routes to the normal row and fresh-review path. Under `## Owed` in [`attn-validate-configuration.md`](specs/attn-validate-configuration.md) | bug |
 | [#1390](https://github.com/mudler/vllm.cpp/issues/1390) | `ENG-CUDAGRAPH-BREAK` | `test_qwen3_5_decode_graph_seam` SIGSEGVs on `main` and every assertion passes. Measured at `5f68e60df`, which is `origin/main` exactly, CPU Release x86_64: 8 cases, 7 passed, 1 failed, an assertion line reading 135 of 135 passed and 0 failed, exit 139, with `W6: two spec shapes of EQUAL S and different q get two graphs` reporting `CRASHED: SIGSEGV`. The number a reader greps says 135/135, so only the exit status and the `CRASHED` line carry the verdict. ORDER-DEPENDENT: `-tc="W6*"` alone passes 9/9 exit 0, so the crash needs state an earlier case in the same process left behind — a doctest binary runs every case in one process, and a leaked pool block, a leaked backend or platform registration, or a driver slot captured under one shape and re-entered under another are all live candidates. `gdb -batch -ex run -ex bt` puts the fault inside `vt::cpu::PagedAttentionKernel` on a `vt::cpu::Threadpool` worker, which is what a block table, slot mapping or sequence length that does not describe the handed KV cache looks like. The crashing case is the one [#1374](https://github.com/mudler/vllm.cpp/issues/1374) added for the `(S, q, spec)` ring key. Found while landing [#1305](https://github.com/mudler/vllm.cpp/issues/1305) and NOT caused by it: reverse-applying that branch's whole source change and rebuilding leaves the same exit 139, and that change touches `qwen3_moe.cpp`, `deepseek_v2.cpp` and three registry translation units, none of which this binary's crashing case executes. NOT fixed in flow, because `AGENTS.md` routes a surprising fix to the normal row, spec and fresh-review path: it is a segmentation fault in another stage's newly landed code, its mechanism is an unlocated cross-case state leak, and `src/vllm/model_executor/models/qwen3_5.cpp` is under concurrent edit for [#1380](https://github.com/mudler/vllm.cpp/issues/1380). Owner: row `ENG-CUDAGRAPH-BREAK`, under `## Owed` in [eng-cudagraph-break.md](specs/eng-cudagraph-break.md) | bug |
 | [#1405](https://github.com/mudler/vllm.cpp/issues/1405) | `ENG-CUDAGRAPH-BREAK` | `scripts/check-gate-commands.py` exists to answer whether a gated row's `## Gates` names a command that CAN FAIL, and it drops `true`, `echo ok` and anything containing a pipe for that reason. It does not drop `git diff`. Measured at `96ed8346f`: `runnable_commands` over `.agents/specs/eng-cudagraph-break.md` returns exactly `['git diff']`, and that one always-zero command is the whole reason `ENG-CUDAGRAPH-BREAK` sits in `RUNNABLE_BASELINE` (`:414`). Its `## Gates` names no test binary, least of all `tests/test_qwen3_5_decode_graph_seam`, whose W6 case exited 139 on `main` ([#1390](https://github.com/mudler/vllm.cpp/issues/1390), [#1394](https://github.com/mudler/vllm.cpp/issues/1394)). The EXECUTION gates were surveyed at the same revision and are sound — `ctest` fails on exit 139 (`tests/CMakeLists.txt:30`, `.github/workflows/ci.yml:1013`), `scripts/mutation-harness.py:99-113` reads the exit code and `Status: FAILURE!` and says the summary line is not the authority, `tools/bench/gdn_packed_component.py:1893-1902` requires `Status: SUCCESS!` beside its pinned totals, and `scripts/music3-vocoder-conv-ab.sh:88` recovers `${PIPESTATUS[0]}` past its grep — so NO execution gate reads a crashed run as green, and the exposure is the record layer plus [#1376](https://github.com/mudler/vllm.cpp/issues/1376). Sharper than it reads: a SIGSEGV TRUNCATES the assertion count (135 crashed vs 138 complete on one unchanged file), so a gate pinning a total reports a crash as count drift. NOT fixed in flow: adding `git diff` to `_CANNOT_FAIL` moves rows out of the runnable population, reds `--check` and the exact `RUNNABLE_BASELINE` pin in `tests/scripts/test_check_gate_commands.py`, and `AGENTS.md` routes a semantic checker change to its own row, spec and fresh review | bug |
 | [#1406](https://github.com/mudler/vllm.cpp/issues/1406) | `ENG-CUDAGRAPH-BREAK` | The block-table bound [#1394](https://github.com/mudler/vllm.cpp/issues/1394) adds is CPU-ONLY. Six backends register `OpId::kPagedAttention` (`cpu_paged_attn.cpp:271`, `cuda_paged_attn.cu:2890`, `rocm_ops.hip:148`, `vulkan_ops.cpp:1604`, `metal_ops.mm:1119`, `tenstorrent_ops.cpp:3188`) and the byte-identical `block_table[r * bt_row + (j / block_size) * bt_col]` appears unguarded at `cuda_paged_attn.cu:230,385,521,676,837` and `rocm_paged_attn.hip:226,424,557,765`, with `j` bounded only by `seq_lens[r]`. The obvious repair — one bound at the shared seam `vt::PagedAttention` — is NOT correct as written, and this was measured rather than reasoned: the seam cannot dereference a device `seq_lens`, and its only host-readable quantity `PagedAttentionArgs::max_seq_len` is documented at `include/vt/ops.h:806-812` as a value where **an upper bound is safe**, so a refusal keyed on it rejects a call the contract permits. `row/ENG-CUDAGRAPH-SEAM-SIGSEGV` carried exactly that arm and WITHDREW it for this reason. Three candidate fixes are named in the issue; the smallest that reaches all six backends narrows `max_seq_len` to exact, which the production caller already passes (`v1/attention/backend.cpp:303`, `:57-58`) and which owes a caller audit and a spec | gap |

.agents/porting-inventory.md

@@ -509,9 +509,14 @@ Examples: `examples/cli` ✅ (C-API client), `examples/server` ✅ (OpenAI serve
    major-10 vs else) and `cpu.py:75-87` (`CpuPlatform`). FLASH_ATTN/GDN_ATTN
    self-register. It reuses (does not extend) the pre-existing CPU FA-NHD-layout
    deviation (§9.1 / `cpu_paged_attn.cpp`): upstream CPU picks CPU_ATTN's
-   `[N,H,block,head]` layout, our CPU paged-attn reuses FlashAttention's NHD
-   layout, so CPU_ATTN is named-but-unregistered and the walk falls through to
-   FLASH_ATTN — behavior-preserving. The concrete attention KERNEL stays selected
+   `[N,H,block,head]` layout while our CPU paged-attn reuses FlashAttention's NHD
+   layout. CPU_ATTN was therefore named-but-unregistered here and the walk fell
+   through to FLASH_ATTN. **Since [#1371](https://github.com/mudler/vllm.cpp/issues/1371)
+   `CPU_ATTN` IS registered** (`src/vllm/v1/attention/backends/cpu_attn.cpp`,
+   ported from `cpu_attn.py:39-110`) and the CPU walk stops on it, which is
+   upstream's answer; the layout deviation is unchanged and is now recorded on the
+   backend itself rather than being the reason it had no registrar.
+   The concrete attention KERNEL stays selected
    at the vt:: op table (§9.1), so this is an organizing engine-SELECTION seam
    over the existing runtime, introducing no new deviation. Adding a backend's
    attention = one self-registering TU + one priority slot, ZERO selector/model/

.agents/specs/attn-validate-configuration.md

@@ -91,6 +91,27 @@ own it already exist.** No new roadmap row is opened.
 4. **The non-MLA sm_100 priority arm gained a `use_non_causal` guard upstream**
    between the two pins; see "Anchor reconciliation" below. Fixed in flow, tracked
    by its own issue, because the anchor being corrected points straight at it.
+5. **This change left three devices declaring a kernel they do not run.**
+   Recorded after the fact, because it was a live regression before it was debt.
+   `FlashAttentionBackend` is registered under `FLASH_ATTN` for `kCPU`, `kMETAL`,
+   `kVULKAN` and `kTENSTORRENT` as well as `kCUDA`
+   (`src/vllm/v1/attention/backend.cpp:381-417`), on the documented precondition
+   that those kernels share its NHD KV layout — a statement about the LAYOUT, not
+   about the kernel. M1 then attached FA2's capability surface to that shared
+   class, including `supports_head_size` = `head_size % 8 == 0 && head_size <= 256`
+   (`flash_attn.py:170-178`). Every one of those four devices thereby inherited a
+   rule about a CUDA kernel it does not run, and since each of the four lists no
+   other dense backend, a head size their own kernels handle correctly matched
+   NOTHING and `SelectAttentionBackendName` threw out of
+   `GPUModelRunner::initialize_kv_cache`. The compute-capability predicate got its
+   precondition in this same change; the head-size one did not, and the difference
+   is only that `CpuPlatform` reports no capability while it does report a head
+   size. CPU is closed by [#1371](https://github.com/mudler/vllm.cpp/issues/1371),
+   which registers `CPU_ATTN` — upstream's own CPU answer at `cpu.py:75-87` — so
+   the CPU stops borrowing another device's declaration. The other three have no
+   upstream backend to mirror and must have their capabilities AUTHORED.
+   **Owner: [#1389](https://github.com/mudler/vllm.cpp/issues/1389), row
+   `BACKEND-ATTN-REGISTRY`.**
 
 ## Anchor reconciliation
 
@@ -194,7 +215,8 @@ its capability by constructor argument.
 | Upstream case | Ported as |
 |---|---|
 | `test_backend_selection` (cuda/cpu arms) | selection under a fully populated config still resolves `FLASH_ATTN` on sm_121 and sm_100 |
-| `test_fp32_fallback` | an `f32` request finds no valid backend and the refusal names `dtype not supported`; upstream lands on `FLEX_ATTENTION`, which this tree does not register |
+| `test_fp32_fallback` (cuda arm) | an `f32` request finds no valid backend and the refusal names `dtype not supported`; upstream lands on `FLEX_ATTENTION`, which this tree does not register |
+| `test_fp32_fallback` (cpu arm, `test_attention_selector.py:230-241`) | ported at the #1371 repair, in `tests/vllm/v1/attention/test_attn_backend_registry.cpp`: on a CPU platform an `f32` request resolves `CPU_ATTN`, which is upstream's assertion verbatim. It could not be ported when this row landed, because `CPU_ATTN` had no registrar and the CPU's only backend declared `{f16, bf16}`. The hip arm stays unported and says why in the case: it asserts a refusal that upstream derives from ROCm's minimum head size of 32, and `RocmAttentionBackend` declares no head-size list, so the throw would arrive for the wrong reason |
 | `test_flash_attn` (upstream `pytest.skip`s it) | its five assertions ported as direct predicate cases: capability `(7,5)`, dtype `fp8`, `kv_cache_dtype="fp8"`, `block_size=8`, `head_size=17` |
 | `test_per_head_quant_scales_backend_selection` | `FLASH_ATTN` + `use_per_head_quant_scales` is refused and the reason names it |
 | `test_non_causal_backend_selection`, `test_non_causal_autoselect_backend` | `FLASH_ATTN` advertises `supports_non_causal()` and a non-causal request selects it |

.agents/specs/extensibility-platform-seam-2026-07-18.md

@@ -348,6 +348,17 @@ but unimplemented; each becomes auto-selected on the appropriate capability the
 moment its self-registering TU lands — ZERO selector edit. The MLA-branch
 priorities are deferred until an MLA model ports (our gate models are non-MLA).
 
+**SUPERSEDED 2026-08-20, on the CPU sentence only**
+([#1371](https://github.com/mudler/vllm.cpp/issues/1371), PR
+[#1392](https://github.com/mudler/vllm.cpp/pull/1392)). The paragraph above
+records what was true when item 4 landed on 2026-07-19, and it is kept for that
+reason. `CPU_ATTN` IS registered now
+(`src/vllm/v1/attention/backends/cpu_attn.cpp`, ported from
+`cpu_attn.py:39-110`), so a CPU walk stops on it instead of falling through to
+FLASH_ATTN. That is upstream's own CPU answer at `cpu.py:75-87`. The NHD-layout
+deviation is unchanged. It is recorded on the backend itself now, rather than
+being the reason `CPU_ATTN` had no registrar.
+
 **Scope discipline.** PURE ENGINE-level SELECTION seam. The concrete attention
 KERNEL stays selected at seam 3 (the vt:: op table, `vt::PagedAttention` →
 `GetOp(kPagedAttention, device.type)`), already device-additive — untouched. No