diff --git a/.agents/issue-index.md b/.agents/issue-index.md index a2d799a74..ef9646ac4 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -458,6 +458,8 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1376](https://github.com/mudler/vllm.cpp/issues/1376) | `ENG-CUDAGRAPH-BREAK` | `main` was red on `tests/scripts/test_check_gate_commands.py`, measured at `601b576c6` in a detached worktree of `origin/main`: 8 failures of 44 tests, every one a comparison between the computed runnable population and `RUNNABLE_BASELINE`. `ENG-CUDAGRAPH-BREAK` was in the first and absent from the second. Cause: W5 of that row ([#1361](https://github.com/mudler/vllm.cpp/issues/1361)) filled its spec's `## Gates` section with runnable evidence, including a named test binary with its case and assertion counts and an exit status, which is exactly what moves a row into the runnable population. The ratchet's own error text instructs a re-pin in the SAME change, and the re-pin was not made. This is the growth case the ratchet exists to force a decision about, not a defect in that row's work. **It landed with no remote verdict**: the continuous integration lane that would have caught it independently has not executed for this repository since roughly 07:43Z on 19 August 2026, with runs queueing and none starting while GitHub reports Actions operational. FIXED IN FLOW while merging `origin/main` into `row/ENG-HF-MODEL-DOWNLOAD` for [#1280](https://github.com/mudler/vllm.cpp/issues/1280), because the fix is small and clear and a red `main` blocks every other row's gate. The entry is added with a justifying comment in the form the neighbouring entries use, no checker semantics change, and no test is weakened. After the re-pin the suite reports 45 tests OK and the audit reads 39 runnable of 119 gated rows | bug | | [#1378](https://github.com/mudler/vllm.cpp/issues/1378) | `ENG-EXPERT-STREAM-DEVICE` | The expert-stream lane fired without asking which RESIDENCY ROUTE the towers would take, which deleted the #1123 refusal on every non-keep-quant arm. The lane condition at `src/vllm/entrypoints/model_loader.cpp` had four terms and all four are properties of the DEVICE and the ARCHITECTURE: the platform stages, the platform can read host slots, the factory declares `streams_routed_experts`, streaming is requested. None of them is a property of the FILE under the PROCESS's policy. `KqExpertSlice`, the seam that serves a slice out of the host slot store, is reached only from the `expert_*_kq` arm, and `LoadExpertsOrNvfp4` (`qwen3_5_gguf_weights.cpp`) routes the same `blk..ffn_{gate,up,down}_exps.weight` tensors to `kNvfp4Fp4` and to `kExpandBf16` as well, both of which reach `MoeBlockFusedMarlinCuda`/`MoeBlockFusedCuda`/`MoeBlockBf16Cuda` and stage every tower through `ResidentWeight`. So on a GB10 with `--device cuda`, a larger-than-pool `qwen35moe` GGUF and `VT_MOE_EXPERT_STREAM=1`, EITHER `VT_GGUF_KEEP_QUANT=0` (a documented, supported opt-out) OR an NVFP4 GGUF turned the lane on, dropped every `*_exps.weight` from the device bound, charged an arena nothing allocates, and put back the 26-minute load and the `cudaMalloc: out of memory` first forward that #1123 exists to prevent. The W0c `expert_streamed` tripwire cannot see it either, because a tower on those arms is never claimed. The UNSAFE direction, unlike the two over-counts the bound documents. FOUND by the fresh review of [#1377](https://github.com/mudler/vllm.cpp/pull/1377) and PROVEN by mutation, not argued: adding `VT_GGUF_KEEP_QUANT=0` to the reach suite's fixture left all 12 cases green. FIXED IN FLOW in the repair of that same pull request: a fifth term, `GgufExpertTowersReachSlotLane`, asks the model loader's own `PeekRoute` about this file under this process's policy and accepts only the residencies that reach the kq arm; it is all-or-nothing in the safe direction, so one tower that would be staged keeps the WHOLE bound. Red first: two new reach cases (`VT_GGUF_KEEP_QUANT=0`, and an NVFP4 tower) go from LET THROUGH to REFUSED at the lane-off bound, 2 of 14 cases and 6 of 66 assertions failing before the term exists and 14/14 with it. Spec [`expert-stream-device-slots.md`](specs/expert-stream-device-slots.md) | bug | | [#1375](https://github.com/mudler/vllm.cpp/issues/1375) | `MODEL-DIFFUSION-LTX25` | First end-to-end per-forward cost for the FULL 21.004 B LTX-2.5 DiT on GB10, measured on run `20260819T150230Z` with binary `0a43a750` built from [`7b9e207b1`](https://github.com/mudler/vllm.cpp/commit/7b9e207b1) (#1252). At 1024x576/25f (2304 latent tokens) the governor resolved **7 forward starts from the GPU busy/idle edge counter** and measured `per_forward ~162.0 s` with `first_dit = 481.5 s`, so the recipe's fixed 60 forwards (30 steps x 2 CFG legs, `ltx2_pipeline.cpp:521-529`) project **10 803 s against the rung's 7 153 s budget** and the rung was refused rather than run to the wall. The same lease then COMPLETED 768x448/25f (1344 tokens) in 2990 s, so the ceiling is geometry against lease length, not a defect. TWO instrument facts belong with the number, because both have already caused a wrong reading: `gpu_edges=0` means the GPU never went idle long enough to sample an edge (SATURATED), not that no work ran — this rung sampled 85% of 3191 samples above 50% utilisation; and `eu-stack` resolves no frames in the rc worker container, so phase attribution came from the cpu%/rss signature rather than from symbols. Owned by the LTX-2.5 row; spec [`ltx-2-5.md`](specs/ltx-2-5.md) | measurement | +| [#1371](https://github.com/mudler/vllm.cpp/issues/1371) | `BACKEND-ATTN-REGISTRY` | CPU had NO attention backend for a head size FLASH_ATTN declines, and the summary line said `0 failed` while it happened. `CpuPlatform::get_attn_backend_priority` returned `{CPU_ATTN, FLASH_ATTN}` where `CPU_ATTN` was a name in a list and nothing else — never registered, so every CPU selection fell through to `FLASH_ATTN`, recorded as behavior-preserving because our CPU paged-attention kernel reads that backend's NHD layout. It was behavior-preserving only while `FLASH_ATTN` accepted everything CPU asked. [#1332](https://github.com/mudler/vllm.cpp/issues/1332) M1 (`369ea7fd4`) gave it `flash_attn.py:170-178`'s `head_size % 8 == 0` rule, and a CPU request with `head_size` 6 then matched no registered backend at all: `SelectAttentionBackendName` throws out of `GPUModelRunner::initialize_kv_cache` with `No valid attention backend for device type 0 from {FLASH_ATTN: [head_size not supported]}`. Attributed by clean before/after on one row's own two shas across two boxes and two architectures (`68a0ff378` on thor sm_110, 13/13 passed; `e35c14d52` on dgx sm_121a, 11 of 13 thrown), so neither box- nor arch-specific. **The reporting shape is the second half of this issue**: assertions collapse `3272 -> 18` while doctest still prints `0 failed`, because a THROWN case runs no assertions — a gate grepping `assertions:` reads eighteen assertions, eighteen passed, zero failed, and calls it clean. Fixed by registering `CPU_ATTN` (`src/vllm/v1/attention/backends/cpu_attn.cpp`), which is the answer upstream `cpu.py:75-87` gives for every CPU request; the FA2 rule is untouched, because it is right about the FA2 kernel and was never about ours | bug | +| [#1389](https://github.com/mudler/vllm.cpp/issues/1389) | `BACKEND-ATTN-REGISTRY` | Metal, Vulkan and Tenstorrent inherit FlashAttention-2's head-size rule from a backend class they share by NAME, so a head size their own kernels run has no backend. Each registers `FlashAttentionBackend` for `FLASH_ATTN` (`src/vllm/v1/attention/backend.cpp:395-417`) and each returns `{"FLASH_ATTN"}` as its entire non-MLA priority list; the documented precondition for those rows is a shared NHD KV layout, not a shared kernel. Since [#1332](https://github.com/mudler/vllm.cpp/issues/1332) M1 that class also carries `supports_head_size` = `head_size % 8 == 0 && head_size <= 256` (`include/vllm/v1/attention/backend.h:394-397`, `flash_attn.py:170-178` @ pin `5559679229`), a true statement about a CUDA kernel none of those three devices runs. Same defect [#1371](https://github.com/mudler/vllm.cpp/issues/1371) reported for CPU, and the refusal names FlashAttention on boards that have never run FlashAttention. `kROCM` is NOT affected — `RocmAttentionBackend` declares no head-size constraint. CPU is fixable by mirroring upstream's own CPU answer; these three have no upstream equivalent, so the fix must AUTHOR each device's declared capabilities and needs its own spec. Found while fixing #1371 and not fixed in that flow: three platforms, no local runnable gate, and an authored capability surface is what `AGENTS.md` routes to the normal row and fresh-review path. Under `## Owed` in [`attn-validate-configuration.md`](specs/attn-validate-configuration.md) | bug | | [#1390](https://github.com/mudler/vllm.cpp/issues/1390) | `ENG-CUDAGRAPH-BREAK` | `test_qwen3_5_decode_graph_seam` SIGSEGVs on `main` and every assertion passes. Measured at `5f68e60df`, which is `origin/main` exactly, CPU Release x86_64: 8 cases, 7 passed, 1 failed, an assertion line reading 135 of 135 passed and 0 failed, exit 139, with `W6: two spec shapes of EQUAL S and different q get two graphs` reporting `CRASHED: SIGSEGV`. The number a reader greps says 135/135, so only the exit status and the `CRASHED` line carry the verdict. ORDER-DEPENDENT: `-tc="W6*"` alone passes 9/9 exit 0, so the crash needs state an earlier case in the same process left behind — a doctest binary runs every case in one process, and a leaked pool block, a leaked backend or platform registration, or a driver slot captured under one shape and re-entered under another are all live candidates. `gdb -batch -ex run -ex bt` puts the fault inside `vt::cpu::PagedAttentionKernel` on a `vt::cpu::Threadpool` worker, which is what a block table, slot mapping or sequence length that does not describe the handed KV cache looks like. The crashing case is the one [#1374](https://github.com/mudler/vllm.cpp/issues/1374) added for the `(S, q, spec)` ring key. Found while landing [#1305](https://github.com/mudler/vllm.cpp/issues/1305) and NOT caused by it: reverse-applying that branch's whole source change and rebuilding leaves the same exit 139, and that change touches `qwen3_moe.cpp`, `deepseek_v2.cpp` and three registry translation units, none of which this binary's crashing case executes. NOT fixed in flow, because `AGENTS.md` routes a surprising fix to the normal row, spec and fresh-review path: it is a segmentation fault in another stage's newly landed code, its mechanism is an unlocated cross-case state leak, and `src/vllm/model_executor/models/qwen3_5.cpp` is under concurrent edit for [#1380](https://github.com/mudler/vllm.cpp/issues/1380). Owner: row `ENG-CUDAGRAPH-BREAK`, under `## Owed` in [eng-cudagraph-break.md](specs/eng-cudagraph-break.md) | bug | | [#1405](https://github.com/mudler/vllm.cpp/issues/1405) | `ENG-CUDAGRAPH-BREAK` | `scripts/check-gate-commands.py` exists to answer whether a gated row's `## Gates` names a command that CAN FAIL, and it drops `true`, `echo ok` and anything containing a pipe for that reason. It does not drop `git diff`. Measured at `96ed8346f`: `runnable_commands` over `.agents/specs/eng-cudagraph-break.md` returns exactly `['git diff']`, and that one always-zero command is the whole reason `ENG-CUDAGRAPH-BREAK` sits in `RUNNABLE_BASELINE` (`:414`). Its `## Gates` names no test binary, least of all `tests/test_qwen3_5_decode_graph_seam`, whose W6 case exited 139 on `main` ([#1390](https://github.com/mudler/vllm.cpp/issues/1390), [#1394](https://github.com/mudler/vllm.cpp/issues/1394)). The EXECUTION gates were surveyed at the same revision and are sound — `ctest` fails on exit 139 (`tests/CMakeLists.txt:30`, `.github/workflows/ci.yml:1013`), `scripts/mutation-harness.py:99-113` reads the exit code and `Status: FAILURE!` and says the summary line is not the authority, `tools/bench/gdn_packed_component.py:1893-1902` requires `Status: SUCCESS!` beside its pinned totals, and `scripts/music3-vocoder-conv-ab.sh:88` recovers `${PIPESTATUS[0]}` past its grep — so NO execution gate reads a crashed run as green, and the exposure is the record layer plus [#1376](https://github.com/mudler/vllm.cpp/issues/1376). Sharper than it reads: a SIGSEGV TRUNCATES the assertion count (135 crashed vs 138 complete on one unchanged file), so a gate pinning a total reports a crash as count drift. NOT fixed in flow: adding `git diff` to `_CANNOT_FAIL` moves rows out of the runnable population, reds `--check` and the exact `RUNNABLE_BASELINE` pin in `tests/scripts/test_check_gate_commands.py`, and `AGENTS.md` routes a semantic checker change to its own row, spec and fresh review | bug | | [#1406](https://github.com/mudler/vllm.cpp/issues/1406) | `ENG-CUDAGRAPH-BREAK` | The block-table bound [#1394](https://github.com/mudler/vllm.cpp/issues/1394) adds is CPU-ONLY. Six backends register `OpId::kPagedAttention` (`cpu_paged_attn.cpp:271`, `cuda_paged_attn.cu:2890`, `rocm_ops.hip:148`, `vulkan_ops.cpp:1604`, `metal_ops.mm:1119`, `tenstorrent_ops.cpp:3188`) and the byte-identical `block_table[r * bt_row + (j / block_size) * bt_col]` appears unguarded at `cuda_paged_attn.cu:230,385,521,676,837` and `rocm_paged_attn.hip:226,424,557,765`, with `j` bounded only by `seq_lens[r]`. The obvious repair — one bound at the shared seam `vt::PagedAttention` — is NOT correct as written, and this was measured rather than reasoned: the seam cannot dereference a device `seq_lens`, and its only host-readable quantity `PagedAttentionArgs::max_seq_len` is documented at `include/vt/ops.h:806-812` as a value where **an upper bound is safe**, so a refusal keyed on it rejects a call the contract permits. `row/ENG-CUDAGRAPH-SEAM-SIGSEGV` carried exactly that arm and WITHDREW it for this reason. Three candidate fixes are named in the issue; the smallest that reaches all six backends narrows `max_seq_len` to exact, which the production caller already passes (`v1/attention/backend.cpp:303`, `:57-58`) and which owes a caller audit and a spec | gap | diff --git a/.agents/porting-inventory.md b/.agents/porting-inventory.md index f20655f11..fb2e93a22 100644 --- a/.agents/porting-inventory.md +++ b/.agents/porting-inventory.md @@ -509,9 +509,14 @@ Examples: `examples/cli` ✅ (C-API client), `examples/server` ✅ (OpenAI serve major-10 vs else) and `cpu.py:75-87` (`CpuPlatform`). FLASH_ATTN/GDN_ATTN self-register. It reuses (does not extend) the pre-existing CPU FA-NHD-layout deviation (§9.1 / `cpu_paged_attn.cpp`): upstream CPU picks CPU_ATTN's - `[N,H,block,head]` layout, our CPU paged-attn reuses FlashAttention's NHD - layout, so CPU_ATTN is named-but-unregistered and the walk falls through to - FLASH_ATTN — behavior-preserving. The concrete attention KERNEL stays selected + `[N,H,block,head]` layout while our CPU paged-attn reuses FlashAttention's NHD + layout. CPU_ATTN was therefore named-but-unregistered here and the walk fell + through to FLASH_ATTN. **Since [#1371](https://github.com/mudler/vllm.cpp/issues/1371) + `CPU_ATTN` IS registered** (`src/vllm/v1/attention/backends/cpu_attn.cpp`, + ported from `cpu_attn.py:39-110`) and the CPU walk stops on it, which is + upstream's answer; the layout deviation is unchanged and is now recorded on the + backend itself rather than being the reason it had no registrar. + The concrete attention KERNEL stays selected at the vt:: op table (§9.1), so this is an organizing engine-SELECTION seam over the existing runtime, introducing no new deviation. Adding a backend's attention = one self-registering TU + one priority slot, ZERO selector/model/ diff --git a/.agents/specs/attn-validate-configuration.md b/.agents/specs/attn-validate-configuration.md index 4a51e4421..b4d7989d0 100644 --- a/.agents/specs/attn-validate-configuration.md +++ b/.agents/specs/attn-validate-configuration.md @@ -91,6 +91,27 @@ own it already exist.** No new roadmap row is opened. 4. **The non-MLA sm_100 priority arm gained a `use_non_causal` guard upstream** between the two pins; see "Anchor reconciliation" below. Fixed in flow, tracked by its own issue, because the anchor being corrected points straight at it. +5. **This change left three devices declaring a kernel they do not run.** + Recorded after the fact, because it was a live regression before it was debt. + `FlashAttentionBackend` is registered under `FLASH_ATTN` for `kCPU`, `kMETAL`, + `kVULKAN` and `kTENSTORRENT` as well as `kCUDA` + (`src/vllm/v1/attention/backend.cpp:381-417`), on the documented precondition + that those kernels share its NHD KV layout — a statement about the LAYOUT, not + about the kernel. M1 then attached FA2's capability surface to that shared + class, including `supports_head_size` = `head_size % 8 == 0 && head_size <= 256` + (`flash_attn.py:170-178`). Every one of those four devices thereby inherited a + rule about a CUDA kernel it does not run, and since each of the four lists no + other dense backend, a head size their own kernels handle correctly matched + NOTHING and `SelectAttentionBackendName` threw out of + `GPUModelRunner::initialize_kv_cache`. The compute-capability predicate got its + precondition in this same change; the head-size one did not, and the difference + is only that `CpuPlatform` reports no capability while it does report a head + size. CPU is closed by [#1371](https://github.com/mudler/vllm.cpp/issues/1371), + which registers `CPU_ATTN` — upstream's own CPU answer at `cpu.py:75-87` — so + the CPU stops borrowing another device's declaration. The other three have no + upstream backend to mirror and must have their capabilities AUTHORED. + **Owner: [#1389](https://github.com/mudler/vllm.cpp/issues/1389), row + `BACKEND-ATTN-REGISTRY`.** ## Anchor reconciliation @@ -194,7 +215,8 @@ its capability by constructor argument. | Upstream case | Ported as | |---|---| | `test_backend_selection` (cuda/cpu arms) | selection under a fully populated config still resolves `FLASH_ATTN` on sm_121 and sm_100 | -| `test_fp32_fallback` | an `f32` request finds no valid backend and the refusal names `dtype not supported`; upstream lands on `FLEX_ATTENTION`, which this tree does not register | +| `test_fp32_fallback` (cuda arm) | an `f32` request finds no valid backend and the refusal names `dtype not supported`; upstream lands on `FLEX_ATTENTION`, which this tree does not register | +| `test_fp32_fallback` (cpu arm, `test_attention_selector.py:230-241`) | ported at the #1371 repair, in `tests/vllm/v1/attention/test_attn_backend_registry.cpp`: on a CPU platform an `f32` request resolves `CPU_ATTN`, which is upstream's assertion verbatim. It could not be ported when this row landed, because `CPU_ATTN` had no registrar and the CPU's only backend declared `{f16, bf16}`. The hip arm stays unported and says why in the case: it asserts a refusal that upstream derives from ROCm's minimum head size of 32, and `RocmAttentionBackend` declares no head-size list, so the throw would arrive for the wrong reason | | `test_flash_attn` (upstream `pytest.skip`s it) | its five assertions ported as direct predicate cases: capability `(7,5)`, dtype `fp8`, `kv_cache_dtype="fp8"`, `block_size=8`, `head_size=17` | | `test_per_head_quant_scales_backend_selection` | `FLASH_ATTN` + `use_per_head_quant_scales` is refused and the reason names it | | `test_non_causal_backend_selection`, `test_non_causal_autoselect_backend` | `FLASH_ATTN` advertises `supports_non_causal()` and a non-causal request selects it | diff --git a/.agents/specs/extensibility-platform-seam-2026-07-18.md b/.agents/specs/extensibility-platform-seam-2026-07-18.md index c44bd6fed..8273f767b 100644 --- a/.agents/specs/extensibility-platform-seam-2026-07-18.md +++ b/.agents/specs/extensibility-platform-seam-2026-07-18.md @@ -348,6 +348,17 @@ but unimplemented; each becomes auto-selected on the appropriate capability the moment its self-registering TU lands — ZERO selector edit. The MLA-branch priorities are deferred until an MLA model ports (our gate models are non-MLA). +**SUPERSEDED 2026-08-20, on the CPU sentence only** +([#1371](https://github.com/mudler/vllm.cpp/issues/1371), PR +[#1392](https://github.com/mudler/vllm.cpp/pull/1392)). The paragraph above +records what was true when item 4 landed on 2026-07-19, and it is kept for that +reason. `CPU_ATTN` IS registered now +(`src/vllm/v1/attention/backends/cpu_attn.cpp`, ported from +`cpu_attn.py:39-110`), so a CPU walk stops on it instead of falling through to +FLASH_ATTN. That is upstream's own CPU answer at `cpu.py:75-87`. The NHD-layout +deviation is unchanged. It is recorded on the backend itself now, rather than +being the reason `CPU_ATTN` had no registrar. + **Scope discipline.** PURE ENGINE-level SELECTION seam. The concrete attention KERNEL stays selected at seam 3 (the vt:: op table, `vt::PagedAttention` → `GetOp(kPagedAttention, device.type)`), already device-additive — untouched. No diff --git a/CMakeLists.txt b/CMakeLists.txt index c67dbc254..3f76d9560 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1115,6 +1115,7 @@ add_library(vllm STATIC src/vllm/v1/attention/backend.cpp src/vllm/v1/attention/registry.cpp src/vllm/v1/attention/backends/utils.cpp + src/vllm/v1/attention/backends/cpu_attn.cpp src/vllm/v1/attention/backends/gdn_attn.cpp src/vllm/v1/structured_output/backend_types.cpp src/vllm/v1/structured_output/backend_native.cpp diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 90d8c8cc8..1ce5d64e1 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -263,7 +263,7 @@ both refuse, naming what is missing. | Backend | vllm.cpp | vLLM | SGLang | llama.cpp | |---|---|---|---|---| | CUDA | ✅ sm_80 to sm_121a | ✅ | ✅ | ✅ | -| CPU (x86, Arm i8mm; A76 assembly correct/default, llama speed gate open, and the closed 20-core floor ran a SUPERSEDED fork denominator rather than the stock `b10451` pin, re-take owed #1003) | ✅ | ◐ | ☐ | ✅ | +| CPU (x86, Arm i8mm; A76 assembly correct/default, llama speed gate open, and the closed 20-core floor ran a SUPERSEDED fork denominator rather than the stock `b10451` pin, re-take owed #1003) | ✅ `CPU_ATTN` registered (#1371/#1392, [spec](../.agents/specs/attn-validate-configuration.md)) | ◐ | ☐ | ✅ | | Metal (Apple Silicon) | ✅ builds under Apple Clang with project warnings promoted to errors, the Qwen3.5 MoE loader included; its layout-refusal path uses the same messages and behavior on every platform (#1054) | ☐ | ☐ | ✅ | | Vulkan | ◐ | ☐ | ☐ | ✅ | | ROCm | W0: 5 gfx archs; dense/GDN all-native; 0.8B dispatch fixed. **M4:** Qwen3-0.6B/3.5-0.8B 16/16 (#41). **M3:** `ROCM_ATTN` registered (#1056/#1065, [spec](../.agents/specs/rocm-attn-backend.md)). CPU parity open (#269) | 44 registered ops including full GDN; ctest-green gfx1151/1103/1100/1201/1200 ([#41](https://github.com/mudler/vllm.cpp/issues/41)). APU managed allocation is unverified. [ROCM.md](ROCM.md) | ✅ | ✅ | diff --git a/docs/USAGE.md b/docs/USAGE.md index dec21f0c6..065b8624b 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -808,6 +808,29 @@ reference engine read the same. `head_size`, `block_size` and the KV-cache dtype come from the geometry the engine has just resolved for your checkpoint, so a refusal is about that checkpoint on this build. +**A device is only ever offered the backends built for it.** On CPU the engine +resolves `CPU_ATTN`, which is what the reference engine resolves on a CPU too. It +is worth saying out loud because it was briefly untrue: `CPU_ATTN` was named as +the CPU's preference while being registered nowhere, so CPU runs quietly fell +through to `FLASH_ATTN` — harmless until `FLASH_ATTN` was taught FlashAttention-2's +rule that a head size must be a multiple of 8. A CPU model with a head size of 6 +then had no backend at all and was refused at initialization, on hardware that +runs it perfectly well ([#1371](https://github.com/mudler/vllm.cpp/issues/1371)). +If you see the refusal above naming `FLASH_ATTN` alone on a device that is not an +NVIDIA GPU, that is the shape to report: the rule quoted at you is about a kernel +your device never runs. + +One consequence is worth stating on its own, because it widens what a CPU run +accepts. `CPU_ATTN` serves **`f32` as well as `f16` and `bf16`**, which is what +the reference engine's CPU backend serves. `FLASH_ATTN` declares the two half +dtypes only, so while the CPU was borrowing it an `f32` model was refused at +initialization with `dtype not supported`. It now runs. The KV-cache dtypes the +CPU accepts are `auto`, `fp8` and `fp8_e4m3`; `fp8_e5m2` is refused by name, +because the CPU kernel's fp8 arm reads e4m3 alone. On an NVIDIA GPU the list is +`auto`, `float16`, `bfloat16`, `fp8` and `fp8_e4m3`, so `fp8_e5m2` is refused +there too. That second refusal is the reference engine's own and is not +something this project trimmed away. + **What this check cannot tell you.** It reports what a backend *claims*, never what your binary contains and never whether the kernel will launch. A backend whose declared floor is compute capability 8.0 is accepted on any newer GPU, even diff --git a/include/vllm/v1/attention/backends/cpu_attn.h b/include/vllm/v1/attention/backends/cpu_attn.h new file mode 100644 index 000000000..f14ef80c6 --- /dev/null +++ b/include/vllm/v1/attention/backends/cpu_attn.h @@ -0,0 +1,132 @@ +// Ported from: vllm/v1/attention/backends/cpu_attn.py @ pin 5559679229 +// (CPUAttentionBackend, :39-110 — the CAPABILITY surface and the name.) +// +// Why this file exists, and what it fixes (issue #1371). +// +// `CpuPlatform::get_attn_backend_priority` has always returned +// {CPU_ATTN, FLASH_ATTN}, mirroring cpu.py:75-87, where CPU_ATTN is the ONLY +// answer upstream ever gives on a CPU. Until now the first name was never +// registered, so every CPU selection fell through to FLASH_ATTN — the backend +// whose NHD KV layout our CPU paged-attention kernel happens to share. That +// fallthrough was recorded as behavior-preserving, and it was, right up to the +// moment FLASH_ATTN grew a capability surface: #1332 M1 (369ea7fd4) ported +// `validate_configuration`, and with it flash_attn.py's `head_size % 8 == 0` +// rule. A CPU request whose head size is not a multiple of 8 then matched NO +// registered backend at all and `SelectAttentionBackendName` threw out of +// `GPUModelRunner::initialize_kv_cache`. The FLASH_ATTN rule is correct and is +// untouched here — it describes the FA2 kernel. The defect was that CPU had +// nothing else to fall back TO, because its own backend was a name in a list. +// +// So this registers it. CPU selection now answers CPU_ATTN, which is what +// upstream answers, and FLASH_ATTN stays registered for kCPU as the second +// entry of the same priority list. +// +// ─── THREE RECORDED DEVIATIONS FROM cpu_attn.py ──────────────────────────── +// +// 1. `get_kv_cache_shape` returns the NHD 5-dim shape +// (num_blocks, 2, block_size, num_kv_heads, head_size), NOT upstream's HND +// (num_blocks, num_kv_heads, block_size, 2 * head_size) at cpu_attn.py:94-104. +// This is the SAME long-standing deviation the CPU kernel header already +// records at src/vt/cpu/cpu_paged_attn.cpp:5-8: our CPU paged-attention +// kernel reads the FlashAttention NHD layout by tensor strides. The shape a +// backend reports is checked against the geometry the engine actually +// allocates (`vllm::v1::CheckKvCacheShape`), so reporting upstream's HND +// here would fail every CPU run at init. The name a backend answers to is +// its identity in this registry — the same footing ROCM_ATTN is registered +// on in backend.h. +// +// 2. `get_supported_head_sizes` declares NO constraint, where cpu_attn.py:58-60 +// lists [32, 64, 80, 96, 112, 128, 160, 192, 224, 256, 512]. That list +// describes upstream's CPU kernel, which dispatches through fixed-width +// vectorized templates. Ours does not: src/vt/cpu/cpu_paged_attn.cpp is a +// per-(request, token, q-head) scalar loop with a two-pass f32 softmax, and +// the ONLY value it specializes on is the K/V cache dtype. It has no +// head-size specialization to miss, and it demonstrably runs head sizes +// outside upstream's list. Declaring upstream's list would refuse work this +// binary performs correctly — which is the same class of untrue capability +// claim #1332 was opened for, pointed the other way. +// +// Say the direction out loud, because it is the unusual one: an EMPTY list is +// WIDER than upstream's, not narrower. This backend accepts any head size, +// including sizes upstream's CPU kernel refuses, and including 0 — which the +// base predicate treats as unconstrained rather than as a value +// (`backend.cpp::AttentionBackend::supports_head_size`, backend.py:158-161). +// That is the base class's own default for a backend with no fixed-width +// dispatch, and it is a true statement about our scalar kernel; it is still a +// wider claim than the file this one is ported from makes, and a future CPU +// kernel that gains head-size specialization owes this list its real entries. +// +// 3. `supported_kv_cache_dtypes` omits "fp8_e5m2", where cpu_attn.py:47-52 lists +// FOUR entries ["auto", "fp8", "fp8_e4m3", "fp8_e5m2"]. Our CPU KV-fp8 arm has +// one encoding: `KvKind::kFp8` resolves to `LoadKvFp8E4M3` and nothing else +// (`src/vt/cpu/cpu_paged_attn.cpp::KvElem`), and both vt entry points +// VT_CHECK-refuse an e5m2 KV cache (`src/vt/ops.cpp`, "fp8_e5m2 CPU compute / +// read is a named later brick"). Claiming e5m2 would select this backend for a +// cache it cannot read, so the entry is dropped rather than claimed. +// +// This is NOT the same omission FlashAttentionBackend makes, and an earlier +// draft of this comment said it was. FLASH_ATTN's list here is +// ["auto", "float16", "bfloat16", "fp8", "fp8_e4m3"], which is +// flash_attn.py:74-80 VERBATIM — upstream's FA backend never claimed e5m2, so +// nothing was trimmed from it and it is no precedent for anything. This +// deviation stands on our own kernel's single encoding, and on that alone. +#ifndef VLLM_V1_ATTENTION_BACKENDS_CPU_ATTN_H_ +#define VLLM_V1_ATTENTION_BACKENDS_CPU_ATTN_H_ + +#include +#include +#include + +#include "vllm/v1/attention/backend.h" +#include "vt/dtype.h" + +namespace vllm::v1 { + +class CpuAttentionBackend final : public AttentionBackend { + public: + // cpu_attn.py:63-65. + static constexpr const char* kName = "CPU_ATTN"; + + std::string get_name() const override { return kName; } + + // DEVIATION 1 in the file header — NHD, not upstream's HND. + std::vector get_kv_cache_shape( + int64_t num_blocks, int64_t block_size, int64_t num_kv_heads, + int64_t head_size, + const std::string& cache_dtype_str = "auto") const override; + + // ─── Capability overrides, ported from cpu_attn.py:39-83 ───────────────── + // cpu_attn.py:42-46 — the CPU backend serves f32 as well as the two half + // dtypes, and our CPU kernel widens every arm to f32 internally anyway. + std::vector supported_dtypes() const override { + return {vt::DType::kF16, vt::DType::kBF16, vt::DType::kF32}; + } + // DEVIATION 3 in the file header — cpu_attn.py:47-52 minus "fp8_e5m2". The + // KV-FP8 arm of our CPU kernel is e4m3 only (`KvKind::kFp8` -> LoadKvFp8E4M3, + // cpu_paged_attn.cpp), so e5m2 is omitted rather than claimed. FLASH_ATTN's + // shorter list is NOT a precedent: it is flash_attn.py's own list verbatim. + std::vector supported_kv_cache_dtypes() const override { + return {"auto", "fp8", "fp8_e4m3"}; + } + // cpu_attn.py:54-56 — MultipleOf(16). + std::vector get_supported_kernel_block_sizes() const override { return {16}; } + // DEVIATION 2 in the file header. An empty list is "no constraint" + // (backend.py:158-161), which is what our scalar CPU kernel actually offers — + // a WIDER claim than upstream's eleven sizes, 0 included. See the header. + std::vector get_supported_head_sizes() const override { return {}; } + // cpu_attn.py:66-68. + bool supports_non_causal() const override { return true; } + // cpu_attn.py:70-72. + bool supports_sliding_window() const override { return true; } + // cpu_attn.py:74-83 — decoder, encoder, encoder-only and encoder-decoder. + bool supports_attn_type(const std::string& attn_type) const override { + return attn_type == AttentionTypeName(AttentionType::kDecoder) || + attn_type == AttentionTypeName(AttentionType::kEncoder) || + attn_type == AttentionTypeName(AttentionType::kEncoderOnly) || + attn_type == AttentionTypeName(AttentionType::kEncoderDecoder); + } +}; + +} // namespace vllm::v1 + +#endif // VLLM_V1_ATTENTION_BACKENDS_CPU_ATTN_H_ diff --git a/include/vllm/v1/attention/registry.h b/include/vllm/v1/attention/registry.h index 49cb94b3b..b03fd4102 100644 --- a/include/vllm/v1/attention/registry.h +++ b/include/vllm/v1/attention/registry.h @@ -63,9 +63,14 @@ std::unique_ptr MakeAttentionBackend(vt::DeviceType device, // or this throws. Otherwise walk the platform's capability-ordered priority list // and return the first name registered for the platform's device (upstream's // min-priority valid backend). Throws if the priority list yields no registered -// backend. Behavior-preserving today: on CUDA (every capability we ship on) and -// CPU this returns "FLASH_ATTN" — the backend whose NHD KV layout the runtime -// already uses. +// backend. On CUDA (every capability we ship on) this returns "FLASH_ATTN". On +// CPU it returns "CPU_ATTN" — upstream's own CPU answer (cpu.py:75-87), which +// this tree finally registers as of issue #1371. Until then CPU_ATTN was a name +// in CpuPlatform's priority list with no registrar behind it and the walk fell +// through to FLASH_ATTN, which was behavior-preserving right up to the moment +// FLASH_ATTN acquired flash_attn.py's `head_size % 8 == 0` rule and the CPU had +// nothing left to answer with. Both names allocate the SAME NHD KV layout, so +// the geometry the runtime uses did not change with the name. // `cfg` (W2) carries upstream's `use_mla` plus the `use_sparse` flag. It selects // which priority list the platform hands back (the MLA vs non-MLA branch of // _get_backend_priorities) AND is applied as a per-candidate FILTER mirroring diff --git a/src/vllm/platforms/cpu.cpp b/src/vllm/platforms/cpu.cpp index eda27075a..0c5b49cf1 100644 --- a/src/vllm/platforms/cpu.cpp +++ b/src/vllm/platforms/cpu.cpp @@ -26,15 +26,20 @@ class CpuPlatform final : public Platform { ResidencyPolicy residency_policy() const override { return {}; } // Attention-backend priority. Upstream cpu.py::get_attn_backend_cls:75-87 @ pin - // e24d1b24 pins CPU to a single backend (AttentionBackendEnum.CPU_ATTN). Our - // CPU paged-attention kernel does NOT implement CPU_ATTN's [N,H,block,head] - // layout; it reuses the FlashAttention NHD layout (num_blocks,2,block,H,D) — - // the recorded deviation in src/vt/cpu/cpu_paged_attn.cpp:6. So CPU_ATTN is - // listed first for upstream fidelity but is unregistered; the walk falls through - // to FLASH_ATTN, the backend our CPU KV layout actually matches. This both - // mirrors upstream's CPU preference AND is behavior-preserving (selection - // returns "FLASH_ATTN", the layout used today), and demonstrates the - // first-registered-in-priority fallthrough that IS the selection mechanism. + // e24d1b24 pins CPU to a single backend (AttentionBackendEnum.CPU_ATTN), and + // since issue #1371 so do we: CPU_ATTN is REGISTERED + // (src/vllm/v1/attention/backends/cpu_attn.cpp) and the walk stops on it. + // + // It used to be a name alone. Our CPU paged-attention kernel does not use + // upstream CPU_ATTN's HND [N,H,block,2*D] layout; it reads the FlashAttention + // NHD layout (num_blocks,2,block,H,D) — the recorded deviation in + // src/vt/cpu/cpu_paged_attn.cpp:5-8 — so CPU_ATTN was listed first for + // upstream fidelity while being unregistered, and the walk fell through to + // FLASH_ATTN. That was behavior-preserving only for as long as FLASH_ATTN + // accepted everything CPU asked of it. #1332 M1 gave FLASH_ATTN flash_attn.py's + // `head_size % 8 == 0` rule, and a CPU request with head_size 6 then matched no + // registered backend at all. FLASH_ATTN stays second, and stays registered for + // kCPU, because that rule is right about the FA2 kernel and wrong about ours. // MLA is not reachable on the CPU platform (no CPU MLA backend upstream at the // pin), so the list is the same for both cfg values — an MLA request simply // finds no is_mla() backend registered and the selector throws, rather than diff --git a/src/vllm/v1/attention/backends/cpu_attn.cpp b/src/vllm/v1/attention/backends/cpu_attn.cpp new file mode 100644 index 000000000..39d3d4acd --- /dev/null +++ b/src/vllm/v1/attention/backends/cpu_attn.cpp @@ -0,0 +1,39 @@ +// Ported from: vllm/v1/attention/backends/cpu_attn.py @ pin 5559679229 +// (CPUAttentionBackend). See the header for the three recorded deviations and +// for why registering this name is the fix for issue #1371. +#include "vllm/v1/attention/backends/cpu_attn.h" + +#include + +#include "vllm/v1/attention/registry.h" + +namespace vllm::v1 { + +// DEVIATION 1 (see cpu_attn.h): upstream cpu_attn.py:94-104 returns the HND +// shape (num_blocks, num_kv_heads, block_size, 2 * head_size). Our CPU +// paged-attention kernel reads the NHD layout by tensor strides +// (src/vt/cpu/cpu_paged_attn.cpp:5-8), and that is the layout the engine +// allocates, so this reports what is really there. Identical, dimension for +// dimension, to FlashAttentionBackend::get_kv_cache_shape — which is precisely +// why CPU could fall through to FLASH_ATTN for as long as it did. +std::vector CpuAttentionBackend::get_kv_cache_shape( + int64_t num_blocks, int64_t block_size, int64_t num_kv_heads, + int64_t head_size, const std::string& /*cache_dtype_str*/) const { + return {num_blocks, 2, block_size, num_kv_heads, head_size}; +} + +namespace { +// CPU_ATTN self-registers for kCPU alone. Upstream registers it the same way +// (@register_backend(AttentionBackendEnum.CPU_ATTN), registry.py) and +// cpu.py:75-87 returns it for every CPU request, so with this line +// SelectAttentionBackendName finally gives upstream's answer on kCPU instead of +// walking past an unregistered name. Retained past the linker by the vllm +// --whole-archive INTERFACE option, like every other backend registrar. +const AttentionBackendRegistrar kCpuAttnCpu{ + vt::DeviceType::kCPU, CpuAttentionBackend::kName, + []() -> std::unique_ptr { + return std::make_unique(); + }}; +} // namespace + +} // namespace vllm::v1 diff --git a/src/vllm/v1/worker/gpu/runner.cpp b/src/vllm/v1/worker/gpu/runner.cpp index afb82a1bf..eb4ee1ba7 100644 --- a/src/vllm/v1/worker/gpu/runner.cpp +++ b/src/vllm/v1/worker/gpu/runner.cpp @@ -718,8 +718,11 @@ void GPUModelRunner::initialize_kv_cache(const KVCacheConfig& kv_cache_config) { // Dense: LOUD. A model with dense full-attention groups needs a dense // backend; a platform whose priority list yields none (how Vulkan and ROCm // started) fails at init instead of silently running unlabelled. On ROCm - // this resolves "ROCM_ATTN" (backend.cpp, M3); on CPU/CUDA/Metal/Vulkan - // "FLASH_ATTN" — the name whose NHD KV layout every device kernel reads. + // this resolves "ROCM_ATTN" (backend.cpp, M3); on CPU "CPU_ATTN" + // (cpu_attn.cpp, issue #1371 — upstream's own CPU answer at cpu.py:75-87); + // on CUDA/Metal/Vulkan "FLASH_ATTN". All four names report the same NHD KV + // layout, which is the layout every one of those device kernels reads — the + // name changed on CPU, the geometry validated below did not. // Mirrors upstream resolving get_attn_backend_cls per attention layer // (gpu_model_runner.py:6994-7099); we group by KV-cache kind because this // engine allocates exactly one layout per kind. diff --git a/tests/vllm/platforms/test_platform.cpp b/tests/vllm/platforms/test_platform.cpp index 1b7d9fb3b..548a99ea7 100644 --- a/tests/vllm/platforms/test_platform.cpp +++ b/tests/vllm/platforms/test_platform.cpp @@ -106,9 +106,10 @@ TEST_CASE("CPU platform is self-registered and advertises CPU capabilities") { CHECK(policy.device_pool_cap_bytes == 0); // Attention-backend priority (item 4): CPU mirrors cpu.py's single-backend - // preference (CPU_ATTN) then our FLASH_ATTN fallthrough (the layout the CPU - // paged-attn kernel actually uses). The registry-driven selection is covered - // in test_attn_backend_registry.cpp. + // preference (CPU_ATTN), followed by FLASH_ATTN, whose NHD layout our CPU + // paged-attn kernel shares. Since #1371 the first entry is registered, so the + // walk stops there; the ORDER is what this case pins, and the registry-driven + // selection is covered in test_attn_backend_registry.cpp. const std::vector cpu_priority{"CPU_ATTN", "FLASH_ATTN"}; CHECK(cpu.get_attn_backend_priority() == cpu_priority); } diff --git a/tests/vllm/v1/attention/test_attn_backend_registry.cpp b/tests/vllm/v1/attention/test_attn_backend_registry.cpp index fd39f21f0..5e2777e1a 100644 --- a/tests/vllm/v1/attention/test_attn_backend_registry.cpp +++ b/tests/vllm/v1/attention/test_attn_backend_registry.cpp @@ -7,8 +7,10 @@ // Covers: (1) the gate backends self-register per DeviceType; (2) MakeAttention- // Backend constructs the named backend / throws when absent; (3) the CPU and // CUDA (per-capability) priority ORDER matches the vLLM-mirrored lists; (4) the -// selection walk returns the first REGISTERED name (behavior-preserving -// FLASH_ATTN on both, incl. CPU's CPU_ATTN→FLASH_ATTN fallthrough); (5) the +// selection walk returns the first REGISTERED name (FLASH_ATTN on CUDA, and +// CPU_ATTN on CPU — where the name used to be unregistered and the walk fell +// through to FLASH_ATTN, until #1332 M1 gave FLASH_ATTN a head-size rule that +// left a head_size-6 CPU request with no backend at all, issue #1371); (5) the // explicit-override path; (6) sm_100 (major 10) vs sm_121 (major 12) select // differently ONLY if their preferred backend is registered — here both resolve // to FLASH_ATTN because FLASHINFER is not implemented, which IS the @@ -35,6 +37,8 @@ using vllm::platforms::Platform; using vllm::platforms::ResidencyPolicy; using vllm::v1::AttentionBackend; using vllm::v1::AttentionImpl; +using vllm::v1::AttentionType; +using vllm::v1::AttentionTypeName; using vllm::v1::AttentionLayer; using vllm::v1::MLACommonMetadata; using vllm::v1::TritonMLAImpl; @@ -114,6 +118,25 @@ class TopIsTestBackendPlatform final : public Platform { } }; +// The real CpuPlatform's selection surface (src/vllm/platforms/cpu.cpp), with +// only the vt::Backend composition faked so no accelerator is resolved. Hoisted +// out of the head-size case at the #1371 repair so the fp32 and capability cases +// below select through the SAME platform the runtime uses, rather than each +// carrying its own copy of the priority list to drift. +class FakeCpuPlatform final : public Platform { + public: + DeviceType device_type() const override { return DeviceType::kCPU; } + vt::Backend& backend() const override { return vt::GetBackend(DeviceType::kCPU); } + DeviceCapability get_device_capability() const override { return {}; } + std::vector supported_dtypes() const override { return {DType::kBF16}; } + ResidencyPolicy residency_policy() const override { return {}; } + std::vector get_attn_backend_priority( + const AttnSelectorConfig& cfg) const override { + (void)cfg; + return {"CPU_ATTN", "FLASH_ATTN"}; + } +}; + } // namespace TEST_CASE("gate backends self-register per DeviceType") { @@ -128,11 +151,18 @@ TEST_CASE("gate backends self-register per DeviceType") { // and the ROCm paged-attn kernel reads the shared NHD layout. CHECK(HasAttentionBackend(DeviceType::kROCM, "ROCM_ATTN")); + // #1371: CPU_ATTN (cpu_attn.cpp) registers for kCPU. It was a name in + // CpuPlatform's priority list and nothing else, which is why a head size + // FLASH_ATTN declines left the CPU with no backend at all. + CHECK(HasAttentionBackend(DeviceType::kCPU, "CPU_ATTN")); + // ...and for kCPU ONLY. It is upstream's CPU answer (cpu.py:75-87); nothing + // registers it elsewhere, so a CUDA walk can never land on it. + CHECK_FALSE(HasAttentionBackend(DeviceType::kCUDA, "CPU_ATTN")); + // Backends we name in the priority lists but do not implement are NOT // registered — the selection walk must skip them. CHECK_FALSE(HasAttentionBackend(DeviceType::kCUDA, "FLASHINFER")); CHECK_FALSE(HasAttentionBackend(DeviceType::kCUDA, "TRITON_ATTN")); - CHECK_FALSE(HasAttentionBackend(DeviceType::kCPU, "CPU_ATTN")); } TEST_CASE("MakeAttentionBackend constructs the named backend / throws when absent") { @@ -481,25 +511,250 @@ TEST_CASE("the runner's KV-shape contract rejects a mis-shaped backend (M3)") { std::invalid_argument); } -TEST_CASE("CPU selection: CPU_ATTN preference falls through to FLASH_ATTN") { - // The real CpuPlatform priority is {CPU_ATTN, FLASH_ATTN}; CPU_ATTN is not - // implemented, so the walk returns FLASH_ATTN (the layout our CPU paged-attn - // kernel uses). Exercised here through a synthetic CPU platform mirroring - // cpu.cpp so no GPU/accelerator resolution is involved. - class FakeCpuPlatform final : public Platform { - public: - DeviceType device_type() const override { return DeviceType::kCPU; } - vt::Backend& backend() const override { return vt::GetBackend(DeviceType::kCPU); } - DeviceCapability get_device_capability() const override { return {}; } - std::vector supported_dtypes() const override { return {DType::kBF16}; } - ResidencyPolicy residency_policy() const override { return {}; } - std::vector get_attn_backend_priority( - const AttnSelectorConfig& cfg) const override { - (void)cfg; - return {"CPU_ATTN", "FLASH_ATTN"}; - } - } cpu; - CHECK(SelectAttentionBackendName(cpu) == "FLASH_ATTN"); +TEST_CASE("CPU selection resolves CPU_ATTN, at every head size the CPU runs") { + // Issue #1371. The real CpuPlatform priority is {CPU_ATTN, FLASH_ATTN} and + // CPU_ATTN is now registered, so the walk stops on it — which is the answer + // upstream cpu.py:75-87 gives for every CPU request. Exercised through a + // synthetic CPU platform mirroring cpu.cpp so no GPU/accelerator resolution + // is involved. + // + // The second half of this case is the regression itself: before CPU_ATTN was + // registered, the walk fell through to FLASH_ATTN, and once FLASH_ATTN + // acquired flash_attn.py's `head_size % 8 == 0` rule (#1332 M1) a CPU request + // with head_size 6 matched NOTHING and SelectAttentionBackendName threw out of + // GPUModelRunner::initialize_kv_cache. Our CPU paged-attention kernel is a + // scalar loop with no head-size specialization and runs head_size 6 fine. + FakeCpuPlatform cpu; + CHECK(SelectAttentionBackendName(cpu) == "CPU_ATTN"); + + // head_size 6: NOT a multiple of 8, so FLASH_ATTN declines it and CPU_ATTN is + // the only thing standing between this request and a throw. This is the exact + // geometry test_nemotron_h_paged_forward builds (kHeadDim = 6). + AttnSelectorConfig odd; + odd.head_size = 6; + odd.block_size = 16; + CHECK(SelectAttentionBackendName(cpu, "", odd) == "CPU_ATTN"); + // ...and the FLASH_ATTN rule that made it necessary is still in force. This + // asserts the guard #1332 M1 added SURVIVES the fix: FLASH_ATTN must still + // refuse head_size 6, on CPU as on CUDA. Repairing #1371 by widening that + // rule would turn this line red. + CHECK_FALSE( + MakeAttentionBackend(DeviceType::kCPU, "FLASH_ATTN")->supports_head_size(6)); + CHECK_FALSE(SelectAttentionBackendName(cpu, "", odd) == "FLASH_ATTN"); + // A head size FLASH_ATTN accepts still resolves to CPU_ATTN, because CPU_ATTN + // is FIRST in the priority list — upstream's order, not a tie-break. + AttnSelectorConfig even; + even.head_size = 128; + even.block_size = 16; + CHECK(SelectAttentionBackendName(cpu, "", even) == "CPU_ATTN"); +} + +TEST_CASE("CPU_ATTN reports the KV shape the CPU engine actually allocates") { + // The recorded deviation (cpu_attn.h): upstream cpu_attn.py:94-104 returns the + // HND (num_blocks, num_kv_heads, block_size, 2 * head_size) shape, ours + // returns the NHD 5-dim shape src/vt/cpu/cpu_paged_attn.cpp reads by strides. + // CheckKvCacheShape compares the SELECTED backend's shape against the engine's + // own geometry, so an HND answer here would fail every CPU run at init — this + // case is what keeps the deviation deliberate rather than drift. + std::unique_ptr cpu_attn = + MakeAttentionBackend(DeviceType::kCPU, "CPU_ATTN"); + REQUIRE(cpu_attn != nullptr); + CHECK(cpu_attn->get_name() == "CPU_ATTN"); + const std::vector shape = cpu_attn->get_kv_cache_shape(10, 16, 2, 6); + const std::vector expected{10, 2, 16, 2, 6}; + CHECK(shape == expected); + // Identical to FLASH_ATTN's, dimension for dimension — which is exactly why + // the CPU could fall through to FLASH_ATTN for as long as it did. + CHECK(shape == MakeAttentionBackend(DeviceType::kCPU, "FLASH_ATTN") + ->get_kv_cache_shape(10, 16, 2, 6)); + CHECK_NOTHROW(CheckKvCacheShape(DeviceType::kCPU, "CPU_ATTN", 10, 16, 2, 6, + /*is_mla=*/false)); + // Not an MLA backend: an MLA request must not land here. + CHECK_FALSE(cpu_attn->is_mla()); +} + +TEST_CASE("fp32 on a CPU platform resolves CPU_ATTN (ported test_fp32_fallback)") { + // PORTED FROM: vllm tests/kernels/attention/test_attention_selector.py:230-241 + // @ pin 5559679229bc961848b121ccdeaa8fa5d79bec98 + // @pytest.mark.parametrize("device", ["cpu", "cuda", "hip"]) + // def test_fp32_fallback(device: str): + // ... + // with patch("vllm.platforms.current_platform", CpuPlatform()): + // backend = get_attn_backend(16, torch.float32, None) + // assert backend.get_name() == "CPU_ATTN" + // + // Upstream's four positional inputs are head_size=16, dtype=torch.float32, + // kv_cache_dtype=None and the default block size; `get_attn_backend` is the + // selector entry point our `SelectAttentionBackendName` ports, so the case is + // carried over field for field. `kv_cache_dtype` is left EMPTY because that is + // what our `supports_kv_cache_dtype` documents as upstream's `None` + // (backend.py:167-173), not because the field was forgotten. + // + // WHY THIS CASE IS THE ONE THE #1371 CHANGE OWED. `supported_dtypes` is the + // one declaration in the new capability surface that changes a live answer: + // FLASH_ATTN declares {f16, bf16} (backend.py:59, which flash_attn.py:73 + // repeats verbatim), so before CPU_ATTN was registered an f32 CPU request was + // REFUSED with "dtype not supported" and there was nothing behind it. Both + // halves are asserted below, so neither the refusal nor the fallback can move + // without a red line. + // + // HARNESS ADAPTATION — the other two arms of the upstream parametrization are + // NOT ported, and neither is a silent omission: + // * "cuda" asserts FLEX_ATTENTION. This tree registers no FLEX_ATTENTION + // (`CHECK_FALSE(HasAttentionBackend(kCUDA, "FLEX_ATTENTION"))` would be the + // whole of it), so the arm has no answer to assert yet. + // * "hip" asserts the selector RAISES, because every ROCm backend upstream + // declares a minimum head size of 32 (rocm_attn.py:193) and 16 is below it. + // Our `RocmAttentionBackend` declares no head-size list at all, so a throw + // here would come from the f32 dtype instead, i.e. the right verdict for + // the wrong reason. Porting it needs ROCM_ATTN's declared surface, which is + // the same authoring job #1389 owns for Metal, Vulkan and Tenstorrent. + FakeCpuPlatform cpu; + AttnSelectorConfig fp32; + fp32.head_size = 16; + fp32.dtype = DType::kF32; + fp32.kv_cache_dtype = ""; // upstream's None + CHECK(SelectAttentionBackendName(cpu, "", fp32) == "CPU_ATTN"); + + // The mechanism, not just the answer: FLASH_ATTN is the name the walk used to + // stop on, and it declines f32 outright. This is the assertion that would go + // red if `supported_dtypes` on CPU_ATTN dropped kF32 and something else + // started answering. + std::unique_ptr flash = + MakeAttentionBackend(DeviceType::kCPU, "FLASH_ATTN"); + REQUIRE(flash != nullptr); + CHECK_FALSE(flash->supports_dtype(DType::kF32)); + CHECK(flash->validate_configuration(fp32, DeviceCapability{}) == + std::vector{"dtype not supported"}); + + // ...and CPU_ATTN accepts the identical request with NO reasons at all, which + // is upstream's definition of a valid backend (backend.py:319-393). + std::unique_ptr cpu_attn = + MakeAttentionBackend(DeviceType::kCPU, "CPU_ATTN"); + REQUIRE(cpu_attn != nullptr); + CHECK(cpu_attn->validate_configuration(fp32, DeviceCapability{}).empty()); + + // The two half dtypes upstream lists beside f32 keep resolving CPU_ATTN too — + // cpu_attn.py:42-46 is a list of three, not an f32 special case. + for (const DType dt : {DType::kF16, DType::kBF16, DType::kF32}) { + AttnSelectorConfig cfg = fp32; + cfg.dtype = dt; + CAPTURE(static_cast(dt)); + CHECK(SelectAttentionBackendName(cpu, "", cfg) == "CPU_ATTN"); + } +} + +TEST_CASE("CPU_ATTN declares cpu_attn.py's capability surface, entry for entry") { + // Every declaration this backend overrides, pinned against + // vllm/v1/attention/backends/cpu_attn.py @ pin 5559679229. Six of the seven + // were unasserted when the class landed, so the reviewer could corrupt them — + // including WIDENING the block-size list from {16} to {1}, which would have let + // CPU_ATTN accept block_size 1 — with every binary still green. A declaration + // nothing reads is a claim, and this registry turns claims into selections. + // + // Each block asserts the DECLARED list and the PREDICATE the base class derives + // from it, because they fail differently: deleting an override moves the list + // to the base default, while corrupting the entries leaves the shape intact. + std::unique_ptr b = + MakeAttentionBackend(DeviceType::kCPU, "CPU_ATTN"); + REQUIRE(b != nullptr); + CHECK(b->get_name() == "CPU_ATTN"); + + // cpu_attn.py:42-46 — [float16, bfloat16, float32], in that order. + CHECK(b->supported_dtypes() == + std::vector{DType::kF16, DType::kBF16, DType::kF32}); + CHECK(b->supports_dtype(DType::kF16)); + CHECK(b->supports_dtype(DType::kBF16)); + CHECK(b->supports_dtype(DType::kF32)); + // ...and nothing else. A widened list would make an integer KV/query dtype + // selectable on a kernel that has no path for it. + CHECK_FALSE(b->supports_dtype(DType::kI8)); + CHECK_FALSE(b->supports_dtype(DType::kQ8_0)); + + // cpu_attn.py:47-52 — ["auto", "fp8", "fp8_e4m3", "fp8_e5m2"], MINUS the last + // (recorded deviation 3 in cpu_attn.h). The refusal is the assertion: our CPU + // KV-fp8 arm resolves KvKind::kFp8 to LoadKvFp8E4M3 alone + // (src/vt/cpu/cpu_paged_attn.cpp), so claiming e5m2 would select this backend + // for a cache it cannot read. + CHECK(b->supported_kv_cache_dtypes() == + std::vector{"auto", "fp8", "fp8_e4m3"}); + CHECK(b->supports_kv_cache_dtype("auto")); + CHECK(b->supports_kv_cache_dtype("fp8")); + CHECK(b->supports_kv_cache_dtype("fp8_e4m3")); + CHECK_FALSE(b->supports_kv_cache_dtype("fp8_e5m2")); + // Upstream's CPU list omits the explicit half-dtype names that FLASH_ATTN's + // carries (flash_attn.py:74-80), so ours does too — the deviation is e5m2 and + // ONLY e5m2. + CHECK_FALSE(b->supports_kv_cache_dtype("float16")); + CHECK_FALSE(b->supports_kv_cache_dtype("bfloat16")); + // An empty name is upstream's None and is accepted outright (backend.py:167-173). + CHECK(b->supports_kv_cache_dtype("")); + + // cpu_attn.py:54-56 — [MultipleOf(16)]. NOT MultipleOf(1): the base default is + // {1} (backend.py:69-71), so losing this override widens the backend silently + // rather than narrowing it, and both lines below are what notice. + CHECK(b->get_supported_kernel_block_sizes() == std::vector{16}); + CHECK(b->supports_block_size(16)); + CHECK(b->supports_block_size(32)); + CHECK_FALSE(b->supports_block_size(1)); + CHECK_FALSE(b->supports_block_size(8)); + CHECK_FALSE(b->supports_block_size(24)); + // 0 is upstream's None (backend.py:176-178), accepted by every backend. + CHECK(b->supports_block_size(0)); + + // DEVIATION 2 (cpu_attn.h): NO head-size constraint, where cpu_attn.py:58-60 + // lists eleven sizes. An empty list is upstream's "unconstrained" + // (backend.py:158-161), and it is WIDER than upstream's CPU backend — which is + // the deliberate part: our CPU kernel is a scalar loop with no head-size + // specialization, so head_size 6 (the geometry #1371 was reported on) and every + // size upstream lists are all accepted here. + CHECK(b->get_supported_head_sizes().empty()); + CHECK(b->supports_head_size(6)); + CHECK(b->supports_head_size(3)); + CHECK(b->supports_head_size(128)); + CHECK(b->supports_head_size(512)); + CHECK(b->supports_head_size(1024)); + + // cpu_attn.py:66-68 / :70-72 — both True, and both are FALSE on the base class + // (backend.py:261-274), so a deleted override reads as a refusal. + CHECK(b->supports_non_causal()); + CHECK(b->supports_sliding_window()); + + // cpu_attn.py:74-83 — decoder, encoder, encoder-only AND encoder-decoder. The + // base supports decoder alone (backend.py:291-298), so the three non-decoder + // lines are the ones that fail if the override goes. + CHECK(b->supports_attn_type(AttentionTypeName(AttentionType::kDecoder))); + CHECK(b->supports_attn_type(AttentionTypeName(AttentionType::kEncoder))); + CHECK(b->supports_attn_type(AttentionTypeName(AttentionType::kEncoderOnly))); + CHECK(b->supports_attn_type(AttentionTypeName(AttentionType::kEncoderDecoder))); + CHECK_FALSE(b->supports_attn_type("not_an_attention_type")); + + // The declarations upstream's CPU backend does NOT override, asserted so a + // future widening has to be deliberate: cpu_attn.py defines no `supports_sink`, + // no `supports_mm_prefix` and no `supports_per_head_quant_scales`, and it is not + // an MLA or a sparse backend. + CHECK_FALSE(b->supports_sink()); + CHECK_FALSE(b->supports_mm_prefix()); + CHECK_FALSE(b->supports_per_head_quant_scales()); + CHECK_FALSE(b->is_mla()); + CHECK_FALSE(b->is_sparse()); + + // Two of the above through the CONSUMER, so the declarations are bound to the + // selection the runtime performs and not only to their own getters. A sliding- + // window CPU request selects CPU_ATTN; a block_size the list refuses selects + // NOTHING, and the walk throws exactly as it did for head_size 6. + FakeCpuPlatform cpu; + AttnSelectorConfig sliding; + sliding.head_size = 6; + sliding.block_size = 16; + sliding.has_sliding_window = true; + sliding.use_non_causal = true; + CHECK(SelectAttentionBackendName(cpu, "", sliding) == "CPU_ATTN"); + + AttnSelectorConfig bad_block = sliding; + bad_block.block_size = 8; + CHECK_THROWS_AS(SelectAttentionBackendName(cpu, "", bad_block), + std::runtime_error); } TEST_CASE("explicit backend override is honored / validated") { diff --git a/tests/vllm/v1/worker/test_runner.cpp b/tests/vllm/v1/worker/test_runner.cpp index 107f2e8a9..d2bb97c2b 100644 --- a/tests/vllm/v1/worker/test_runner.cpp +++ b/tests/vllm/v1/worker/test_runner.cpp @@ -453,11 +453,13 @@ TEST_CASE("runner: KV allocation from KVCacheConfig (full-attn + GDN state)") { // M3: the runner resolves the ENGINE-level attention backend at init, per // attention group. On CPU the dense priority walk (cpu.cpp) is - // [CPU_ATTN (unregistered), FLASH_ATTN] so it lands on FLASH_ATTN — - // behavior-preserving, and the proof that selection is now part of the - // runtime path, not just the registry test. One name per attention layer. + // [CPU_ATTN, FLASH_ATTN] and, since #1371 registered it, lands on CPU_ATTN — + // upstream's own CPU answer (cpu.py:75-87). This is the proof that selection + // is part of the runtime path, not just the registry test: the name below is + // resolved inside GPUModelRunner::initialize_kv_cache. One name per attention + // layer. REQUIRE(runner.attn_backend_names().size() == 1); - CHECK(runner.attn_backend_names()[0] == "FLASH_ATTN"); + CHECK(runner.attn_backend_names()[0] == "CPU_ATTN"); // One PagedKvCache per full-attn layer (config has exactly 1). REQUIRE(runner.attn_kv().size() == 1);