diff --git a/.agents/issue-index.md b/.agents/issue-index.md index a585ac0be..e5bb29945 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -427,6 +427,8 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1325](https://github.com/mudler/vllm.cpp/issues/1325) | `ENG-RECORD-ANCHOR-RATCHET` | `scripts/record-anchor-baseline.json` stores a `"total"` that no code reads. `load_record_anchor_baseline` in `scripts/check-agent-record.py` returns `{bucket: int(data["buckets"][bucket]) for bucket in RECORD_ANCHOR_BUCKETS}` with `RECORD_ANCHOR_BUCKETS = ("stale", "broken")`, and it is the file's only reader; `check_record_anchors` iterates those two buckets, and `write_record_anchor_baseline`'s refusal compares `result.total > sum(previous.values())` — the buckets, not the stored `total`. Measured at `af87251c5`: mutating `"total": 38` to `39` (a file whose total disagrees with `32 + 6`) leaves `check-agent-record.py` exit 0, tree restored byte-for-byte by sha256. So this row's own budget file carries the exact shape the row exists to name: a recorded figure no gate reads, sitting beside the figures that are read and presenting as if it were checked. `--write-baseline` compounds it by printing `-> 38`, which reads as the value it stored and is the one no later run consults. TWO candidate resolutions, deliberately not chosen here because choosing belongs to the fixing row: read it and assert `total == stale + broken` on load, or drop the field and derive it at read time, which is the shape AGENTS.md §Records prefers. Either is a semantic change to `check-agent-record.py` owing a spec and a red-before case in `tests/scripts/test_agent_record.py` `RecordAnchorRatchet`. Distinct from [#1287](https://github.com/mudler/vllm.cpp/issues/1287) and [#1270](https://github.com/mudler/vllm.cpp/issues/1270), neither of which reaches the unread field. Also under `## Owed` in [`record-anchor-ratchet.md`](specs/record-anchor-ratchet.md) | bug | | [#1316](https://github.com/mudler/vllm.cpp/issues/1316) | — | `scripts/main-baseline.py` renders a scheduled run that executed ZERO jobs as `RED` with all 11 covered jobs `missing`, so `NEWEST BASELINE: RED at ` names a tree the run never checked out. Measured at `origin/main` `250db75a2`: runs `32206456661` and `32140419182` both return `startedAt: null` for every job, because GitHub cancelled them while they were pending in the single `ci-schedule-refs/heads/main-mudler/vllm.cpp` group, whose queue holds one run ([#274](https://github.com/mudler/vllm.cpp/issues/274)). Fail-closed, and the `missing (expected, never ran)` line is accurate about the jobs; the defect is the verdict word, because a run that executed nothing is NOT RUN rather than RED, and the newest verdict should fall through to the newest run that actually ran. NOT fixed in flow: `test_an_expected_job_the_payload_never_mentions_is_red` and `test_a_narrowed_run_reports_red_and_names_what_never_ran` deliberately assert missing-is-red so a narrowed run cannot pass, and separating "narrowed" from "never started" changes what the verdict means, which owes its own spec, red-before evidence and a fresh reviewer. Owed under `## Owed` of [baseline-lane-eviction.md](specs/baseline-lane-eviction.md), which removes the only observed producer of a zero-job run | bug | | [#1314](https://github.com/mudler/vllm.cpp/issues/1314) | `SPEC-DFLASH2` | **DFlash2 (`DFlash2DraftModel`) is unported**, and one config rule would run the published checkpoint wrong in silence. Upstream carries DFlash2 as a SECOND architecture beside DFlash rather than as a change to it ([vllm#52816](https://github.com/vllm-project/vllm/pull/52816), OPEN at head `19c9351904df4c63042671bc67a866ca48dc7d6f`, base `9842d701`, 755+/5-, 11 files, plus the stacked guard fix [vllm#52883](https://github.com/vllm-project/vllm/pull/52883)): DFlash1 gains two subclass seams and keeps every behaviour, and the new architecture adds a GROUPED DYNAMIC DEPTHWISE CONVOLUTION around each attention and each MLP sublayer plus a CANDIDATE SELECTOR that replaces the independent per-slot argmax with a scored path walk over the target head's top-K. Shapes taken from the published checkpoint rather than from the diff: `z-lab/Qwen3.8-27B-DFlash2`, safetensors header range-read 2026-08-19, 81 tensors -- DFlash1's set plus `layers.N.{attention,mlp}_conv.{base_kernel (2,2,5120), kernel_projection.weight (1280,5120)}` x5 and `candidate_selector.{hidden_projection.weight (256,5120), predecessor_codebook, successor_codebook}` at `(248320,256)` bf16 each, ~254 MB resident the DFlash1 lane never allocates; `conv_kernel_size 2`, `conv_group_size 16`, `selector_rank 256`, `selector_top_k 16`, `block_size 8`. **The silent one:** that config declares all five layers `sliding_attention` AND `is_causal false`, while our resolution mirrors the OLD upstream rule (causal iff SWA, unless `dflash_config.causal`, `include/vllm/model_executor/models/qwen3_dflash.h:22-24`), so every layer would run CAUSAL -- plausible tokens, a token gate against our own output sees nothing, and only ACCEPTANCE moves, which the lossless verify hides. Upstream changes `_dflash_layer_causal` to read `is_causal` first, in the same commit. Three further things are owed and none of them is silent: no route for the `DFlash2DraftModel` architecture string (the same classification code as the open `DSparkDraftModel` gap, [#1193](https://github.com/mudler/vllm.cpp/issues/1193)); no top-k that EMITS the surviving (id, value) pairs, where the decision is to extend the sort-free pivot-bracket threshold search already ported from the same FlashInfer approach at `src/vt/cuda/cuda_sample.cu:297-506` rather than port FlashInfer's 3380-line general radix kernel; and the path walk must run ON DEVICE from the first landing, because the identical sequential shape in DSpark shipped host-side and measured 28% of the 27B draft step ([#436](https://github.com/mudler/vllm.cpp/issues/436)) before `SampleSequentialDevice` moved it. Already reusable unchanged: `vt::DFlashBlockAttention`, the DFlash runner/rejection/GDN-rollback lane, and the loader's target-shared `embed_tokens`+`lm_head`, which is already what a DFlash2 checkpoint needs. BEYOND-PIN by developer decision 2026-08-19 (mirror the open PR now, reconcile if review moves it), in the same posture `SPEC-DSPARK-QWEN3-ROUTING` takes toward vllm#52197; the parity pin `555967922` does not carry the architecture and is NOT advanced. Gate arm is bf16 27B plus the GGUF drafter arm in the same wave, oracle = vLLM built at the PR head, acceptance measured SAME-TRAJECTORY because `SPEC-DFLASH` D8 spent a campaign on a divergent-trajectory confound that D9 refuted. Spec [dflash2-spec-decode.md](specs/dflash2-spec-decode.md) | feature | +| [#1332](https://github.com/mudler/vllm.cpp/issues/1332) | `BACKEND-ATTN-REGISTRY` | The attention-backend selector routes nothing, and a capability check is not a runnability check. `vllm::v1::SelectAttentionBackendName` (`src/vllm/v1/attention/registry.cpp`) mirrors vLLM's priority walk faithfully, and its result reaches only `attn_backend_names_`, a `VT_ATTN_SELECT_LOG` print and `CheckKvCacheShape` in `runner.cpp::GpuModelRunner::InitializeKvCache`; `dense_attn::AttnBlock` calls `vt::PagedAttention` unconditionally and the real arm choice is an env-flag + shape + dtype ladder in `src/vt/cuda/cuda_paged_attn.cu`. Deleting the whole selector would leave every emitted token identical, which is `.agents/reachability.md`'s "unselected branch" and "unpassed parameter" at once. Separately, measured on a GB10 (capability 12,1): vLLM selects `FLASH_ATTN` because `supports_compute_capability` is `capability >= (8,0)`, while the shipped FA2 binary carries `sm_80` SASS plus `compute_80` PTX alone, so every launch needs a driver JIT that fails with `cudaErrorUnsupportedPtxVersion` — `grep -rn get_arch_list vllm/` returns zero hits, so vLLM never asks what its own fatbins contain. We have written the same class of check: `src/vllm/platforms/cuda.cpp::CudaPlatform::supports_fa2_attention` returns true for every CUDA device while `CMakeLists.txt` defaults `VLLM_CPP_CUDA_ARCHITECTURES` to `121a` alone. The invariant: no backend may be declared valid on the strength of a property of the DEVICE alone; every predicate naming a compute capability must be paired with one naming the binary. M0 (reconcile) and M1 (the `validate_configuration` capability surface) land in [`attn-validate-configuration.md`](specs/attn-validate-configuration.md); the compiled-arch manifest, the launch probe, the dispatch wiring and the ABI override remain owed there under `## Owed` | bug | +| [#1333](https://github.com/mudler/vllm.cpp/issues/1333) | `BACKEND-ATTN-REGISTRY` | The non-MLA sm_100 attention priority row misses the `use_non_causal` guard the pin added. `include/vllm/platforms/cuda_attn_priority.h::AttnPriorityTable` keys rows on `(use_mla, major)` and cites `vllm/platforms/cuda.py:143-166` `@ pin e24d1b24`; at the tree pin `5559679229` that arm reads `if device_capability.major == 10 and not use_non_causal` (`cuda.py:148`) with `use_non_causal` added as a fifth `_get_backend_priorities` parameter (`cuda.py:88`), because SM100f's non-causal cutlass path — the one DFlash attention uses — is known-bad (`cuda.py:145-147`). No selected name changes in this tree today, since `FLASHINFER` is registered for no device and both orderings fall through to `FLASH_ATTN`; it becomes a real divergence the moment a FlashInfer backend registers, and it is already a divergence in the record because the header claims a faithful, complete port of that function. Found while reconciling the four `@ pin e24d1b24` header anchors under [#1332](https://github.com/mudler/vllm.cpp/issues/1332) M0 and fixed in the same flow, since `use_non_causal` is a field M1 adds to `AttnSelectorConfig` regardless. Spec [`attn-validate-configuration.md`](specs/attn-validate-configuration.md) | bug | | [#1309](https://github.com/mudler/vllm.cpp/issues/1309) | `MUSIC3-DEPTH-DEVICE` | MiniMax-Music3's 0.646 B RVQ depth decoder is **48.4 % of a run** on `thor:gpu0` and is the last large stage still on the host, so a 0.646 B model costs **6.3x** the 8.6 B language model beside it on the same box. §11.4's last owed device row: the vocoder closed in §13, the DiT in §14, and §14.5 blocked this one on a dtype rather than on the work. **The dtype is settled in [§19.2](specs/minimax-music3.md): bf16 storage, f32 accumulation.** The oracle settles it by declaring nothing — `MiniMaxMusic3RVQDepthDecoder` takes no `dtype` parameter and contains no `torch.float32` literal and no `.float()` call (`diffusers` @ `c6da9936`, `models/transformers/minimax_music3_rvq_depth_decoder.py:101-125`); its one cast is the **down**-cast `:51` `.to(query.dtype)`; and `tools/oracle/music3_oracle.py:91-95,103-108` resolves `rvq_depth_decoder: torch.bfloat16` under both policies. `vt::MatmulBT` already carries exactly that contract (`include/vt/ops.h:1281-1283`), so §14.5's objection — that an **f32** `vt::MatmulBT` would drop the bf16 rounding every gated number was taken with — is answered by using the **bf16** one. A finding falls out that no gate here can see: the host arm keeps bf16-exact weights in `std::vector`, so every golden, token gate and WAV hash passes while the path moves twice the oracle's bytes ([§19.2a](specs/minimax-music3.md)) | feature | | [#1322](https://github.com/mudler/vllm.cpp/issues/1322) | `MUSIC3-DEPTH-DEVICE` | `vt` cannot express torch's **per-op bf16 rounding**, and no caller can build it. A bf16 torch module rounds at every op boundary; `vt::RmsNorm` keeps full f32 across the weight multiply (`include/vt/ops.h` says so) where `normalization.py:600-606` casts back to the weight dtype first, and `vt::SiluAndMul` computes `silu(g)*u` wholly in f32 where `F.silu(gate)` produces a **bf16 tensor** that is then multiplied. MEASURED on MiniMax-Music3's depth decoder ([#1309](https://github.com/mudler/vllm.cpp/issues/1309), spec §19.4a): the device arm reads worst **110 bf16 ULP** / mean **2.095** against the host reference, and collapsing those two host roundings to the seam's shape takes it to worst 8 / mean 0.0596 and the composed stage to **ZERO — bit-identical**. So ~97 % of the deviation is rounding POLARITY inside two shared ops, not dtype, accumulator or reduction order. A call site cannot split them, because `vt` has no elementwise or row-broadcast multiply — `kMulScalar` takes a scalar and `kMulColVecF32` is an f32 in-place column scale. Closing it needs either a `vt::Mul` with `vt::Add`'s two operand shapes or rounding-faithful modes on the two ops, with CPU and CUDA providers; #1309 records the measurement and stops. A token gate cannot see any of this | bug | | [#1351](https://github.com/mudler/vllm.cpp/issues/1351) | `MUSIC3-DEPTH-DEVICE` | `scripts/check-fusion-consistency.py` Check 2 is satisfied by a **COMMENT**: it runs `_MERGED_GEMM_SEAM` over `path.read_text()` with no comment stripping, so a model TU that merely *mentions* `MlpGateUpMethod` in prose is exempt from the check forever. Measured on `minimax_music3_depth_device.cpp` ([#1309](https://github.com/mudler/vllm.cpp/issues/1309), spec §19.7): replacing the `layers::UnquantizedMlpGateUpMethod` call with an inline hand-rolled `vt::MatmulBT` + `vt::SiluAndMul` path while leaving the two comment mentions gives `uses_merged_gemm_seam = True` and the checker returns **rc=0 green**, with the numbers bit-identical so no other gate sees it; stripping comments flips it to `False`. The only comment-aware helper in the file, `allowlisted_names`, strips `#` comments from the allowlist FILE rather than from the scanned source. NOT a defect: `MlpGateUp[A-Za-z]*Method` matching a renamed `MlpGateUpXX` is deliberate, to cover `UnquantizedMlpGateUpGeluMethod`. Fixing it is a semantic checker change — spec, red-before test, green-after evidence — and must not be done by widening the regex; stripping comments also moves the site count, so a fixture must pin both numbers | bug | diff --git a/.agents/kernel-matrix.md b/.agents/kernel-matrix.md index b90469977..c4e96c5a5 100644 --- a/.agents/kernel-matrix.md +++ b/.agents/kernel-matrix.md @@ -137,7 +137,7 @@ host/sched. Detail: state `KERNEL-FA2-GQA-SWAP-FLIP`. | `KERNEL-ATTN-FA3-FA4` | FlashAttention-3/4 | FA3/FA4 build `setup.py:1113-1124`; runtime version selection `vllm/v1/attention/backends/fa_utils.py:132-250` | - | - | [inventory](specs/kernel-family-inventory.md) | `INVENTORIED` | - | | `KERNEL-ATTN-FLASHINFER-TRTLLM` | FlashInfer paged attention, XQA, and TensorRT-LLM generation kernels | availability/dispatch `vllm/utils/flashinfer.py:206-342,373-511`; SM90 decode and SM100+ prefill/decode gates | - | - | [inventory](specs/kernel-family-inventory.md) | `INVENTORIED` | - | | `KERNEL-ATTN-TRITON-FLEX-HPC` | Triton, FlexAttention, TurboQuant, and HPC attention families | registry `registry.py:34-120`; Triton contract `triton_attn.py:250-383`; HPC gate `hpc_attn.py:235-247` | - | - | [inventory](specs/kernel-family-inventory.md) | `INVENTORIED` | - | -| `KERNEL-ATTN-MLA-SPARSE` | MLA and sparse attention: CUTLASS, FlashMLA, FlashInfer, FA, Triton, MSA **W6: the MLA attention BLOCK + WEIGHT ABSORPTION — the layer that COMPOSES W3+W4+W5** — [mla_attention.h](../include/vllm/model_executor/models/mla_attention.h) + [mla_attention.cpp](../src/vllm/model_executor/layers/attention/mla_attention.cpp) <- `mla.py:119-181` (`MultiHeadLatentAttentionWrapper.forward`) over `mla_attention.py:553-620` (the cache-update-BEFORE-attention order), `:624-874` (`forward_impl`: the dispatch + the absorbed decode) and `:2344-2425` (`forward_mha`); `AbsorbKvBProjBf16` <- `:875-962 process_weights_after_loading` (split `:892-900`, permutes `:959-962`); `MakeMlaUpProjectFn` <- `:2141-2170` (the `kv_b_proj` callback W5 left open); `BuildDeepseekRopeCosSinCache` <- `deepseek_scaling_rope.py:76-118` over `rotary_embedding/common.py:34-70`; `MlaAttentionScale` <- `deepseek_v2.py:995,1067-1075` (the mscale^2 correction, kept SEPARATE from the rope's own rotation mscale). **Absorption is a LOAD-TIME weight transform plus TWO batched GEMMs, not a fused kernel**, so it needed only two new general primitives: **`vt::BatchedMatmul`** <- `torch.bmm` at `mla_attention.py:789` (q-side W_UK fold) and `:1034` (`_v_up_proj`), CUDA impl = cuBLASLt STRIDED-BATCHED [cuda_matmul.cu](../src/vt/cuda/cuda_matmul.cu) (the cuBLASLt form of the cuBLAS `gemmStridedBatchedEx` torch.bmm resolves to; the only upstream alternatives are ROCm-only aiter fp8/fp4 bmm branches) + CPU ref [cpu_ops.cpp](../src/vt/cpu/cpu_ops.cpp), stride-driven because BOTH call sites pass `.transpose(0,1)` views; and **`vt::ConcatMlaNopeRope`** <- `ConcatMLAQKernel` (`csrc/libtorch_stable/concat_mla_q.cuh`) + wrapper `cache_kernels.cu:1555-1600`, GENERALIZED to arbitrary nope/rope widths and a head-BROADCAST rope operand so one op also serves `_concat_k_nope_k_pe` (`:2063-2092`) — CUDA [cuda_mla_attn.cu](../src/vt/cuda/cuda_mla_attn.cu), CPU [cpu_ops.cpp](../src/vt/cpu/cpu_ops.cpp). Two ADDITIVE relaxations of existing ops, integer-identical for contiguous tensors: `vt::RopeFromCache` stride-driven on q/k (DeepSeek rotates the TRAILING 64-dim slice and its `k_pe` is a column block of the fused kv_a projection) and `vt::MatmulBT` accepting a row-strided ACTIVATION (`kv_b_proj` applied to a 512-column slice of the 576-wide workspace, `:2160`) | CUDA priority `vllm/platforms/cuda.py:84-176` (`_get_backend_priorities`, both branches); MLA classes `vllm/v1/attention/backends/mla/*.py`; MLA prefill selector `mla/prefill/selector.py:47-76`; capability filter `vllm/v1/attention/backend.py:307-360`; CUTLASS build `CMakeLists.txt:1037-1061` **W6** [test_mla_attention_block.cpp](../tests/vllm/model_executor/layers/attention/test_mla_attention_block.cpp) **10/10 cases / 2,372,644 assertions** and [test_ops_mla_absorb.cpp](../tests/vt/test_ops_mla_absorb.cpp) **9/9 / 1,644,807 assertions** on dgx sm_121 — ports of `tests/kernels/test_concat_mla_q.py` (BOTH arms incl. the NON-CONTIGUOUS transposed-nope case, compared bit-exactly since a concat is a pure copy), the MLA-geometry sweep of `tests/v1/attention/test_mla_backends.py`, and the two-pass-oracle discipline of `tests/kernels/attention/test_mla_decode_cpu.py`. **THE ABSORBED-vs-UNABSORBED EQUIVALENCE IS PROVEN NUMERICALLY, THREE WAYS:** an INDEPENDENT double-precision block oracle computing the attention BOTH ways agrees to **< 1e-11** (the identity itself, at both query branches); our absorbed decode reproduces the UNABSORBED oracle to **< 2e-4** (f32); and the SAME batch driven once through the ABSORBED MQA decode kernel and once through the UNABSORBED materialized-MHA prefill path agrees to **< 3e-4** (CPU f32) / **< 4e-2** (CUDA bf16) — two code paths sharing nothing but the weights. Real geometry throughout (V2-Lite 512/128/64/128/16-head, plus V3's 7168 / 128-head / `q_lora_rank=1536` for the lora branch, which has NO e2e coverage and says so). Decode-only / prefill-only-no-context / chunked-prefill-with-context / MIXED (decode packed FIRST) all gated; NaN-poisoned outputs; run-to-run BIT-exact; CUDA cases proven to EXECUTE (124,941 + 290,835 assertions when run alone). memcheck **0 errors**, racecheck **0 hazards**, synccheck **0 errors** (the last requires `--num-cuda-barriers 65536`: the default table OVERFLOWS on a binary driving this many kernel families and the tool then emits a bogus `unspecified launch failure`). Clean CUDA build 0 warn/0 err; regression set UNCHANGED (27B 235/235, 35B 315/315, Coder 138/138, Qwen3-dense 664/664, OPT 36/36). No speed number — W9 owns tuning | **SELECTION (W2) + the DECODE KERNEL (W4) + the PREFILL PATH and CHUNKED-CONTEXT LOOP (W5). What is still absent is the MLA attention BLOCK and MODEL (W6/W7).** Priority TABLE [cuda_attn_priority.h:49](../include/vllm/platforms/cuda_attn_priority.h#L49) (both branches, one row per upstream arch arm) + lookup [cuda_attn_priority.h:86](../include/vllm/platforms/cuda_attn_priority.h#L86); `is_mla()`/`is_sparse()` filter [registry.cpp:63](../src/vllm/v1/attention/registry.cpp#L63); `TritonMLABackend` NAME + 3-D `get_kv_cache_shape` [backend.h:271](../include/vllm/v1/attention/backend.h#L271), [backend.cpp:83](../src/vllm/v1/attention/backend.cpp#L83), registration [backend.cpp:108](../src/vllm/v1/attention/backend.cpp#L108) — **W4: `vt::MlaDecodeAttention`** — `OpId::kMlaDecodeAttention` + args/validation [ops.h](../include/vt/ops.h), [ops.cpp](../src/vt/ops.cpp); CPU single-pass REFERENCE [cpu_mla_attn.cpp](../src/vt/cpu/cpu_mla_attn.cpp) (numerics from `csrc/cpu/mla_decode.cpp`); CUDA two-stage split-KV [cuda_mla_attn.cu](../src/vt/cuda/cuda_mla_attn.cu) — `MlaDecodeStage1` <- `_fwd_grouped_kernel_stage1` (`triton_decode_attention.py:278-458`, IS_MLA `v = tl.trans(k)` branch `:424-431`), `MlaDecodeStage2` <- `_fwd_kernel_stage2` (`:575-639`), `ComputeNumKvSplits` <- `_compute_num_kv_splits` (`triton_mla.py:40-47`), split workspace via the house grow-only per-stream scratch (upstream's `_reserve_attn_logits_workspace` `:57-78`). Deterministic by construction: fixed ASCENDING split merge, NO atomicAdd. `TritonMLABackend::get_impl_cls()` now returns a real `TritonMLAImpl` [backend.h](../include/vllm/v1/attention/backend.h), [backend.cpp](../src/vllm/v1/attention/backend.cpp); PREFILL remains W5 and `TritonMLAImpl::forward` refuses a prefill-shaped batch by name. **W5: `vt::MlaPrefillAttention` + `vt::GatherMlaCache` + `vt::MergeAttnStates` + the chunked-context driver** — `vt::MlaPrefillAttention` [cuda_mla_prefill.cu](../src/vt/cuda/cuda_mla_prefill.cu) / CPU ref [cpu_mla_prefill.cpp](../src/vt/cpu/cpu_mla_prefill.cpp) <- `mla/prefill/flash_attn.py:153-248` `FlashAttnPrefillBackend` (the ONLY MLA prefill backend reachable on sm_121 per `mla/prefill/selector.py:66-76`, and it HARD-RAISES with no fallback at `:191-194`), running over the vendored FA-2 through the NEW launcher entry `LaunchMlaPrefillFA2Bf16` [cuda_flash_attn_fa2.cu](../src/vt/cuda/cuda_flash_attn_fa2.cu) plus two new explicit instantiations of the UNCHANGED generic template (`flash_fwd_split_hdim192_bf16{,_causal}_sm80.cu`). V is ZERO-PADDED 128->192 and the output sliced back, exactly as upstream's `requires_v_padding` path does (`flash_attn.py:88-99,164-168,196-197`) — which is WHY the asymmetric QK 192 / V 128 pair needs no asymmetric kernel. `vt::GatherMlaCache` <- `csrc/libtorch_stable/cache_kernels.cu:992-1064`; `vt::MergeAttnStates` <- `csrc/libtorch_stable/attention/merge_attn_states.cu:18-192` (BOTH `-inf` edge cases ported verbatim). The workspace-bounded loop is [mla_chunked_context.h](../include/vllm/model_executor/layers/attention/mla_chunked_context.h) <- `mla_attention.py:1422-1451,1667-1745,2094-2199,2344-2425`. **The paged launcher `LaunchPrefillFA2Bf16` that every non-MLA prefill calls is textually UNTOUCHED** (211 insertions / 0 deletions in that TU; 2 new vendored files) | [test_attn_backend_registry.cpp:146](../tests/vllm/v1/attention/test_attn_backend_registry.cpp#L146) (GB10 MLA list), [:203](../tests/vllm/v1/attention/test_attn_backend_registry.cpp#L203) (`use_mla=true` -> `TRITON_MLA`, matching the W0 oracle observation), [:230](../tests/vllm/v1/attention/test_attn_backend_registry.cpp#L230) (the DSA seam, proven both directions with a stand-in sparse backend) — ports of `test_attention_backends_selection.py` (MLA cases), `test_mla_prefill_selector.py`, `test_mla_prefill_registry.py`; **W4** [test_ops_mla_attn.cpp](../tests/vt/test_ops_mla_attn.cpp) — port of `tests/kernels/attention/test_mla_decode_cpu.py` (`ref_mla` as a TWO-PASS oracle, its bs=4/mean_seq_len=256/h_q=16/d=576/dv=512/block=16 parametrization, BOTH varlen arms, and its NaN-padding out-of-bounds detector) plus the `test_mla_backends.py` shape sweep: ragged, multi-block, single-block/single-token, EVERY num_kv_splits in {1,2,3,4,5,8,16,17,64,300,512} (incl. splits > seq_len, the empty-split path both stages must skip), 128-head DeepSeek-V3 geometry, head counts 1/3/17 that do not fill a BLOCK_H tile, a 288/256 block-32 non-V2-Lite geometry, bf16 + f32, and run-to-run BIT-exactness over 5 runs. Gated on dgx/sm_121: 11/11 cases, 2,303,193 assertions; `compute-sanitizer` memcheck **0 errors**, racecheck **0 hazards**, synccheck **0 errors**; clean CUDA build 0 warn/0 err; regression set UNCHANGED (27B 235/235, 35B 315/315, Coder 6/6, Qwen3-dense 16/16, OPT 6/6). NO speed number yet — decode perf is W9. **W5** [test_ops_mla_prefill.cpp](../tests/vt/test_ops_mla_prefill.cpp) **4/4 cases / 2,377,052 assertions** and [test_ops_mla_chunked_context.cpp](../tests/vt/test_ops_mla_chunked_context.cpp) **5/5 / 306,037 assertions** on dgx sm_121 — ports of `tests/v1/attention/test_mla_backends.py` and `tests/v1/attention/test_mla_prefill_quant_output.py` (its fp8 arms NOT ported: they need device-capability family 100, unreachable on sm_121 — recorded, not dropped). REAL V2-Lite prefill geometry (QK 192 / V 128 / latent 576, block 16, mscale^2 scale) against an INDEPENDENT double-precision TWO-PASS oracle, plus — for the chunked loop — a SINGLE-SHOT whole-sequence oracle that never chunks: exact / +1 / -1 chunk boundaries, a request with NO context, a chunk in which a request contributes ZERO keys, ragged multi-chunk, 128-head V3, single-token queries, ADVERSARIAL reverse-interleaved block tables, NaN-poisoned outputs, run-to-run BIT-exact over 5 runs. memcheck **0 errors**, racecheck **0 hazards**, synccheck **0 errors** on both binaries; clean CUDA build 0 warn/0 err; regression set UNCHANGED (27B 235/235, 35B 315/315, Coder 138/138, Qwen3-dense 664/664, OPT 36/36). Prefill perf is W9 | [MLA campaign spike](specs/mla-deepseek-campaign.md) | `PARTIAL` | `CLAIM-MLA-DEEPSEEK` | +| `KERNEL-ATTN-MLA-SPARSE` | MLA and sparse attention: CUTLASS, FlashMLA, FlashInfer, FA, Triton, MSA **W6: the MLA attention BLOCK + WEIGHT ABSORPTION — the layer that COMPOSES W3+W4+W5** — [mla_attention.h](../include/vllm/model_executor/models/mla_attention.h) + [mla_attention.cpp](../src/vllm/model_executor/layers/attention/mla_attention.cpp) <- `mla.py:119-181` (`MultiHeadLatentAttentionWrapper.forward`) over `mla_attention.py:553-620` (the cache-update-BEFORE-attention order), `:624-874` (`forward_impl`: the dispatch + the absorbed decode) and `:2344-2425` (`forward_mha`); `AbsorbKvBProjBf16` <- `:875-962 process_weights_after_loading` (split `:892-900`, permutes `:959-962`); `MakeMlaUpProjectFn` <- `:2141-2170` (the `kv_b_proj` callback W5 left open); `BuildDeepseekRopeCosSinCache` <- `deepseek_scaling_rope.py:76-118` over `rotary_embedding/common.py:34-70`; `MlaAttentionScale` <- `deepseek_v2.py:995,1067-1075` (the mscale^2 correction, kept SEPARATE from the rope's own rotation mscale). **Absorption is a LOAD-TIME weight transform plus TWO batched GEMMs, not a fused kernel**, so it needed only two new general primitives: **`vt::BatchedMatmul`** <- `torch.bmm` at `mla_attention.py:789` (q-side W_UK fold) and `:1034` (`_v_up_proj`), CUDA impl = cuBLASLt STRIDED-BATCHED [cuda_matmul.cu](../src/vt/cuda/cuda_matmul.cu) (the cuBLASLt form of the cuBLAS `gemmStridedBatchedEx` torch.bmm resolves to; the only upstream alternatives are ROCm-only aiter fp8/fp4 bmm branches) + CPU ref [cpu_ops.cpp](../src/vt/cpu/cpu_ops.cpp), stride-driven because BOTH call sites pass `.transpose(0,1)` views; and **`vt::ConcatMlaNopeRope`** <- `ConcatMLAQKernel` (`csrc/libtorch_stable/concat_mla_q.cuh`) + wrapper `cache_kernels.cu:1555-1600`, GENERALIZED to arbitrary nope/rope widths and a head-BROADCAST rope operand so one op also serves `_concat_k_nope_k_pe` (`:2063-2092`) — CUDA [cuda_mla_attn.cu](../src/vt/cuda/cuda_mla_attn.cu), CPU [cpu_ops.cpp](../src/vt/cpu/cpu_ops.cpp). Two ADDITIVE relaxations of existing ops, integer-identical for contiguous tensors: `vt::RopeFromCache` stride-driven on q/k (DeepSeek rotates the TRAILING 64-dim slice and its `k_pe` is a column block of the fused kv_a projection) and `vt::MatmulBT` accepting a row-strided ACTIVATION (`kv_b_proj` applied to a 512-column slice of the 576-wide workspace, `:2160`) | CUDA priority `vllm/platforms/cuda.py:84-176` (`_get_backend_priorities`, both branches); MLA classes `vllm/v1/attention/backends/mla/*.py`; MLA prefill selector `mla/prefill/selector.py:47-76`; capability filter `vllm/v1/attention/backend.py:307-360`; CUTLASS build `CMakeLists.txt:1037-1061` **W6** [test_mla_attention_block.cpp](../tests/vllm/model_executor/layers/attention/test_mla_attention_block.cpp) **10/10 cases / 2,372,644 assertions** and [test_ops_mla_absorb.cpp](../tests/vt/test_ops_mla_absorb.cpp) **9/9 / 1,644,807 assertions** on dgx sm_121 — ports of `tests/kernels/test_concat_mla_q.py` (BOTH arms incl. the NON-CONTIGUOUS transposed-nope case, compared bit-exactly since a concat is a pure copy), the MLA-geometry sweep of `tests/v1/attention/test_mla_backends.py`, and the two-pass-oracle discipline of `tests/kernels/attention/test_mla_decode_cpu.py`. **THE ABSORBED-vs-UNABSORBED EQUIVALENCE IS PROVEN NUMERICALLY, THREE WAYS:** an INDEPENDENT double-precision block oracle computing the attention BOTH ways agrees to **< 1e-11** (the identity itself, at both query branches); our absorbed decode reproduces the UNABSORBED oracle to **< 2e-4** (f32); and the SAME batch driven once through the ABSORBED MQA decode kernel and once through the UNABSORBED materialized-MHA prefill path agrees to **< 3e-4** (CPU f32) / **< 4e-2** (CUDA bf16) — two code paths sharing nothing but the weights. Real geometry throughout (V2-Lite 512/128/64/128/16-head, plus V3's 7168 / 128-head / `q_lora_rank=1536` for the lora branch, which has NO e2e coverage and says so). Decode-only / prefill-only-no-context / chunked-prefill-with-context / MIXED (decode packed FIRST) all gated; NaN-poisoned outputs; run-to-run BIT-exact; CUDA cases proven to EXECUTE (124,941 + 290,835 assertions when run alone). memcheck **0 errors**, racecheck **0 hazards**, synccheck **0 errors** (the last requires `--num-cuda-barriers 65536`: the default table OVERFLOWS on a binary driving this many kernel families and the tool then emits a bogus `unspecified launch failure`). Clean CUDA build 0 warn/0 err; regression set UNCHANGED (27B 235/235, 35B 315/315, Coder 138/138, Qwen3-dense 664/664, OPT 36/36). No speed number — W9 owns tuning | **SELECTION (W2) + the DECODE KERNEL (W4) + the PREFILL PATH and CHUNKED-CONTEXT LOOP (W5). What is still absent is the MLA attention BLOCK and MODEL (W6/W7).** Priority TABLE [cuda_attn_priority.h:49](../include/vllm/platforms/cuda_attn_priority.h#L49) (both branches, one row per upstream arch arm) + lookup [cuda_attn_priority.h:86](../include/vllm/platforms/cuda_attn_priority.h#L86); `is_mla()`/`is_sparse()` filter [registry.cpp:63](../src/vllm/v1/attention/registry.cpp#L63); `TritonMLABackend` NAME + 3-D `get_kv_cache_shape` [backend.h:509](../include/vllm/v1/attention/backend.h#L509), [backend.cpp:83](../src/vllm/v1/attention/backend.cpp#L83), registration [backend.cpp:108](../src/vllm/v1/attention/backend.cpp#L108) — **W4: `vt::MlaDecodeAttention`** — `OpId::kMlaDecodeAttention` + args/validation [ops.h](../include/vt/ops.h), [ops.cpp](../src/vt/ops.cpp); CPU single-pass REFERENCE [cpu_mla_attn.cpp](../src/vt/cpu/cpu_mla_attn.cpp) (numerics from `csrc/cpu/mla_decode.cpp`); CUDA two-stage split-KV [cuda_mla_attn.cu](../src/vt/cuda/cuda_mla_attn.cu) — `MlaDecodeStage1` <- `_fwd_grouped_kernel_stage1` (`triton_decode_attention.py:278-458`, IS_MLA `v = tl.trans(k)` branch `:424-431`), `MlaDecodeStage2` <- `_fwd_kernel_stage2` (`:575-639`), `ComputeNumKvSplits` <- `_compute_num_kv_splits` (`triton_mla.py:40-47`), split workspace via the house grow-only per-stream scratch (upstream's `_reserve_attn_logits_workspace` `:57-78`). Deterministic by construction: fixed ASCENDING split merge, NO atomicAdd. `TritonMLABackend::get_impl_cls()` now returns a real `TritonMLAImpl` [backend.h](../include/vllm/v1/attention/backend.h), [backend.cpp](../src/vllm/v1/attention/backend.cpp); PREFILL remains W5 and `TritonMLAImpl::forward` refuses a prefill-shaped batch by name. **W5: `vt::MlaPrefillAttention` + `vt::GatherMlaCache` + `vt::MergeAttnStates` + the chunked-context driver** — `vt::MlaPrefillAttention` [cuda_mla_prefill.cu](../src/vt/cuda/cuda_mla_prefill.cu) / CPU ref [cpu_mla_prefill.cpp](../src/vt/cpu/cpu_mla_prefill.cpp) <- `mla/prefill/flash_attn.py:153-248` `FlashAttnPrefillBackend` (the ONLY MLA prefill backend reachable on sm_121 per `mla/prefill/selector.py:66-76`, and it HARD-RAISES with no fallback at `:191-194`), running over the vendored FA-2 through the NEW launcher entry `LaunchMlaPrefillFA2Bf16` [cuda_flash_attn_fa2.cu](../src/vt/cuda/cuda_flash_attn_fa2.cu) plus two new explicit instantiations of the UNCHANGED generic template (`flash_fwd_split_hdim192_bf16{,_causal}_sm80.cu`). V is ZERO-PADDED 128->192 and the output sliced back, exactly as upstream's `requires_v_padding` path does (`flash_attn.py:88-99,164-168,196-197`) — which is WHY the asymmetric QK 192 / V 128 pair needs no asymmetric kernel. `vt::GatherMlaCache` <- `csrc/libtorch_stable/cache_kernels.cu:992-1064`; `vt::MergeAttnStates` <- `csrc/libtorch_stable/attention/merge_attn_states.cu:18-192` (BOTH `-inf` edge cases ported verbatim). The workspace-bounded loop is [mla_chunked_context.h](../include/vllm/model_executor/layers/attention/mla_chunked_context.h) <- `mla_attention.py:1422-1451,1667-1745,2094-2199,2344-2425`. **The paged launcher `LaunchPrefillFA2Bf16` that every non-MLA prefill calls is textually UNTOUCHED** (211 insertions / 0 deletions in that TU; 2 new vendored files) | [test_attn_backend_registry.cpp:146](../tests/vllm/v1/attention/test_attn_backend_registry.cpp#L146) (GB10 MLA list), [:203](../tests/vllm/v1/attention/test_attn_backend_registry.cpp#L203) (`use_mla=true` -> `TRITON_MLA`, matching the W0 oracle observation), [:230](../tests/vllm/v1/attention/test_attn_backend_registry.cpp#L230) (the DSA seam, proven both directions with a stand-in sparse backend) — ports of `test_attention_backends_selection.py` (MLA cases), `test_mla_prefill_selector.py`, `test_mla_prefill_registry.py`; **W4** [test_ops_mla_attn.cpp](../tests/vt/test_ops_mla_attn.cpp) — port of `tests/kernels/attention/test_mla_decode_cpu.py` (`ref_mla` as a TWO-PASS oracle, its bs=4/mean_seq_len=256/h_q=16/d=576/dv=512/block=16 parametrization, BOTH varlen arms, and its NaN-padding out-of-bounds detector) plus the `test_mla_backends.py` shape sweep: ragged, multi-block, single-block/single-token, EVERY num_kv_splits in {1,2,3,4,5,8,16,17,64,300,512} (incl. splits > seq_len, the empty-split path both stages must skip), 128-head DeepSeek-V3 geometry, head counts 1/3/17 that do not fill a BLOCK_H tile, a 288/256 block-32 non-V2-Lite geometry, bf16 + f32, and run-to-run BIT-exactness over 5 runs. Gated on dgx/sm_121: 11/11 cases, 2,303,193 assertions; `compute-sanitizer` memcheck **0 errors**, racecheck **0 hazards**, synccheck **0 errors**; clean CUDA build 0 warn/0 err; regression set UNCHANGED (27B 235/235, 35B 315/315, Coder 6/6, Qwen3-dense 16/16, OPT 6/6). NO speed number yet — decode perf is W9. **W5** [test_ops_mla_prefill.cpp](../tests/vt/test_ops_mla_prefill.cpp) **4/4 cases / 2,377,052 assertions** and [test_ops_mla_chunked_context.cpp](../tests/vt/test_ops_mla_chunked_context.cpp) **5/5 / 306,037 assertions** on dgx sm_121 — ports of `tests/v1/attention/test_mla_backends.py` and `tests/v1/attention/test_mla_prefill_quant_output.py` (its fp8 arms NOT ported: they need device-capability family 100, unreachable on sm_121 — recorded, not dropped). REAL V2-Lite prefill geometry (QK 192 / V 128 / latent 576, block 16, mscale^2 scale) against an INDEPENDENT double-precision TWO-PASS oracle, plus — for the chunked loop — a SINGLE-SHOT whole-sequence oracle that never chunks: exact / +1 / -1 chunk boundaries, a request with NO context, a chunk in which a request contributes ZERO keys, ragged multi-chunk, 128-head V3, single-token queries, ADVERSARIAL reverse-interleaved block tables, NaN-poisoned outputs, run-to-run BIT-exact over 5 runs. memcheck **0 errors**, racecheck **0 hazards**, synccheck **0 errors** on both binaries; clean CUDA build 0 warn/0 err; regression set UNCHANGED (27B 235/235, 35B 315/315, Coder 138/138, Qwen3-dense 664/664, OPT 36/36). Prefill perf is W9 | [MLA campaign spike](specs/mla-deepseek-campaign.md) | `PARTIAL` | `CLAIM-MLA-DEEPSEEK` | | `KERNEL-ATTN-DFLASH-BLOCK` | **DFlash in-block attention — the project's FIRST non-causal / bidirectional attention primitive** (SPEC-DFLASH D2, DF-DRAFT-MODEL). Per-request uniform (1+k) query block attends within its own block: FULL-attention layers BIDIRECTIONAL (`causal=false`, no mask), SWA layers causal-within-window. f32 online softmax, GQA broadcast. A SEPARATE `vt::` op from the causal `kAttention`/`kPagedAttention` so every other model stays byte-identical | `vllm/model_executor/models/qwen3_dflash.py:86-146` (`_resolve_layer_attention`: full layers default non-causal, SWA causal) + `:149-263` (`DFlashQwen3Attention`); flashinfer non-causal path (vllm#48167 Blackwell non-causal attn, in-pin) | `OpId::kDFlashBlockAttention` + `DFlashBlockAttentionArgs` + decl [ops.h:1713](../include/vt/ops.h#L1713) + wrapper/validation [ops.cpp:2069](../src/vt/ops.cpp#L2069); CPU REFERENCE `DFlashBlockAttentionKernel` [cpu_ops.cpp:1843](../src/vt/cpu/cpu_ops.cpp#L1843) (three-pass block-local softmax, the authoritative impl); CUDA `DFlashBlockAttentionKernelCuda` [cuda_ops.cu:1300](../src/vt/cuda/cuda_ops.cu#L1300) mirroring the causal `AttentionKernel` block-reduction recurrence with per-block bounds + the bidirectional/window mask; the draft model that consumes it [qwen3_dflash.cpp:52](../src/vllm/model_executor/models/qwen3_dflash.cpp#L52) | **CPU GATE GREEN** [test_ops_dflash_block_attn.cpp:79](../tests/vt/test_ops_dflash_block_attn.cpp#L79) 5 cases / 12 assertions — hand-checked non-causal (query 0 sees the future key), the RED causal-vs-non-causal separation (the mask is load-bearing), per-request cu_seqlens block isolation, SWA window bound, GQA; model forward [test_qwen3_dflash_forward.cpp:116](../tests/vllm/models/test_qwen3_dflash_forward.cpp#L116) 5 cases / 95 assertions (RED full-layer-causal-flip); existing causal `test_ops_attention` 9/9·23 UNCHANGED. **GPU GATE GREEN on dgx (2026-07-26, GB10 sm_121a):** CUDA `-Werror=all-warnings` build clean (kernel compiles as-written, no change); CUDA==CPU parity [test_ops_dflash_block_attn CUDA case](../tests/vt/test_ops_dflash_block_attn.cpp#L153) 198412/198412 within the 1e-4 f32-softmax envelope over all 5 corners; `compute-sanitizer --tool memcheck` 0 errors; consumed by the draft-forward parity gate ([test_qwen3_dflash_draft_parity](../tests/parity/test_qwen3_dflash_draft_parity.cpp), fc rel-L2 0.46% / hidden ≤1.3% vs the real vLLM draft). **DONE 2026-07-27 with the DFlash block (`CLAIM-DFLASH-D14`):** the D2 non-causal in-block primitive is the CPU/materialized reference the D12+ paged/warp kernels are gated against; closure [ledger](parity-ledger.md#L722). | [DFlash spec](specs/dflash-spec-decode.md) §1.3/§6 D2 | `DONE` | `489a7544` | | `KERNEL-ATTN-DFLASH-PAGED-BLOCK` | **DFlash PAGED in-block attention — the CAPTURE-SAFE form of `KERNEL-ATTN-DFLASH-BLOCK`** (SPEC-DFLASH D12 Part B, the CUDA-graph draft-attention primitive). The (1+k) block queries attend over `[PAGED context ; their own (1+k) block]`: the growing context enters as DATA (paged K/V cache `[pages,block_size,Hkv,D]` + per-request `seq_lens` + `block_table`, mirroring `PagedAttentionKernel`) instead of a variable-size materialized combined buffer, so the launch grid is STATIC over the fixed `Nq=(1+k)*num_reqs` rows and EVERY metadata input is a persistent DEVICE tensor read in place — NO `cudaMallocAsync`/`cudaMemcpyAsync` of a function-local host `cu_seqlens` (the [[cudagraph-capture-bakes-stack-addresses]] UAF class the eager `LaunchDFlashBlockAttention` had). Same f32 online softmax + D2 in-block mask over the COMBINED index; bit-identical to `DFlashBlockAttention` over the materialized `[context;block]` buffer | vLLM full CG `dflash/cudagraph.py` + `speculator.py:411-458` + `precompute_and_store_context_kv` (`qwen3_dflash.py:548-619`) @ `555967922`; paged read mirrors our `PagedAttentionKernel` [cuda_paged_attn.cu:184](../src/vt/cuda/cuda_paged_attn.cu#L184) | `OpId::kDFlashPagedBlockAttention` + `DFlashPagedBlockAttentionArgs` + decl [ops.h](../include/vt/ops.h) + wrapper/validation [ops.cpp](../src/vt/ops.cpp); CPU REFERENCE `DFlashPagedBlockAttentionKernel` [cpu_ops.cpp](../src/vt/cpu/cpu_ops.cpp); CUDA `DFlashPagedBlockAttentionKernelCuda` [cuda_ops.cu](../src/vt/cuda/cuda_ops.cu#L1452) (static grid, persistent device metadata) + D14 WARP variant [DFlashPagedBlockAttentionWarpKernel](../src/vt/cuda/cuda_ops.cu#L1433) | **GPU GATE GREEN on dgx (2026-07-27, GB10 sm_121a):** CUDA `-Werror` clean (0 warnings); [test_ops_dflash_paged_block_attn.cpp](../tests/vt/test_ops_dflash_paged_block_attn.cpp#L79) cross-checks CPU-paged == materialized `DFlashBlockAttention` across 6 corners (non-causal, causal-SWA, block isolation, GQA, multi-page, zero-context) + CUDA==CPU (f32+bf16) = **795648/795648 assertions**; `compute-sanitizer --tool memcheck` **0 errors**. **D13 (2026-07-27, `CLAIM-DFLASH-D13`): WIRED INTO PRODUCTION** — the single-request DFlash draft forward (`ForwardPagedBody`, `qwen3_dflash.cpp`) now runs the (1+k) block through this kernel reading a fixed-capacity paged `DflashDeviceKVStore`, and the whole draft step is captured into a per-request CUDA graph + replayed (the growing context enters only via the in-place `seq_lens`). Capture-correctness PROVEN: `test_qwen27_dflash_spec_decode` 27/27 with the graph BIT-IDENTICAL to eager (same tokens + acceptance 19/39/29/25); c1 throughput NEAR-PARITY with vLLM-DFlash-ON (ours 0.978×, ~2% below the tight 3-rep band; gap closed 0.917×→0.978× via the paged read, the CG is perf-neutral) — the kernel is landed + wired + gated; STAYS `ACTIVE` with the engine feature (the ~2% ≥vLLM residual is per-step compute for an nsys). **D14 (2026-07-27, `CLAIM-DFLASH-D14`): the residual WAS this kernel → WARP-scoped variant added → SPEED GATE MET, `DONE`.** An nsys (`--cuda-graph-trace=node`) attributed the D13 ~2% residual to THIS kernel: `DFlashPagedBlockAttentionKernel` = 242.9 ms = 1.8% of the graphed step's GPU time, median ~460 us/call (grid `(nq=17,hq=32)` × kBlock=256 threads looping SERIALLY over C~500-640 keys with a 256-wide shared-mem tree reduction + 2 `__syncthreads` PER key — the latency/sync storm the ViT tower fixed with `AttentionDenseFast`), vs vLLM's fused flash draft-attn ~0.15%. Added `DFlashPagedBlockAttentionWarpKernel` ([cuda_ops.cu](../src/vt/cuda/cuda_ops.cu)): ONE WARP per (block-query,head), `__shfl_xor` butterfly head_dim reduction, register accumulator, NO `__syncthreads`; SAME paged/block combined-index read + causal/SWA mask + GQA (copied verbatim from the block kernel), mirroring the shipped `AttentionWarpKernel`. Default ON; `VT_DFLASH_ATTN_BLOCK=1` keeps the bit-identical D12/D13 block kernel. Draft attn **242.9 → 77.9 ms (3.1×)**; our-ON c1 **28.60 → 29.32 tok/s**; FINAL 3-rep A/B our-ON 29.32 ≥ vLLM-ON 29.240 (non-overlapping bands, 1.003×) ⇒ **≥vLLM MET**. Not bit-identical to the block kernel but same f32-online-softmax math within envelope; CUDA==CPU `test_ops_dflash_paged_block_attn` **795648/795648** (f32 1e-4/bf16 3e-2) + **compute-sanitizer 0**; e2e 27/27 graph==eager, acceptance 19/39/29/25 unchanged (1629 accepted identical warp-vs-block); SACRED 235/235 + MTP 9/9 inert; `-Werror` clean; closure [ledger](parity-ledger.md#L738) | [DFlash spec](specs/dflash-spec-decode.md) §0 D12/D13/D14 | `DONE` | `489a7544` | | `KERNEL-ATTN-DSA-SPARSE-INDEX` | **DeepSeek-V4 DSA "Lightning Indexer" sparse-attention SELECTION — the project's FIRST sparse candidate-selection primitive** (DeepSeek-V4-Flash W3). Two ops: (1) the weighted-MQA INDEXER LOGIT `logit[t,s] = Σ_h w[t,h]·ReLU(q[t,h]·k[s])` over the causal candidate window (the per-head **ReLU** is load-bearing — it is what makes the indexer a learned sparse SELECTOR, not a plain attention score), where `w[t,h] = weights_proj[t,h]·index_head_dim^-0.5·index_n_heads^-0.5`; and (2) the per-row **causal top-k** that keeps the `index_topk=512` highest-logit keys (short-context: every candidate, ascending; else top-k with -1 padding). Distinct from every dense/paged/MLA family, which score ALL keys — this one PICKS a sparse key subset the downstream MLA then attends over. W3 also lands the two 512-wide-MLA OUTPUT seams V2/V3 lack (per-head attention-**sink** softmax + **grouped output-LoRA** `wo_a` bmm→`wo_b`) as portable host references beside it | MQA logit `vllm/v1/attention/ops/triton_fp8_mqa_logits.py:120-156` (dot→×kv_scale→ReLU→×weights→Σheads); weight fold `vllm/model_executor/layers/sparse_attn_indexer.py:203-207`; top-k `sparse_attn_indexer.py:488-497` + short-context `vllm/models/deepseek_v4/attention.py:70-86,:813-831`; sinks `deepseek_v4/nvidia/flashinfer_sparse.py:777,:896`; grouped output-LoRA `deepseek_v4/nvidia/ops/o_proj.py:58-73` @ `555967922` | Portable host reference (device kernel is a W7 residual) [deepseek_v4_dsa.cpp](../src/vllm/model_executor/models/deepseek_v4_dsa.cpp) + [deepseek_v4_dsa.h](../include/vllm/model_executor/models/deepseek_v4_dsa.h): `DsaIndexerWeightFold` / `DsaIndexerLogits` / `DsaTopkSelect` / `SoftmaxWithSink` / `GroupedOutputLora` | **CPU UNIT GATE GREEN (2026-07-28, `-Wall -Werror -Wextra` 0-warn):** [test_deepseek_v4_dsa.cpp](../tests/vllm/models/test_deepseek_v4_dsa.cpp) **13/13 cases · 38 assertions** — hand-derived literal cases (the ReLU clip, the weight fold, short-context all-select, full top-k, tie→smaller-index, causal-window offset, sink probability mass, sink numerical stability, grouped-LoRA) + from-first-principles double-precision references on randomized shapes (indexer logits + grouped output-LoRA rel-L2 < 1e-6). Full-model gate is multi-Spark-blocked (156.7 GiB); MHC (W5) + sqrtsoftplus/hash MoE (W6) + device kernel + forward integration (W7) are named residuals | [deepseek-v4-flash spike](specs/deepseek-v4-flash.md) §W3 | `SPIKE` | `CLAIM-DEEPSEEK-V4-W3` | diff --git a/.agents/specs/attn-validate-configuration.md b/.agents/specs/attn-validate-configuration.md new file mode 100644 index 000000000..31c74dcf9 --- /dev/null +++ b/.agents/specs/attn-validate-configuration.md @@ -0,0 +1,278 @@ +# SPEC — `validate_configuration` and the `supports_*` capability surface + +Issue: [#1332](https://github.com/mudler/vllm.cpp/issues/1332) +Owning rows: `BACKEND-ATTN-REGISTRY` (`.agents/backend-matrix.md`, `ANCHOR-BACKFILL`, +claim `CLAIM-ATTN-REGISTRY-1`) owns the selector seam; `BACKEND-CUDA-COMP-FA` +(`PARTIAL`) owns the FA2 arms. +Scope of this change: **M0 and M1 of #1332 only.** + +## Now + +`BACKEND-ATTN-REGISTRY` stays `ANCHOR-BACKFILL`. This change adds the declared +capability layer the row recorded as deferred and re-anchors four headers onto +the tree pin. It does not move the row's lifecycle state, because the row's +open item — that the resolved name does not drive dispatch — is untouched and is +owed to M4. + +## M0 — reconciliation, performed before any code + +Checked at `origin/main` `f06b9e93d`: + +- `git log --oneline --grep 'BACKEND-ATTN-REGISTRY'` returns one commit, + `b3a4ca963` (the seam itself). No later landing. +- `git log -S'validate_configuration' --oneline` returns six commits, every one + of them a spec, audit or spike that *names* the upstream method. `git log + -S'supports_compute_capability'` returns two, both records-only spikes. The + capability surface has never been implemented. +- `gh pr list --state open --limit 100`: 24 open pull requests, none touching + `src/vllm/v1/attention/**`, `include/vllm/v1/attention/**` or + `include/vllm/platforms/**`. +- `git branch -r | grep -iE 'attn|validate|registry'`: 25 remote branches match, + every one of them a kernel or model-factory branch (`perf/attn-prefill-*`, + `opt/decode-paged-attn`, `wmma-attn`, `row/MODEL-FACTORY-registry`, + `row/KERNEL-ATTN-DECODE-D128`, `row/PERF-CPU-ATTN-DTYPE`). None is a selector + branch. +- `.agents/NOW.md` names no attention-selector gate. + +**Outcome: the gap is real, unowned by any branch in flight, and both rows that +own it already exist.** No new roadmap row is opened. + +## Owed — recorded debt this change does NOT close + +1. **The selector still routes nothing.** `SelectAttentionBackendName`'s result + reaches exactly three places in + `src/vllm/v1/worker/gpu/runner.cpp::GPUModelRunner::initialize_kv_cache`: + `attn_backend_names_` (`:1002`), the `VT_ATTN_SELECT_LOG` debug print + (`:1003-1024`), and `vllm::v1::CheckKvCacheShape` (`:1025-1029`). + `dense_attn::AttnBlock` calls `vt::PagedAttention` unconditionally + (`include/vllm/model_executor/models/dense_attn_block.h:532-536`) and + `AttentionBackend::get_impl_cls()` returns `nullptr` for every dense backend. + The real arm choice is the env-flag + shape + dtype ladder in + `src/vt/cuda/cuda_paged_attn.cu:2696-2848`. Deleting the selector today would + leave every emitted token identical. **Owner: #1332 M4, row + `BACKEND-ATTN-REGISTRY`.** +2. **This layer is necessary and NOT sufficient.** M1 reproduces upstream's + *declared* capability layer, and that layer is precisely what passed on a + GB10 (capability 12,1) and then failed at launch: upstream's + `FlashAttentionBackend.supports_compute_capability` is `capability >= (8,0)` + (`vllm/v1/attention/backends/flash_attn.py:200-202` @ `5559679229`) while the + shipped fatbin carries `sm_80` SASS plus `compute_80` PTX and nothing else, so + every launch needs a driver JIT that fails with + `cudaErrorUnsupportedPtxVersion`. **A predicate over the DEVICE cannot answer a + question about the BINARY.** Landing M1 while claiming it fixes selection would + be the defect #1332 exists to correct. The compiled-arch manifest (M2) and the + launch probe (M3) are what make the layer sound. **Owner: #1332 M2/M3, row + `BACKEND-CUDA-COMP-FA`.** + + **The measurement that settles this, and it is not a thought experiment.** + Run through the pinned oracle on a GB10 board in the fleet (compute capability + 12,1), same wheel, same prompt, one variable: + + | Requested backend | Result | + |---|---| + | `FLASHINFER` | `GENERATED: ' Paris. The capital of France is also the capital of the Republic of France.'`, `GEN_RC=0` | + | default (resolves `FLASH_ATTN`) | dies at the FIRST attention call, `cudaErrorUnsupportedPtxVersion` | + + vLLM's own priority list puts `FLASH_ATTN` at position 0 and `FLASHINFER` at + position 1 for this device (`cuda.py:156-163`, the `else` arm our table row + mirrors). **The priority-0 choice is unrunnable on the board and the + priority-1 choice works, and every filter in this file passes both.** Both + declare a compute-capability floor at or below 12,1; neither is asked what its + own fatbin contains. That is the gap, stated as a pair of observations rather + than as an argument, and it is why this row's M1 must not be reported as + fixing selection. Motivation only — M2 and M3 close it and are NOT in this + change. +3. **`AttnSelectorConfig::dtype` is not supplied by the runner.** The production + call site fills `head_size`, `block_size` and `kv_cache_dtype` from the + resolved per-layer KV geometry, but the model/query dtype is not available at + `initialize_kv_cache` (the runner resolves only `ResolveKvCacheDType()` there), + so `dtype` keeps its `kBF16` default in production and the `supports_dtype` + predicate is exercised by tests alone. **Owner: #1332 M4.** +4. **The non-MLA sm_100 priority arm gained a `use_non_causal` guard upstream** + between the two pins; see "Anchor reconciliation" below. Fixed in flow, tracked + by its own issue, because the anchor being corrected points straight at it. + +## Anchor reconciliation + +Verified against a checkout of the tree pin. `git rev-parse HEAD` in that +checkout returns `5559679229bc961848b121ccdeaa8fa5d79bec98`, which equals the +`vllm_commit` in the `parity-pin` block of `.agents/upstream-sync.md`. Four +headers cited `@ pin e24d1b24`, the pin retired at W5 (`bc415a3e`). + +| File | Cited at `e24d1b24` | Verified at `5559679229` | +|---|---|---| +| `include/vllm/v1/attention/backend.h:1` | `vllm/v1/attention/backend.py` | same file; `validate_configuration` is `:320-393`, the `supports_*` predicates `:154-317` | +| `include/vllm/v1/attention/registry.h:3` | `cuda.py:361-470` (`get_valid_backends` / `get_attn_backend_cls`) | `get_valid_backends` is `:359-394`, `get_attn_backend_cls` is `:397-492` | +| `include/vllm/platforms/cuda_attn_priority.h:2` | `cuda.py:84-176` `_get_backend_priorities`, MLA `:93-142`, non-MLA `:143-166`; `mla/prefill/selector.py:47-76` | `_get_backend_priorities` is `:83-163`, MLA `:93-143`, non-MLA `:144-163`; `selector.py:48-77` | +| `include/vllm/platforms/interface.h:2` | `interface.py:134-229` (`class Platform`) | `class Platform` is `:134-1290`; the cited `:409-439` capability accessors and `:181-187` `supported_dtypes` moved | + +**One content drift, not just a line shift.** At `e24d1b24` the non-MLA sm_100 +arm was `if device_capability.major == 10`. At `5559679229` it is +`if device_capability.major == 10 and not use_non_causal` (`cuda.py:148`), with +`use_non_causal` added as a fifth `_get_backend_priorities` parameter +(`cuda.py:88`); the comment at `:145-147` gives the reason — SM100f's non-causal +CUTLASS path is known-bad for DFlash. Our table keyed rows on `(use_mla, major)` +alone, so a non-causal request on sm_100 got upstream's causal ordering. Today +this changes no selected name in this tree, because `FLASHINFER` is not +registered and both orderings fall through to `FLASH_ATTN`; it would change one +the moment FlashInfer lands. Fixed here rather than deferred: `use_non_causal` is +a field this change adds to `AttnSelectorConfig` regardless, the fix is one row +plus one predicate in the same header whose anchor is being corrected, and it is +CPU-gateable through the existing `FakeCudaPlatform`. + +The MLA rows, the sm_12x two-entry row, and the MLA-prefill lists are unchanged +in content at the new pin and were re-read line by line to confirm it. + +**Not reconciled here, recorded instead.** `FlashAttentionBackend::get_kv_cache_shape` +mirrors `flash_attn.py::get_kv_cache_shape` at `e24d1b24` +(`num_blocks, 2, block_size, num_kv_heads, head_size`); at `5559679229` upstream +returns `(num_blocks, num_kv_heads, block_size, 2 * head_size)`. That is the KV +memory format the whole engine allocates and every paged-attention kernel reads, +so re-anchoring it is a kernel campaign, not a comment edit. Left at its +`e24d1b24` anchor with the divergence named in the header. Owner: +`BACKEND-ATTN-REGISTRY`. + +## Design + +Mirror upstream's structure exactly: predicates are virtual member functions +returning `bool`, `validate_configuration` collects reason strings, and an empty +list means valid. + +**Where the config lives.** Upstream's `AttentionSelectorConfig` +(`vllm/v1/attention/selector.py:24-39`) is one named tuple that feeds both +`_get_backend_priorities` and `validate_configuration`. Our equivalent is +`vllm::platforms::AttnSelectorConfig` (`include/vllm/platforms/interface.h`), +already threaded through `Platform::get_attn_backend_priority`. It gains the +remaining upstream fields, every one with upstream's default, so no existing call +site changes meaning. + +**`attn_type` is a string.** Upstream's `AttentionType` is a `str` enum and +`supports_attn_type` takes a `str` (`backend.py:292-298`). Our `AttentionType` is +an `enum class` in `vllm::v1`, and `platforms/` must not depend on `v1/`. The +config therefore carries upstream's string, which is the faithful shape, and +`backend.h` gains `AttentionTypeName()` to convert. + +**Compute capability is checked only when the platform reports one.** Upstream +runs `validate_configuration` from `CudaPlatform.get_attn_backend_cls`, which +asserts `device_capability is not None` first (`cuda.py:403-404`); `CpuPlatform` +has its own selector (`cpu.py:75-87`) and never reaches this code. Our selector +is shared across every `DeviceType`, and `DeviceCapability::present()` is already +false (`major == -1`) for every non-CUDA platform. The predicate is therefore +applied only when `present()` is true, which reproduces upstream's precondition +rather than inventing a new rule. Recorded because it is the one structural +adaptation in this change. + +**The refusal names what failed.** `SelectAttentionBackendName` keeps its two +paths. An explicit override that fails validation throws with the full reason +list, mirroring `cuda.py:416-420`. The priority walk skips an invalid candidate +and, on exhaustion, throws naming each candidate with its reasons, mirroring +`cuda.py:432-446`. + +**Per-backend overrides ported.** `FlashAttentionBackend` gets upstream's +`supported_kv_cache_dtypes`, `get_supported_kernel_block_sizes` (`MultipleOf(16)`), +`supports_head_size` (`% 8 == 0 && <= 256`; upstream's FA4 `<= 512` arm is +unreachable here and says so), `supports_compute_capability` (`>= 8.0`), +`supports_sliding_window`, `supports_batch_invariance`, `supports_non_causal`, +`supports_attn_type` (all four types) and the `supports_combination` sink rule. +`TritonMLABackend` keeps `MultipleOf(16)` and its existing `is_mla()`. + +**One deliberate divergence from upstream's per-backend data.** +`FlashAttentionBackend::supports_per_head_quant_scales()` is upstream's +`fa_version >= 3`; this tree ships FA2 only (`CMakeLists.txt` +`VLLM_CPP_CUDA_ARCHITECTURES`, `BACKEND-CUDA-COMP-FA` records FA3 as unported), +so it returns `false` — which is upstream's own answer for FA2 and is what makes +the ported `test_per_head_quant_scales_backend_selection` case assert a refusal. + +## Tests + +`tests/vllm/v1/attention/test_attn_validate_configuration.cpp`, a port of +`tests/kernels/attention/test_attention_selector.py` @ `5559679229`. That file +monkeypatches the device capability and asserts a chosen name, so every case +selected below runs on CPU against the existing `FakeCudaPlatform`, which takes +its capability by constructor argument. + +| Upstream case | Ported as | +|---|---| +| `test_backend_selection` (cuda/cpu arms) | selection under a fully populated config still resolves `FLASH_ATTN` on sm_121 and sm_100 | +| `test_fp32_fallback` | an `f32` request finds no valid backend and the refusal names `dtype not supported`; upstream lands on `FLEX_ATTENTION`, which this tree does not register | +| `test_flash_attn` (upstream `pytest.skip`s it) | its five assertions ported as direct predicate cases: capability `(7,5)`, dtype `fp8`, `kv_cache_dtype="fp8"`, `block_size=8`, `head_size=17` | +| `test_per_head_quant_scales_backend_selection` | `FLASH_ATTN` + `use_per_head_quant_scales` is refused and the reason names it | +| `test_non_causal_backend_selection`, `test_non_causal_autoselect_backend` | `FLASH_ATTN` advertises `supports_non_causal()` and a non-causal request selects it | +| `test_flash_attn_rejects_unhandled_kv_cache_dtypes` | all six upstream parameters asserted false | +| `test_flash_attn_accepts_handled_fp8_variants` | both upstream parameters asserted true | +| `test_invalid_backend`, `test_auto_backend_string` | not applicable: our override is a `std::string` with `""` for automatic, so there is no enum lookup to fail | + +Plus, beyond upstream: the reason list is asserted to be a LIST — a request that +violates three predicates at once produces three reasons, so a refusal message +cannot collapse to the first failure; the sm_100 `use_non_causal` ordering; and +`AttnSelectorConfig{}` (every default) still validates, which is the +behavior-preserving control for the existing production call sites. + +## Gates + +`scripts/agent-preflight.sh --fail-on-skip`, plus the focused +`test_attn_validate_configuration` and `test_attn_backend_registry` cases. No GPU: +every predicate is host code and the capability is injected. + +## Evidence + +CPU Debug build, `-DVLLM_CPP_CUDA=OFF`. Every run below is a direct binary +invocation, never through a pipe, so the reported status is the binary's. + +**RED before.** `tests/vllm/v1/attention/test_attn_validate_configuration.cpp` +written first against the config fields alone, with no predicate implemented: + +``` +[doctest] test cases: 12 | 2 passed | 10 failed | 0 skipped +[doctest] assertions: 22 | 10 passed | 12 failed | +[doctest] Status: FAILURE! (exit 1) +``` + +The two that passed are the behavior control (`FLASH_ATTN` still selected) and +the case asserting the sm_100 CAUSAL ordering, which was already correct — the +non-causal half of that same case is one of the ten reds. + +**GREEN after**, with the predicate-level cases added: + +| Binary | cases | assertions | status | +|---|---:|---:|---| +| `test_attn_validate_configuration` | 21 | 76 | SUCCESS | +| `test_attn_backend_registry` | 17 | 62 | SUCCESS | +| `test_runner` | 19 | 543 | SUCCESS | + +**Mutations.** Each applied to the worktree, built, run, then restored from a +`tar` snapshot taken before the pass — not `git checkout`, which reads the index +and would have destroyed the untracked new test file. `git diff --stat` was +printed before and after each one and the line counts returned to their +pre-mutation values, and `compile_rc` was printed for each, because a mutation +that fails to build and a mutation that never applied both read as a pass. + +| # | Mutation | `compile_rc` | Result | +|---|---|---:|---| +| M1 | `CandidateInvalidReasons` returns `{}` — the selector stops asking | 0 | RED: `validate_configuration` 12/21, `registry` 15/17 | +| M2 | drop the `capability.present()` precondition | 0 | RED: both files, 1 case each | +| M3 | `supports_block_size` compares equality instead of a multiple | 0 | RED: 4 cases | +| M4 | remove the #1333 causal-only guard | 0 | RED: 1 case | +| M5 | **reachability**: the production geometry stops flowing into the config (`cfg.head_size = 17`) | 0 | RED: `test_runner` **19/19 cases fail** | +| M6 | **reachability**: delete the production config fill entirely | 0 | GREEN: `test_runner` 543/543 | + +**M5 is the reachability proof.** Corrupting one field of the config that +`GPUModelRunner::initialize_kv_cache` builds fails every one of the 19 +`test_runner` cases, because the refusal propagates out of the runner's own +initialization. The capability layer is therefore reached from a production +entry point with production data, not only from its own unit test. + +**M6 is the honest negative, and it is recorded rather than explained away.** +Deleting the fill leaves `test_runner` green, because the defaults still +validate. The layer is reached and executed; no test's failure depends on the +fill being PRESENT, only on its being CORRECT. That is the expected shape while +the selected name still dispatches nothing (`## Owed` item 1), and it is the +second reason this change must not be read as fixing selection. + +## Stop conditions + +- Stop and report `NEEDS_DECISION` if a ported predicate would refuse a + configuration the gate models use today, because that is a behavior change, not + a capability port. +- Stop if `validate_configuration` cannot be reached from the production call + site without also wiring dispatch; wiring dispatch is M4 and out of scope. diff --git a/docs/USAGE.md b/docs/USAGE.md index 3e19debd0..c52189761 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -734,6 +734,47 @@ block-wise FP8 scale tensor the checkpoint had never contained ([#1256](https://github.com/mudler/vllm.cpp/issues/1256)). A message that blames the wrong side costs more than the failure does. +### A refusal that names the attention backend, and what it cannot tell you + +Starting an engine resolves an attention backend for each KV-cache group, and +that backend is now asked whether it can serve the request before it is chosen. +When none of the backends this build registers can, the engine refuses at +initialization rather than later, and the message names every candidate with +every reason it lost: + +```text +No valid attention backend for device type 1 from +{FLASH_ATTN: [head_size not supported, block_size not supported]} +(use_mla=false, use_sparse=false) +``` + +The reason strings are vLLM's own, so a refusal here and a refusal from the +reference engine read the same. `head_size`, `block_size` and the KV-cache dtype +come from the geometry the engine has just resolved for your checkpoint, so a +refusal is about that checkpoint on this build. + +**What this check cannot tell you.** It reports what a backend *claims*, never +what your binary contains and never whether the kernel will launch. A backend +whose declared floor is compute capability 8.0 is accepted on any newer GPU, even +when the build carries no compiled code for that GPU. + +That is a real failure mode, not a hypothetical one, and it surfaces as a launch +error rather than as the refusal above. It has been measured on a GB10 board +(compute capability 12,1) against the reference engine, same wheel and same +prompt: asking for its `FLASHINFER` backend generates text and exits cleanly, +while the default — which resolves `FLASH_ATTN`, the reference engine's *first* +preference for that device — dies at the first attention call with +`cudaErrorUnsupportedPtxVersion`. The first preference could not run and the +second could, and no capability check on either side could tell them apart. + +So if a run dies inside attention rather than being refused before it starts, +the backend was accepted on a claim your build does not honour. Confirming which +architectures a build actually targets is a separate question, answered under +"Confirming which CUDA architecture a build targets" above. Tracked as +[#1332](https://github.com/mudler/vllm.cpp/issues/1332). + +Selecting a backend by name is not exposed yet; the engine always resolves one. + ### Architectures that resolve but refuse to run A few architectures are registered so their config and weight layout are diff --git a/include/vllm/platforms/cuda_attn_priority.h b/include/vllm/platforms/cuda_attn_priority.h index 28934cdb8..6faf734ca 100644 --- a/include/vllm/platforms/cuda_attn_priority.h +++ b/include/vllm/platforms/cuda_attn_priority.h @@ -1,9 +1,15 @@ // The CUDA attention-backend priority TABLE — a faithful, complete port of -// vllm/platforms/cuda.py:84-176 `_get_backend_priorities` @ pin e24d1b24 (BOTH -// the MLA branch `:93-142` and the non-MLA branch `:143-166`), plus -// vllm/v1/attention/backends/mla/prefill/selector.py:47-76 +// vllm/platforms/cuda.py:83-163 `_get_backend_priorities` @ pin 5559679229 (BOTH +// the MLA branch `:93-143` and the non-MLA branch `:144-163`), plus +// vllm/v1/attention/backends/mla/prefill/selector.py:48-77 // `_get_mla_prefill_backend_priorities`. // +// Re-anchored from `cuda.py:84-176 @ e24d1b24`, the pin retired at W5. The MLA +// rows, the sm_12x two-entry row and the MLA-prefill lists are unchanged in +// CONTENT at the new pin and were re-read line by line to confirm it. The +// non-MLA sm_100 row was NOT: it gained a `use_non_causal` guard, which is +// issue #1333 and is ported below. +// // It lives in a HEADER, not inside the CUDA-only cuda.cpp TU, for two reasons: // * it is pure DATA + a lookup — no CUDA, no device — so the CPU test tier can // assert the REAL table instead of a hand-copied duplicate (the pre-W2 @@ -41,6 +47,11 @@ struct AttnPriorityRow { int major; // exact compute-capability major, or kAnyMajor for upstream `else` std::vector backends; SparseTailOrder sparse_tail; + // cuda.py:148 `and not use_non_causal` — issue #1333. When true, this row + // matches only a CAUSAL request and a non-causal one falls through to the + // `else` row. Exactly one upstream arm carries the guard, so it is a per-row + // field rather than a second key. + bool causal_only = false; }; // Upstream names are the AttentionBackendEnum MEMBER names @@ -70,12 +81,17 @@ inline const std::vector& AttnPriorityTable() { {"FLASH_ATTN_MLA", "FLASHMLA", "FLASHINFER_MLA", "TRITON_MLA", "FLASH_ATTN_MLA_SPARSE", "FLASHMLA_SPARSE"}, SparseTailOrder::kFixed}, - // ── non-MLA (unchanged behavior; previously an inline if-chain) ──────── - // cuda.py:144-152 — sm_100. + // ── non-MLA ─────────────────────────────────────────────────────────── + // cuda.py:145-155 — sm_100, CAUSAL only. The guard is upstream's, and its + // reason is upstream's comment at `:145-147`: SM100f defaults to FlashInfer + // for TRTLLM causal attention, but its non-causal cutlass path — the one + // DFlash attention uses — is known to have problems, so FlashAttention is + // preferred when non-causal on SM100f. Issue #1333. {false, 10, {"FLASHINFER", "FLASH_ATTN", "TRITON_ATTN", "FLEX_ATTENTION", "TURBOQUANT"}, - SparseTailOrder::kFixed}, - // cuda.py:153-166 — everything else, INCLUDING GB10 sm_121 (major 12). + SparseTailOrder::kFixed, /*causal_only=*/true}, + // cuda.py:156-163 — everything else, INCLUDING GB10 sm_121 (major 12), and + // INCLUDING sm_100 for a non-causal request. {false, kAnyMajor, {"FLASH_ATTN", "FLASHINFER", "TRITON_ATTN", "FLEX_ATTENTION", "TURBOQUANT"}, SparseTailOrder::kFixed}, @@ -89,6 +105,9 @@ inline std::vector LookupAttnPriority(int major, const AttnPriorityRow* fallback = nullptr; for (const AttnPriorityRow& row : AttnPriorityTable()) { if (row.use_mla != cfg.use_mla) continue; + // cuda.py:148 — a causal-only row does not match a non-causal request, so + // the walk continues to the `else` row (#1333). + if (row.causal_only && cfg.use_non_causal) continue; if (row.major == major) { match = &row; break; diff --git a/include/vllm/platforms/interface.h b/include/vllm/platforms/interface.h index ebfb15439..fc67ac0d3 100644 --- a/include/vllm/platforms/interface.h +++ b/include/vllm/platforms/interface.h @@ -1,5 +1,8 @@ -// Faithful 1:1 port of vllm/platforms/interface.py:134-229 (class Platform) @ -// pin e24d1b24 — the device-capability / memory-model seam. See +// Faithful 1:1 port of vllm/platforms/interface.py:134-1290 (class Platform) @ +// pin 5559679229 — the device-capability / memory-model seam. (Re-anchored from +// `:134-229 @ e24d1b24`, the pin retired at W5: the class is unchanged in the +// parts we mirror, and the cited `supported_dtypes` / `get_device_capability` +// members moved rather than changing.) See // .agents/porting-inventory.md §9 note 8 (the platforms/ tree is a faithful // mirror of the upstream seam, NOT a deviation) and // .agents/specs/extensibility-platform-seam-2026-07-18.md. @@ -117,15 +120,24 @@ inline ResidencyPolicy CudaResidencyPolicy(size_t device_memory_total_bytes) { return p; } -// The selection inputs of vllm/platforms/cuda.py::_get_backend_priorities @ pin -// e24d1b24 (`use_mla`, `device_capability`, `num_heads`, `kv_cache_dtype`) plus -// the sparse flag that `AttentionBackend.is_sparse()` / -// `vllm/v1/attention/backend.py:307-360 validate_configuration` keys on. The -// capability itself is NOT a field — it is the platform's own -// `get_device_capability()`, exactly as upstream passes it in. +// A 1:1 port of `vllm/v1/attention/selector.py::AttentionSelectorConfig` +// (`:24-39`) @ pin `5559679229` — the ONE request description upstream feeds to +// BOTH `vllm/platforms/cuda.py::_get_backend_priorities` (`:83-163`) and +// `vllm/v1/attention/backend.py::AttentionBackend.validate_configuration` +// (`:320-393`). The capability itself is NOT a field — it is the platform's own +// `get_device_capability()`, exactly as upstream passes it in beside the config +// (`cuda.py:381-384`). // -// Defaults reproduce today's non-MLA dense selection EXACTLY, so every existing -// caller (`get_attn_backend_priority()` with no argument) is unchanged. +// It lives under `platforms/` rather than under `v1/` because +// `Platform::get_attn_backend_priority` takes it and `platforms/` must not depend +// on `v1/`. That is also why `attn_type` below is a STRING: upstream's +// `AttentionType` is a `str` enum (`backend.py:38-46`) and `supports_attn_type` +// takes a `str` (`:292-298`), so the string IS the faithful shape; +// `vllm::v1::AttentionTypeName` converts our enum to it. +// +// Every default is upstream's, so every existing caller +// (`get_attn_backend_priority()` with no argument, and the runner's dense walk) +// keeps its exact selection behavior. struct AttnSelectorConfig { // model_config.use_mla — the MLA branch of _get_backend_priorities:93-142. // On our gate models (Qwen3 dense / GDN) this is false. @@ -139,9 +151,44 @@ struct AttnSelectorConfig { // cuda.py:105 `num_heads is not None and num_heads <= 16` (sm_100 sparse-MLA // ordering only). 0 == unknown, upstream's `None`. int num_heads = 0; - // cuda.py:96 `is_quantized_kv_cache(kv_cache_dtype)` (sm_100 sparse-MLA - // ordering only). Our KV cache is bf16/f32 today, so false. + // cuda.py:98 `is_quantized_kv_cache(kv_cache_dtype)` (sm_100 sparse-MLA + // ordering only). Our KV cache is bf16/f32 today, so false. Upstream DERIVES + // this from `kv_cache_dtype` below; it stays a separate field because the MLA + // priority rows and their tests already read it, and the two are asserted + // consistent nowhere — recorded as such rather than silently unified. bool quantized_kv_cache = false; + + // ─── selector.py:25-28 — the four inputs upstream makes REQUIRED ─────────── + // The attention head dimension. 0 is accepted by every predicate whose + // supported list is empty, which is upstream's answer for an unconstrained + // backend (`backend.py:159-161`). + int head_size = 0; + // The model/query dtype (`supports_dtype`, backend.py:163-165). Upstream has + // no `None` here. The runner does not yet supply it (issue #1332 M4), so on + // the production path it keeps this default. + DType dtype = DType::kBF16; + // The KV-cache quantization name ("auto", "fp8", "fp8_e4m3", ...). EMPTY is + // upstream's `None`, which `supports_kv_cache_dtype` accepts outright + // (`backend.py:167-173`). + std::string kv_cache_dtype; + // The framework block size. 0 is upstream's `None`, accepted outright + // (`backend.py:176-178`). + int block_size = 0; + + // ─── selector.py:30-39 — the request flags, upstream defaults ───────────── + bool has_sink = false; + bool use_mm_prefix = false; + bool use_per_head_quant_scales = false; + // `AttentionType` is a `str` enum upstream; "decoder" is `AttentionType.DECODER`. + std::string attn_type = "decoder"; + bool has_sliding_window = false; + // Non-causal (bidirectional) decoder attention — DFlash's drafter sets it. + // Also a `_get_backend_priorities` input at this pin (`cuda.py:88,148`), which + // is issue #1333. + bool use_non_causal = false; + bool use_batch_invariant = false; + bool use_kv_connector = false; + bool use_pcp = false; }; // Faithful port of vllm/platforms/interface.py:134-229 `class Platform`. Exposes diff --git a/include/vllm/v1/attention/backend.h b/include/vllm/v1/attention/backend.h index d0dfd329a..1fb3b578b 100644 --- a/include/vllm/v1/attention/backend.h +++ b/include/vllm/v1/attention/backend.h @@ -1,4 +1,13 @@ -// Ported from: vllm/v1/attention/backend.py @ e24d1b24 +// Ported from: vllm/v1/attention/backend.py @ pin 5559679229 (re-anchored from +// `e24d1b24`, the pin retired at W5; `validate_configuration` is `:320-393` and +// the `supports_*` predicates `:154-317` there). +// +// ONE anchor is deliberately NOT advanced: FlashAttentionBackend::get_kv_cache_shape +// below still mirrors flash_attn.py @ `e24d1b24`. At `5559679229` upstream returns +// (num_blocks, num_kv_heads, block_size, 2 * head_size) instead. That is the KV +// memory format the whole engine allocates and every paged-attention kernel +// reads, so re-anchoring it is a kernel campaign rather than a comment edit. +// Named here rather than left to be discovered; owner: BACKEND-ATTN-REGISTRY. // // Scope (M1.6 Task 1): the per-step attention metadata + the attention backend // interface the paged-attention path (Task 2/3) and the GDN metadata (Task 4) @@ -23,13 +32,11 @@ // * The deprecated lazy _seq_lens_cpu accessor and the unpadded()/replace() // spec-decode helpers. The host-equivalent num_computed_tokens_cpu array is // ported because chunked-local virtual batching consumes and rewrites it. -// * AttentionBackend: the whole supports_*/validate_configuration capability -// surface, get_kv_cache_stride_order / get_kv_cache_block_dim, cudagraph -// capture/drafting hooks, MLA / sparse-MLA impls. AttentionCGSupport itself -// is ported for the chunked-local wrapper. Ported here: the T0 core -// (CommonAttentionMetadata, the AttentionBackend/AttentionImpl/ -// AttentionMetadataBuilder ABCs, get_name + get_kv_cache_shape, the forward -// signature). +// * AttentionBackend: get_kv_cache_stride_order / get_kv_cache_block_dim, +// cudagraph capture/drafting hooks, MLA / sparse-MLA impls. +// AttentionCGSupport itself is ported for the chunked-local wrapper. +// The supports_*/validate_configuration capability surface is NO LONGER +// deferred — it is ported below (issue #1332 M1). #ifndef VLLM_V1_ATTENTION_BACKEND_H_ #define VLLM_V1_ATTENTION_BACKEND_H_ @@ -39,6 +46,7 @@ #include #include +#include "vllm/platforms/interface.h" #include "vllm/v1/worker/gpu/prepare_inputs.h" #include "vt/ops.h" @@ -53,6 +61,26 @@ enum class AttentionType { kEncoderDecoder // "encoder_decoder" }; +// Upstream's AttentionType IS its string value (backend.py:38-46 is a class of +// `str` constants), and `supports_attn_type` takes that string (`:292-298`). +// `platforms::AttnSelectorConfig::attn_type` therefore carries the string, and +// this is the conversion. Returns the exact upstream spelling. +const char* AttentionTypeName(AttentionType type); + +// The `CacheDType` name (vllm/config/cache.py:19-36) for a KV-cache storage +// dtype, which is what `supports_kv_cache_dtype` matches against. +// +// Upstream's "auto" means "the model dtype, unquantized" (`cache.py:77-78`), and +// an f32/f16/bf16 KV cache is exactly that — so all three map to "auto" rather +// than to their own `CacheDType` spellings, which upstream uses only when the +// user pins the cache to a dtype the model does not have. `vt::DType` carries no +// fp8 member yet, so the quantized names are unreachable from here; when it +// does, this is the one place the mapping goes. +const char* KvCacheDTypeName(vt::DType dtype); + +// vllm/utils/torch_utils.py:75-80 `is_quantized_kv_cache`, over the same name. +bool IsQuantizedKvCacheName(const std::string& kv_cache_dtype); + // Cudagraph support level advertised by an attention metadata builder. // Values preserve upstream ordering; chunked-local explicitly returns kNever. enum class AttentionCGSupport { @@ -233,6 +261,104 @@ class AttentionBackend { // W2 keeps its exact selection behavior. virtual bool is_mla() const { return false; } virtual bool is_sparse() const { return false; } + + // ─── The declared-capability surface (backend.py:154-393) ───────────────── + // + // READ THIS BEFORE TRUSTING A GREEN RESULT FROM IT (issue #1332). Every + // predicate below describes what a backend CLAIMS. None of them can see what + // the shipped binary contains. Upstream's + // FlashAttentionBackend.supports_compute_capability is `>= (8,0)`; it returned + // true on a GB10 (capability 12,1) whose FA2 fatbin holds sm_80 SASS plus + // compute_80 PTX and nothing else, and every launch then failed a driver JIT + // with cudaErrorUnsupportedPtxVersion. `grep -rn get_arch_list vllm/` returns + // zero hits: vLLM never asks what its own fatbins contain, and neither does + // this. Measured on that board through the pinned oracle, same wheel and same + // prompt: asking for FLASHINFER generates text and exits 0, while the default + // — which resolves FLASH_ATTN, upstream's priority 0 for this device — dies at + // the FIRST attention call. The priority-0 choice is unrunnable and the + // priority-1 choice works, and every predicate below passes BOTH. + // + // The invariant #1332 states is that no backend may be declared valid on + // the strength of a property of the DEVICE alone, and satisfying it needs the + // build-derived compiled-arch manifest (M2) and the launch probe (M3). This + // layer is NECESSARY AND NOT SUFFICIENT, and a reader who takes it for a + // runnability check will reproduce exactly the failure that opened the issue. + // + // Structure mirrors upstream: static declarations returning bool, collected by + // validate_configuration into a list of reason strings; an EMPTY list means + // valid. Ours are virtual member functions because our backends are instances + // rather than classes (MakeAttentionBackend), which is the same adaptation + // get_impl_cls above already records. + + // backend.py:155-156 / :59-64 — the declared value lists. A backend overrides + // the list, not the predicate, wherever upstream does. + virtual std::vector supported_dtypes() const { + return {vt::DType::kF16, vt::DType::kBF16}; // backend.py:59 + } + virtual std::vector supported_kv_cache_dtypes() const { + return {"auto", "float16", "bfloat16"}; // backend.py:60-64 + } + virtual std::vector get_supported_head_sizes() const { return {}; } + // backend.py:69-71 — upstream returns [MultipleOf(1)]. Every entry here IS a + // MultipleOf: supports_block_size accepts any multiple of any entry, which is + // upstream's hybrid_blocks rule at :184-192, and MultipleOf(1) accepts all. + virtual std::vector get_supported_kernel_block_sizes() const { return {1}; } + + // backend.py:158-161. + virtual bool supports_head_size(int head_size) const; + // backend.py:163-165. + virtual bool supports_dtype(vt::DType dtype) const; + // backend.py:167-173 — an EMPTY name is upstream's `None`, accepted outright. + virtual bool supports_kv_cache_dtype(const std::string& kv_cache_dtype) const; + // backend.py:175-192 — 0 is upstream's `None`, accepted outright. + virtual bool supports_block_size(int block_size) const; + + // backend.py:237-298 — the plain feature flags, upstream defaults. + virtual bool supports_sink() const { return false; } // :241-243 + virtual bool supports_mm_prefix() const { return false; } // :249-251 + virtual bool supports_per_head_quant_scales() const { return false; } // :257-259 + virtual bool supports_sliding_window() const { return false; } // :261-263 + virtual bool supports_non_causal() const { return false; } // :265-274 + virtual bool supports_batch_invariance() const { return false; } // :276-278 + virtual bool supports_kv_connector() const { return true; } // :280-282 + virtual bool supports_pcp() const { return false; } // :284-289 + // :291-298 — the base supports DECODER only. + virtual bool supports_attn_type(const std::string& attn_type) const { + return attn_type == AttentionTypeName(AttentionType::kDecoder); + } + // :300-302. + virtual bool supports_compute_capability( + const platforms::DeviceCapability& capability) const { + (void)capability; + return true; + } + + // :304-317 — the cross-field rule a per-field predicate cannot express. + // Returns the reason when the COMBINATION is invalid, nullopt when it is fine. + // + // ADAPTATION: upstream passes nine positional arguments, every one of them a + // field of the AttentionSelectorConfig the caller already holds + // (cuda.py:381-384 splats that config). We pass the config itself, so a new + // upstream field does not change nine signatures. + virtual std::optional supports_combination( + const platforms::AttnSelectorConfig& cfg, + const platforms::DeviceCapability& capability) const { + (void)cfg; + (void)capability; + return std::nullopt; + } + + // backend.py:319-393. Collects one reason per failed predicate and returns + // them ALL: upstream builds a list and prints the whole list (cuda.py:416-420, + // :432-446), because a message that stopped at the first failure would send a + // reader chasing one cause of four. An EMPTY result means the backend is valid + // for this request. + // + // The reason STRINGS are upstream's, byte for byte, so a refusal here and a + // refusal from the oracle read the same. + virtual std::vector validate_configuration( + const platforms::AttnSelectorConfig& cfg, + const platforms::DeviceCapability& capability) const; }; // The T0 concrete full-attention backend. Ports the FlashAttention V1 paged KV @@ -253,6 +379,54 @@ class FlashAttentionBackend final : public AttentionBackend { int64_t num_blocks, int64_t block_size, int64_t num_kv_heads, int64_t head_size, const std::string& cache_dtype_str = "auto") const override; + + // ─── Capability overrides, ported from flash_attn.py:72-239 ────────────── + // flash_attn.py:74-80. + std::vector supported_kv_cache_dtypes() const override { + return {"auto", "float16", "bfloat16", "fp8", "fp8_e4m3"}; + } + // flash_attn.py:82-84 — MultipleOf(16). + std::vector get_supported_kernel_block_sizes() const override { return {16}; } + // flash_attn.py:170-178. Upstream raises the ceiling to 512 when FlashAttention + // v4 resolves (`is_fa_version_supported(4)`); this tree ships FA2 only + // (BACKEND-CUDA-COMP-FA records FA3/FA4 as unported), so the 512 arm is + // unreachable and is deliberately not written as an always-false branch. + bool supports_head_size(int head_size) const override { + if (head_size % 8 != 0) return false; + return head_size <= 256; + } + // flash_attn.py:200-202. SEE THE WARNING ON THE BASE CLASS: this is the exact + // predicate that returned true on a GB10 whose FA2 fatbin had no sm_121 code + // (issue #1332). It is upstream's, and it is not a runnability check. + bool supports_compute_capability( + const platforms::DeviceCapability& capability) const override { + return capability.major > 8 || (capability.major == 8 && capability.minor >= 0); + } + bool supports_sliding_window() const override { return true; } // :98-100 + bool supports_batch_invariance() const override { return true; } // :102-104 + bool supports_non_causal() const override { return true; } // :106-108 + // :110-118 — FlashAttention serves every attention type. + bool supports_attn_type(const std::string& attn_type) const override { + return attn_type == AttentionTypeName(AttentionType::kDecoder) || + attn_type == AttentionTypeName(AttentionType::kEncoder) || + attn_type == AttentionTypeName(AttentionType::kEncoderOnly) || + attn_type == AttentionTypeName(AttentionType::kEncoderDecoder); + } + // :120-123 `get_flash_attn_version() >= 3`. This tree ships FA2, so the answer + // is upstream's own answer for FA2 rather than a divergence — and it is what + // makes the ported test_per_head_quant_scales case assert a refusal. + bool supports_per_head_quant_scales() const override { return false; } + // :204-239. Only the sink rule is expressible here: the fp8-KV and mm_prefix + // rules both key on a resolved FA version >= 3, which this tree never reaches, + // and supported_kv_cache_dtypes / supports_mm_prefix already refuse them. + std::optional supports_combination( + const platforms::AttnSelectorConfig& cfg, + const platforms::DeviceCapability& capability) const override { + if (cfg.has_sink && capability.present() && capability.major < 9) { + return std::string("sink not supported on compute capability < 9.0"); + } + return std::nullopt; + } }; // The dense ROCm paged-attention backend (issue #41 M3). Upstream ROCM_ATTN @@ -348,10 +522,14 @@ class TritonMLABackend final : public AttentionBackend { // makes FLASHINFER_MLA_SPARSE_SM120 lose and TRITON_MLA win on GB10). bool is_mla() const override { return true; } - // triton_mla.py:100-103 supports_block_size — block_size % 16 == 0. Exposed as - // a plain predicate (the full supports_* surface is still deferred, see the - // header note) because get_kv_cache_shape enforces it. - static bool supports_block_size(int64_t block_size) { return block_size % 16 == 0; } + // triton_mla.py:100-103 supports_block_size — block_size % 16 == 0. Declared as + // the SUPPORTED-SIZE LIST rather than as an overridden predicate, which is + // upstream's shape and which the base supports_block_size (backend.py:175-192) + // turns into the same `% 16 == 0` answer. Was a `static` member before #1332; + // a static of that name would have HIDDEN the base virtual, so a caller + // through an AttentionBackend& and a caller through a TritonMLABackend& would + // have got different answers. + std::vector get_supported_kernel_block_sizes() const override { return {16}; } // W4: the MLA decode impl. Upstream `get_impl_cls` returns the CLASS // (triton_mla.py:126-128 -> TritonMLAImpl); here a factory returning an diff --git a/include/vllm/v1/attention/registry.h b/include/vllm/v1/attention/registry.h index 1a1aaebb4..49cb94b3b 100644 --- a/include/vllm/v1/attention/registry.h +++ b/include/vllm/v1/attention/registry.h @@ -1,7 +1,9 @@ // Ported from: vllm/v1/attention/backends/registry.py (AttentionBackendEnum + // @register_backend self-registration) and the platform-driven selection in -// vllm/platforms/cuda.py:361-470 (get_valid_backends / get_attn_backend_cls) @ -// pin e24d1b24 — the attention-backend REGISTRY + platform-priority SELECTION +// vllm/platforms/cuda.py:359-394 (get_valid_backends) and :397-492 +// (get_attn_backend_cls) @ pin 5559679229 — re-anchored from `:361-470 @ +// e24d1b24`, the pin retired at W5 — the attention-backend REGISTRY + +// platform-priority SELECTION // seam (extensibility item 4). This is the ENGINE-level "which AttentionBackend" // seam; the concrete attention KERNEL stays selected at the vt:: op-table level // (vt::PagedAttention -> GetOp(kPagedAttention, device.type)), which is already diff --git a/src/vllm/v1/attention/backend.cpp b/src/vllm/v1/attention/backend.cpp index d1236a99c..7b4eb0b70 100644 --- a/src/vllm/v1/attention/backend.cpp +++ b/src/vllm/v1/attention/backend.cpp @@ -1,5 +1,6 @@ -// Ported from: vllm/v1/attention/backend.py @ e24d1b24 -// (get_kv_cache_shape from vllm/v1/attention/backends/flash_attn.py @ e24d1b24) +// Ported from: vllm/v1/attention/backend.py @ pin 5559679229 +// (get_kv_cache_shape from vllm/v1/attention/backends/flash_attn.py @ e24d1b24 — +// deliberately NOT re-anchored; see the divergence note at the top of backend.h) #include "vllm/v1/attention/backend.h" #include @@ -70,6 +71,159 @@ CommonAttentionMetadata MakeCommonAttentionMetadata( return cam; } +const char* AttentionTypeName(AttentionType type) { + // backend.py:38-46 — the exact upstream string values. + switch (type) { + case AttentionType::kDecoder: + return "decoder"; + case AttentionType::kEncoder: + return "encoder"; + case AttentionType::kEncoderOnly: + return "encoder_only"; + case AttentionType::kEncoderDecoder: + return "encoder_decoder"; + } + return "decoder"; +} + +// vllm/config/cache.py:19-36 CacheDType. See the header for why f32/f16/bf16 all +// map to "auto". +const char* KvCacheDTypeName(vt::DType dtype) { + switch (dtype) { + case vt::DType::kF32: + case vt::DType::kF16: + case vt::DType::kBF16: + return "auto"; + default: + // No other vt::DType is a KV-cache storage dtype in this tree. "auto" is + // the unquantized answer, which is what a caller with an unexpected dtype + // needs the selector to keep doing until the mapping above grows a row. + return "auto"; + } +} + +// vllm/utils/torch_utils.py:75-80. +bool IsQuantizedKvCacheName(const std::string& kv_cache_dtype) { + const auto ends_with = [&](const std::string& suffix) { + return kv_cache_dtype.size() >= suffix.size() && + kv_cache_dtype.compare(kv_cache_dtype.size() - suffix.size(), + suffix.size(), suffix) == 0; + }; + return kv_cache_dtype.rfind("fp8", 0) == 0 || ends_with("per_token_head") || + kv_cache_dtype == "nvfp4"; +} + +// backend.py:158-161 — an EMPTY supported list means "no constraint". +bool AttentionBackend::supports_head_size(int head_size) const { + const std::vector supported = get_supported_head_sizes(); + if (supported.empty()) return true; + return std::find(supported.begin(), supported.end(), head_size) != supported.end(); +} + +// backend.py:163-165. +bool AttentionBackend::supports_dtype(vt::DType dtype) const { + const std::vector supported = supported_dtypes(); + return std::find(supported.begin(), supported.end(), dtype) != supported.end(); +} + +// backend.py:167-173. An EMPTY name is upstream's `None`. +bool AttentionBackend::supports_kv_cache_dtype( + const std::string& kv_cache_dtype) const { + if (kv_cache_dtype.empty()) return true; + const std::vector supported = supported_kv_cache_dtypes(); + if (supported.empty()) return true; + return std::find(supported.begin(), supported.end(), kv_cache_dtype) != + supported.end(); +} + +// backend.py:175-192. Every entry of get_supported_kernel_block_sizes() is an +// upstream MultipleOf, so a framework block size is supported when it is a +// multiple of ANY declared size — upstream's hybrid_blocks rule at :187-191, +// which is why 32 passes a backend declaring MultipleOf(16) and 8 does not. +bool AttentionBackend::supports_block_size(int block_size) const { + if (block_size == 0) return true; // upstream `None` + const std::vector supported = get_supported_kernel_block_sizes(); + if (supported.empty()) return true; + for (const int size : supported) { + if (size != 0 && block_size % size == 0) return true; + } + return false; +} + +// backend.py:319-393, in upstream's order, with upstream's reason strings. +std::vector AttentionBackend::validate_configuration( + const platforms::AttnSelectorConfig& cfg, + const platforms::DeviceCapability& capability) const { + std::vector invalid_reasons; + if (!supports_head_size(cfg.head_size)) { + invalid_reasons.emplace_back("head_size not supported"); + } + if (!supports_dtype(cfg.dtype)) { + invalid_reasons.emplace_back("dtype not supported"); + } + if (!supports_kv_cache_dtype(cfg.kv_cache_dtype)) { + invalid_reasons.emplace_back("kv_cache_dtype not supported"); + } + if (!supports_block_size(cfg.block_size)) { + invalid_reasons.emplace_back("block_size not supported"); + } + if (cfg.use_mm_prefix && !supports_mm_prefix()) { + invalid_reasons.emplace_back( + "partial multimodal token full attention not supported"); + } + if (cfg.use_mla != is_mla()) { + invalid_reasons.emplace_back(cfg.use_mla ? "MLA not supported" + : "non-MLA not supported"); + } + if (cfg.has_sink && !supports_sink()) { + invalid_reasons.emplace_back("attention sinks not supported"); + } + if (cfg.use_sparse != is_sparse()) { + invalid_reasons.emplace_back(cfg.use_sparse ? "sparse not supported" + : "non-sparse not supported"); + } + if (cfg.use_per_head_quant_scales && !supports_per_head_quant_scales()) { + invalid_reasons.emplace_back("per-head quant scales not supported"); + } + // backend.py:366-367, with upstream's own PRECONDITION rather than a new rule: + // CudaPlatform.get_attn_backend_cls asserts `device_capability is not None` + // before it calls this (cuda.py:403-404), and CpuPlatform has a separate + // selector that never reaches it (cpu.py:75-87). Our selector is shared across + // every DeviceType, and DeviceCapability::present() is already false for every + // platform that cannot answer the question, so the predicate applies exactly + // where upstream applies it. Without this, FLASH_ATTN — which this tree also + // registers for kCPU/kMETAL/kVULKAN/kTENSTORRENT — would be refused on every + // one of them by a rule about NVIDIA compute capability. + if (capability.present() && !supports_compute_capability(capability)) { + invalid_reasons.emplace_back("compute capability not supported"); + } + if (!supports_attn_type(cfg.attn_type)) { + invalid_reasons.emplace_back("attention type " + cfg.attn_type + + " not supported"); + } + if (cfg.has_sliding_window && !supports_sliding_window()) { + invalid_reasons.emplace_back("sliding window not supported"); + } + if (cfg.use_non_causal && !supports_non_causal()) { + invalid_reasons.emplace_back("non-causal attention not supported"); + } + if (cfg.use_batch_invariant && !supports_batch_invariance()) { + invalid_reasons.emplace_back("batch invariance not supported"); + } + if (cfg.use_kv_connector && !supports_kv_connector()) { + invalid_reasons.emplace_back("KV connector not supported"); + } + if (cfg.use_pcp && !supports_pcp()) { + invalid_reasons.emplace_back("PCP not supported"); + } + if (const std::optional combination = + supports_combination(cfg, capability); + combination.has_value()) { + invalid_reasons.push_back(*combination); + } + return invalid_reasons; +} + std::vector FlashAttentionBackend::get_kv_cache_shape( int64_t num_blocks, int64_t block_size, int64_t num_kv_heads, int64_t head_size, const std::string& /*cache_dtype_str*/) const { @@ -98,8 +252,9 @@ std::vector RocmAttentionBackend::get_kv_cache_shape( std::vector TritonMLABackend::get_kv_cache_shape( int64_t num_blocks, int64_t block_size, int64_t num_kv_heads, int64_t head_size, const std::string& /*cache_dtype_str*/) const { - // triton_mla.py:100-103 supports_block_size. - if (!supports_block_size(block_size)) { + // triton_mla.py:100-103 supports_block_size — now the base predicate reading + // this class's get_supported_kernel_block_sizes() == {16}. + if (!supports_block_size(static_cast(block_size))) { throw std::invalid_argument("Block size must be a multiple of 16."); } // mla_attention.py:1219 — "num_kv_heads ... assumed to be 1 for MLA". Upstream diff --git a/src/vllm/v1/attention/registry.cpp b/src/vllm/v1/attention/registry.cpp index c068b8f60..4dd4960c0 100644 --- a/src/vllm/v1/attention/registry.cpp +++ b/src/vllm/v1/attention/registry.cpp @@ -1,7 +1,9 @@ // Ported from: vllm/v1/attention/backends/registry.py + vllm/platforms/cuda.py -// :361-470 (get_valid_backends / get_attn_backend_cls) @ pin e24d1b24 — the -// attention-backend registry storage + platform-priority selection. See -// registry.h for the design; this file is the registry table and the selector. +// :359-394 (get_valid_backends) and :397-492 (get_attn_backend_cls) @ pin +// 5559679229 — the attention-backend registry storage + platform-priority +// selection. (Re-anchored from `:361-470 @ e24d1b24`, the pin retired at W5.) +// See registry.h for the design; this file is the registry table and the +// selector. #include "vllm/v1/attention/registry.h" #include @@ -55,15 +57,32 @@ std::unique_ptr MakeAttentionBackend(vt::DeviceType device, namespace { -// The capability half of vllm/v1/attention/backend.py:307-360 -// validate_configuration: a candidate must agree with the request on is_mla() -// and is_sparse(). Backends are stateless descriptors, so constructing one to -// ask is cheap and mirrors upstream querying the CLASS. This is the ONLY place -// sparse/DSA needs to be understood — see registry.h "the DSA seam". -bool CandidateMatchesConfig(vt::DeviceType device, const std::string& name, - const platforms::AttnSelectorConfig& cfg) { +// vllm/v1/attention/backend.py:319-393 `validate_configuration`, asked of one +// candidate. Backends are stateless descriptors, so constructing one to ask is +// cheap and mirrors upstream querying the CLASS. Returns upstream's list of +// reasons; EMPTY means the candidate is valid for this request. +// +// Before #1332 M1 this asked two questions inline (is_mla / is_sparse). Those +// two are still asked, by validate_configuration, in upstream's order and with +// upstream's reason strings — the sparse/DSA seam registry.h describes is +// unchanged, it just no longer has its own private copy of the rule. +std::vector CandidateInvalidReasons( + vt::DeviceType device, const std::string& name, + const platforms::Platform& platform, + const platforms::AttnSelectorConfig& cfg) { const std::unique_ptr backend = MakeAttentionBackend(device, name); - return backend->is_mla() == cfg.use_mla && backend->is_sparse() == cfg.use_sparse; + return backend->validate_configuration(cfg, platform.get_device_capability()); +} + +// `[reason, reason, ...]`, upstream's own rendering (cuda.py:433-438). +std::string JoinReasons(const std::vector& reasons) { + std::string out = "["; + for (size_t i = 0; i < reasons.size(); ++i) { + if (i != 0) out += ", "; + out += reasons[i]; + } + out += "]"; + return out; } } // namespace @@ -84,12 +103,13 @@ std::string SelectAttentionBackendName(const platforms::Platform& platform, "' is not registered for device type " + std::to_string(static_cast(device))); } - if (!CandidateMatchesConfig(device, selected, cfg)) { + const std::vector reasons = + CandidateInvalidReasons(device, selected, platform, cfg); + if (!reasons.empty()) { + // cuda.py:416-420 — the error names every reason, not the first. throw std::invalid_argument( - std::string("selected attention backend '") + selected + - "' does not satisfy the request (use_mla=" + - (cfg.use_mla ? "true" : "false") + - ", use_sparse=" + (cfg.use_sparse ? "true" : "false") + ")"); + std::string("Selected backend ") + selected + + " is not valid for this configuration. Reason: " + JoinReasons(reasons)); } return selected; } @@ -99,17 +119,24 @@ std::string SelectAttentionBackendName(const platforms::Platform& platform, // backend). An unregistered name is skipped exactly as an ImportError-ing // backend is in get_valid_backends; a registered-but-mismatched one is skipped // exactly as validate_configuration's "invalid reasons" reject it. + std::string all_invalid_reasons; for (const std::string& name : platform.get_attn_backend_priority(cfg)) { if (!HasAttentionBackend(device, name)) continue; - if (!CandidateMatchesConfig(device, name, cfg)) continue; - return name; + const std::vector reasons = + CandidateInvalidReasons(device, name, platform, cfg); + if (reasons.empty()) return name; + // cuda.py:387-388 — a rejected candidate is REMEMBERED with its reasons, so + // the exhaustion message can say why each one lost rather than only that + // none won. + if (!all_invalid_reasons.empty()) all_invalid_reasons += ", "; + all_invalid_reasons += name + ": " + JoinReasons(reasons); } + // cuda.py:440-446. throw std::runtime_error( - std::string("no valid attention backend registered for device type ") + - std::to_string(static_cast(device)) + - " (priority list yielded no registered backend; use_mla=" + - (cfg.use_mla ? "true" : "false") + + std::string("No valid attention backend for device type ") + + std::to_string(static_cast(device)) + " from {" + all_invalid_reasons + + "} (use_mla=" + (cfg.use_mla ? "true" : "false") + ", use_sparse=" + (cfg.use_sparse ? "true" : "false") + ")"); } diff --git a/src/vllm/v1/worker/gpu/runner.cpp b/src/vllm/v1/worker/gpu/runner.cpp index 4ac0cebf5..9478c25b3 100644 --- a/src/vllm/v1/worker/gpu/runner.cpp +++ b/src/vllm/v1/worker/gpu/runner.cpp @@ -976,11 +976,31 @@ void GPUModelRunner::initialize_kv_cache(const KVCacheConfig& kv_cache_config) { // A future backend with a different layout fails LOUDLY here at init. const bool is_mla = mla_layer_mask[static_cast(i)] != 0; + // #1332 M1: the selector now applies the full validate_configuration + // capability surface, so the request it is asked has to BE the request. The + // three fields this site can answer come straight from the geometry it just + // resolved. `dtype` (the model/query dtype) is NOT available here — the + // runner resolves only ResolveKvCacheDType() — so it keeps its bf16 default; + // that is owed to #1332 M4 and recorded under `## Owed` in + // .agents/specs/attn-validate-configuration.md. + // + // AND READ THIS BEFORE READING A GREEN SELECTION AS A WORKING BACKEND: the + // name resolved here still reaches only attn_backend_names_, the + // VT_ATTN_SELECT_LOG print below and CheckKvCacheShape. dense_attn::AttnBlock + // calls vt::PagedAttention unconditionally. Nothing DISPATCHES on this. #1332 + // M4 owns that, and until it lands a valid name is a claim, not a route. + vllm::platforms::AttnSelectorConfig cfg; + cfg.head_size = static_cast(fa_dims[i].head_size); + cfg.num_heads = static_cast(fa_dims[i].num_kv_heads); + cfg.block_size = static_cast(fa_block_size); + cfg.kv_cache_dtype = vllm::v1::KvCacheDTypeName(fa_dims[i].dtype); + cfg.quantized_kv_cache = vllm::v1::IsQuantizedKvCacheName(cfg.kv_cache_dtype); + std::string name; if (is_mla) { if (!mla_backend_resolved) { mla_backend_resolved = true; - vllm::platforms::AttnSelectorConfig mla_cfg; + vllm::platforms::AttnSelectorConfig mla_cfg = cfg; mla_cfg.use_mla = true; try { mla_backend = vllm::v1::SelectAttentionBackendName( @@ -995,7 +1015,7 @@ void GPUModelRunner::initialize_kv_cache(const KVCacheConfig& kv_cache_config) { if (!dense_backend_resolved) { dense_backend_resolved = true; dense_backend = vllm::v1::SelectAttentionBackendName( - vllm::platforms::GetPlatform(queue_.device.type)); + vllm::platforms::GetPlatform(queue_.device.type), "", cfg); } name = dense_backend; } diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 5dc7b4c64..74e1eb1b1 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -1257,6 +1257,8 @@ vllm_cpp_add_test(test_runner vllm/v1/worker/test_runner.cpp) target_include_directories(test_runner PRIVATE ${CMAKE_SOURCE_DIR}/src) vllm_cpp_add_test(test_common_attn_metadata vllm/v1/attention/test_common_attn_metadata.cpp) vllm_cpp_add_test(test_attn_backend_registry vllm/v1/attention/test_attn_backend_registry.cpp) +vllm_cpp_add_test(test_attn_validate_configuration + vllm/v1/attention/test_attn_validate_configuration.cpp) vllm_cpp_add_test(test_attention_window vllm/model_executor/layers/attention/test_attention.cpp) vllm_cpp_add_test(test_chunked_local_attention vllm/v1/attention/test_chunked_local_attention.cpp) diff --git a/tests/vllm/v1/attention/test_attn_validate_configuration.cpp b/tests/vllm/v1/attention/test_attn_validate_configuration.cpp new file mode 100644 index 000000000..9986b1ed4 --- /dev/null +++ b/tests/vllm/v1/attention/test_attn_validate_configuration.cpp @@ -0,0 +1,481 @@ +// Ported from: tests/kernels/attention/test_attention_selector.py @ pin +// 5559679229 — the executable spec of +// `vllm/v1/attention/backend.py::AttentionBackend.validate_configuration` +// (`:320-393`) and the `supports_*` predicates it collects (`:154-317`), as +// applied by `vllm/platforms/cuda.py::get_valid_backends` (`:359-394`) and +// `get_attn_backend_cls` (`:397-492`). +// +// Every upstream case in that file monkeypatches the device capability and then +// asserts a chosen backend NAME, so the whole file is CPU-runnable: the only +// thing it needs from a GPU is a number, and `FakeCudaPlatform` below takes that +// number as a constructor argument. +// +// WHAT THIS FILE DOES NOT MEASURE (issue #1332, and the reason that issue +// exists). These predicates describe what a backend CLAIMS. They cannot see what +// the shipped binary contains, and they cannot see which kernel actually runs: +// `dense_attn::AttnBlock` calls `vt::PagedAttention` unconditionally, so the +// name selected here dispatches nothing. Upstream's +// `FlashAttentionBackend.supports_compute_capability` is `>= (8,0)`; it passed on +// a GB10 (12,1) whose FA2 fatbin holds `sm_80` SASS and `compute_80` PTX alone, +// and every launch then failed a driver JIT. A green run of this file is +// necessary and NOT sufficient. See `.agents/specs/attn-validate-configuration.md`. +#include + +#include +#include +#include +#include + +#include "vllm/platforms/cuda_attn_priority.h" +#include "vllm/platforms/interface.h" +#include "vllm/v1/attention/backend.h" +#include "vllm/v1/attention/registry.h" +#include "vt/backend.h" +#include "vt/device.h" +#include "vt/dtype.h" + +using vllm::platforms::AttnSelectorConfig; +using vllm::platforms::DeviceCapability; +using vllm::platforms::Platform; +using vllm::platforms::ResidencyPolicy; +using vllm::v1::AttentionBackend; +using vllm::v1::MakeAttentionBackend; +using vllm::v1::SelectAttentionBackendName; +using vt::DeviceType; +using vt::DType; + +namespace { + +// The capability-injecting platform of the upstream test's +// `patch("vllm.platforms.current_platform", CudaPlatform())` plus +// `monkeypatch.setattr(torch.cuda, "get_device_capability", ...)`. It delegates +// to the REAL shared priority table so no list is hand-copied here. +class FakeCudaPlatform final : public Platform { + public: + explicit FakeCudaPlatform(int major, int minor) : cap_{major, minor} {} + DeviceType device_type() const override { return DeviceType::kCUDA; } + vt::Backend& backend() const override { return vt::GetBackend(DeviceType::kCPU); } + DeviceCapability get_device_capability() const override { return cap_; } + std::vector supported_dtypes() const override { return {DType::kBF16}; } + ResidencyPolicy residency_policy() const override { return {}; } + std::vector get_attn_backend_priority( + const AttnSelectorConfig& cfg) const override { + return vllm::platforms::LookupAttnPriority(cap_.major, cfg); + } + + private: + DeviceCapability cap_; +}; + +// The request the gate models actually make on GB10: Qwen3-dense geometry, +// bf16, framework block size 16, plain causal decoder attention. Every case +// below starts from this and changes ONE field, so a refusal is attributable. +AttnSelectorConfig GateModelConfig() { + AttnSelectorConfig cfg; + cfg.head_size = 128; + cfg.dtype = DType::kBF16; + cfg.kv_cache_dtype = "auto"; + cfg.block_size = 16; + return cfg; +} + +} // namespace + +TEST_CASE("the gate-model request still selects FLASH_ATTN (behavior control)") { + // THE control for every refusal below: a fully populated, valid config + // resolves exactly what the runner resolves today. If this case ever goes red, + // the capability layer has changed production selection, which is a behavior + // change and not a capability port. + FakeCudaPlatform sm121(12, 1); + CHECK(SelectAttentionBackendName(sm121, "", GateModelConfig()) == "FLASH_ATTN"); + + // And the all-defaults config, which is what every pre-existing call site + // passes, is unchanged too. + CHECK(SelectAttentionBackendName(sm121, "", AttnSelectorConfig{}) == "FLASH_ATTN"); +} + +TEST_CASE("test_flash_attn: an unsupported compute capability refuses FLASH_ATTN") { + // test_attention_selector.py:277-280 — `get_device_capability -> (7, 5)` then + // `assert backend.get_name() != "FLASH_ATTN"`. Upstream's predicate is + // `capability >= DeviceCapability(8, 0)` (flash_attn.py:200-202). sm_75 is + // Turing, the last generation before FlashAttention's floor. + FakeCudaPlatform sm75(7, 5); + CHECK_THROWS_AS(SelectAttentionBackendName(sm75, "", GateModelConfig()), + std::runtime_error); +} + +TEST_CASE("test_flash_attn: an unsupported model dtype refuses FLASH_ATTN") { + // test_attention_selector.py:285-287 (`torch.float8_e4m3fn`) and + // test_fp32_fallback:242-247. `supported_dtypes` is `[float16, bfloat16]` + // (backend.py:59, flash_attn.py:73). Upstream's fp32 request lands on + // FLEX_ATTENTION; this tree registers no FLEX_ATTENTION, so the walk is + // exhausted and refuses — which is the correct answer, not a fallback. + FakeCudaPlatform sm121(12, 1); + AttnSelectorConfig f32 = GateModelConfig(); + f32.dtype = DType::kF32; + CHECK_THROWS_AS(SelectAttentionBackendName(sm121, "", f32), std::runtime_error); + + // HARNESS ADAPTATION: upstream's second parameter here is + // `torch.float8_e4m3fn`, and `vt::DType` carries no fp8 member, so the second + // "outside the supported list" value is `kI8`. The predicate under test is a + // list membership either way, and keeping a second value proves the check is + // the list rather than an `== kF32` special case. + AttnSelectorConfig i8 = GateModelConfig(); + i8.dtype = DType::kI8; + CHECK_THROWS_AS(SelectAttentionBackendName(sm121, "", i8), std::runtime_error); +} + +TEST_CASE("test_flash_attn: an unsupported kv-cache dtype refuses FLASH_ATTN") { + // test_attention_selector.py:289-291. FLASH_ATTN's + // `supported_kv_cache_dtypes` is ["auto","float16","bfloat16","fp8","fp8_e4m3"] + // (flash_attn.py:74-80), so `fp8_e5m2` is outside it. + FakeCudaPlatform sm121(12, 1); + AttnSelectorConfig cfg = GateModelConfig(); + cfg.kv_cache_dtype = "fp8_e5m2"; + CHECK_THROWS_AS(SelectAttentionBackendName(sm121, "", cfg), std::runtime_error); +} + +TEST_CASE("test_flash_attn: an unsupported block size refuses FLASH_ATTN") { + // test_attention_selector.py:293-296 (`block_size = 8`). FLASH_ATTN declares + // `MultipleOf(16)` (flash_attn.py:82-84), and `supports_block_size` accepts any + // MULTIPLE of a declared size (backend.py:184-192), so 32 is fine and 8 is not. + FakeCudaPlatform sm121(12, 1); + AttnSelectorConfig bad = GateModelConfig(); + bad.block_size = 8; + CHECK_THROWS_AS(SelectAttentionBackendName(sm121, "", bad), std::runtime_error); + + AttnSelectorConfig ok = GateModelConfig(); + ok.block_size = 32; + CHECK(SelectAttentionBackendName(sm121, "", ok) == "FLASH_ATTN"); +} + +TEST_CASE("test_flash_attn: an unsupported head size refuses FLASH_ATTN") { + // test_attention_selector.py:313-315 (`head_size = 17`). Upstream: + // `head_size % 8 != 0 -> False`, `<= 256 -> True` (flash_attn.py:170-178). + FakeCudaPlatform sm121(12, 1); + AttnSelectorConfig odd = GateModelConfig(); + odd.head_size = 17; + CHECK_THROWS_AS(SelectAttentionBackendName(sm121, "", odd), std::runtime_error); + + // 264 is a multiple of 8 and ABOVE the 256 ceiling — the second half of the + // predicate, which a `% 8` test alone would not reach. Upstream raises the + // ceiling to 512 only when FlashAttention v4 resolves, and this tree ships FA2. + AttnSelectorConfig big = GateModelConfig(); + big.head_size = 264; + CHECK_THROWS_AS(SelectAttentionBackendName(sm121, "", big), std::runtime_error); + + // 256 itself is inside it. + AttnSelectorConfig edge = GateModelConfig(); + edge.head_size = 256; + CHECK(SelectAttentionBackendName(sm121, "", edge) == "FLASH_ATTN"); +} + +TEST_CASE("test_per_head_quant_scales_backend_selection: FA2 refuses per-head scales") { + // test_attention_selector.py:369-427, the `("FLASH_ATTN", 2, False)` row: + // FA2 does not support per-head quant scales. Upstream's predicate is + // `get_flash_attn_version() >= 3` (flash_attn.py:120-123); this tree ships FA2 + // only, so the answer is upstream's own answer for FA2. + FakeCudaPlatform sm121(12, 1); + AttnSelectorConfig cfg = GateModelConfig(); + cfg.kv_cache_dtype = "fp8"; + cfg.use_per_head_quant_scales = true; + CHECK_THROWS_AS(SelectAttentionBackendName(sm121, "", cfg), std::runtime_error); +} + +TEST_CASE("test_non_causal_backend_selection: FLASH_ATTN serves a non-causal request") { + // test_attention_selector.py:430-487, the two FLASH_ATTN rows: FlashAttention + // supports non-causal (flash_attn.py:106-108), so a drafter's non-causal + // request selects it, and so does an ordinary causal one. + FakeCudaPlatform sm121(12, 1); + AttnSelectorConfig non_causal = GateModelConfig(); + non_causal.use_non_causal = true; + CHECK(SelectAttentionBackendName(sm121, "", non_causal) == "FLASH_ATTN"); +} + +TEST_CASE("a sliding-window request refuses a backend that does not declare it") { + // backend.py:370-371 `has_sliding_window and not supports_sliding_window()`. + // FLASH_ATTN declares it (flash_attn.py:98-100), so the positive control + // selects; GDN_ATTN inherits the base `false`, so a platform preferring GDN + // for a sliding-window request finds nothing valid. + FakeCudaPlatform sm121(12, 1); + AttnSelectorConfig sw = GateModelConfig(); + sw.has_sliding_window = true; + CHECK(SelectAttentionBackendName(sm121, "", sw) == "FLASH_ATTN"); + + class GdnOnlyPlatform final : public Platform { + public: + DeviceType device_type() const override { return DeviceType::kCPU; } + vt::Backend& backend() const override { return vt::GetBackend(DeviceType::kCPU); } + DeviceCapability get_device_capability() const override { return {}; } + std::vector supported_dtypes() const override { return {DType::kBF16}; } + ResidencyPolicy residency_policy() const override { return {}; } + std::vector get_attn_backend_priority( + const AttnSelectorConfig& cfg) const override { + (void)cfg; + return {"GDN_ATTN"}; + } + } gdn; + CHECK(SelectAttentionBackendName(gdn, "", GateModelConfig()) == "GDN_ATTN"); + CHECK_THROWS_AS(SelectAttentionBackendName(gdn, "", sw), std::runtime_error); +} + +TEST_CASE("the refusal names every failed predicate, not only the first") { + // Upstream builds a LIST (`invalid_reasons`) and prints all of it + // (cuda.py:416-420, :432-446). A message that collapsed to the first failure + // would send a reader chasing one cause of four. Three violations at once: + // head size, block size and kv-cache dtype. + FakeCudaPlatform sm121(12, 1); + AttnSelectorConfig bad = GateModelConfig(); + bad.head_size = 17; + bad.block_size = 8; + bad.kv_cache_dtype = "fp8_e5m2"; + + std::string message; + try { + SelectAttentionBackendName(sm121, "", bad); + } catch (const std::exception& e) { + message = e.what(); + } + REQUIRE_FALSE(message.empty()); + CHECK(message.find("head_size not supported") != std::string::npos); + CHECK(message.find("block_size not supported") != std::string::npos); + CHECK(message.find("kv_cache_dtype not supported") != std::string::npos); +} + +TEST_CASE("an explicit override that fails validation is refused by reason") { + // cuda.py:407-420 — a `selected_backend` is validated too, and the error names + // the reasons rather than silently falling through to the priority walk. + FakeCudaPlatform sm121(12, 1); + AttnSelectorConfig odd = GateModelConfig(); + odd.head_size = 17; + + std::string message; + try { + SelectAttentionBackendName(sm121, "FLASH_ATTN", odd); + } catch (const std::exception& e) { + message = e.what(); + } + REQUIRE_FALSE(message.empty()); + CHECK(message.find("FLASH_ATTN") != std::string::npos); + CHECK(message.find("head_size not supported") != std::string::npos); +} + +TEST_CASE("#1333: sm_100 prefers FLASH_ATTN for a non-causal request") { + // cuda.py:144-163 @ pin 5559679229. The sm_100 non-MLA arm is guarded by + // `and not use_non_causal`, because SM100f's non-causal cutlass path — the one + // DFlash attention uses — is known-bad (`:145-147`). Causal keeps FlashInfer + // first; non-causal falls to the `else` arm, FlashAttention first. + FakeCudaPlatform sm100(10, 0); + const std::vector causal{"FLASHINFER", "FLASH_ATTN", "TRITON_ATTN", + "FLEX_ATTENTION", "TURBOQUANT"}; + const std::vector non_causal{"FLASH_ATTN", "FLASHINFER", "TRITON_ATTN", + "FLEX_ATTENTION", "TURBOQUANT"}; + AttnSelectorConfig plain; + CHECK(sm100.get_attn_backend_priority(plain) == causal); + + AttnSelectorConfig nc; + nc.use_non_causal = true; + CHECK(sm100.get_attn_backend_priority(nc) == non_causal); + + // sm_121 is in the `else` arm either way, so the guard moves nothing there. + FakeCudaPlatform sm121(12, 1); + CHECK(sm121.get_attn_backend_priority(plain) == non_causal); + CHECK(sm121.get_attn_backend_priority(nc) == non_causal); +} + +// ─── The predicate-level upstream cases, asked of the backend directly ─────── + +TEST_CASE("test_flash_attn_rejects_unhandled_kv_cache_dtypes") { + // test_attention_selector.py:524-540, all six upstream parameters. The point + // upstream makes in its own docstring: FlashAttentionBackend must not CLAIM + // support for kv_cache dtypes it cannot handle. Our list is upstream's + // (flash_attn.py:74-80), so the same six fall outside it. + const std::unique_ptr fa = + MakeAttentionBackend(DeviceType::kCUDA, "FLASH_ATTN"); + REQUIRE(fa != nullptr); + for (const char* name : {"fp8_e5m2", "fp8_ds_mla", "fp8_inc", "nvfp4", + "fp8_per_token_head", "int8_per_token_head"}) { + CAPTURE(name); + CHECK_FALSE(fa->supports_kv_cache_dtype(name)); + } +} + +TEST_CASE("test_flash_attn_accepts_handled_fp8_variants") { + // test_attention_selector.py:543-556, both upstream parameters: "fp8" (the + // alias for fp8_e4m3fn) and "fp8_e4m3" are the two FlashAttention can handle. + // Upstream needs an `is_xpu` monkeypatch to reach the answer through + // fa_utils; our predicate is the plain list membership it resolves to. + const std::unique_ptr fa = + MakeAttentionBackend(DeviceType::kCUDA, "FLASH_ATTN"); + REQUIRE(fa != nullptr); + CHECK(fa->supports_kv_cache_dtype("fp8")); + CHECK(fa->supports_kv_cache_dtype("fp8_e4m3")); + // backend.py:169-170 — upstream's `None`, which our empty string models. + CHECK(fa->supports_kv_cache_dtype("")); +} + +TEST_CASE("validate_configuration returns an EMPTY list for a valid request") { + // backend.py:339,393 — the empty list IS the "valid" answer, and the selector + // reads emptiness rather than a bool. A predicate that returned a one-element + // list of "" would select nothing while looking correct here, so the size is + // asserted, not just the truthiness. + const std::unique_ptr fa = + MakeAttentionBackend(DeviceType::kCUDA, "FLASH_ATTN"); + REQUIRE(fa != nullptr); + const std::vector reasons = + fa->validate_configuration(GateModelConfig(), DeviceCapability{12, 1}); + CHECK(reasons.empty()); +} + +TEST_CASE("validate_configuration collects ONE reason per failed predicate") { + // backend.py:339-393. Four independent violations produce four reasons, in + // upstream's order, with upstream's strings. A short-circuiting implementation + // returns one; a mis-ordered one fails the sequence. + const std::unique_ptr fa = + MakeAttentionBackend(DeviceType::kCUDA, "FLASH_ATTN"); + REQUIRE(fa != nullptr); + AttnSelectorConfig bad = GateModelConfig(); + bad.head_size = 17; // :340-341 + bad.dtype = DType::kF32; // :342-343 + bad.kv_cache_dtype = "nvfp4"; // :344-345 + bad.block_size = 8; // :346-347 + const std::vector reasons = + fa->validate_configuration(bad, DeviceCapability{12, 1}); + const std::vector expected{ + "head_size not supported", "dtype not supported", + "kv_cache_dtype not supported", "block_size not supported"}; + CHECK(reasons == expected); +} + +TEST_CASE("validate_configuration keys MLA and sparse on equality, both ways") { + // backend.py:352-363 — the check is `use_mla != is_mla()`, not "backend is + // MLA", so a dense backend asked for MLA and an MLA backend asked for dense + // BOTH fail, with different strings. This is the pair the pre-#1332 selector + // asked inline, so it is the behavior that must not have moved. + const std::unique_ptr fa = + MakeAttentionBackend(DeviceType::kCUDA, "FLASH_ATTN"); + const std::unique_ptr mla = + MakeAttentionBackend(DeviceType::kCUDA, "TRITON_MLA"); + REQUIRE(fa != nullptr); + REQUIRE(mla != nullptr); + + AttnSelectorConfig want_mla = GateModelConfig(); + want_mla.use_mla = true; + const std::vector fa_reasons = + fa->validate_configuration(want_mla, DeviceCapability{12, 1}); + CHECK(fa_reasons == std::vector{"MLA not supported"}); + + // TRITON_MLA asked for a DENSE request says the opposite thing. Its head size + // and block size are the MLA geometry, so the request is otherwise valid. + AttnSelectorConfig dense = GateModelConfig(); + dense.head_size = 576; + const std::vector mla_reasons = + mla->validate_configuration(dense, DeviceCapability{12, 1}); + CHECK(mla_reasons == std::vector{"non-MLA not supported"}); + + AttnSelectorConfig sparse = want_mla; + sparse.head_size = 576; + sparse.use_sparse = true; + const std::vector sparse_reasons = + mla->validate_configuration(sparse, DeviceCapability{12, 1}); + CHECK(sparse_reasons == std::vector{"sparse not supported"}); +} + +TEST_CASE("the compute-capability predicate is skipped when there is none") { + // backend.py:366-367 applies `supports_compute_capability` unconditionally, + // and CudaPlatform.get_attn_backend_cls guarantees a capability before calling + // (cuda.py:403-404) while CpuPlatform never calls it at all (cpu.py:75-87). + // Our selector is shared, and this tree registers FLASH_ATTN for kCPU too, so + // a default-constructed (absent) DeviceCapability must NOT refuse it — that + // would refuse the CPU tier on a rule about NVIDIA compute capability. + const std::unique_ptr fa = + MakeAttentionBackend(DeviceType::kCPU, "FLASH_ATTN"); + REQUIRE(fa != nullptr); + CHECK_FALSE(DeviceCapability{}.present()); + CHECK(fa->validate_configuration(GateModelConfig(), DeviceCapability{}).empty()); + + // And a capability that IS present is still enforced. + const std::vector turing = + fa->validate_configuration(GateModelConfig(), DeviceCapability{7, 5}); + CHECK(turing == std::vector{"compute capability not supported"}); + // 8.0 is the floor itself, so it passes (flash_attn.py:200-202 is `>=`). + CHECK(fa->validate_configuration(GateModelConfig(), DeviceCapability{8, 0}).empty()); +} + +TEST_CASE("attn_type is upstream's STRING, and the base serves decoder alone") { + // backend.py:291-298 + :38-46. FLASH_ATTN overrides to all four + // (flash_attn.py:110-118); GDN_ATTN inherits the base, which is decoder-only. + CHECK(std::string(vllm::v1::AttentionTypeName(vllm::v1::AttentionType::kDecoder)) == + "decoder"); + CHECK(std::string(vllm::v1::AttentionTypeName( + vllm::v1::AttentionType::kEncoderOnly)) == "encoder_only"); + + const std::unique_ptr fa = + MakeAttentionBackend(DeviceType::kCUDA, "FLASH_ATTN"); + const std::unique_ptr gdn = + MakeAttentionBackend(DeviceType::kCPU, "GDN_ATTN"); + REQUIRE(fa != nullptr); + REQUIRE(gdn != nullptr); + for (const char* type : {"decoder", "encoder", "encoder_only", "encoder_decoder"}) { + CAPTURE(type); + CHECK(fa->supports_attn_type(type)); + } + CHECK(gdn->supports_attn_type("decoder")); + CHECK_FALSE(gdn->supports_attn_type("encoder_only")); + + AttnSelectorConfig encoder = GateModelConfig(); + encoder.attn_type = "encoder_only"; + const std::vector reasons = + gdn->validate_configuration(encoder, DeviceCapability{}); + CHECK(reasons == std::vector{"attention type encoder_only not supported"}); +} + +TEST_CASE("supports_combination is asked, and FLASH_ATTN refuses a pre-Hopper sink") { + // flash_attn.py:204-218 — the cross-field rule no per-field predicate can + // express: sinks need compute capability >= 9.0. supports_sink() is false in + // this tree (no sink kernel), so the sink flag is refused by the per-field + // predicate first and BOTH reasons appear — which is the list shape, again. + const std::unique_ptr fa = + MakeAttentionBackend(DeviceType::kCUDA, "FLASH_ATTN"); + REQUIRE(fa != nullptr); + AttnSelectorConfig sink = GateModelConfig(); + sink.has_sink = true; + const std::vector reasons = + fa->validate_configuration(sink, DeviceCapability{8, 6}); + const std::vector expected{ + "attention sinks not supported", + "sink not supported on compute capability < 9.0"}; + CHECK(reasons == expected); + + // On Blackwell the combination rule is silent, so only the per-field one + // remains — proving the combination hook is genuinely consulted rather than + // being a constant. + const std::vector blackwell = + fa->validate_configuration(sink, DeviceCapability{10, 0}); + CHECK(blackwell == std::vector{"attention sinks not supported"}); +} + +TEST_CASE("block_size accepts any MULTIPLE of a declared size (hybrid blocks)") { + // backend.py:184-192 — the rule is `block_size % supported == 0`, NOT + // equality, because the framework block size only has to be a multiple of the + // kernel's requirement. A `== 16` implementation passes 16 and fails 32. + const std::unique_ptr fa = + MakeAttentionBackend(DeviceType::kCUDA, "FLASH_ATTN"); + REQUIRE(fa != nullptr); + CHECK(fa->supports_block_size(16)); + CHECK(fa->supports_block_size(32)); + CHECK(fa->supports_block_size(64)); + CHECK_FALSE(fa->supports_block_size(8)); + CHECK_FALSE(fa->supports_block_size(24)); + // backend.py:177-178 — upstream's `None`. + CHECK(fa->supports_block_size(0)); + + // TRITON_MLA declares the same MultipleOf(16) through the LIST, and it is now + // the base virtual rather than a static that hid it: the answer must be the + // same through an AttentionBackend& as it was through the concrete type. + const std::unique_ptr mla = + MakeAttentionBackend(DeviceType::kCUDA, "TRITON_MLA"); + REQUIRE(mla != nullptr); + CHECK(mla->supports_block_size(16)); + CHECK_FALSE(mla->supports_block_size(24)); +}