diff --git a/.agents/specs/gemma4-rocm-fp8-moe.md b/.agents/specs/gemma4-rocm-fp8-moe.md index deb6d0a08..bc371ba0a 100644 --- a/.agents/specs/gemma4-rocm-fp8-moe.md +++ b/.agents/specs/gemma4-rocm-fp8-moe.md @@ -58,3 +58,85 @@ and env-gated MoE paths do not. Revisit as a **separate** PR with CUDA golden. `VT_GEMMA4_MLP_MOE_PARALLEL` is documented but **not** wired in this tip (was only in the deferred layer-loop path). + +## Measured KEEP (2026-08-13, contributor lab) + +Hardware: 2x R9700 gfx1201, ROCm 7.2.4, kernel 7.0.0-29-generic. +Model: Gemma-4-26B-A4B-it FP8. Fair protocol: `PREFIX_CACHE=0` + unique pads. + +Recipe as run: SharedK-WMMA on, FLASH/FMHA/scoreless off, +`VT_GEMMA4_PREFILL_GEMM_M=2048`, `VT_GEMMA4_PREFILL_PEER_ACT=1`, batch MoE +`T>=64`, decode `KV_SPLITS=16` / `SLIDE_SPLITS=8` / `SLIDE_WARPS=16` / +`SPLIT_WARPS=12`. Speculative, ngram, layer-split: **off**. + +**Four of those names are not read by any product code in this tree, so this +recipe is not reproducible as written.** `VT_ATTN_DECODE_KV_SPLITS`, +`VT_ATTN_DECODE_SLIDE_SPLITS` and `VT_ATTN_DECODE_SPLIT_WARPS` appear only in +`tests/vt/test_gemma4_rocm_fp8_seams.cpp`, and `VT_ATTN_DECODE_SLIDE_WARPS` +appears nowhere at all; `git grep` over `src/` and `include/` returns zero hits +for all four. That is [#845](https://github.com/mudler/vllm.cpp/issues/845): the +seam test asserts `EnvInt(name, 16) == 16` with the variable unset, a tautology +that passes whether or not the knob exists, which is how three names that +nothing reads came to look real. The first two knob names above were also +abbreviated and are spelled here as the product spells them +(`PEER_ACT` is `VT_GEMMA4_PREFILL_PEER_ACT`, `gemma4_moe.cpp:1026`; +`PREFILL_GEMM_M` is `VT_GEMMA4_PREFILL_GEMM_M`, `:1016`, and `2048` is its +default, not an override). + +This file's `**Status:**` line records the run as `PR tip +feat/gemma4-rocm-fp8-split` with **no commit SHA**, so which tree produced these numbers cannot be +established from this repository. Either the four decode splits were live on that +tree and it is not this one, or they were inert and the numbers were produced by +the shipped defaults. Owed to the contributor: the tree SHA, and which of the two +it was. Until then the decode figure in particular has no recipe behind it, and +`docs/USAGE.md` publishes only the knobs a reader can actually set. + +| Depth | median prefill t/s | notes | +|------:|-------------------:|-------| +| ~3k | 2112 | 3 reps, Paris | +| ~11k | 2014 | first rep 1170 outlier dropped from claim; median 2014 | +| ~18k | 1705 | 3 reps | +| ~42k | 1099 | 2 reps | + +Decode stream 64 tok: **55.5 t/s** temp=0, **49.1 t/s** temp=0.7. +Quality: Paris, arith `63`, `gemma4` parser `tool_calls`, `/health` `/metrics`. +No engine-fatal / hipError in the closeout log. + +Same-box Vulkan Q8 unique-pad bar: **3503 @11k / 2714 @42k**. KEEP gap ~1.74x / 2.47x. +This row ships the reliable ROCm plateau. It does **not** close that bar. + +### Rejected (do not default-on) + +FMHA_WMMA2 (quality fail + 0.67x @11k). Isolated P1 cm1 wg256 (~1.13x KEEP; need ~3.35x isolated). Layer-split FIFO (~0.60x vs the Q8 bar). Head-TP peer-read (peer BW). hipBLASLt dual-GPU (fatal or slower). + +### Residual compiler / ACO + +Faithful HIP cm1 hsaco on gfx1201: 192 VGPR, 339 spills, 940 B private, vs KEEP SharedK 35 spills / 120 B. Output-slicing bottoms out at 235 spills / 736 B (D_PT=1). Mechanism of the isolated 1.13x, not by itself a named LLVM defect. + +Matched RADV/ACO (Mesa 26.0.3, discrete R9700, llama.cpp coopmat1 spec Br=16 Bc=64 wg=128 sg=32 row_split=4): official pipeline stats are **0 spilled VGPR, 0 scratch** at VGPR=256 (f16 and q8_0 d=512). That is an (a)-leaning HIP-LLVM vs ACO disparity on equivalent ownership. Smallest LLVM/rocWMMA component is not named. Phase-2 rebuild stays blocked until that component and a measurable prediction exist. A general codegen fix, if one exists, is a separate claim from the 3.35x bar. + +## Owed + +- **A vLLM-ROCm denominator for every number in "Measured KEEP".** They are + engine-side figures with nothing on the other side, so under AGENTS.md "Gates" + they are not a throughput result at all, and `docs/BENCHMARKS.md` keeps this + backend at `PENDING: no binding throughput number`. The oracle is not + hypothetical: `docs/ROCM.md` §5 documents two working Docker vLLM-ROCm recipes + on this hardware family, tier 2 building this project's pinned commit + `555967922` inside `rocm/vllm-dev:base` in about 6.5 minutes. Owed: the same + model, quantization, prompt set, request shape, concurrency and cache policy on + both sides of one idle 2x R9700 box, and the ratio per depth. Only the + contributor has that hardware. +- **The tree SHA behind the 2026-08-13 run**, and whether the four decode splits + named in the recipe were live on it. See the note under "Measured KEEP" and + [#845](https://github.com/mudler/vllm.cpp/issues/845). +- **`.agents/benchmark-record.md` is deliberately not appended here.** It is the + append-only measurement log, and a number whose tree is unidentified and whose + denominator is missing must not enter it: once logged, a figure gets quoted as + measured. It goes in when the two items above are discharged. +- **The Vulkan Q8 comparison stays open, not closed.** Same box, ~1.74x at 11k + and ~2.47x at 42k in Vulkan's favour. AGENTS.md forbids reading that as a + ceiling. The next traceable hypothesis is the one this file already names: the + HIP-LLVM versus ACO register-allocation disparity, 339 spills against ACO's 0, + and it needs the smallest LLVM/rocWMMA component named before a phase-2 + rebuild. diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md index d18bd64fa..e8874f7aa 100644 --- a/docs/ENVIRONMENT.md +++ b/docs/ENVIRONMENT.md @@ -229,7 +229,7 @@ on CUDA/CPU builds beyond the documented behavior. | `VT_GEMMA4_PREFILL_BATCH_MOE` | auto / `1` in lab recipe | `=1` group-by-expert prefill GEMM for `T>=64`; `=0` serial M=1 (slow). Unset = auto | | `VT_GEMMA4_MLP_MOE_PARALLEL` | off | `=1` run Gemma4 MLP and MoE on two HIP streams (lab; wall ~flat on R9700). Not wired in this PR tip (decode-graph-free split) | | `VT_ATTN_PREFILL_FLASH` | off | `=1` SGLang-style BM×BN GQA flash prefill (lab A/B) | -| `VT_GEMMA4_PREFILL_GEMM_M` | `256` | Tokens per expert in prefill-batch GEMM chunks (`16..2048`). Larger M → fewer launches; lab `512` ~+37% prefill vs `64` | +| `VT_GEMMA4_PREFILL_GEMM_M` | `2048` | Tokens per expert in prefill-batch GEMM chunks (`16..8192`; out-of-range values are ignored and the default is used). Larger M → fewer launches; lab `512` ~+37% prefill vs `64`, and `512`→`2048` ~+80 eng @11k vs the WMMA baseline (2026-08-10), which is why the default is `2048`. Lab KEEP on dual R9700 uses the default | | `VT_GEMMA4_HOST_EXPERT_MB` | `512` | Host-side expert staging budget (MiB) for non-resident paths | | `VT_GEMMA4_LAYER_TRACE` | off | `=1` layer GPU-synced phase timers; `=2` per-layer heartbeats | | `VLLM_CPP_HTTP_FIXED_POOL` | `1` (fixed) | `=0` reverts the HTTP worker pool to the legacy dynamic mode. Production uses the capacity-derived fixed pool; the opt-out exists for same-binary A/B attribution | diff --git a/docs/ROCM.md b/docs/ROCM.md index cc2f708c2..f12d0b258 100644 --- a/docs/ROCM.md +++ b/docs/ROCM.md @@ -9,8 +9,11 @@ Gemma-3-1B-it is 48/48 token-identical to two independent vLLM-ROCm oracles; Qwen3-0.6B exposed a deterministic cross-version near-tie. Qwen3.5-0.8B also runs all-native through the GDN stack, but its CPU/ROCm divergence is still an open correctness gap. The Gemma-4 FP8 MoE and SharedK-WMMA path has contributor -runtime evidence on 2x R9700; this repository has only CPU link coverage for -that path, and no matched oracle performance result. +runtime evidence on 2x R9700 (KEEP fair ~2014 t/s @~11k / ~1099 @~42k, +`PREFIX_CACHE=0`; decode ~55 t/s). This repository has only CPU link coverage +for that path, and no matched vLLM-ROCm oracle performance result. Vulkan Q8 +prefill on the same box is faster (~3503 / ~2714); the ROCm path is the +reliable recipe, not that bar. This page exists because several people offered hardware in [issue #41](https://github.com/mudler/vllm.cpp/issues/41), and it answers the diff --git a/docs/USAGE.md b/docs/USAGE.md index b52c781ae..546999cb7 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -3899,9 +3899,27 @@ ENVIRONMENT.md (`VT_GEMMA4_RESIDENT_*`, `VT_ATTN_*`). Defaults stay safe off RDN GetBlas keeps two per-thread hipBLAS handles (`tls_slots[2]`, device 1 → slot 1) so a 0→1 hop does not destroy GPU0's handle. `ProductGetBlasHandle` is the test accessor for that file-local `GetBlas`. HIP live probe is a separate CTest -target (exit 77 if `HIP_VISIBLE_DEVICES` empty); it enters capture so production `StreamIsCapturing` is load-bearing. No new env. This PR does **not** -restructure the Gemma-4 layer loop or enable decode hipGraph (those stay lab-only -until a CUDA token-exact gate can land them). +target (exit 77 if `HIP_VISIBLE_DEVICES` empty); it enters capture so production `StreamIsCapturing` is load-bearing. No new env. +This path does **not** restructure the Gemma-4 layer loop or enable decode hipGraph +(those stay lab-only until a CUDA token-exact gate can land them). + +Contributor KEEP recipe (2x R9700 gfx1201, ROCm 7.2.4, `PREFIX_CACHE=0`, unique +pads, 2026-08-13): SharedK-WMMA on, FLASH/FMHA off, `VT_GEMMA4_PREFILL_GEMM_M=2048` +(the default), `VT_GEMMA4_PREFILL_PEER_ACT=1` (the default), batch MoE `T>=64`. +Fair median prefill **2014 t/s @~11k** and **1099 t/s @~42k**; stream decode +**55 t/s** temp=0. Paris / arith `63` / `gemma4` tool_calls held. Speculative, +ngram, FMHA, and layer-split are **out of this recipe**. Details: +[spec](../.agents/specs/gemma4-rocm-fp8-moe.md). + +These are contributor-lab numbers against **no denominator**: no pinned vLLM-ROCm +run on the same box, same model, same quantization and same request shape exists +for them, so `docs/BENCHMARKS.md` still records this backend as `PENDING: no +binding throughput number` and this recipe does not change that. The decode +figure is also not reproducible from the knobs above: the as-run recipe set four +further decode splits that no product code in this tree reads +([#845](https://github.com/mudler/vllm.cpp/issues/845)), and they are recorded in +the spec rather than here, because a recipe on this page has to be one a reader +can follow. ## LTX-2.5 text conditioning