record(BACKEND-ROCM): index the decode profile, which refutes the transfer hypothesis

.agents/issue-index.md

@@ -485,3 +485,4 @@ rather than merged. `scripts/check-agent-record.py` gates both.
 | [#1447](https://github.com/mudler/vllm.cpp/issues/1447) | `ENG-EXPERT-STREAM` | **`docs/USAGE.md` said EVERY Qwen3.8 decode figure came from the W0e C ABI harness; the 66.7 s/token streaming-off row of 16 August 2026 came from `vllm-server`, the same binary the section tells the reader to run.** Introduced by #1211 and fixed in the same flow: the sentence is scoped to the W0e and W0f runs and the exception is named, in `docs/USAGE.md` and in the spec paragraph that mirrors it. Provenance read at the source, `.agents/specs/expert-streaming.md:837` (server entry point) and `:905`. | record |
 | [#1403](https://github.com/mudler/vllm.cpp/issues/1403) | `ENG-CUDAGRAPH-BREAK` | **`test_qwen3_5_decode_graph_seam` SIGSEGVs — a DUPLICATE of [#1394](https://github.com/mudler/vllm.cpp/issues/1394) that was already fixed when it was filed, so no code change was owed.** Filed against `96ed8346f`, which PREDATES the fix `7dec1d990` (PR #1393, merged 2026-08-20 00:49); `git merge-base --is-ancestor` puts the fix outside the reported tree. Re-measured at `b537a5344`, CPU-only Release x86_64: the test passes 5 runs of 5, `8 cases`/`8 passed`, `assertions: 138`, `Status: SUCCESS!`, exit 0. The cause was re-derived by single-variable mutation rather than inherited from #1394's report, and NAMED by a sanitizer rather than reasoned. Debug + `VLLM_CPP_SANITIZE=address,undefined` (NDEBUG OFF) with BOTH halves of `7dec1d990` reverted to their `96ed8346f` bytes: `AddressSanitizer: SEGV on unknown address 0x5045f5f84900`, `The signal is caused by a READ memory access`, at `src/vt/cpu/cpu_paged_attn.cpp:59` in `KvElem<KvKind::K>` reached from `:224`, on threadpool worker T1, inside case `W6: two spec shapes of EQUAL S and different q get two graphs` (`tests/vllm/models/test_qwen3_5_decode_graph_seam.cpp:800`). **It is NOT a use-after-free and there is no free site**, and `AddressSanitizer can not provide additional info` is the tell: the out-of-bounds block-table read is itself IN-BOUNDS for ASan because it lands inside the neighbouring pooled allocation, so this is not a `heap-buffer-overflow` either — the VALUE read becomes a wild block index that `KvElem` multiplies by the KV block stride and dereferences, which is why the fault address is unmapped rather than a redzone and why the crash is order-dependent instead of deterministic. Both halves are load-bearing: reverting ONLY the `SpecAttnMeta` fixture leaves the kernel bound to refuse by name at `src/vt/cpu/cpu_paged_attn.cpp:152` (exit 1, the case THROWS); reverting the bound as well restores the crash (Release exit 139, 3 runs of 3; ASan exit 1); restoring both gives exit 0, 8/8, zero sanitizer findings. The issue's `5f68e60df` pointer was right about the commit and wrong about the half — `git log -S'SpecAttnMeta'` and `git log -S'W6: two spec shapes of EQUAL S'` each return that commit ALONE, and both hits sit in the TEST fixture, not in the graph eligibility it widened. [#1405](https://github.com/mudler/vllm.cpp/issues/1405)'s truncation reproduced (`assertions: 135` crashed vs `138` complete) and is SHARPER under a threadpool fault: doctest printed the summary TWICE with different totals, `135` then `141`, because the main thread kept running after a worker died. That a red reached `main` unreported is [#1285](https://github.com/mudler/vllm.cpp/issues/1285)/[#1376](https://github.com/mudler/vllm.cpp/issues/1376), not this row | bug |
 | [#1458](https://github.com/mudler/vllm.cpp/issues/1458) | `VT-ACT-ROUND-POLARITY` | **`4712dac40` reds FOUR suites on `main` — `test_ltx2_text_encoder`, `test_muse_glimmer_text`, `test_muse_glimmer_text_fallback`, `test_minimax_music3_ar` — by exceeding bf16 error floors none of them had re-derived.** Found while gating [#1403](https://github.com/mudler/vllm.cpp/issues/1403) (PR #1457) and filed in flow, not that row's defect. Deterministic, not a load artifact: first seen under `-j4` and re-measured SERIALLY, 101 s across the four. `test_ltx2_text_encoder.cpp:2407` reads `CHECK( 0.1323 <= 0.109394 )` and `:2409` reads `CHECK( 0.0752773 <= 0.0573374 )`, both over by 20-30%, with the file at `26 passed / 1 failed`, `4118 assertions`, `Status: FAILURE!`. ATTRIBUTED BY MUTATION rather than inferred, both directions, in one build directory changing only `src/vt/cpu/cpu_ops.cpp`, compile rc 0 on every arm and `sha256sum` taken before and after: at `0adeb8b0e` four fail; reverted to `4712dac40^` four pass and `test_ltx2_text_encoder` is 27/27 `SUCCESS!`; restored to a re-matched `sha256` four fail again. They also passed at `b537a5344`, three commits earlier, in a full 567-test `ctest` run at 3-6 s each. `4712dac40` is +42/-3 in `src/vt/cpu/cpu_ops.cpp` plus a new `tests/vt/test_ops_activation.cpp`, and touches none of the four. NOT FIXED IN FLOW, deliberately: the question is a numerics decision, not a defect with an obvious repair — either the floors were calibrated against the rounding polarity that commit corrected and need re-deriving against the oracle, or the narrowing is wider than upstream's — and editing the four floors to pick the first branch is the scope-widening `AGENTS.md` prohibits | bug |
+| [#1294](https://github.com/mudler/vllm.cpp/issues/1294) | `BACKEND-ROCM` | ROCm decode on gfx1200 is kernel-bound, not transfer-bound, and the profile names which kernels. `rocprofv3` (rocprofiler-sdk 7.2.3) on RX 9060 XT with `Qwen3.6-14B-A3B-VibeForged-v2-Q4_K_M.gguf`, decode isolated by differencing `--max-tokens 4` against `--max-tokens 36` and dividing by 32, idle host: wall 88.7 ms/token (11.3 tok/s, matching the unprofiled 11.06-11.18 so the profiler is not the observer), GPU kernel dispatch 59.99 ms/token over 1738 dispatches, and **real H2D+D2H transfer 0.885 ms/token over 160 copies**. `hipMemcpyAsync` reports 59.31 ms/token of API time across 689 calls, which is the CPU blocking on unfinished GPU work rather than PCIe cost — so the host round-trips in the MoE reference path (`qwen3_5.cpp:6736`) are NOT the cost, which refutes the obvious hypothesis. By class: `vt::` kernels 44.92 ms/token (74.9%), hipBLASLt GEMM 13.81 (23.0%), copy/fill plumbing 1.26 (2.1%). Two leads, both measured: `QuantizeQ8KK` at 281 calls/token, 21.25 ms/token, 75.6 us per call, **35% of GPU time**; and `GdnPostConvK` (`rocm_gdn_postconv.hip:57`) at 30 calls/token, 11.24 ms/token, 375 us per call, 19%. Lead one has a named cause ON main and backend-agnostic: `MoeBlock` hands the SAME activation buffer to `KqGrouped` twice (`qwen3_5.cpp:6832-6833`, gate then up), so the identical activation is quantized twice per layer per token, and CPU and CUDA pay it too. NOT FIXED IN FLOW: the quantizer itself is `rocm_grouped_gemm.hip`, which is not on main and arrives with [#523](https://github.com/mudler/vllm.cpp/pull/523), so the 35% figure is feedback on that pull request rather than a main defect, and `GdnPostConvK` has no hypothesis attached yet — both want their own red-first evidence rather than a drive-by. Dispatch counts were byte-identical (281/30/60/20/21) across a contended and an idle run, so the structure is deterministic and only durations moved. Measured on `main` @ `4ee5f4a6` plus #523 plus the #559/#570 `AttnQkNormRopeGate` fix; no pinned-oracle comparison, one model, one prompt, batch 1 | record |