perf(rocm): stop GdnPostConvK putting the whole value_dim copy in one thread - #1402
Merged
Conversation
… thread The kernel decomposes by (token, head) into t*(hk+1) items, and the single item with head == hk copies the entire value_dim row while the other hk items do about 4*dk element-ops each. On Qwen3.5/3.6 geometry (Hk=16, Hv=32, Dk=Dv=128) at decode that is 17 items in one workgroup, 17 of 256 threads, with one lane doing 4096 element-ops against sixteen doing 512. The kernel's duration is set by that lane: 413-479 us per call, 30 calls per token, 11.24 ms/token, 19% of ROCm GPU decode time and the second largest kernel in the profile. Unlike the activation quantizer there is nothing to hoist here. Ld/St are compile-time overloads and the kernel is already templated on all three dtypes, so this is purely a decomposition defect. Give a token hk q/k slots, then value_dim/dk copy chunks, then one gate slot. That takes max per-thread element-ops from 4096 to 512 and items from 17 to 49. Every element operation is identical and independent, because the v copy is elementwise and the gates are per-h, so this changes WHICH thread does a given element and never the arithmetic. VT_ROCM_GDN_POSTCONV_CHUNK=0 restores the original kernel in the same binary. It selects a decomposition, not a behaviour a user would reach for, so it is classified kernel-internal on scripts/env-doc-allowlist.txt beside its VT_GDN_* siblings rather than documented in docs/ENVIRONMENT.md. Measured on RX 9060 XT (gfx1200), ROCm 7.2.3, arms interleaved on a near-idle host, rocprofv3 --kernel-trace --stats for the per-call figures: Qwen3.5-4B Q4_K_M, qwen35 dense 21.551 -> 25.492 tok/s (1.183x) Ornith-1.5-9B Q4_K_M, qwen35 dense 16.275 -> 18.451 tok/s (1.134x), 413.2 -> 76.5 us Qwen3.6-14B-A3B Q4_K_M, qwen35moe 12.560 -> 14.170 tok/s (1.128x), 468.6 -> 84.1 us Qwen3.6-35B-A3B UD-Q4_K_S, offloaded 7.846 -> 8.579 tok/s (1.093x), 479.1 -> 83.0 us The kernel lands at 76-84 us on every model regardless of dense or MoE and resident or offloaded, which is what the geometry predicts: per-call cost is set by Hk/Hv/Dk/Dv, identical across all four. Output byte-identical on every model, both arms. The dense arms reach the kernel through Qwen3_5ForCausalLM with no MoE block, so nothing else in the path accounts for the win, and the 35B's smaller ratio is expected because its decode is PCIe-bound from offloaded experts. Gated by the existing cross-device oracle comparison, and the gate detects a broken decomposition: mutating the chunk bound to drop one element per chunk fails test_backend_cross_device.cpp:1560 on Nmse(ref_v, ...) <= 5e-4, 20/21 cases, restored green after revert. Not addressed, stated rather than implied: occupancy is unchanged at one workgroup because items only goes 17 -> 49. Full element-parallelism, about 8192 items, needs a wave-per-head reduction for the q/k sums and is a separate change. The effect is decode-shaped and will not appear in a throughput benchmark, because larger t spreads the original across more items. Closes mudler#1401 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-sonnet-5 [Claude Code]
joral
force-pushed
the
fix/gdn-postconv-chunk
branch
from
August 20, 2026 12:43
a24440b to
bf85ac3
Compare
Collaborator
|
The focused ROCm change is currently conflicting with |
Brings the branch to current main so the union merge driver resolves the issue-index append locally. GitHub's server-side merge does not apply custom merge drivers, which is the only reason the pull request read CONFLICTING. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The kernel decomposes by (token, head) into t*(hk+1) items, and the single item
with head == hk copies the entire value_dim row while the other hk items do
about 4*dk element-ops each. On Qwen3.5/3.6 geometry (Hk=16, Hv=32, Dk=Dv=128)
at decode that is 17 items in one workgroup, 17 of 256 threads, with one lane
doing 4096 element-ops against sixteen doing 512. The kernel's duration is set
by that lane: 413-479 us per call, 30 calls per token, 11.24 ms/token, 19% of
ROCm GPU decode time and the second largest kernel in the profile.
Unlike the activation quantizer there is nothing to hoist here. Ld/St are
compile-time overloads and the kernel is already templated on all three dtypes,
so this is purely a decomposition defect.
Give a token hk q/k slots, then value_dim/dk copy chunks, then one gate slot.
That takes max per-thread element-ops from 4096 to 512 and items from 17 to 49.
Every element operation is identical and independent, because the v copy is
elementwise and the gates are per-h, so this changes WHICH thread does a given
element and never the arithmetic.
VT_ROCM_GDN_POSTCONV_CHUNK=0 restores the original kernel in the same binary.
Measured on RX 9060 XT (gfx1200), ROCm 7.2.3, arms interleaved on a near-idle
host, rocprofv3 --kernel-trace --stats for the per-call figures:
Qwen3.5-4B Q4_K_M, qwen35 dense 21.551 -> 25.492 tok/s (1.183x)
Ornith-1.5-9B Q4_K_M, qwen35 dense 16.275 -> 18.451 tok/s (1.134x), 413.2 -> 76.5 us
Qwen3.6-14B-A3B Q4_K_M, qwen35moe 12.560 -> 14.170 tok/s (1.128x), 468.6 -> 84.1 us
Qwen3.6-35B-A3B UD-Q4_K_S, offloaded 7.846 -> 8.579 tok/s (1.093x), 479.1 -> 83.0 us
The kernel lands at 76-84 us on every model regardless of dense or MoE and
resident or offloaded, which is what the geometry predicts: per-call cost is set
by Hk/Hv/Dk/Dv, identical across all four. Output byte-identical on every model,
both arms. The dense arms reach the kernel through Qwen3_5ForCausalLM with no
MoE block, so nothing else in the path accounts for the win, and the 35B's
smaller ratio is expected because its decode is PCIe-bound from offloaded
experts.
Gated by the existing cross-device oracle comparison, and the gate detects a
broken decomposition: mutating the chunk bound to drop one element per chunk
fails test_backend_cross_device.cpp:1560 on Nmse(ref_v, ...) <= 5e-4, 20/21
cases, restored green after revert.
Not addressed, stated rather than implied: occupancy is unchanged at one
workgroup because items only goes 17 -> 49. Full element-parallelism, about 8192
items, needs a wave-per-head reduction for the q/k sums and is a separate
change. The effect is decode-shaped and will not appear in a throughput
benchmark, because larger t spreads the original across more items.
Closes #1401
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude:claude-sonnet-5 [Claude Code]