Skip to content

perf(rocm): stop GdnPostConvK putting the whole value_dim copy in one thread - #1402

Merged
localai-bot merged 2 commits into
mudler:mainfrom
joral:fix/gdn-postconv-chunk
Aug 21, 2026
Merged

perf(rocm): stop GdnPostConvK putting the whole value_dim copy in one thread#1402
localai-bot merged 2 commits into
mudler:mainfrom
joral:fix/gdn-postconv-chunk

Conversation

@joral

@joral joral commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

The kernel decomposes by (token, head) into t*(hk+1) items, and the single item
with head == hk copies the entire value_dim row while the other hk items do
about 4*dk element-ops each. On Qwen3.5/3.6 geometry (Hk=16, Hv=32, Dk=Dv=128)
at decode that is 17 items in one workgroup, 17 of 256 threads, with one lane
doing 4096 element-ops against sixteen doing 512. The kernel's duration is set
by that lane: 413-479 us per call, 30 calls per token, 11.24 ms/token, 19% of
ROCm GPU decode time and the second largest kernel in the profile.

Unlike the activation quantizer there is nothing to hoist here. Ld/St are
compile-time overloads and the kernel is already templated on all three dtypes,
so this is purely a decomposition defect.

Give a token hk q/k slots, then value_dim/dk copy chunks, then one gate slot.
That takes max per-thread element-ops from 4096 to 512 and items from 17 to 49.
Every element operation is identical and independent, because the v copy is
elementwise and the gates are per-h, so this changes WHICH thread does a given
element and never the arithmetic.

VT_ROCM_GDN_POSTCONV_CHUNK=0 restores the original kernel in the same binary.

Measured on RX 9060 XT (gfx1200), ROCm 7.2.3, arms interleaved on a near-idle
host, rocprofv3 --kernel-trace --stats for the per-call figures:

Qwen3.5-4B Q4_K_M, qwen35 dense 21.551 -> 25.492 tok/s (1.183x)
Ornith-1.5-9B Q4_K_M, qwen35 dense 16.275 -> 18.451 tok/s (1.134x), 413.2 -> 76.5 us
Qwen3.6-14B-A3B Q4_K_M, qwen35moe 12.560 -> 14.170 tok/s (1.128x), 468.6 -> 84.1 us
Qwen3.6-35B-A3B UD-Q4_K_S, offloaded 7.846 -> 8.579 tok/s (1.093x), 479.1 -> 83.0 us

The kernel lands at 76-84 us on every model regardless of dense or MoE and
resident or offloaded, which is what the geometry predicts: per-call cost is set
by Hk/Hv/Dk/Dv, identical across all four. Output byte-identical on every model,
both arms. The dense arms reach the kernel through Qwen3_5ForCausalLM with no
MoE block, so nothing else in the path accounts for the win, and the 35B's
smaller ratio is expected because its decode is PCIe-bound from offloaded
experts.

Gated by the existing cross-device oracle comparison, and the gate detects a
broken decomposition: mutating the chunk bound to drop one element per chunk
fails test_backend_cross_device.cpp:1560 on Nmse(ref_v, ...) <= 5e-4, 20/21
cases, restored green after revert.

Not addressed, stated rather than implied: occupancy is unchanged at one
workgroup because items only goes 17 -> 49. Full element-parallelism, about 8192
items, needs a wave-per-head reduction for the q/k sums and is a separate
change. The effect is decode-shaped and will not appear in a throughput
benchmark, because larger t spreads the original across more items.

Closes #1401

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude:claude-sonnet-5 [Claude Code]

… thread

The kernel decomposes by (token, head) into t*(hk+1) items, and the single item
with head == hk copies the entire value_dim row while the other hk items do
about 4*dk element-ops each. On Qwen3.5/3.6 geometry (Hk=16, Hv=32, Dk=Dv=128)
at decode that is 17 items in one workgroup, 17 of 256 threads, with one lane
doing 4096 element-ops against sixteen doing 512. The kernel's duration is set
by that lane: 413-479 us per call, 30 calls per token, 11.24 ms/token, 19% of
ROCm GPU decode time and the second largest kernel in the profile.

Unlike the activation quantizer there is nothing to hoist here. Ld/St are
compile-time overloads and the kernel is already templated on all three dtypes,
so this is purely a decomposition defect.

Give a token hk q/k slots, then value_dim/dk copy chunks, then one gate slot.
That takes max per-thread element-ops from 4096 to 512 and items from 17 to 49.
Every element operation is identical and independent, because the v copy is
elementwise and the gates are per-h, so this changes WHICH thread does a given
element and never the arithmetic.

VT_ROCM_GDN_POSTCONV_CHUNK=0 restores the original kernel in the same binary.
It selects a decomposition, not a behaviour a user would reach for, so it is
classified kernel-internal on scripts/env-doc-allowlist.txt beside its VT_GDN_*
siblings rather than documented in docs/ENVIRONMENT.md.

Measured on RX 9060 XT (gfx1200), ROCm 7.2.3, arms interleaved on a near-idle
host, rocprofv3 --kernel-trace --stats for the per-call figures:

  Qwen3.5-4B Q4_K_M, qwen35 dense       21.551 -> 25.492 tok/s (1.183x)
  Ornith-1.5-9B Q4_K_M, qwen35 dense    16.275 -> 18.451 tok/s (1.134x), 413.2 -> 76.5 us
  Qwen3.6-14B-A3B Q4_K_M, qwen35moe     12.560 -> 14.170 tok/s (1.128x), 468.6 -> 84.1 us
  Qwen3.6-35B-A3B UD-Q4_K_S, offloaded   7.846 ->  8.579 tok/s (1.093x), 479.1 -> 83.0 us

The kernel lands at 76-84 us on every model regardless of dense or MoE and
resident or offloaded, which is what the geometry predicts: per-call cost is set
by Hk/Hv/Dk/Dv, identical across all four. Output byte-identical on every model,
both arms. The dense arms reach the kernel through Qwen3_5ForCausalLM with no
MoE block, so nothing else in the path accounts for the win, and the 35B's
smaller ratio is expected because its decode is PCIe-bound from offloaded
experts.

Gated by the existing cross-device oracle comparison, and the gate detects a
broken decomposition: mutating the chunk bound to drop one element per chunk
fails test_backend_cross_device.cpp:1560 on Nmse(ref_v, ...) <= 5e-4, 20/21
cases, restored green after revert.

Not addressed, stated rather than implied: occupancy is unchanged at one
workgroup because items only goes 17 -> 49. Full element-parallelism, about 8192
items, needs a wave-per-head reduction for the q/k sums and is a separate
change. The effect is decode-shaped and will not appear in a throughput
benchmark, because larger t spreads the original across more items.

Closes mudler#1401

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude:claude-sonnet-5 [Claude Code]
@joral
joral force-pushed the fix/gdn-postconv-chunk branch from a24440b to bf85ac3 Compare August 20, 2026 12:43
@localai-org-maint-bot

Copy link
Copy Markdown
Collaborator

The focused ROCm change is currently conflicting with main, and the present head has several red shared gates. Please rebase onto current main and rerun the ROCm-specific correctness and performance evidence before merge review.

Brings the branch to current main so the union merge driver resolves the
issue-index append locally. GitHub's server-side merge does not apply custom
merge drivers, which is the only reason the pull request read CONFLICTING.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude:claude-opus-5 [Claude Code]
@localai-bot
localai-bot merged commit f4ccabb into mudler:main Aug 21, 2026
3 of 18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GdnPostConvK: one thread copies the whole value_dim row, setting the kernel's duration (19% of ROCm decode)

4 participants