record(BACKEND-ROCM): index the decode profile, which refutes the transfer hypothesis - #1295
Open
joral wants to merge 1 commit into
Open
record(BACKEND-ROCM): index the decode profile, which refutes the transfer hypothesis#1295joral wants to merge 1 commit into
joral wants to merge 1 commit into
Conversation
Collaborator
|
The record addition is coherent and correctly separates API blocking time from measured H2D/D2H transfer time, but this branch is currently conflicting with |
…nsfer hypothesis gfx1200 decode is kernel-bound. Profiled with rocprofv3 on an idle host, decode isolated by differencing --max-tokens 4 against 36: wall 88.7 ms/token, GPU kernel dispatch 59.99 ms/token, and real host/device transfer 0.885 ms/token. The 59.31 ms/token that hipMemcpyAsync reports is the CPU blocking on unfinished GPU work, not PCIe, so the MoE reference path's host round-trips are not the cost anyone would have guessed they were. Two measured leads: QuantizeQ8KK at 35% of GPU time, and GdnPostConvK at 19%. The first has a named backend-agnostic cause on main, where MoeBlock quantizes the same activation twice for gate and up. Filed and not fixed, so the row names BACKEND-ROCM as owner: the quantizer is not on main, and GdnPostConvK has no hypothesis yet. Both want red-first evidence rather than a drive-by change. Closes nothing; mudler#1294 stays open as the tracked gap. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-sonnet-5 [Claude Code]
joral
force-pushed
the
record/rocm-decode-profile
branch
from
August 19, 2026 12:53
b9b545e to
9a85d08
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
gfx1200 decode is kernel-bound. Profiled with rocprofv3 on an idle host, decode
isolated by differencing --max-tokens 4 against 36: wall 88.7 ms/token, GPU
kernel dispatch 59.99 ms/token, and real host/device transfer 0.885 ms/token.
The 59.31 ms/token that hipMemcpyAsync reports is the CPU blocking on unfinished
GPU work, not PCIe, so the MoE reference path's host round-trips are not the
cost anyone would have guessed they were.
Two measured leads: QuantizeQ8KK at 35% of GPU time, and GdnPostConvK at 19%.
The first has a named backend-agnostic cause on main, where MoeBlock quantizes
the same activation twice for gate and up.
Filed and not fixed, so the row names BACKEND-ROCM as owner: the quantizer is
not on main, and GdnPostConvK has no hypothesis yet. Both want red-first
evidence rather than a drive-by change.
Closes nothing; #1294 stays open as the tracked gap.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude:claude-sonnet-5 [Claude Code]