[DSv4][bring-up] FP8 MI325X SGLang benchmark / DSv4 FP8 MI325X SGLang 基准（bring-up）#2195
[DSv4][bring-up] FP8 MI325X SGLang benchmark / DSv4 FP8 MI325X SGLang 基准（bring-up）#2195Oseltamivir wants to merge 17 commits into
Conversation
Oseltamivir
requested review from
1am9trash,
billishyahao,
chunfangamd,
seungrokj and
yctseng0211
as code owners
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
2 similar comments
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
New SKU — no prior DeepSeek-V4-Pro data on gfx942. Extrapolated per recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Pro from the same-model MI355X entry (search-space shape, deepseek_v4 flags), the same-SKU dsr1-fp8-mi325x-sglang (gfx942 AITER infra), and the H200 dsv4 FP8 path (--quantization deepseek_v4_fp8). gfx942 has no native FP4, so the ~960GB mixed checkpoint runs in FP8 (~1.05TB), which only fits TP8 — MI355X TP4 bands dropped. 8x256GB (2TB) leaves ~1TB free after weights. Sweep: TP-only low-latency (conc 1-32) + DP-attn/EP8 throughput (conc 64-2048), 1k1k + 8k1k. Unvalidated on-cluster: needs debug-runs to confirm the mi30x image has the DeepseekV4 class + --attention-backend dsv4 / --quantization deepseek_v4_fp8 on gfx942. 中文：搭建 MI325X 上 DeepSeek-V4-Pro FP8 单节点 SGLang 脚手架（依据 recipes.vllm.ai 及同型号 MI355X、同 SKU dsr1-mi325x 外推；FP8 仅 TP8 可 放下故去掉 TP4）。尚未集群验证。
Oseltamivir
force-pushed
the
dsv4-mi325x-fp8-bringup
branch
from
da87721 to
4559de9
Compare
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
LGTM — bring-up scaffold PR that mirrors well-established sibling recipes (dsr1_fp8_mi325x.sh, dsr1_fp8_mi300x.sh, dsv4_fp4_mi355x_sglang.sh) with correct sizing math and no logic changes to shared code. The only finding is a comment-only provenance nit with zero runtime impact.
Extended reasoning...
Overview
This PR adds a new benchmark recipe (dsv4-fp8-mi325x-sglang): a new shell script under benchmarks/single_node/fixed_seq_len/, a new config block in configs/amd-master.yaml, and a corresponding perf-changelog.yaml entry. It is explicitly scoped as a bring-up scaffold that extrapolates config from three existing sibling recipes rather than introducing new logic.
Security risks
None. This is benchmark tooling (shell script that launches an sglang server and runs a benchmark client) — no auth, crypto, or permission-sensitive code is touched. Inputs (MODEL, TP, CONC, etc.) come from the internal sweep-generation pipeline, not untrusted external input.
Level of scrutiny
Low-to-moderate is appropriate. The script is copy-derived from three existing, already-reviewed files (dsr1_fp8_mi325x.sh for AITER env vars, dsr1_fp8_mi300x.sh for the MEC-firmware scratch-reclaim guard, dsv4_fp4_mi355x_sglang.sh for search-space shape and deepseek_v4 flags), and I confirmed the guard logic and env-var exports are copied correctly even though one citation comment points to the wrong source file (flagged inline as a nit). The YAML config block's sizing rationale (~960GB mixed checkpoint → ~1.05TB FP8, fits only TP8) is internally consistent with the stated 8x256GB=2TB capacity. The PR description itself flags that on-cluster validation (debug-runs) is still required before this is considered production-ready — that validation gate is a CI/process step, not something a code review can substitute for, so it doesn't change the code-review verdict.
Other factors
No prior reviews exist on this PR. The change is additive only (new file + new config entries), doesn't modify any existing recipe's behavior, and follows the same shape as numerous prior sibling PRs in this repo's history. The one bug found is a comment/provenance-only nit with no runtime effect.
| # shape, DP-attention + EP path, deepseek_v4 model flags, SWA/page-size) | ||
| # * same SKU, different model: dsr1_fp8_mi325x.sh (gfx942 infra: AITER, | ||
| # MEC-firmware scratch-reclaim guard) | ||
| # * same model, FP8 path: the H200 dsv4 recipe (--quantization |
There was a problem hiding this comment.
🟡 The header comment (lines 9-10) and the matching amd-master.yaml comment cite dsr1_fp8_mi325x.sh as the source of the MEC-firmware scratch-reclaim guard, but that file only sets SGLANG_USE_AITER/SGLANG_AITER_MLA_PERSIST and has no such guard - the guard actually originates in dsr1_fp8_mi300x.sh. This is a comment-only provenance error with no runtime effect, but it will send a future maintainer to the wrong file when tracing the pattern.
Extended reasoning...
The bug: This PR's header comment (dsv4_fp8_mi325x.sh:9-10) states:
# * same SKU, different model: dsr1_fp8_mi325x.sh (gfx942 infra: AITER,
# MEC-firmware scratch-reclaim guard)
and the inline comment right above the guard itself (near line 42-43) says (see dsr1_fp8_mi325x.sh). The matching configs/amd-master.yaml comment block repeats the same attribution for the dsv4-fp8-mi325x-sglang entry. All three point to dsr1_fp8_mi325x.sh as the source of the "MEC-firmware scratch-reclaim guard" (the rocm-smi --showfw | grep MEC ...; HSA_NO_SCRATCH_RECLAIM=1 block).
Why this is wrong: I read both candidate source files directly. dsr1_fp8_mi325x.sh contains only:
export SGLANG_USE_AITER=1
export SGLANG_AITER_MLA_PERSIST=1with no MEC/scratch-reclaim logic anywhere in the file. The guard - the version=$(rocm-smi --showfw | grep MEC | head -n 1 | awk '{print $NF}') line, the if [[ "$version" == "" || $version -lt 177 ]] check, and export HSA_NO_SCRATCH_RECLAIM=1 - actually lives in dsr1_fp8_mi300x.sh (lines 23-31), copied verbatim (including the "MEC FW older than 177, RCCL cannot reclaim" phrasing) into the new dsv4_fp8_mi325x.sh.
Step-by-step proof:
- Open
benchmarks/single_node/fixed_seq_len/dsr1_fp8_mi325x.sh- grep forMECorHSA_NO_SCRATCH_RECLAIM: no matches. The file only exports the two AITER env vars. - Open
benchmarks/single_node/fixed_seq_len/dsr1_fp8_mi300x.sh- the exact guard block is present at lines ~23-31, including the same comment wording ("If the machine runs a MEC FW older than 177, RCCL cannot reclaim some memory"). - Compare to the new
dsv4_fp8_mi325x.shguard (lines ~41-45): identical logic and near-identical comment wording to thedsr1_fp8_mi300x.shversion, confirming the copy-source wasmi300x, notmi325x. - Therefore the citation "same SKU, different model: dsr1_fp8_mi325x.sh (... MEC-firmware scratch-reclaim guard)" is factually incorrect for the MEC-guard portion (the AITER-infra portion of that same citation is correct, since
dsr1_fp8_mi325x.shdoes have the AITER flags).
Impact: Purely a documentation/provenance issue - the guard logic itself in dsv4_fp8_mi325x.sh is correct and functions properly regardless of the miscited source. The only consequence is that a maintainer later tracing "where did this scratch-reclaim guard pattern come from" by opening dsr1_fp8_mi325x.sh will find nothing there, costing them a few minutes of confusion, and the same wrong pointer is echoed in the amd-master.yaml config comment. There is no functional/runtime risk.
Suggested fix: Split the citation so the MEC-guard attribution points to dsr1_fp8_mi300x.sh instead of (or in addition to) dsr1_fp8_mi325x.sh, e.g.:
# * same SKU, different model: dsr1_fp8_mi325x.sh (gfx942 AITER infra);
# MEC-firmware scratch-reclaim guard borrowed from dsr1_fp8_mi300x.sh
and update the inline (see dsr1_fp8_mi325x.sh) comment plus the amd-master.yaml comment block to match.
All three independent verifiers confirmed this against the actual source files and agreed on nit severity, since it is comment-only with zero runtime impact.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29326767592 |
sglang argparse rejected --quantization deepseek_v4_fp8 (a vLLM-only method), crashing the canary before server start (run 29326767592). Remove the flag: sglang reads the modelopt quant config from the checkpoint and runs it via the AITER MoE path, matching the H200 and MI355X dsv4 sglang recipes (which pass no --quantization). 中文：删除 MI325X sglang 脚本中无效的 --quantization deepseek_v4_fp8 （该方法仅 vLLM 支持，sglang argparse 直接报错）。改为不传 --quantization， 由 sglang 从 checkpoint 读取 modelopt 量化配置并经 AITER MoE 执行。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29328296892 |
Same fix as MI300X: the prior fix cleared argparse but the server was killed at CUDA-graph capture (run 29328296892) because --attention-backend dsv4 compiles its FP8 MLA kernel via TileLang, whose InjectSoftwarePipeline pass fails on gfx942. Set SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton (+ SGLANG_USE_ROCM700A=0) per dsv4-fp4-mi355x-sglang to route MLA through the unified-KV triton path. Drop SGLANG_AITER_MLA_PERSIST; keep SGLANG_USE_AITER for the MoE GEMMs. 中文：与 MI300X 相同的修复——设置 SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton 改走统一 KV triton FlashMLA 路径，绕开在 gfx942 上编译失败的 TileLang FP8 MLA 内核。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29329280346 |
Same as MI300X: after the unified_kv_triton MLA fix, the server was killed at CUDA-graph capture (run 29328296892 lineage) in the AITER Composable-Kernel fused-MoE (ck_moe_stage1) — "Unsupported kernel config for moe heuristic dispatch", no CK tuned config for dsv4's FP8 MoE shapes on gfx942. Add --moe-runner-backend triton (JIT per-shape, no pretuned table). 中文：与 MI300X 相同——AITER CK fused-MoE 在 gfx942 上缺少 dsv4 FP8 MoE 形状 的调优配置，改用 --moe-runner-backend triton。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29330109620 |
Same pivot as MI300X: sglang has no gfx942 build of the dsv4 nvfp4 MoE / TileLang-MLA kernels. vLLM runs the checkpoint in FP8 via --quantization deepseek_v4_fp8 (dequant FP4 MoE -> FP8). Recipe mirrors dsv4-fp4-mi355x-vllm + gfx942 AITER infra; TP8 conc 4-512. Config key renamed dsv4-fp8-mi325x-sglang -> dsv4-fp8-mi325x-vllm. 中文：MI325X dsv4 与 MI300X 一样由 sglang 改为 vLLM，通过 --quantization deepseek_v4_fp8 将 FP4 MoE 反量化为 FP8 运行。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29333451101 |
…fx942 Same fix as MI300X: drop --moe-backend aiter so vLLM's select_deepseek_v4_mxfp4_moe_backend takes its ROCm+DeepseekV4 branch and prefers AITER_MXFP4_BF16 (W4A16, dequantizes mxfp4 weights) instead of the gfx942-unsupported AITER_MXFP4_MXFP4 (W4A4 native mxfp4). 中文：与 MI300X 相同的修复——去掉 --moe-backend aiter，让 vLLM 在 ROCm+DeepseekV4 分支选 AITER_MXFP4_BF16（反量化），避免 gfx942 不支持的 AITER_MXFP4_MXFP4。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29335838194 |
dsv4-fp8-mi325x-vllm-mtp mirrors the normal recipe + DeepSeek-V4 built-in MTP (--speculative-config mtp, num_speculative_tokens=2) with --dsv4 chat-template encoding. Routes to dsv4_fp8_mi325x_mtp.sh. 256GB keeps default eval concurrency. 中文：新增 MI325X vLLM MTP 变体（--speculative-config mtp + --dsv4）。
origin/main advanced (#2191 merged); reset perf-changelog.yaml to origin/main and re-append the dsv4-fp8-mi325x-vllm and dsv4-fp8-mi325x-vllm-mtp entries. 中文：将 MI325X 分支 perf-changelog 重置到 origin/main 并重新追加其条目。
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29353690389 |
8k1k KV fits only ~20x concurrency (9472-tok requests); conc512 is ~25x oversubscribed -> request timeouts (MTP c512 8k1k: 55.8% failures). conc256 passed for both normal and MTP on MI300X/MI325X, so cap 8k1k conc-end at 256 (1k1k holds at 512). Also add EVAL_CONCURRENT_REQUESTS=8 (as MI300X already has) so the gsm8k eval runs at a safe in-flight count instead of crashing the server at high concurrency.
MTP 8k1k at conc256 is borderline (MI300X 19.2% req failures, flaked pass->fail across runs) because the MTP draft model shrinks KV below normal's. conc128 passed cleanly on the memory-tightest SKU. Normal 8k1k holds at 256 (passed twice), 1k1k holds at 512.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29365630161 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29374064314 |
The mi325x-amds runner pool is down (canary never ran, sweep never spawned). Switch both dsv4-fp8-mi325x-vllm and -vllm-mtp to the mi325x-tw cluster.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29382843054 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29383406543 |
dsv4 MI325X was routed to cluster:mi325x-tw (mi325x-amds down), but the per-runner launch script was missing (canary failed with 'launch_mi325x-tw.sh: No such file'). Copy the amds launcher — same MI325X hardware + _mi325x.sh benchmark; assumes the same /raid + compute-partition layout.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29383844587 |
The mi325x-tw runners execute on the GPU node itself with docker + ROCm and no SLURM (prev attempt failed: 'salloc: command not found'). Run the container directly like launch_h100-cr.sh but AMD (--device=/dev/kfd,/dev/dri, --group-add video/render) on the node's 8 GPUs; runs the same _mi325x.sh benchmark. First run will pull the ROCm image and hf-download the model.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29384519189 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29384519189 |
The amds enroot launcher used --export=ALL; the docker launcher passes an explicit -e list that omitted DP_ATTENTION (required by _mi325x.sh's check_env_vars) plus EP_SIZE/DP_SIZE/EVAL_MAX_MODEL_LEN/SPEC_DECODING/ NUM_SPEC_TOKENS. Canary failed 'required environment variables not set: DP_ATTENTION'.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29385618855 |
1 participant
Summary
Bring-up scaffold — extrapolated config, not yet validated on-cluster. One of three per-SKU PRs adding DeepSeek-V4-Pro to the missing SKUs (MI325X here; MI300X is #2194, H100 separately). No prior dsv4 data exists on gfx942.
Extrapolated per https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Pro plus:
dsv4-fp4-mi355x-sglang: search-space shape (TP-only low-latency + DP-attn/EP throughput),deepseek_v4model flags, SWA/page-size.dsr1-fp8-mi325x-sglang: gfx942 AITER infra.--quantization deepseek_v4_fp8. gfx942 has no native FP4, so the ~960 GB mixed checkpoint runs in FP8.Sizing: ~960 GB mixed → ~1.05 TB in FP8, which only fits TP8 (MI355X TP4 bands dropped). 8×256 GB (2 TB) leaves ~1 TB free after FP8 weights, so the throughput band could later extend past conc 2048 — kept at parity with the MI300X entry for a first bring-up. Sweep: TP-only low-latency (conc 1–32) + DP-attn/EP8 throughput (conc 64–2048), for 1k1k and 8k1k (24 configs).
Image:
lmsysorg/sglang-rocm:v0.5.15-rocm720-mi30x-20260713.DeepseekV4model class, and that--attention-backend dsv4(elseaiter) and--quantization deepseek_v4_fp8work on gfx942.Local checks:
generate_sweep_configs.py --runner-type mi325x→ 24 configs;validate_perf_changelog.py --base-ref mainpasses; script passesbash -n.中文说明
Bring-up 脚手架——外推配置，尚未在集群验证。 为缺失 SKU 补充 DeepSeek-V4-Pro 的三个按 SKU 拆分 PR 之一（此为 MI325X；MI300X 为 #2194，H100 另开）。gfx942 此前无 dsv4 数据。
外推依据 https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Pro 及：
dsv4-fp4-mi355x-sglang：扫描空间形态、deepseek_v4模型参数、SWA/page-size。dsr1-fp8-mi325x-sglang：gfx942 AITER 基础设施。--quantization deepseek_v4_fp8。gfx942 无原生 FP4，故 ~960 GB 混合精度权重以 FP8 运行。显存测算： ~960 GB 混合 → FP8 约 1.05 TB，仅 TP8 可放下（去掉 MI355X 的 TP4 频段）。8×256 GB（2 TB）在 FP8 权重之外约剩 1 TB，吞吐频段后续可扩展至 conc 2048 以上——首次 bring-up 暂与 MI300X 保持一致。扫描：TP-only 低延迟（conc 1–32）+ DP-attn/EP8 吞吐（conc 64–2048），覆盖 1k1k 与 8k1k（24 个配置）。
镜像：
lmsysorg/sglang-rocm:v0.5.15-rocm720-mi30x-20260713。DeepseekV4模型类，且--attention-backend dsv4（否则aiter）与--quantization deepseek_v4_fp8在 gfx942 上可用。本地检查：
generate_sweep_configs.py --runner-type mi325x→ 24 个配置；validate_perf_changelog.py --base-ref main通过；脚本通过bash -n。