[AMD][AgentX] Add DeepSeek-V4.1-Flash ATOM TP2/TP4 / 新增 DeepSeek-V4.1-Flash ATOM TP2/TP4 - #3387
Conversation
新增 DeepSeek-V4.1-Flash 在 MI355X 上的 ATOM TP2/TP4 AgentX 配方，使用指定 nightly 镜像、5-token DSpark 和 golden AL 3.51；同步中英文文档及性能变更记录。
回填 DeepSeek-V4.1-Flash ATOM 配方的实际 PR 链接。
|
Thanks for the contribution!
中文感谢你的贡献！
|
将性能变更记录指向 upstream 分支提交的 PR 3387。
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=36123267823 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=36123267823 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
I reviewed this PR and didn't find any bugs. Because it adds a new AMD ATOM benchmark recipe touching CODEOWNER-protected files (configs/amd-master.yaml, MODELS.md) with ROCm/ATOM-specific server flags that still lack GPU validation, a human (ideally one of the listed AMD codeowners) look would still be worthwhile.
What was reviewed: the new dsv41flash_fp4_mi355x_atom_mtp.sh benchmark script (env-var checks, TP/EP/DP validation, capture-size logic, spec-decode args, server startup/cleanup traps) against the closely-analogous existing dsv4_fp4_mi355x_atom_mtp.sh and dsv41flash_fp4_mi355x_vllm_mtp.sh scripts; confirmed the MI355X launcher's _mtp suffix routing for draft_model/mtp covers the new script; checked that configs/amd-master.yaml's new matrix entry, the append-only perf-changelog.yaml entry, and the English/Chinese docs all describe the same TP2/TP4 concurrency lists, image, and DSpark settings consistently.
Extended reasoning...
The change adds a new AMD MI355X ATOM AgentX recipe: a new benchmark shell script, a config matrix entry, docs (EN/ZH), MODELS.md/MODELS_zh.md updates, and an append-only perf-changelog entry — no application logic, auth, or data-handling code is touched, so there is no injection/auth/data-exposure surface. The script closely mirrors an existing sibling script (dsv4_fp4_mi355x_atom_mtp.sh) in structure, env-var validation, cleanup traps, and capture-size handling, and cross-checking it against that pattern and against benchmark_lib.sh's helper functions turned up no discrepancies. The decisive factor for deferring rather than approving is that configs/amd-master.yaml and MODELS.md are both explicitly CODEOWNER-protected files, and the recipe encodes ROCm/ATOM-specific tuning (capture sizes, kv/index cache dtypes, DSpark AL) that the PR itself says still awaits GPU execution and checkpoint-loading evidence.
This review covers commit 7f94c5f, which is no longer the latest commit on this pull request; later commits are not covered by it.
将 DeepSeek-V4.1-Flash ATOM 配方镜像更新为 rocm/atom-dev:nightly_202609241513（digest sha256:d04ec105），同步中英文文档及性能变更记录。 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
解决 perf-changelog.yaml 追加冲突，本 PR 条目移至末尾。 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
将 DeepSeek-V4.1-Flash ATOM 配方镜像更新为 rocm/atom-dev:nightly_202609250902（digest sha256:9aaeed59），同步中英文文档及性能变更记录。 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
解决 perf-changelog.yaml 追加冲突，本 PR 条目移至末尾。 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
/reuse-sweep-run 36123267823 |
…-Flash PR #3387 adds the DeepSeek-V4.1-Flash FP4 ATOM AgentX recipe on MI355X (TP2/TP4, five-token DSpark, BF16 KV, FP8 index cache, no EP/KV offload). This commit adds a changelog note recording the data-type-preserved audit: DSpark draft layers 37-39 keep their checkpoint precision. The recipe passes no --online_quant_config, so make_v4_quant_config builds the same per-layer quant spec for the drafter as the target; the mtp.0/1/2 weights load inline from the same checkpoint (spec_decode=True) and config.get_layer_quant_config has no mtp.* special case, so ATOM neither re-quantizes nor up/down-casts them. --kv_cache_dtype/--index-cache-dtype affect cache storage only, not draft/MTP weights. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…4 DSpark) Add the DeepSeek-V4.1-Flash FP4 ATOM AgentX recipe on MI355X following the upstream TP2/TP4 recipe: TP2 conc [1,2,8,16,32,64], TP4 conc [2,8,16,32,64], no expert parallelism or KV offload. Uses rocm/atom-dev:nightly_202609250902, BF16 KV, FP8 index cache, max-num-seqs 128, 16K batching/prefill chunks, prefix caching with block size 16, 8K state checkpoints, level 3 + FULL graphs. Five-token DSpark with thinking_on golden AL 3.51 for throughput, real acceptance for eval, dsml_v41 tool parsing. Data-type preserved: DSpark draft layers 37-39 keep their checkpoint precision. The recipe passes no --online_quant_config, so make_v4_quant_config builds the same per-layer quant spec for the drafter as the target; the mtp.0/1/2 weights load inline from the same checkpoint (spec_decode=True) and config.get_layer_quant_config has no mtp.* special case, so ATOM neither re-quantizes nor up/down-casts them. --kv_cache_dtype/--index-cache-dtype affect cache storage only, not draft/MTP weights. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that
SGLANG_NVFP4_CKPT_FP8_NEXTN_MOEis not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison. - For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If this PR uses
append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
- Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.
Additional detail section:
- insert any additional info here
- recipe at https://github.com/ROCm/ATOM/blob/main/recipes/DeepSeek-V4.1-Flash-Agentic.md
- "DSpark draft layers 37-39 keep their checkpoint precision: the recipe passes no --online_quant_config, so online_quant_config resolves to None and make_v4_quant_config builds the same per-layer quant spec for the drafter as for the target. The MTP weights (mtp.0/1/2) load inline from the same checkpoint with spec_decode=True, and config.get_layer_quant_config has no mtp.* special case, so layers 37-39 are neither re-quantized nor up/down-cast by ATOM and retain their on-disk dtype. --kv_cache_dtype bf16 and --index-cache-dtype fp8 touch cache storage only, not draft/MTP weights."
Signed: seungrokj
|
|
overriding warning as the points seems reasonably spaced out 
|
3 participants
Summary
Add
dsv41flash-fp4-mi355x-atom-agentic-dsparkfollowing the upstream ATOM TP2 / TP4 recipe, using the integration pattern from #2668.rocm/atom-dev:nightly_202609250902and servedeepseek-ai/DeepSeek-V4.1-Flashoncluster:mi355x-amds. TP2 concurrency:[1, 2, 8, 16, 32, 64]; TP4:[2, 8, 16, 32, 64]. No expert parallelism or KV offload.max-num-seqs=128, 16K batching/prefill chunks, prefix caching with block size 16, 8K state checkpoints, level 3, FULL graphs, anddsml_v41. At c32, capture every size through 32 plus 48/64/128; otherwise use the upstream sparse list.thinking_ongolden AL 3.51 for throughput. Eval-only runs use real acceptance. Canonical AgentX runs replaysemianalysis_cc_traces_weka_062126for 3600 seconds with five warmup requests per lane. Retain the shared harness's explicit duration overrides and fast mode.The existing MI355X launcher routes
draft_modelto the new executable_atom_mtp.shscript, mounts the model cache and repository at/ix, and supplies the GPU allocation. The script preserves that allocation and uses the shared replay, metrics, eval, and cleanup helpers. English/Chinese documentation and the append-only performance changelog are updated.Image and draft
Docker Hub manifest:
sha256:9aaeed59d78decfc51eaabda072ba095c1d64d6cfbdb64ecb28ea74cf3deb18a. Image labels identify ATOM0.1.7.dev35+g4685e3cf7, source revision4685e3cf7da95036126f94749258b9ed3b4cb3ed, and ROCm 7.2.4.Use the checkpoint's embedded DSpark head through the pinned image's default loader. No draft checkpoint replacement, online quantization configuration, draft precision override, or engine patch is applied. The pinned weight loader retains its default draft tensor handling, including
wo_adequantization. These are source/registry checks; GPU execution and checkpoint-loading evidence remain pending.Validation
bash -nandgit diff --checkpassed.EVAL_ONLYfail before server launch.AI model disclosure
Prepared with OpenAI GPT-6 via Codex for implementation, source inspection, and local validation. The exact runtime model/version identifier was not exposed and could not be verified. No delegated agents were used.
中文
概要
参考 #2668 的接入方式，按照 ATOM 上游 TP2 / TP4 配方 新增
dsv41flash-fp4-mi355x-atom-agentic-dspark。rocm/atom-dev:nightly_202609250902，在cluster:mi355x-amds上运行deepseek-ai/DeepSeek-V4.1-Flash。TP2 并发为[1, 2, 8, 16, 32, 64]，TP4 为[2, 8, 16, 32, 64]，不启用专家并行或 KV 卸载。max-num-seqs=128、16K batching/prefill chunk、block size 16 的前缀缓存、8K 状态检查点、level 3、FULL graphs 和dsml_v41。c32 捕获 1–32 的全部尺寸及 48/64/128，其他点使用上游稀疏列表。thinking_ongolden AL 3.51，eval-only 使用真实 acceptance。标准 AgentX 测试回放semianalysis_cc_traces_weka_062126，每点 3600 秒，每条 lane 预热 5 个请求；保留公共 harness 的显式时长覆盖和 fast mode。现有 MI355X launcher 已能将
draft_model路由到新增的可执行_atom_mtp.sh脚本，将模型缓存和仓库挂载至/ix，并传入 GPU 分配。脚本保留该分配，复用公共 replay、metrics、eval 和清理函数；同步更新中英文文档，性能 changelog 仅在末尾追加。镜像与 draft
Docker Hub manifest 为
sha256:9aaeed59d78decfc51eaabda072ba095c1d64d6cfbdb64ecb28ea74cf3deb18a。镜像标签记录 ATOM0.1.7.dev35+g4685e3cf7、源码 revision4685e3cf7da95036126f94749258b9ed3b4cb3ed和 ROCm 7.2.4。通过镜像默认 loader 使用检查点内置的 DSpark head，不替换 draft 检查点、不设置在线量化或 draft 精度覆盖，也不修改 engine。已检查固定版本的 weight loader，保留包括
wo_a反量化在内的默认处理。这些证据来自源码和 registry，GPU 执行及检查点加载证据仍待补充。验证
bash -n、git diff --check、YAML 解析、精确配置及过滤矩阵生成、正式 changelog validator 均通过。两个矩阵都只有预期的 11 个点，时长均为 3600 秒；历史 changelog 字节保持不变。EVAL_ONLY均在启动 server 前失败。AI 模型披露
由 OpenAI GPT-6 通过 Codex 完成实现、源码检查和本地验证。运行环境未暴露确切的模型／版本标识，因此无法核实。未使用委派代理。