Skip to content

port MI355X DeepSeek V4 ATOM disagg to srt-slurm - #2627

Draft
cquil11 wants to merge 5 commits into
agent/atom-infera-validationfrom
agent/port-dsv4-mi355x-atom-srt-slurm
Draft

port MI355X DeepSeek V4 ATOM disagg to srt-slurm#2627
cquil11 wants to merge 5 commits into
agent/atom-infera-validationfrom
agent/port-dsv4-mi355x-atom-srt-slurm

Conversation

@cquil11

@cquil11 cquil11 commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

Ports the active dsv4-fp4-mi355x-atom-disagg 8k/1k submission from the configuration-specific AMD launcher to the ATOM/Infera implementation in SemiAnalysisAI/srt-slurm#3.

Scope

  • preserve the original 1P1D TP8 points: c4, c8, c16, c32, c64, c128
  • preserve the original 2P1D DPA+TP8 points: c256, c512, c768, c1024, c2048
  • run the unchanged InferenceX utils/bench_serving/benchmark_serving.py through the custom benchmark contract
  • use Infera discovery and KV-aware routing with ATOM KV events
  • use the hardware-proven Mooncake TCP path initially; RDMA validation follows with the updated ATOM/Mooncake runtime
  • remove the exact legacy wrapper and obsolete deprecated 1k/1k entry that depended on it

Dependency structure

Local validation

  • both recipes pass srtctl dry-run
  • generated InferenceX matrix contains exactly two topology jobs
  • master config validation passes
  • 171 InferenceX matrix/runner/gating tests pass
  • 13 targeted srt-slurm AMD/ATOM/Infera tests pass
  • YAML parsing, shell syntax, and git diff --check pass

Hardware validation

WIP. Exact workflow links and Slurm/artifact evidence will be added here as the full sweep runs.

@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs


感谢你的贡献!请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人,你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板,包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have

如需进行 PR 验证,请为此 PR 添加 full-sweep-fail-fast 标签(强烈推荐)— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动(flake),重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

@cquil11

cquil11 commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator Author

Live 2P/1D DPA validation update:\n\n- Slurm 39443 completed all 64 model shards and CUDA-graph capture on both prefills and the decode.\n- Startup then failed deterministically in every logical worker because each ATOM data-parallel EngineCore tried to bind the same KV-event ZMQ endpoint. One DP rank bound successfully; the others raised Address already in use from Infera BlockManager hook initialization, so the worker could not register.\n- This is an Infera multi-DP KV-event publisher bug, not Slurm, MI355X hardware, model loading, Mooncake, or the srt-slurm topology.\n- The upstream fix is open as AMD-AGI/Infera#121. It gives every DP rank a local PUB ingress and relays all rank streams through the one endpoint advertised for the logical worker, retaining complete KV-aware cache visibility.\n- Focused engine/router validation is green: 50 tests passed, including two simultaneous EngineCore publishers through one advertised endpoint; Ruff, pre-commit, and diff checks passed.\n- The current exact-head workflow was not cancelled. Its 1P/1D arm is still running naturally as Slurm 39448. I will pin the reviewed fix and launch a fresh exact-head validation only after this attempt closes.

Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

1 participant