Skip to content

Commit 972ef1e

Browse files
functionstackxclaude
functionstackx
and
claude
committed
Add DSV4 MI355X vLLM disagg smoke test (8k1k conc=32)
DeepSeek-V4-Pro disaggregated P/D on MI355X via vLLM + MoRI-IO: new dsv4-fp4-mi355x-vllm-disagg config (1P1D, TP8/EP1, 8k1k conc=32 only), thin multi_node launcher, and DeepSeek-V4-Pro entry in models_vllm.yaml reusing the validated aggregated serving flags. Uses the latest vLLM ROCm nightly (2026-07-04). Refreshes closed #1707 against current main, which already carries the #1585 patch-free MoRI-IO path. 中文：新增 DSV4 MI355X vLLM 分离式冒烟测试（8k1k conc=32）- 新配置 dsv4-fp4-mi355x-vllm-disagg（1P1D、TP8/EP1）、轻量多节点启动脚本，以及 models_vllm.yaml 中复用已验证聚合配方 serving 参数的 DeepSeek-V4-Pro 条目；采用最新 vLLM ROCm nightly 镜像。基于当前 main（已含 #1585 的 免补丁 MoRI-IO 路径）重制已关闭的 #1707。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent 78085ac commit 972ef1e

4 files changed

Lines changed: 173 additions & 0 deletions

File tree

‎benchmarks/multi_node/amd_utils/models_vllm.yaml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,3 +59,27 @@ MiniMax-M3-MXFP8:
5959
decode_flags: "--tensor-parallel-size 8 --block-size 128 --language-model-only --kv-cache-dtype fp8 --attention-backend TRITON_ATTN --no-enable-prefix-caching --gpu-memory-utilization 0.90 --tool-call-parser minimax_m3 --reasoning-parser minimax_m3 --enable-auto-tool-choice"
6060
env: "VLLM_USE_V1=1 VLLM_ROCM_USE_AITER=1 VLLM_USE_BREAKABLE_CUDAGRAPH=0 VLLM_ENGINE_READY_TIMEOUT_S=3600"
6161
hf_dir: "models--MiniMaxAI--MiniMax-M3-MXFP8"
62+
63+
DeepSeek-V4-Pro:
64+
# DeepSeek-V4-Pro is mixed-precision FP4+FP8 (FP4 MoE expert weights dominate
65+
# the ~960 GB footprint; FP8 on attention/norm/router; FP8 KV cache at runtime).
66+
# InferenceX classifies this as the fp4 variant.
67+
#
68+
# Serving flags reuse the validated single-node MI355X recipe
69+
# (benchmarks/single_node/fixed_seq_len/dsv4_fp4_mi355x_vllm.sh, from
70+
# vllm-project/recipes#433) so the per-node engine config is identical to the
71+
# known-good aggregated run; disaggregation only adds the MoRIIO kv-transfer
72+
# role (injected by server_vllm.sh). Each P/D worker is a full TP=8 node, EP=1
73+
# — matching the aggregated recipe, which runs DSv4 on TP=8 without expert
74+
# parallelism. DEP decode is a follow-up.
75+
#
76+
# --moe-backend triton_unfused is REQUIRED for the FP4 MoE expert weight format;
77+
# the auto backend doesn't register the FP4 scale params and safetensors load
78+
# raises KeyError. --enforce-eager (no CUDA graphs) keeps the first disagg recipe
79+
# robust against cudagraph/MoRIIO-hook interactions; FULL/PIECEWISE capture is a
80+
# follow-up. --async-scheduling is intentionally omitted (not used by the kimi /
81+
# minimax vllm-disagg recipes).
82+
prefill_flags: "--tensor-parallel-size 8 --distributed-executor-backend mp --kv-cache-dtype fp8 --moe-backend triton_unfused --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4 --no-enable-prefix-caching --gpu-memory-utilization 0.9 --max-num-batched-tokens 8192 --enforce-eager"
83+
decode_flags: "--tensor-parallel-size 8 --distributed-executor-backend mp --kv-cache-dtype fp8 --moe-backend triton_unfused --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4 --no-enable-prefix-caching --gpu-memory-utilization 0.9 --max-num-batched-tokens 8192 --enforce-eager"
84+
env: "VLLM_USE_V1=1 VLLM_ROCM_USE_AITER=1 VLLM_ENGINE_READY_TIMEOUT_S=3600"
85+
hf_dir: "models--deepseek-ai--DeepSeek-V4-Pro"

‎benchmarks/multi_node/dsv4_fp4_mi355x_vllm-disagg.sh‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
#!/usr/bin/env bash
2+
3+
# DeepSeek-V4-Pro disaggregated prefill/decode on MI355X via vLLM + MoRI-IO.
4+
# Thin, model-agnostic launcher: all serving flags / env live in the
5+
# DeepSeek-V4-Pro entry of amd_utils/models_vllm.yaml, and topology (P/D node
6+
# counts, TP/EP) comes from amd-master.yaml. Identical in shape to the kimi /
7+
# minimax vllm-disagg wrappers; only the model differs.
8+
9+
source "$(dirname "$0")/../benchmark_lib.sh"
10+
11+
check_env_vars \
12+
CONC_LIST \
13+
ISL \
14+
OSL \
15+
IMAGE \
16+
SPEC_DECODING \
17+
MODEL_PATH \
18+
PREFILL_NUM_WORKERS \
19+
PREFILL_TP \
20+
PREFILL_EP \
21+
PREFILL_DP_ATTN \
22+
DECODE_NUM_WORKERS \
23+
DECODE_TP \
24+
DECODE_EP \
25+
DECODE_DP_ATTN \
26+
PREFILL_NODES \
27+
DECODE_NODES \
28+
RANDOM_RANGE_RATIO \
29+
FRAMEWORK
30+
31+
if [[ -n "$SLURM_JOB_ID" ]]; then
32+
echo "JOB $SLURM_JOB_ID running on $SLURMD_NODENAME"
33+
fi
34+
35+
set -x
36+
37+
cd "$GITHUB_WORKSPACE/benchmarks/multi_node/amd_utils" || exit 1
38+
39+
export TIME_LIMIT="08:00:00"
40+
export MODEL_PATH=$MODEL_PATH
41+
export MODEL_NAME=$MODEL_NAME
42+
export CONTAINER_IMAGE=$IMAGE
43+
44+
# Same EP/DP booleans as kimik2.5_fp4_mi355x_vllm-disagg.sh → amd_utils/submit.sh
45+
if [[ "${PREFILL_EP:-1}" -eq 1 ]]; then
46+
export PREFILL_ENABLE_EP=false
47+
else
48+
export PREFILL_ENABLE_EP=true
49+
fi
50+
51+
if [[ "$PREFILL_DP_ATTN" == "true" ]]; then
52+
export PREFILL_ENABLE_DP=true
53+
else
54+
export PREFILL_ENABLE_DP=false
55+
fi
56+
57+
if [[ "${DECODE_EP:-1}" -eq 1 ]]; then
58+
export DECODE_ENABLE_EP=false
59+
else
60+
export DECODE_ENABLE_EP=true
61+
fi
62+
63+
if [[ "$DECODE_DP_ATTN" == "true" ]]; then
64+
export DECODE_ENABLE_DP=true
65+
else
66+
export DECODE_ENABLE_DP=false
67+
fi
68+
69+
# Parameter order matches SGLang disagg submit.sh; arg 16 is optional NODELIST.
70+
JOB_ID=$(bash ./submit.sh $PREFILL_NODES \
71+
$PREFILL_NUM_WORKERS \
72+
$DECODE_NODES \
73+
$DECODE_NUM_WORKERS \
74+
$ISL $OSL "${CONC_LIST// /x}" inf \
75+
${PREFILL_ENABLE_EP} ${PREFILL_ENABLE_DP} \
76+
${DECODE_ENABLE_EP} ${DECODE_ENABLE_DP} \
77+
${PREFILL_TP} ${DECODE_TP} \
78+
${RANDOM_RANGE_RATIO} \
79+
"${NODELIST:-}")
80+
81+
if [[ $? -ne 0 ]]; then
82+
echo "Failed to submit job" >&2
83+
exit 1
84+
fi
85+
86+
echo "$JOB_ID"

‎configs/amd-master.yaml‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1326,6 +1326,60 @@ kimik2.5-fp4-mi355x-vllm-disagg:
13261326
additional-settings:
13271327
- "DECODE_NODES=2"
13281328

1329+
# DeepSeek-V4-Pro disaggregated prefill/decode on MI355X via vLLM + MoRI-IO.
1330+
# Combines the validated single-node DSv4 vLLM serving recipe
1331+
# (dsv4-fp4-mi355x-vllm, vllm-project/recipes#433) with the vLLM-disagg
1332+
# framework (the kimi / minimax vllm-disagg recipes). Routes to
1333+
# benchmarks/multi_node/dsv4_fp4_mi355x_vllm-disagg.sh; per-node serving flags
1334+
# + env live in the DeepSeek-V4-Pro entry of amd_utils/models_vllm.yaml.
1335+
#
1336+
# DeepSeek-V4-Pro is FP4+FP8 mixed (FP4 MoE expert weights, FP8 for the rest);
1337+
# InferenceX classifies this as fp4 — same as dsv4-fp4-mi355x-{sglang,vllm,atom}.
1338+
#
1339+
# Image: latest vLLM ROCm nightly (2026-07-04). The patch-free MoRI-IO path
1340+
# (vllm#40344) and the DSv4 model class (vllm#40871) are both long since in
1341+
# nightlies; the release tags still trail on #40344, hence nightly.
1342+
#
1343+
# Topology 1P1D: each prefill/decode worker is a full TP=8 node (EP=1),
1344+
# matching the aggregated recipe which runs DSv4 on TP=8 without expert
1345+
# parallelism (no all2all backend needed). DEP decode + 1P2D are follow-ups
1346+
# once the base disagg path validates.
1347+
dsv4-fp4-mi355x-vllm-disagg:
1348+
image: vllm/vllm-openai-rocm:nightly-f329ce405b12623fb8b1cf1830f12e5a712523be
1349+
model: deepseek-ai/DeepSeek-V4-Pro
1350+
model-prefix: dsv4
1351+
runner: mi355x-disagg
1352+
precision: fp4
1353+
framework: vllm-disagg
1354+
multinode: true
1355+
disagg: true
1356+
scenarios:
1357+
fixed-seq-len:
1358+
# Smoke test: a single ISL/OSL (8k/1k) at a single concurrency (conc=32) to
1359+
# validate the DSv4 vLLM-disagg path (image, MoRIIO transport, serving
1360+
# flags, model staging) end-to-end before expanding to the full 1k1k + 8k1k
1361+
# conc sweep that the kimi / minimax vllm-disagg recipes run.
1362+
- isl: 8192
1363+
osl: 1024
1364+
search-space:
1365+
# 1P1D: 1 prefill node (co-located with proxy) + 1 decode node = 2 nodes
1366+
- spec-decoding: "none"
1367+
conc-list: [ 32 ]
1368+
prefill:
1369+
num-worker: 1
1370+
tp: 8
1371+
ep: 1
1372+
dp-attn: false
1373+
additional-settings:
1374+
- "PREFILL_NODES=1"
1375+
decode:
1376+
num-worker: 1
1377+
tp: 8
1378+
ep: 1
1379+
dp-attn: false
1380+
additional-settings:
1381+
- "DECODE_NODES=1"
1382+
13291383
dsr1-fp4-mi355x-sglang-disagg:
13301384
image: lmsysorg/sglang-rocm:v0.5.12-rocm720-mi35x-20260519
13311385
model: amd/DeepSeek-R1-0528-MXFP4-v2

‎perf-changelog.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4455,3 +4455,12 @@
44554455
description:
44564456
- "Update SGLang image from qwen3.5-fp8-h200-sglang: lmsysorg/sglang:v0.5.12-cu130 / qwen3.5-fp8-h200-sglang-mtp: lmsysorg/sglang:v0.5.12 to lmsysorg/sglang:v0.5.14-cu130"
44574457
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2061
4458+
4459+
- config-keys:
4460+
- dsv4-fp4-mi355x-vllm-disagg
4461+
description:
4462+
- "New recipe: DeepSeek-V4-Pro disaggregated prefill/decode on MI355X via vLLM + MoRI-IO, combining the validated single-node DSv4 vLLM serving recipe (dsv4-fp4-mi355x-vllm, vllm-project/recipes#433) with the existing vllm-disagg framework (MoRI-IO P/D, standalone router)."
4463+
- "New benchmarks/multi_node/dsv4_fp4_mi355x_vllm-disagg.sh launcher + DeepSeek-V4-Pro entry in amd_utils/models_vllm.yaml. Per-node flags reuse the aggregated recipe verbatim (--moe-backend triton_unfused for the FP4 expert format, deepseek_v4 tokenizer/reasoning parser, fp8 KV cache, --enforce-eager); the framework adds the MoRIIO kv-transfer role with the patch-free read_mode path already on main."
4464+
- "Image: latest vLLM ROCm nightly vllm/vllm-openai-rocm:nightly-f329ce405b12623fb8b1cf1830f12e5a712523be (2026-07-04)."
4465+
- "1P1D topology, TP8/EP1 prefill+decode. Smoke test: single ISL/OSL (8k/1k) at conc=32 to validate the path end-to-end before expanding to the full sweep."
4466+
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2081

0 commit comments

Comments
 (0)