Skip to content

Commit bc4ad29

Browse files
YukioZzzYukioZzz
committed
fix: account MPND workers by role TP size
1 parent f58f704 commit bc4ad29

4 files changed

Lines changed: 305 additions & 1 deletion

File tree

‎.github/configs/amd-kimi-mi355x-mpnd-normal.yaml‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
kimik2.5-fp4-mi355x-vllm-disagg-mpnd-normal:
2+
image: vllm/vllm-openai-rocm:nightly-09663abde0f50944a8d5ea30120666024b503faa
3+
model: amd/Kimi-K2.5-MXFP4
4+
model-prefix: kimik2.5
5+
runner: mi355x-disagg
6+
precision: fp4
7+
framework: vllm-disagg
8+
multinode: true
9+
disagg: true
10+
scenarios:
11+
fixed-seq-len:
12+
- isl: 1024
13+
osl: 1024
14+
search-space:
15+
- spec-decoding: "none"
16+
conc-list: [ 512, 1024 ]
17+
prefill:
18+
num-worker: 1
19+
tp: 4
20+
ep: 4
21+
dp-attn: false
22+
additional-settings:
23+
- "PREFILL_NODES=1"
24+
- "VLLM_MORIIO_CONNECTOR_READ_MODE=1"
25+
decode:
26+
num-worker: 1
27+
tp: 8
28+
ep: 8
29+
dp-attn: false
30+
additional-settings:
31+
- "DECODE_NODES=1"
32+
- "VLLM_MORIIO_CONNECTOR_READ_MODE=1"
33+
- spec-decoding: "none"
34+
conc-list: [ 1024, 2048 ]
35+
prefill:
36+
num-worker: 1
37+
tp: 4
38+
ep: 4
39+
dp-attn: false
40+
additional-settings:
41+
- "PREFILL_NODES=1"
42+
- "VLLM_MORIIO_CONNECTOR_READ_MODE=1"
43+
decode:
44+
num-worker: 2
45+
tp: 8
46+
ep: 8
47+
dp-attn: false
48+
additional-settings:
49+
- "DECODE_NODES=2"
50+
- "VLLM_MORIIO_CONNECTOR_READ_MODE=1"
51+
52+
- isl: 8192
53+
osl: 1024
54+
search-space:
55+
- spec-decoding: "none"
56+
conc-list: [ 512, 1024 ]
57+
prefill:
58+
num-worker: 2
59+
tp: 4
60+
ep: 4
61+
dp-attn: false
62+
additional-settings:
63+
- "PREFILL_NODES=2"
64+
- "VLLM_MORIIO_CONNECTOR_READ_MODE=1"
65+
decode:
66+
num-worker: 1
67+
tp: 8
68+
ep: 8
69+
dp-attn: false
70+
additional-settings:
71+
- "DECODE_NODES=1"
72+
- "VLLM_MORIIO_CONNECTOR_READ_MODE=1"
73+
- spec-decoding: "none"
74+
conc-list: [ 1024 ]
75+
prefill:
76+
num-worker: 4
77+
tp: 4
78+
ep: 4
79+
dp-attn: false
80+
additional-settings:
81+
- "PREFILL_NODES=4"
82+
- "VLLM_MORIIO_CONNECTOR_READ_MODE=1"
83+
decode:
84+
num-worker: 1
85+
tp: 8
86+
ep: 8
87+
dp-attn: false
88+
additional-settings:
89+
- "DECODE_NODES=1"
90+
- "VLLM_MORIIO_CONNECTOR_READ_MODE=1"

‎benchmarks/multi_node/amd_utils/job.slurm‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -451,6 +451,19 @@ echo \"Rank \$SLURM_PROCID on \$(hostname)\"
451451
eval \"\$DOCKER_CMD_DETECT\"
452452
echo \"[docker-detect] rank \$SLURM_PROCID: DOCKER_CMD=\$DOCKER_CMD\"
453453
454+
# Role-local GPU visibility. The allocation can still be whole-node, but MPND
455+
# prefill workers such as TP4 should only see four GPUs inside the container.
456+
_ROLE_TP_SIZE=\"\$DECODE_TP_SIZE\"
457+
if (( SLURM_PROCID < xP )); then
458+
_ROLE_TP_SIZE=\"\$PREFILL_TP_SIZE\"
459+
fi
460+
if (( _ROLE_TP_SIZE < 1 || _ROLE_TP_SIZE > GPUS_PER_NODE )); then
461+
echo \"ERROR: role TP size \$_ROLE_TP_SIZE must be in [1, \$GPUS_PER_NODE] on \$(hostname)\" >&2
462+
exit 1
463+
fi
464+
_ROLE_VISIBLE_DEVICES=\"\$(seq -s, 0 \$(( _ROLE_TP_SIZE - 1 )))\"
465+
echo \"[gpu-visible] rank=\$SLURM_PROCID role_tp_size=\$_ROLE_TP_SIZE visible_devices=\$_ROLE_VISIBLE_DEVICES\"
466+
454467
# Enable out-of-tree RDMA library mounts for atom-disagg (mooncake requires host RDMA stack)
455468
RDMA_MOUNTS=()
456469
if [[ "$ENGINE" == "atom-disagg" ]]; then
@@ -598,6 +611,9 @@ fi
598611
\${RDMA_MOUNTS[@]+"\${RDMA_MOUNTS[@]}"} \
599612
${DOCKER_ENV_COMMON[*]} \
600613
${DOCKER_ENV_ENGINE[*]} \
614+
-e HIP_VISIBLE_DEVICES=\"\$_ROLE_VISIBLE_DEVICES\" \
615+
-e ROCR_VISIBLE_DEVICES=\"\$_ROLE_VISIBLE_DEVICES\" \
616+
-e CUDA_VISIBLE_DEVICES=\"\$_ROLE_VISIBLE_DEVICES\" \
601617
--name \"$DOCKER_CONT_NAME\" \
602618
--entrypoint \"\" \
603619
\"$DOCKER_IMAGE_NAME\" bash -lc '

‎benchmarks/multi_node/amd_utils/server_vllm.sh‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -441,7 +441,10 @@ PY
441441
cd $WS_PATH
442442

443443
export ROUTER_PORT=$ROUTER_PORT
444-
BENCH_CMD="bash $WS_PATH/bench.sh ${xP} ${yD} $((GPUS_PER_NODE*xP)) $((GPUS_PER_NODE*yD)) \
444+
PREFILL_BENCH_GPUS=$((PREFILL_TP_SIZE * xP))
445+
DECODE_BENCH_GPUS=$((DECODE_TP_SIZE * yD))
446+
echo "Benchmark GPU accounting: prefill=${PREFILL_BENCH_GPUS} decode=${DECODE_BENCH_GPUS} (TP-sized workers)"
447+
BENCH_CMD="bash $WS_PATH/bench.sh ${xP} ${yD} ${PREFILL_BENCH_GPUS} ${DECODE_BENCH_GPUS} \
445448
$MODEL_DIR $MODEL_NAME /run_logs/slurm_job-${SLURM_JOB_ID} ${BENCH_INPUT_LEN} \
446449
${BENCH_OUTPUT_LEN} \"${BENCH_MAX_CONCURRENCY}\" ${BENCH_REQUEST_RATE} \
447450
${BENCH_RANDOM_RANGE_RATIO} ${BENCH_NUM_PROMPTS_MULTIPLIER}"

‎kimi_mi355x_vllm_disagg_notes.md‎

Lines changed: 195 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,195 @@
1+
# Kimi K2.5 MI355X vLLM Disagg Notes
2+
3+
## Goal
4+
5+
Make `kimik2.5-fp4-mi355x-vllm-disagg` beat the single-node MI355X baseline on per-GPU throughput, not just total throughput from using more GPUs.
6+
7+
## B200/GB200 Recipe Lessons
8+
9+
The B200/GB200 Kimi disagg recipes use worker counts, not 8 GPUs per `P` or `D`.
10+
11+
Examples from `benchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k2.5-fp4`:
12+
13+
- `1p1d dep4-dep8`: prefill = 1 worker x 4 GPUs, decode = 1 worker x 8 GPUs.
14+
- `1p1d dep4-dep16`: prefill = 1 worker x 4 GPUs, decode = 1 worker x 16 GPUs.
15+
- `3p1d dep4-dep16`: prefill = 3 workers x 4 GPUs, decode = 1 worker x 16 GPUs.
16+
- `6p1d dep4-dep16`: prefill = 6 workers x 4 GPUs, decode = 1 worker x 16 GPUs.
17+
- `8p1d dep4-dep16`: prefill = 8 workers x 4 GPUs, decode = 1 worker x 16 GPUs.
18+
19+
So the useful lesson is the P:D GPU ratio and worker granularity, not the literal P/D count.
20+
21+
## MI355X Mapping
22+
23+
Current MI355X launcher appears node-granular: one worker normally consumes an 8-GPU MI355X node.
24+
25+
The first sub-node slicing layer is implemented by role-local visibility in
26+
`benchmarks/multi_node/amd_utils/job.slurm`: prefill workers see
27+
`0..PREFILL_TP_SIZE-1`, decode workers see `0..DECODE_TP_SIZE-1`. Benchmark
28+
accounting in `server_vllm.sh` now reports `PREFILL_TP_SIZE * xP` and
29+
`DECODE_TP_SIZE * yD`, instead of assuming 8 GPUs per worker.
30+
31+
With sub-node GPU slicing via `HIP_VISIBLE_DEVICES` / `ROCR_VISIBLE_DEVICES`,
32+
the closest mapping is:
33+
34+
- GB200 `dep4` prefill -> MI355X 4-GPU prefill worker.
35+
- GB200 `dep8` decode -> MI355X 8-GPU decode worker.
36+
- GB200 `dep16` decode -> MI355X 2 x 8-GPU decode workers.
37+
38+
If sub-node slicing is not available, use coarser whole-node sweeps:
39+
40+
- `1k1k`: start with `1P1D` and `1P2D`; this is decode-heavy.
41+
- `8k1k`: start with `2P1D`, then possibly `3P1D`; this is prefill-heavy.
42+
43+
Avoid interpreting `3P1D` as a universally good first target on MI355X. With whole nodes, `3P1D` already means 32 GPUs.
44+
45+
## DP-Attention / FP8 KV Finding
46+
47+
Validated on `mia1-p01-g07`:
48+
49+
- `DP8 + EP8 + TP1 + --kv-cache-dtype fp8` fails.
50+
- The failure enters AITER MLA `mla_a8w8_qh64_qseqlen1_gqaratio64_v3_ps`.
51+
- With vLLM persistent metadata enabled, this hits GPU memory access faults.
52+
- With persistent metadata disabled via a local monkey patch, AITER reports:
53+
`fp8/fp8 with gqa_ratio=64 only supports decode_qlen=1 in persistent mode`.
54+
- `DP8 + EP8 + TP1` with auto/bf16 KV starts successfully and returns a chat completion.
55+
56+
Therefore the current safe recipe rule is:
57+
58+
- DP-attn can be used for high-concurrency sweeps.
59+
- DP-attn must strip `--kv-cache-dtype fp8` and run auto/bf16 KV on MI355X.
60+
- Keep TP8 + fp8 KV as a separate non-DP baseline for low/mid concurrency.
61+
62+
## Candidate MI355X Recipe Sweep
63+
64+
Minimal first pass before DP-attn is stable:
65+
66+
1. `1k1k`, `1P1D`, prefill TP4/EP4 + decode TP8/EP8, conc `[512, 1024]`.
67+
2. `1k1k`, `1P2D`, prefill TP4/EP4 + decode TP8/EP8, conc `[1024, 2048]`.
68+
3. `8k1k`, `2P1D`, prefill TP4/EP4 + decode TP8/EP8, conc `[512, 1024]`.
69+
4. `8k1k`, `4P1D`, prefill TP4/EP4 + decode TP8/EP8, conc `[1024]`.
70+
71+
The corresponding normal-only CI config is
72+
`.github/configs/amd-kimi-mi355x-mpnd-normal.yaml`. It excludes DP-attn rows so
73+
the run can finish green and produce ingestible artifacts.
74+
75+
Older whole-node fallback if sub-node slicing regresses:
76+
77+
1. `1k1k`, `1P1D`, TP8 prefill + TP8/EP8 decode, conc `[512, 1024]`.
78+
2. `1k1k`, `1P2D`, TP8 prefill + TP8/EP8 decode, conc `[1024, 2048]`.
79+
3. `1k1k`, `1P2D`, DP8/EP8 auto-KV, conc `[1024, 2048]`.
80+
4. `8k1k`, `2P1D`, TP8 prefill + TP8/EP8 decode, conc `[512, 1024]`.
81+
5. `8k1k`, `3P1D`, TP8 prefill + TP8/EP8 decode, conc `[1024, 2048]`.
82+
6. `8k1k`, `3P1D`, DP8/EP8 auto-KV, conc `[1024, 2048]`.
83+
84+
If sub-node 4-GPU workers become available, prefer matching GB200 ratios more directly:
85+
86+
- `1k1k`: P=4G, D=8G or 16G.
87+
- `8k1k`: P=3x4G, D=16G.
88+
89+
## CI Run 29200792444 / 29200792335 Diagnosis
90+
91+
Both pinned Kimi K2.5 MPND runs produced successful normal heterogeneous TP
92+
jobs and failed only on DP-attn comparison jobs.
93+
94+
Normal successful rows were present in artifacts:
95+
96+
- `29200792444` (`1k1k`): TP4/EP4 prefill + TP8/EP8 decode succeeded for
97+
`1P1D` and `1P2D`.
98+
- `29200792335` (`8k1k`): TP4/EP4 prefill + TP8/EP8 decode succeeded for
99+
`2P1D` and `4P1D`.
100+
101+
The missing 1k1k rows in the unofficial UI were not because artifacts were
102+
absent. `collect-results` succeeded, but the whole workflow conclusion was
103+
failure because DP-attn jobs failed, so the unofficial ingestion path likely
104+
ignored or did not refresh partial data from the failed run.
105+
106+
The old artifacts also show why the first-layer accounting fix is needed:
107+
108+
- `1k1k 1P1D` reported `num_prefill_gpu=8`, but should be 4 for TP4.
109+
- `1k1k 1P2D` reported `num_prefill_gpu=8`, but should be 4 for TP4.
110+
- `8k1k 2P1D` reported `num_prefill_gpu=16`, but should be 8 for 2 x TP4.
111+
- `8k1k 4P1D` reported `num_prefill_gpu=32`, but should be 16 for 4 x TP4.
112+
113+
DP-attn failures in those runs are separate from heterogeneous TP. Server and
114+
MoRI proxy readiness completed, but benchmark reported:
115+
116+
```text
117+
Successful requests: 0
118+
FAIL: request failure rate 100.0% exceeds 5% threshold (0/10240 completed)
119+
```
120+
121+
Those failed jobs did not preserve per-request error bodies, so the next DP-attn
122+
debug pass should enable detailed benchmark output or server access/error logs.
123+
124+
## B200 Parameters Worth Adapting
125+
126+
Applicable ideas:
127+
128+
- Prefill: `enforce-eager=true`.
129+
- Decode: `FULL_DECODE_ONLY` cudagraph.
130+
- Disable prefix caching for fixed-seq throughput sweeps.
131+
- Consider disabling chunked prefill for fixed-seq disagg.
132+
- Split `max-num-seqs` by workload:
133+
- `1k1k`: larger, e.g. 512/1024.
134+
- `8k1k` prefill: smaller, e.g. 64/128.
135+
- `8k1k` decode: 256/512.
136+
- Tune decode `max-cudagraph-capture-size` separately, starting with 256/512.
137+
- Increase frontend/router capacity at high concurrency. B200/GB200 recipes often enable multiple frontends for high-throughput cases.
138+
139+
Not directly portable to MI355X:
140+
141+
- `FLASHINFER_MLA`.
142+
- `flashinfer_nvlink_one_sided`.
143+
- NVIDIA NCCL MNNVL/NVLS knobs.
144+
- NIXL connector assumptions.
145+
146+
## CI Run 28083945960 Diagnosis
147+
148+
Run: `https://github.com/SemiAnalysisAI/InferenceX/actions/runs/28083945960`
149+
150+
Job `83145269559` is `multi-node eval`, not benchmark:
151+
152+
- The job name includes `eval-only`.
153+
- `benchmark-multinode-tmpl.yml` skips benchmark result checks when `inputs.eval-only == true`.
154+
- The log says `EVAL_ONLY mode: skipping throughput benchmark`.
155+
- It then runs `lm_eval` on `utils/evals/gsm8k.yaml`.
156+
- The job passed GSM8K:
157+
- strict match: `0.9310`
158+
- flexible extract: `0.9431`
159+
160+
Most failed jobs did not run servers or benchmarks. They failed during `actions/checkout` cleanup:
161+
162+
```text
163+
File was unable to be removed Error: EACCES: permission denied,
164+
rmdir '/it-share/gharunners2/gharunner06/actions-runner/_work/InferenceX/InferenceX/LOGS/agentic'
165+
```
166+
167+
This is a stale root-owned or otherwise non-runner-owned workspace artifact. Because checkout runs before repo code is available, the existing pre/post launch cleanup in runner scripts cannot fix this class of failure.
168+
169+
The successful eval job also shows workspace permission problems after Slurm completion:
170+
171+
```text
172+
cp .../eval_results/meta_env.json .../InferenceX/meta_env.json: Permission denied
173+
cp .../eval_results/results_*.json .../InferenceX/results_*.json: Permission denied
174+
cp .../eval_results/samples_gsm8k_*.jsonl .../InferenceX/samples_*.jsonl: Permission denied
175+
```
176+
177+
Despite those errors, the result file was present later and upload/verification passed. The copy loop is misleading because it echoes "Copied" even when `cp` failed.
178+
179+
## CI Cleanup Recommendations
180+
181+
1. Add a pre-checkout cleanup step in the reusable workflow before `actions/checkout`, using `sudo rm -rf` for known stale paths:
182+
- `$GITHUB_WORKSPACE/LOGS`
183+
- `$GITHUB_WORKSPACE/benchmark_logs`
184+
- `$GITHUB_WORKSPACE/benchmark_artifacts`
185+
- root-level `results*.json`, `samples*.jsonl`, `meta_env.json`
186+
187+
2. Fix eval artifact extraction in `runners/launch_mi355x-amds.sh`:
188+
- Check `cp` return codes before printing "Copied".
189+
- If copying into `$GITHUB_WORKSPACE` needs sudo due to ownership drift, use `sudo cp` followed by `sudo chown "$USER":"$USER"` on copied artifacts.
190+
191+
3. Ensure containers do not write under `$GITHUB_WORKSPACE` as root where possible. Prefer `benchmark_logs` and then copy artifacts back with normalized ownership.
192+
193+
4. Manual immediate cleanup for current runners:
194+
- On affected runner hosts, remove stale workspace paths with sudo:
195+
`sudo rm -rf /it-share/gharunners2/gharunner*/actions-runner/_work/InferenceX/InferenceX/LOGS`

0 commit comments

Comments
 (0)