[AMD] Optimize kimik2.5-fp4-mi355x-vllm-disagg: image bump + serve sync + all-TP4 P/D (EP off) / 优化 kimik2.5-fp4-mi355x-vllm-disagg：镜像升级 + serve 参数对齐 + 全 TP4 P/D（关闭 EP）

[hongxiayang]

Description

Retunes the kimik2.5-fp4-mi355x-vllm-disagg disaggregated config for MI355X, and fixes two
bugs that were failing the sweep. Every engine change below was A/B'd locally on real weights
(gfx950, TP4, 8k/1k, on the pinned CI image) before being adopted.

Recipe (configs/amd-master.yaml, amd_utils/models_vllm.yaml)

• Image bump to vllm/vllm-openai-rocm:nightly-2afa3f7e950264bb179d030c23a1ed1f46558fd9.

• All-TP4 P/D layout (TP8 dropped) — a real-weight sweep showed TP8 decode gives no per-GPU
  benefit over TP4. 1P(TP4)/1D(TP4) covers the full curve; 1P(TP4)/2D(TP4) adds decode KV
  headroom for the KV-bound 8k/1k tail (conc 256/512). All layouts keep prefill+decode ≤ 3 nodes.

• Expert parallelism off (ep:1) — a single-node TP8 real-weight sweep showed EP is 14–27% slower.

• Serve-flag/env sync with the single-node recipe (VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4,
  HSA_NO_SCRATCH_RECLAIM=1, AITER defaults).

• Dropped the cudagraph_mode: PIECEWISE pin (carried over from bring-up [AMD] vLLM Kimi MXFP4 & MiniMax M2.5 FP8 disaggregated prefill-decode for MI355X #1569), taking vLLM's
  default FULL_AND_PIECEWISE as the single-node recipe does. Measured vs the PIECEWISE pin, output tok/s:

  conc	1	4	16	64
  FULL_AND_PIECEWISE	+15.9%	+11.1%	+12.6%	+1.7%
  FULL_DECODE_ONLY	+18.5%	+13.6%	+11.9%	+1.5%

  PIECEWISE loses at every concurrency. The two full modes tie within run-to-run noise, so the
  default wins the tiebreak by also matching single-node. The gap decays to ~0 by conc 128.

• Added --kv-cache-dtype fp8, --max-model-len 32768, --max-num-seqs 256,
  --max-num-batched-tokens 32768. Measured vs the flags above, total tok/s:

  conc	16	64	128
  kv fp8 only	+9.7%	+16.1%	+20.5%
  all four (adopted)	+16.2%	+19.1%	+27.4%

  TTFT −18% and TPOT −22% at conc 128. fp8 KV does most of the work on two counts: it halves KV
  read bandwidth (a win at every concurrency, not only when KV-bound) and doubles resident capacity
  (1,358,885 → 2,730,007 tokens), relieving the KV-bound decode tail. No accuracy cost: local
  TP4 GSM8K scored 0.9697 flexible / 0.9712 strict — identical to the bf16-KV run and clear of the
  0.90 gate in utils/evals/thresholds.json.

Bug fix: the multi-node eval never produced results (amd_utils/server_vllm.sh)

server_vllm.sh never set EVAL_MAX_MODEL_LEN, so run_lm_eval fell back to its 16384 default and
derived max_tokens = 16384 − 4096 = 12288 — against engines served with --max-model-len 9472.
Every gsm8k request was rejected. From run 29522769567:

--model_args '...,max_length=16384' --gen_kwargs max_tokens=12288,...
Requesting API:   0%|          | 0/1319
... 4203 x "Retry attempt 1"
ERROR: run_eval exited rc=1; skipping metadata write and eval artifact staging
ERROR: eval failed; exiting node-0 with rc=1

Node-0 exits 1 → "Launch multi-node job script" fails → nothing staged → "Upload eval results" and
"Verify eval scores" fail as consequences, and eval_results_all comes back []. This is why the
throughput jobs passed while both eval arms failed: throughput never touches that path.

Now EVAL_MAX_MODEL_LEN is derived via setup_eval_context (ISL+OSL+256, capped at the model's
native max) and clamped to the engine's served --max-model-len, mirroring the single-node recipe.
A pre-flight probe at the real output budget also runs before the 1319-question eval, so a rejected
token budget fails in seconds with the engine's own error instead of burning ~20 minutes of 3-node
time over 4203 retries. Both fixes apply to every *-vllm-disagg config, not just kimik2.5.

Bug fix: the pinned vllm-router image was garbage-collected (amd_utils/job.slurm)

vllm/vllm-router retains only ~16 nightlies on Docker Hub; the pinned
nightly-20260629-e667ebb now 404s, so rank 0's docker run -d fails with "manifest unknown"
after the allocation is already up. Re-pinned to nightly-20260716-1fbcde7.

Results

Test plan

• cudagraph_mode A/B (PIECEWISE vs FULL_AND_PIECEWISE vs FULL_DECODE_ONLY), 8k/1k, real weights.
• Serve-flag A/B (baseline vs kv fp8 vs all four), 8k/1k, real weights.
• GSM8K with the adopted recipe: 0.9697 flexible / 0.9712 strict vs the 0.90 gate.
• Router image verified live on Docker Hub.
• full-sweep-fail-fast green for kimik2.5-fp4-mi355x-vllm-disagg.
• 8k/1k multi-node eval produces results and clears the GSM8K threshold in CI.

---

中文说明

针对 MI355X 重新调优分离式(disagg)配置 kimik2.5-fp4-mi355x-vllm-disagg，并修复了两个导致扫描
(sweep)失败的缺陷。下述所有引擎参数改动，均已在本地以真权重(gfx950、TP4、8k/1k、CI 固定镜像)
完成 A/B 实测后才予以采用。

配方改动（configs/amd-master.yaml、amd_utils/models_vllm.yaml）

• 镜像升级至 vllm/vllm-openai-rocm:nightly-2afa3f7e950264bb179d030c23a1ed1f46558fd9。
• 全部改为 TP4（移除 TP8）：真权重实测显示 TP8 解码相比 TP4 无每 GPU 收益。1P(TP4)/1D(TP4)
  覆盖完整并发曲线；1P(TP4)/2D(TP4) 为受 KV 限制的 8k/1k 高并发尾部（并发 256/512）提供额外
  解码 KV 容量。所有布局的预填充+解码节点数 ≤ 3。
• 关闭专家并行(ep:1)：单节点 TP8 实测显示 EP 相比 dense 慢 14%~27%。
• serve 参数/环境变量对齐单节点配方（VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4、
  HSA_NO_SCRATCH_RECLAIM=1、AITER 默认值）。
• 移除 cudagraph_mode: PIECEWISE 固定值（自 bring-up [AMD] vLLM Kimi MXFP4 & MiniMax M2.5 FP8 disaggregated prefill-decode for MI355X #1569 沿用至今），改用 vLLM 默认的
  FULL_AND_PIECEWISE，与单节点配方一致。相较 PIECEWISE，输出吞吐提升见上表：PIECEWISE 在所有
  并发点均落后；两种 full 模式的差异在运行间噪声范围内，因此默认值凭借"同时与单节点保持一致"
  胜出。该优势在并发 128 时衰减至约 0。
• 新增 --kv-cache-dtype fp8、--max-model-len 32768、--max-num-seqs 256、
  --max-num-batched-tokens 32768：相较上述参数，总吞吐在并发 16/64/128 分别提升
  +16.2%/+19.1%/+27.4%，并发 128 时 TTFT 降低 18%、TPOT 降低 22%。其中 fp8 KV 贡献最大，
  原因有二：将 KV 读取带宽减半（在所有并发下均有收益，而非仅在受 KV 限制时），以及将常驻容量
  翻倍（1,358,885 → 2,730,007 tokens），从而缓解受 KV 限制的解码尾部。精度无损失：本地 TP4
  GSM8K 得分 0.9697（flexible）/ 0.9712（strict），与 bf16 KV 结果一致，且远高于
  utils/evals/thresholds.json 中 0.90 的门槛。

缺陷修复：多节点评估始终无法产出结果（amd_utils/server_vllm.sh）

server_vllm.sh 从未设置 EVAL_MAX_MODEL_LEN，导致 run_lm_eval 回退到默认的 16384，并推导出
max_tokens = 16384 − 4096 = 12288——而引擎实际以 --max-model-len 9472 启动，因此每个 gsm8k
请求都被拒绝（详见上方英文部分引用的 run 29522769567 日志）。node-0 以 rc=1 退出，导致
"Launch multi-node job script" 步骤失败、评估产物未暂存，"Upload eval results" 与
"Verify eval scores" 随之连带失败，eval_results_all 返回 []。这也解释了为何吞吐任务全部通过
而两个评估任务均失败：吞吐路径根本不会走到这段代码。

现在改为通过 setup_eval_context 推导 EVAL_MAX_MODEL_LEN（ISL+OSL+256，上限为模型原生最大值），
并按引擎实际提供的 --max-model-len 进行钳制，与单节点配方保持一致。此外，在执行 1319 道题的
完整评估之前，会先以真实输出预算发送一次预检请求：一旦 token 预算被拒绝，可在数秒内带着引擎自身
的错误信息失败，而不是在 4203 次重试中白白消耗约 20 分钟的三节点机时。以上两项修复对所有
*-vllm-disagg 配置均生效，不限于 kimik2.5。

缺陷修复：固定的 vllm-router 镜像已被回收（amd_utils/job.slurm）

vllm/vllm-router 在 Docker Hub 上仅保留约 16 个 nightly 版本；此前固定的
nightly-20260629-e667ebb 现已返回 404，导致 rank 0 在资源已分配之后才因
"manifest unknown" 拉取失败。现重新固定至 nightly-20260716-1fbcde7。

结果

[claude]

Additional findings (outside current diff — PR may have been updated during review):

• 🟡 perf-changelog.yaml:4800 — The new perf-changelog.yaml entry for kimik2.5-fp4-mi355x-vllm-disagg has 'pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX' — the placeholder was never replaced with the actual PR number (2247). Every other entry in the file uses a real PR number, so this should be updated before merge to avoid a dead/wrong link.

  Extended reasoning...

  The bug: the newly-added perf-changelog.yaml entry for kimik2.5-fp4-mi355x-vllm-disagg ends with:

  pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX

  XXX is a literal, unfilled placeholder rather than a real PR number. Per the PR metadata, this change is PR #2247, so the intended value is https://github.com/SemiAnalysisAI/InferenceX/pull/2247.

  Why this happened / why nothing catches it: perf-changelog.yaml is a plain data file with no schema validation or CI check enforcing that pr-link contains a numeric PR reference — it's free-form text, so a placeholder string is syntactically valid YAML and passes any lint/parse step. The convention of using a real PR number is purely a human convention established by every prior entry in the file, not something enforced in code.

  Proof / concrete walkthrough:

  1. Look at the five entries immediately preceding this one in the diff context: they use pull/2197, pull/2198, pull/2222, etc. — all real, resolvable PR numbers.
  2. The new entry for kimik2.5-fp4-mi355x-vllm-disagg (added in this PR) instead has pull/XXX.
  3. The actual PR is #2247 (per PR metadata: title '[AMD] Optimize kimik2.5-fp4-mi355x-vllm-disagg...').
  4. Once this PR merges, anyone reading the changelog and clicking that link gets a 404 (GitHub has no PR literally numbered "XXX"), whereas every other row in the file resolves to the actual PR that introduced the change.

  Impact: This is confined to documentation/metadata — it does not affect any benchmark config, runner behavior, or CI. It breaks the changelog's traceability convention (each entry should let a reader jump straight to the PR that made the change) and would need a separate follow-up commit to fix if merged as-is.

  Fix: Replace XXX with 2247 in the pr-link field before merging:

  pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2247

  All three verifiers independently confirmed this via direct diff inspection and agreed on nit severity — none refuted it, and there's no reasonable argument that XXX is intentional given the unbroken pattern of real PR numbers throughout the rest of the file.

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29522769567
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29522769567

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29547977874
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29547977874