Skip to content

Enable DeepSeek-V3 Wide-EP disaggregated inference + fix DP=32 correctness - #181

Open
raviguptaamd wants to merge 6 commits into
ROCm:developfrom
raviguptaamd:dsv3-wideep-disagg-enablement
Open

Enable DeepSeek-V3 Wide-EP disaggregated inference + fix DP=32 correctness#181
raviguptaamd wants to merge 6 commits into
ROCm:developfrom
raviguptaamd:dsv3-wideep-disagg-enablement

Conversation

@raviguptaamd

@raviguptaamd raviguptaamd commented Jul 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Enable DeepSeek-V3 Wide-EP disaggregated inference + fix DP=32 correctness

Related: #176 (complementary; this PR is separate and surgical — #176 is untouched).

Summary

Enables DeepSeek-V3 (671B FP8) on the vllm_disagg MoRI-EP wide-EP path and fixes a
silent MoE token-routing corruption at 32-way expert parallelism (DP=32). Inspired by
and complementary to #176; kept as a separate, surgical PR (7 files, +164/−32) so #176 is
untouched.

Validated end-to-end with NIAH long-context accuracy on MI300X (gfx942) across 1P/1D, 2P/2D,
4P/4D. At DP=32 the fix restores accuracy to the DP≤16 baseline (~8/10, zero empty answers).

What's in here

Area Change Why
DP=32 correctness (headline) Default prefill MoRI backend to mori_low_latency when PREFILL_DP_SIZE ≥ 32 (connectors/moriio.sh) The mori_high_throughput (InterNodeV1) prefill dispatch silently mis-routes MoE tokens at ≥32-way EP → corrupt prefill KV → empty/wrong decode. low_latency (which decode already uses) is clean at DP=32. DP≤16 unchanged; any explicit override still wins.
Boot / OOM Bound the attention workspace via --max-model-len (connectors/moriio.sh + models.yaml MAX_MODEL_LEN) Without a cap, vLLM sizes the attention workspace for the model's full 163840 context → ~128 GiB alloc that OOMs alongside EP-sharded weights.
Image Dockerfile builds from the shikpate ROCm/vLLM/MoRI-1.2.3 base + a GLM-stripped vLLM branch (docker/vllm_disagg_inference.ubuntu.amd.Dockerfile) The validated Wide-EP MoRI-IO stack for DeepSeek-V3 (standard MLA; GLM sparse-DSA commits removed as unused).
Recipe env 3-tier models.yaml env precedence (image-baked < yaml < submit -e) + submit-time protect-list (vllm_disagg.sh, run_xPyD_models.slurm) Lets a model's models.yaml env: recipe override image-baked ENV while a genuine submit-time -e still wins.
Accuracy harness niah benchmark selector + generalized NIAH harness (multi-seed, warmup/readiness, timeout-vs-wrong, model-neutral) (run_xPyD_models.slurm, benchmark_niah.{py,sh}) Reproducible long-context accuracy gate used to validate the above.

Backend / image refs

  • BASE_IMAGE: rocmshared/pytorch-private:vllm-rocm_07_22_2026_shikpate_mori1.2.3 (shikpate MoRI-1.2.3 base)
  • VLLM_REPO/REF: https://github.com/raviguptaamd/vllm.git @ dsv3-wideep-clean
    (Wide-EP MoRI-IO base + a generic crash-safe rmsnorm probe fix; GLM sparse-DSA commits stripped —
    DeepSeek-V3 uses standard MLA. Delta vs base = only vllm/_aiter_ops.py, 2 lines.)
    This fork/branch is public.
  • MoRI v1.2.1, AITER 0.1.16.post3, router ravgupta/discovery-dp-rank-roundrobin.

Validation (NIAH mean /10, 2k/8k/20k/35k words, 3 seeds)

Topology Prefill DP Backend Result
1P/1D 8 HT 9.3 / 8.7 / 7.7 / 7.7 ✅
2P/2D 16 HT 9.3 / 8.3 / 8.3 / 8.0 ✅
4P/4D (before) 32 HT 0.3 / 5.3 / 7.3 / 2.0 ❌
4P/4D (this PR) 32 LL 7.7 / 7.3 / 8.0 / 7.7 ✅

Notes / follow-ups

  • The DP=32 fix is a config-only default (backend selection), not a vLLM code change. No rebuild
    needed to toggle it; explicit PREFILL_MORI_BACKEND / VLLM_ALL2ALL_BACKEND still override.
  • Root cause is a pre-existing MoRI HT (InterNodeV1) internode-dispatch bug at EP≥32 (reproduces on
    stock develop). The low_latency prefill path is the low-risk production fix here. The full HT-kernel
    fix is characterized for the MoRI team separately; open cost is the LL-vs-HT prefill throughput trade,
    quantifiable with a perf sweep.

Test plan

  • Build the image from the Dockerfile (WITH_NIXL=0 lean is sufficient for the MoRI-EP path).
  • BENCHMARK_SCRIPT=niah, MODEL_NAME=DeepSeek-V3, CONNECTOR=moriio WIDE_EP=1 EP_BACKEND=mori.
  • Run 1P/1D, 2P/2D, 4P/4D; confirm NIAH means ≥ baseline and no found=0/10.
  • Confirm 4P/4D log shows PREFILL_DP_SIZE=32 >= 32: defaulting prefill backend to mori_low_latency.

🤖 Generated with Claude Code

raviguptaamd and others added 6 commits July 25, 2026 20:40
@raviguptaamd @claude
vllm_dissag: generalize NIAH harness (multi-seed, warmup, thinking-mo…
390b9aa 
…del-neutral)

Carry the model-agnostic NIAH improvements from the GLM branch onto mainline:
multi-seed variance reporting, cold-start JIT warmup + readiness poll, and
timeout-vs-wrong-answer distinction. Thinking-model handling is ungated (no
GLM name-check) so it applies to any reasoning model.

Co-Authored-By: Claude <noreply@anthropic.com>
@raviguptaamd @claude
vllm_dissag: 3-tier models.yaml env precedence (image-baked < yaml < …
98a616f 
…submit -e)

Let models.yaml env: override image-baked container ENV defaults while a genuine
submit-time -e still wins, via MODELS_YAML_PROTECT. Generic mechanism (comments
model-neutral); needed so a model's recipe knobs aren't shadowed by a tuned image.

Co-Authored-By: Claude <noreply@anthropic.com>
@raviguptaamd @claude
vllm_dissag: capture submit-time recipe-env protect-list + add niah b…
c80cad2 
…enchmark selector

Slurm captures which recipe knobs the user set at submit (MODELS_YAML_PROTECT,
consumed by the driver's 3-tier precedence) and adds BENCHMARK_SCRIPT=niah to
select benchmark_niah.sh alongside sweep/long_context.

Co-Authored-By: Claude <noreply@anthropic.com>
@raviguptaamd @claude
vllm_dissag: bound DeepSeek-V3 attention workspace with --max-model-len
d7d8cc3 
Without --max-model-len vLLM sizes the attention workspace for the model's full
163840 context, producing a ~128 GiB allocation that OOMs alongside the EP-sharded
weights (96.5 GiB/GPU at DP=8). Wire MAX_MODEL_LEN (models.yaml recipe) through the
moriio wideEP serve path; set 65536 for DeepSeek-V3 (covers the NIAH 35k-word sweep).

Co-Authored-By: Claude <noreply@anthropic.com>
@raviguptaamd @claude
vllm_disagg docker: build from shikpate MoRI-1.2.3 base + GLM-strippe…
3db2228 
…d vLLM

Repin BASE_IMAGE to the shikpate ROCm/vLLM/MoRI-1.2.3 base and VLLM_REPO/REF to
raviguptaamd/vllm@dsv3-wideep-clean (the Wide-EP MoRI-IO base minus GLM sparse-DSA
commits; DeepSeek-V3 uses standard MLA so DSA is unused). Validated end-to-end:
DSv3 serves + NIAH passes at 1P/1D wideEP.

Co-Authored-By: Claude <noreply@anthropic.com>
@raviguptaamd @claude
vllm_dissag: default prefill MoRI backend to low_latency at EP>=32
a5f074f 
The mori_high_throughput (InterNodeV1) prefill all2all dispatch silently
mis-routes MoE tokens at >=32-way expert parallelism, corrupting prefill KV so
decode emits empty/wrong output. Validated on DeepSeek-V3 NIAH: 4P/1D and 4P/4D
(PREFILL_DP=32) collapse to ~3-4/10 with empty answers on high_throughput, and
recover to ~8/10 (matching the DP<=16 baseline) on low_latency; 1P/4D (DECODE_DP=32,
PREFILL_DP=8) is clean, isolating the fault to the prefill HT path. Pre-existing in
rocm/MAD develop (stock mainline image reproduces it) and NOT fixed by MoRI tip, so
select low_latency for prefill when PREFILL_DP_SIZE>=32. Any explicit
PREFILL_MORI_BACKEND/VLLM_ALL2ALL_BACKEND override still wins.

Co-Authored-By: Claude <noreply@anthropic.com>
@raviguptaamd raviguptaamd mentioned this pull request Jul 27, 2026
Open
raviguptaamd added a commit to raviguptaamd/MAD that referenced this pull request Aug 16, 2026
@raviguptaamd @claude
[GLM-5.1] v0.27 level-set: 3.4x decode speedup, 200K context, RDMA + …
8d0ecab 
…launcher fixes

Validated on MI300X, 8 nodes, image rocmshared/pytorch-private:glm5.1-vllm027-b8
(base ci_base-dedbf6be8b + vLLM raviguptaamd/vllm@glm5.1-dsa-wideEP_on_vllm-v0.27
+ aiter e03fa6040 + MoRI 42e895472b08 + router ROCm#181).

PERF FIX (models.yaml decode.dp) -- the headline change:
  --max-num-batched-tokens 2048 on the DECODE role only.
  max_num_batched_tokens is a chunked-prefill SCHEDULER knob, but it also sizes the
  MoRI EP dispatch buffer (fused_moe/layer.py -> all2all_utils.py -> all2all.py
  max_num_inp_token_per_rank). At the 8192 default a decode instance ran an
  8192-token-wide all2all every step, per layer, x78 layers, while decoding a handful
  of tokens: a fixed ~302ms/step floor, ~320x this model's HBM-bandwidth bound.
  Prefill keeps 8192 (it genuinely dispatches wide batches).

  1024/64 con=8, warm:      TPOT        TTFT        out tok/s
    1P/1D  302 ->  88.0 ms   2431 -> 906 ms   24.9 -> 78.8
    2P/2D  302 ->  94.1 ms          1633 ms          66.7
  Published reference: 1P/1D ~89ms, 2P/2D ~91ms -> matched within 3%.
  Accuracy unaffected: NIAH 2k-200k clean on both topologies, no length collapse,
  memfault=0, latencies equal-or-better at every length. 200K validated (5.7x beyond
  the previously published 35K ceiling).

Dockerfile: base -> ci_base-dedbf6be8b (matches the fork's upstream base), VLLM_REF ->
  the v0.27 branch, and WITH_MORI_BUILD/WITH_AITER_BUILD now default to 1 so a plain
  `docker build` reproduces the validated stack. Previously they defaulted to 0, which
  silently used the base's bundled aiter 0.1.19 -- that GPU-faults on the GLM DSA decode
  kernel. The pinned aiter e03fa6040 / MoRI 42e895472b08 must not be bumped without
  re-running long-context NIAH.

connectors/moriio.sh: per-role env split (PREFILL_*/DECODE_* -> VLLM_MORI_*), mirroring
  the existing PREFILL/DECODE_MORI_BACKEND pattern -- models.yaml env: applies to BOTH
  roles, but prefill and decode need opposite values here. Also injects
  use_inductor_graph_partition (pairs with the vLLM splitting_ops fix).

connectors/moriio.env: RDMA fabric -- MORI_IB_GID_INDEX=3 (RoCEv2 IPv4),
  MORI_RDMA_DEVICES/NCCL_IB_HCA restricted to the 8 GPU-local NICs (leaving the mgmt
  NICs in makes QPs form over a non-routable fabric -> ibverbs.cpp:189 timeouts),
  NCCL/GLOO control sockets on eth0.

run_xPyD_models.slurm: libionic bind-mount requires a regular file after symlink resolve
  (a dangling symlink gave "OCI runtime create ... not a directory", container exit 125);
  prefer FABRIC_SUBNET over `hostname -I` first IP (nodes list a 10.224 overlay first,
  which made the socket_barrier advertise an unreachable NIC -> "Waiting for nodes" hang);
  GLM_KERNEL_PATCH/GLM_BACKEND_PATCH bind-mount hooks to test .py fixes without a rebuild;
  forward the new per-role env keys.

vllm_disagg.sh: same FABRIC_SUBNET IP-selection fix for host_ip.

benchmark_xPyD.sh: per-shape warmup at the REAL isl/osl before each shape's cells. The
  global warmup is isl=osl=32/con=1, which never exercises a 1024/8192/28672 prefill path
  or the decode cudagraph batch sizes, so the first measured cell absorbed residual JIT
  (observed 302ms vs ~88ms steady-state). Warmup output goes to a separate
  _SHAPEWARMUP.log so it cannot pollute the CSV.

models.yaml (GLM-5.1-FP8): decode.dp perf fix above; recipe = prefill eager +
  mori_high_throughput, decode PIECEWISE cudagraph + mori_low_latency; VLLM_USE_LAYERNAME=0;
  VLLM_SPARSE_INDEXER_MAX_LOGITS_MB=64; NCCL heartbeat/timeout knobs for long-context
  collectives.

Full operational playbook (including the dead ends) in skills_vllm_disagg.md.

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gargrahul gargrahul Awaiting requested review from gargrahul gargrahul is a code owner
@amathews-amd amathews-amd Awaiting requested review from amathews-amd amathews-amd is a code owner
@coketaste coketaste Awaiting requested review from coketaste coketaste is a code owner
@ppalaniappan-amd ppalaniappan-amd Awaiting requested review from ppalaniappan-amd ppalaniappan-amd is a code owner
@Rohan138 Rohan138 Awaiting requested review from Rohan138 Rohan138 is a code owner

At least 2 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@raviguptaamd