vllm_dissag: GLM-5.1/5.2-FP8 (MLA+DSA) MoRI-EP WideEP disaggregated enablement by raviguptaamd · Pull Request #176 · ROCm/MAD

raviguptaamd · 2026-07-08T06:43:16Z

Summary

Adds GLM-5.1-FP8 (GlmMoeDsaForCausalLM = MLA + DeepSeek Sparse Attention) to the MoRI-EP WideEP disaggregated serving path, stacked on the #171 unified launcher. Fully isolated from DeepSeek-V3/R1 — GLM gets its own image and a MODEL_NAME-gated runtime path, so existing models are byte-identical to develop.

Defects fixed

Validated on 1P/1D (EP8) and 2P/2D (EP16): NIAH 2k–35k = 10/10, no crash; DeepSeek-V3 non-regressed.

Long-context accuracy collapse (garbage >~30k tokens): vLLM #47766 cache-key fix keeps the persistent sparse-MLA kernel ON (keys attention metadata on per-request context+query length instead of a colliding key under chunked prefill).
8k disagg prefill crash: DSA adds a second (indexer) KV cache per layer that the single-geometry MoRIIO connector never transferred → decode unmap MISS → 60s deferred-write expiry → prefill EngineCore crash. Now the indexer cache is paired with its main cache and shipped prefill→decode. Plus the DSA invalid-token kernel fix (#45324) and the shik-latest DP-notify/hash-failsafe path.

Changes

docker/vllm_disagg_inference.glmv5.1.ubuntu.amd.Dockerfile — NEW per-model image (GLM vLLM + aiter e03fa6040 + mori 42e895472b08 + router). The base vllm_disagg_inference Dockerfile (DSV3/R1) is left untouched. Future models (e.g. Kimi-2.6) add their own Dockerfile the same way.
models.json — card pyt_vllm_disagg_mori_glm-5.1-fp8 → the new Dockerfile (GLM_SKIP_PATCHERS=1: the image carries the DSA fixes in-source).
models.yaml — GLM-5.1-FP8 recipe (block=1, AITER MLA on, eager, per-role mori backends). Also DeepSeek-V3 dp: caps (--max-num-seqs 64 --max-model-len 32768) to bound the newer base's decode split-KV logits workspace — isolated to the DSV3 entry.
connectors/moriio.sh — MODEL_NAME-gated GLM DSA runtime patchers (pure no-op for other models); GLM_SKIP_PATCHERS switch for baked-fix images.
9 GLM DSA patcher scripts — idempotent, anchor-based, self-skipping, py_compile-checked.

Isolation

Every GLM addition is either a new file, a new catalog entry, or gated on MODEL_NAME=GLM-5.1-FP8. DeepSeek-V3/R1 run on the unchanged upstream Dockerfile and an unchanged connector code path.

Known open defect (future work)

4P/4D EP32 emits corrupted tokens at all context lengths (garbage even at 2k), distinct from the long-context bug; prime suspect is the moriep all-to-all combine at EP32 scale. Use 1P/1D and 2P/2D; EP32 is documented in the GLM Dockerfile STATUS.

Test plan

GLM 1P/1D EP8 — NIAH 2k,8k,16k,20k,28k,35k = 10/10, no crash
GLM 2P/2D EP16 — NIAH 2k–35k = 10/10, no crash
DeepSeek-V3 1P/1D + 2P/2D regression on shared base — no regression
4P/4D EP32 — known corruption defect (future work)
Perf sweep + multi-seed accuracy (follow-up)

🤖 Generated with Claude Code

…ement Adds GLM-5.1-FP8 (GlmMoeDsaForCausalLM = MLA + DeepSeek Sparse Attention) to the MoRI-EP WideEP disaggregated serving path, stacked on the ROCm#171 unified launcher. Fully isolated from DeepSeek-V3/R1: GLM gets its own image + a MODEL_NAME-gated runtime path, so existing models are byte-identical to develop. Defects fixed (validated 1P/1D EP8 + 2P/2D EP16, NIAH 2k-35k = 10/10, no crash): - Long-context accuracy collapse: vLLM #47766 cache-key fix keeps the persistent sparse-MLA kernel ON (keys metadata on per-request context+query len). - 8k disagg prefill crash: DSA adds a 2nd (indexer) KV cache per layer that the single-geometry MoRIIO connector never transferred; paired + shipped prefill-> decode. Plus DSA invalid-token kernel fix (#45324) and shik-latest DP-notify. Changes: - docker/vllm_disagg_inference.glmv5.1.ubuntu.amd.Dockerfile: NEW per-model image (raviguptaamd/vllm glm5.1-dsa-wideEP_on_shik_latest + aiter e03fa6040 + mori 42e895472b08 + router). The base vllm_disagg_inference Dockerfile (DSV3/R1) is left untouched. Future models add their own Dockerfile the same way. - models.json: card pyt_vllm_disagg_mori_glm-5.1-fp8 (GLM_SKIP_PATCHERS=1: image carries the DSA fixes in-source). - models.yaml: GLM-5.1-FP8 recipe (block=1, AITER MLA on, eager, mori backends). DeepSeek-V3 dp: caps (--max-num-seqs 64 --max-model-len 32768) to bound the newer base's decode logits workspace (isolated to the DSV3 entry). - connectors/moriio.sh: MODEL_NAME-gated GLM DSA runtime patchers (pure no-op for other models); GLM_SKIP_PATCHERS switch for baked-fix images. - 9 idempotent, anchor-based, self-skipping GLM DSA patcher scripts. KNOWN OPEN DEFECT (future work): 4P/4D EP32 emits corrupted tokens at all context lengths (suspect moriep all-to-all combine at scale); use 1P/1D and 2P/2D. Co-Authored-By: Claude <noreply@anthropic.com>

benchmark_niah.py mis-scored thinking models: it never disabled thinking and read only content + reasoning_content. GLM-5.1 emits chain-of-thought into the `reasoning` field and leaves `content` empty until the final answer, so with a small max_tokens the answer never lands in content -> a false 0/10 even when generation is correct. - Add chat_template_kwargs.enable_thinking=false so the answer goes to content. - Also score the `reasoning` field as a fallback. Verified against GLM-5.1-FP8: correct 9-10/10 retrieval across 2k-35k on all tested topologies (EP8/EP16/EP32) after the fix. Co-Authored-By: Claude <noreply@anthropic.com>

…accuracy Needle layout is seeded, so a single run is deterministic (bit-exact on the same stack) but can't tell a real accuracy dip from single-needle variance. Add NIAH_SEEDS (default 0,1,2) to run each context length across multiple needle layouts; the summary now reports mean/min/max across seeds. Backward compatible: NIAH_SEEDS=0 reproduces the prior single-seed behavior. Co-Authored-By: Claude <noreply@anthropic.com>

…robe) On a freshly-booted node the first request of each context shape pays the full JIT/kernel-autotune compile (minutes). The NIAH harness scored the FIRST request, so cold compile landed on a scored/gated request -> false 0/10 or timeout, failing the accuracy gate and skipping the perf sweep. Root-caused by reproducing on a cold boot (0 results) vs a warm server (all pass) on the same image. Fixes: - benchmark_niah.py: add a warmup pass (NIAH_WARMUP=1 default) — one throwaway request per context length before scoring, with a generous timeout, failures tolerated. Scored requests are then always warm. - benchmark_niah.py: distinguish TIMEOUT/ERROR from a wrong answer. Timeouts return a sentinel (excluded from mean, never counted as 0/10); summary flags NO-RESULT with guidance instead of silently reporting 0. - benchmark_niah.sh: replace the blind `sleep 10` with a /v1/models readiness poll (up to 5 min), and forward NIAH_WARMUP. Verified: patched harness on the warm server passes 10/10; cold-boot repro no longer produces false 0/10 because compile happens in the warmup pass. Co-Authored-By: Claude <noreply@anthropic.com>

…launcher fixes Validated on MI300X, 8 nodes, image rocmshared/pytorch-private:glm5.1-vllm027-b8 (base ci_base-dedbf6be8b + vLLM raviguptaamd/vllm@glm5.1-dsa-wideEP_on_vllm-v0.27 + aiter e03fa6040 + MoRI 42e895472b08 + router ROCm#181). PERF FIX (models.yaml decode.dp) -- the headline change: --max-num-batched-tokens 2048 on the DECODE role only. max_num_batched_tokens is a chunked-prefill SCHEDULER knob, but it also sizes the MoRI EP dispatch buffer (fused_moe/layer.py -> all2all_utils.py -> all2all.py max_num_inp_token_per_rank). At the 8192 default a decode instance ran an 8192-token-wide all2all every step, per layer, x78 layers, while decoding a handful of tokens: a fixed ~302ms/step floor, ~320x this model's HBM-bandwidth bound. Prefill keeps 8192 (it genuinely dispatches wide batches). 1024/64 con=8, warm: TPOT TTFT out tok/s 1P/1D 302 -> 88.0 ms 2431 -> 906 ms 24.9 -> 78.8 2P/2D 302 -> 94.1 ms 1633 ms 66.7 Published reference: 1P/1D ~89ms, 2P/2D ~91ms -> matched within 3%. Accuracy unaffected: NIAH 2k-200k clean on both topologies, no length collapse, memfault=0, latencies equal-or-better at every length. 200K validated (5.7x beyond the previously published 35K ceiling). Dockerfile: base -> ci_base-dedbf6be8b (matches the fork's upstream base), VLLM_REF -> the v0.27 branch, and WITH_MORI_BUILD/WITH_AITER_BUILD now default to 1 so a plain `docker build` reproduces the validated stack. Previously they defaulted to 0, which silently used the base's bundled aiter 0.1.19 -- that GPU-faults on the GLM DSA decode kernel. The pinned aiter e03fa6040 / MoRI 42e895472b08 must not be bumped without re-running long-context NIAH. connectors/moriio.sh: per-role env split (PREFILL_*/DECODE_* -> VLLM_MORI_*), mirroring the existing PREFILL/DECODE_MORI_BACKEND pattern -- models.yaml env: applies to BOTH roles, but prefill and decode need opposite values here. Also injects use_inductor_graph_partition (pairs with the vLLM splitting_ops fix). connectors/moriio.env: RDMA fabric -- MORI_IB_GID_INDEX=3 (RoCEv2 IPv4), MORI_RDMA_DEVICES/NCCL_IB_HCA restricted to the 8 GPU-local NICs (leaving the mgmt NICs in makes QPs form over a non-routable fabric -> ibverbs.cpp:189 timeouts), NCCL/GLOO control sockets on eth0. run_xPyD_models.slurm: libionic bind-mount requires a regular file after symlink resolve (a dangling symlink gave "OCI runtime create ... not a directory", container exit 125); prefer FABRIC_SUBNET over `hostname -I` first IP (nodes list a 10.224 overlay first, which made the socket_barrier advertise an unreachable NIC -> "Waiting for nodes" hang); GLM_KERNEL_PATCH/GLM_BACKEND_PATCH bind-mount hooks to test .py fixes without a rebuild; forward the new per-role env keys. vllm_disagg.sh: same FABRIC_SUBNET IP-selection fix for host_ip. benchmark_xPyD.sh: per-shape warmup at the REAL isl/osl before each shape's cells. The global warmup is isl=osl=32/con=1, which never exercises a 1024/8192/28672 prefill path or the decode cudagraph batch sizes, so the first measured cell absorbed residual JIT (observed 302ms vs ~88ms steady-state). Warmup output goes to a separate _SHAPEWARMUP.log so it cannot pollute the CSV. models.yaml (GLM-5.1-FP8): decode.dp perf fix above; recipe = prefill eager + mori_high_throughput, decode PIECEWISE cudagraph + mori_low_latency; VLLM_USE_LAYERNAME=0; VLLM_SPARSE_INDEXER_MAX_LOGITS_MB=64; NCCL heartbeat/timeout knobs for long-context collectives. Full operational playbook (including the dead ends) in skills_vllm_disagg.md. Co-Authored-By: Claude <noreply@anthropic.com>

GLM-5.1 is served over MoRI-EP (all2all) + MoRI-IO (KV transfer); the rixl connector's UCX/RIXL/rocSHMEM/DeepEP stack is never selected by this recipe, so building it only lengthens the build and ships unused transports. 0 is also how the validated image was actually built: glm5.1-vllm027-b8 used --build-arg WITH_NIXL=0, so the previous default of 1 meant a plain `docker build` did NOT reproduce the tested artifact. With this change, and with WITH_MORI_BUILD/WITH_AITER_BUILD already defaulting to 1, a no-flag build now matches the validated stack exactly. Set --build-arg WITH_NIXL=1 if you need the rixl connector from this same Dockerfile. Co-Authored-By: Claude <noreply@anthropic.com>

…ybook niah_200k.py: needle-in-a-haystack sweep that validated GLM-5.1-FP8 to 200,049 tokens on both 1P/1D (EP8) and 2P/2D (EP16). Reports found/10, latency, and the server-reported prompt_tokens per length, and writes JSON. Model id is overridable via NIAH_MODEL so it is not GLM-specific. The existing benchmark_niah.* stop well short of this range; this covers the 64k-200k band. skills_vllm_disagg.md: operational playbook for vLLM PD-disaggregated WideEP on MI300X (MoRI-EP + MoRI-IO), written from this enablement. Documents, with measurements: - benchmarking method: ALWAYS discard the first post-boot run (cold Triton JIT made TTFT read 13.4s vs 906ms warm; with prefill eager the JIT cost lands in TTFT, not TPOT), and sanity-check against the HBM-bandwidth bound before blaming a kernel - the max_num_batched_tokens trap: a chunked-prefill SCHEDULER knob also sizes the MoRI EP dispatch buffer, so decode ran an 8192-token-wide all2all every step (302ms -> 88ms TPOT once sized for the real batch) - the DSA sentinel landmine: the invalid sparse-index sentinel must be 0, not -1, because aiter's mla_decode_fwd dereferences it (only bites at disagg long context) - three documented DEAD ENDS so they are not retried, including why mori's max_total_recv_tokens cannot decouple recv from send capacity (the clamp is a min()) - cache/boot behaviour (three caches with different rules, the aiter baton lock, measured boot times), readiness signals for multi-node topologies, per-role env plumbing, and RDMA fabric verification (a node can be SLURM-'alloc' with a dead fabric - verify with ping/ib_write_bw before blaming code) Co-Authored-By: Claude <noreply@anthropic.com>

…ments Review cleanup. The VLLM_MORI_MAX_TOKENS_PER_RANK / VLLM_MORI_MAX_TOTAL_RECV_TOKENS per-role plumbing was written while chasing the decode-TPOT bug and is NOT what fixed it (the fix is `--max-num-batched-tokens 2048` on decode.dp). Worse, the comments asserted that max_total_recv_tokens keeps recv capacity large enough for vLLM's profiling dummy run -- which is false and was disproved by measurement: mori's MaxNumTokensToRecvPerRank() = min(ceil(maxTotalRecvTokens / worldSize), maxNumInpTokenPerRank) is a min(), so maxTotalRecvTokens can only LOWER recv capacity, never raise it above the send width. Anyone following those comments and setting the knobs would hit "Total recv token overflow" at boot (observed at 512, 2048, and with recv=65536). Removed: the per-role export block in moriio.sh, the stale models.yaml comment block, and the six dead keys from _RECIPE_ENV_KEYS. Replaced with a short NOTE in moriio.sh pointing at the real fix and at skills_vllm_disagg.md for the measurements and dead ends. No functional change to the validated configuration: the knobs defaulted to 0/unset, so the b8 runs never exercised them. Co-Authored-By: Claude <noreply@anthropic.com>

…16) faster decode Second perf lever, on top of the decode --max-num-batched-tokens fix. Switching the decode role from PIECEWISE to FULL_AND_PIECEWISE captures a FULL graph for the uniform-decode shapes and falls back to PIECEWISE for the rest. Measured on MI300X, ISL/OSL 1024/64, warm (first post-boot run discarded), image rocmshared/pytorch-private:glm5.1-pr176-verify: PIECEWISE FULL_AND_PIECEWISE gain 1P/1D EP8 TPOT c8 88.0 ms 41.8 ms 2.11x 1P/1D EP8 TPOT c16 91.5 ms 45.8 ms 2.00x 2P/2D EP16 TPOT c8 94.1 ms 55.4 ms 1.70x 2P/2D EP16 TPOT c16 96.2 ms 60.3 ms 1.60x 1P/1D outTP c8 78.8 tok/s 143.7 tok/s 1.82x 2P/2D outTP c8 66.7 tok/s 113.0 tok/s 1.69x 2P/2D TTFT c16 2221 ms 1400 ms 1.59x Both topologies now sit well below the published ~89 ms (EP8) / ~91 ms (EP16) reference. Accuracy is unaffected. NIAH 2k-35k: 1P/1D 51/60, 2P/2D 55/60 - both inside the PIECEWISE band (52-53 and 53 respectively; 2P/2D is in fact the best NIAH run of the campaign). No length collapse, no '!!!', memfault=0, latencies equal-or-better at every length. Caveats, stated rather than buried: - The gain SHRINKS as EP widens (2.11x at EP8 -> 1.70x at EP16). FULL captures the whole decode step including the cross-node all2all, which does not compress. This is validated at EP8 and EP16 ONLY; do not assume it holds at EP32 (which has a separate, unrelated correctness regression). - Capture costs ~92-94 s and ~3.0-3.5 GiB vs ~5 s / 7.2 GiB for PIECEWISE (fewer but larger graphs), so boot is longer. Irrelevant for a long-lived server, notable for CI. - Set DECODE_CUDAGRAPH_MODE=PIECEWISE to fall back. Two code paths branch on CUDAGraphMode.FULL (sparse_attn_indexer.py:411 and the MoRIIO READ-mode barrier); both are guards that skip host-side work under FULL, so enabling it is safe here - the MoRIIO one is already a no-op in WRITE mode. skills_vllm_disagg.md gains a section 2b with the table, the EP-width caveat, the capture cost, and the observation that FULL is far less cold-JIT sensitive (its warmup run already reads steady-state TPOT, where PIECEWISE's first run showed 13.7 s TTFT). Co-Authored-By: Claude <noreply@anthropic.com>

MIR-AMD · 2026-08-18T04:59:06Z

Opened #206 as a rebased successor to this PR, since this branch is now CONFLICTING against develop (it predates #197, which moved model cards into per-directory catalogs).

#206 carries the same GLM-5.1-FP8 enablement rebased onto current develop, plus three trims so the change is scoped to GLM:

Drops the .nfs* artifact and the lab-only keepalive_bench.sh / niah_200k.py / skills_vllm_disagg.md.
Reverts the DeepSeek-V3 --max-num-seqs 64 --max-model-len 32768 caps (they would cut DSV3 context from ~163k to 32k on the untouched DSV3 image) and the cluster-wide moriio.env GID/NIC pins (they applied to every moriio model).
Makes apply_glm_dsa_persistent_kernel_gate_fix.py and apply_glm_dsa_kernel_fix.py opt-in, matching what vllm_disagg: GLM-5.2 (FP8 + MXFP4) on MI355X/gfx950 + AI NIC, 1P/1D EP8 [stacked on #176] #205 found: on the v0.27 image they abort at asm_mla.cu:945 and hipErrorIllegalAddress respectively. The validated card sets GLM_SKIP_PATCHERS=1 regardless.

@raviguptaamd happy to fold in anything I trimmed if you want it kept.

basemam · 2026-08-18T20:40:51Z

Just opened raviguptaamd#4 to merge Wei's topk fix and niah 4P$D good test

raviguptaamd requested review from Rohan138, amathews-amd, coketaste, gargrahul and ppalaniappan-amd as code owners July 8, 2026 06:43

raviguptaamd and others added 3 commits July 8, 2026 15:51

raviguptaamd changed the title ~~vllm_dissag: GLM-5.1-FP8 (MLA+DSA) MoRI-EP WideEP disaggregated enablement~~ vllm_dissag: GLM-5.1/5.2-FP8 (MLA+DSA) MoRI-EP WideEP disaggregated enablement Jul 21, 2026

raviguptaamd mentioned this pull request Jul 27, 2026

Enable DeepSeek-V3 Wide-EP disaggregated inference + fix DP=32 correctness #181

Open

4 tasks

i-kosarev previously approved these changes Aug 13, 2026

View reviewed changes

i-kosarev requested review from i-kosarev and removed request for i-kosarev August 13, 2026 14:51

raviguptaamd dismissed i-kosarev’s stale review via 8d0ecab August 16, 2026 04:17

raviguptaamd and others added 3 commits August 16, 2026 04:24

MIR-AMD mentioned this pull request Aug 18, 2026

vllm_dissag: GLM-5.1-FP8 (MLA+DSA) MoRI-EP WideEP disaggregated enablement #206

Open

7 tasks

Navigation Menu

Uh oh!

vllm_dissag: GLM-5.1/5.2-FP8 (MLA+DSA) MoRI-EP WideEP disaggregated enablement - #176

vllm_dissag: GLM-5.1/5.2-FP8 (MLA+DSA) MoRI-EP WideEP disaggregated enablement#176
raviguptaamd wants to merge 9 commits into
ROCm:developfrom
raviguptaamd:glm5.1-wideep-clean

raviguptaamd commented Jul 8, 2026

Uh oh!

MIR-AMD commented Aug 18, 2026

Uh oh!

basemam commented Aug 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

raviguptaamd commented Jul 8, 2026

Summary

Defects fixed

Changes

Isolation

Known open defect (future work)

Test plan

Uh oh!

MIR-AMD commented Aug 18, 2026

Uh oh!

basemam commented Aug 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants