vllm_dissag: GLM-5.1-FP8 (MLA+DSA) MoRI-EP WideEP disaggregated enablement by MIR-AMD · Pull Request #206 · ROCm/MAD

MIR-AMD · 2026-08-18T04:58:41Z

Rebased, trimmed successor to #176 (that branch is now CONFLICTING against develop).

Adds GLM-5.1-FP8 (GlmMoeDsaForCausalLM, MLA + DeepSeek Sparse Attention) as a MoRI-EP wideEP disaggregated model. Validated 1P/1D EP8 and 2P/2D EP16; EP32 is a documented open defect.

What lands

New per-model image docker/vllm_disagg_inference.glmv5.1.ubuntu.amd.Dockerfile. The DeepSeek image is untouched. Pins: VLLM_REF=glm5.1-dsa-wideEP_on_vllm-v0.27, AITER_REF=e03fa6040, MORI_REF=42e895472b08, WITH_NIXL=0.
Card pyt_vllm_disagg_mori_glm-5.1-fp8 in scripts/vllm_dissag/models.json (per-directory catalog, post-Migrated models from global to per directory #197), shipping GLM_SKIP_PATCHERS=1.
Recipe GLM-5.1-FP8 in scripts/vllm_dissag/models.yaml: block-size 1 (the DSA sparse indexer requires it), AITER MLA on, decode --max-num-batched-tokens 2048, DECODE_CUDAGRAPH_MODE=FULL_AND_PIECEWISE.
DSA runtime patchers in connectors/moriio.sh, gated on GLM-5.* so every other model is a byte-identical no-op.
NIAH accuracy harness selectable via BENCHMARK_SCRIPT=niah.

Deliberately kept out

Relative to #176, this branch drops the lab leftovers and reverts two changes that would have affected unrelated models:

Removed an .nfs* silly-rename artifact, keepalive_bench.sh, niah_200k.py, and skills_vllm_disagg.md.
Reverted DeepSeek-V3's --max-num-seqs 64 --max-model-len 32768 caps, which would have cut DSV3 context from ~163k to 32k on the untouched DSV3 image.
Reverted the cluster-wide connectors/moriio.env GID/NIC pins, which applied to every moriio model. Site-specific fabric settings stay as submit-time env.

Patcher defaults

Two patchers that #176 defaulted ON were later shown to crash this exact v0.27 image, so they are now opt-in:

apply_glm_dsa_persistent_kernel_gate_fix.py (ports vLLM #47567, since superseded by the merged #47766) aborts at asm_mla.cu:945 for fp8 with gqa_ratio=64. Enable with GLM_PERSIST_GATE=1.
apply_glm_dsa_kernel_fix.py (ports the still-open vLLM #45324 sentinel change) yields hipErrorIllegalAddress, because aiter's mla_decode_fwd dereferences the index and the image ships 0 on purpose. Enable with GLM_DSA_SENTINEL_FIX=1.

The validated card sets GLM_SKIP_PATCHERS=1, so none of the patchers run on the intended image; they remain as fallbacks for older images.

Which image to serve this on

The recipe comment previously named a pre-v0.27 tag as "validated". That is corrected here to rocmshared/pytorch-private:glm5.1-vllm027-b8, the image the Dockerfile builds. This is not cosmetic: serving the recipe on the older rocm/pytorch-private:glm-dockerimage-built-09072026 boots cleanly and then returns empty completions (NIAH 2k scored 0/10, no crash), which is an easy trap to fall into.

Test plan

scripts/vllm_dissag/tests/argv_assert.sh: 25 passed, 0 failed.
DRY_RUN=1 for GLM-5.1-FP8 emits expert-parallel, --block-size 1, FULL_AND_PIECEWISE, decode --max-num-batched-tokens 2048.
DRY_RUN=1 for DeepSeek-V3 is unchanged versus develop.
OCI 1P/1D (EP8) serving smoke on amd-rccl, 2 nodes, rocmshared/pytorch-private:glm5.1-vllm027-b8:
- [glm] GLM_SKIP_PATCHERS=1: image carries DSA fixes in-source; skipping runtime patchers.
- prefill + decode engines ready, Proxy (vllm_router) ready, router registration complete, disaggregated prefill to decode KV transfer verified end to end.
- NIAH 2k retrieval 10/10 (crocodile, dolphin, elephant, giraffe, kangaroo, octopus, panda, penguin, rhinoceros, tiger).
- No EngineDeadError, no hipErrorIllegalAddress, no asm_mla.cu:945. Job exited cleanly.

2P/2D EP16 was validated on #176 and no GLM recipe knob changed here. EP32 remains a known defect and is not claimed.

…ement Adds GLM-5.1-FP8 (GlmMoeDsaForCausalLM = MLA + DeepSeek Sparse Attention) to the MoRI-EP WideEP disaggregated serving path, stacked on the ROCm#171 unified launcher. Fully isolated from DeepSeek-V3/R1: GLM gets its own image + a MODEL_NAME-gated runtime path, so existing models are byte-identical to develop. Defects fixed (validated 1P/1D EP8 + 2P/2D EP16, NIAH 2k-35k = 10/10, no crash): - Long-context accuracy collapse: vLLM #47766 cache-key fix keeps the persistent sparse-MLA kernel ON (keys metadata on per-request context+query len). - 8k disagg prefill crash: DSA adds a 2nd (indexer) KV cache per layer that the single-geometry MoRIIO connector never transferred; paired + shipped prefill-> decode. Plus DSA invalid-token kernel fix (#45324) and shik-latest DP-notify. Changes: - docker/vllm_disagg_inference.glmv5.1.ubuntu.amd.Dockerfile: NEW per-model image (raviguptaamd/vllm glm5.1-dsa-wideEP_on_shik_latest + aiter e03fa6040 + mori 42e895472b08 + router). The base vllm_disagg_inference Dockerfile (DSV3/R1) is left untouched. Future models add their own Dockerfile the same way. - models.json: card pyt_vllm_disagg_mori_glm-5.1-fp8 (GLM_SKIP_PATCHERS=1: image carries the DSA fixes in-source). - models.yaml: GLM-5.1-FP8 recipe (block=1, AITER MLA on, eager, mori backends). DeepSeek-V3 dp: caps (--max-num-seqs 64 --max-model-len 32768) to bound the newer base's decode logits workspace (isolated to the DSV3 entry). - connectors/moriio.sh: MODEL_NAME-gated GLM DSA runtime patchers (pure no-op for other models); GLM_SKIP_PATCHERS switch for baked-fix images. - 9 idempotent, anchor-based, self-skipping GLM DSA patcher scripts. KNOWN OPEN DEFECT (future work): 4P/4D EP32 emits corrupted tokens at all context lengths (suspect moriep all-to-all combine at scale); use 1P/1D and 2P/2D. Co-Authored-By: Claude <noreply@anthropic.com>

benchmark_niah.py mis-scored thinking models: it never disabled thinking and read only content + reasoning_content. GLM-5.1 emits chain-of-thought into the `reasoning` field and leaves `content` empty until the final answer, so with a small max_tokens the answer never lands in content -> a false 0/10 even when generation is correct. - Add chat_template_kwargs.enable_thinking=false so the answer goes to content. - Also score the `reasoning` field as a fallback. Verified against GLM-5.1-FP8: correct 9-10/10 retrieval across 2k-35k on all tested topologies (EP8/EP16/EP32) after the fix. Co-Authored-By: Claude <noreply@anthropic.com>

…accuracy Needle layout is seeded, so a single run is deterministic (bit-exact on the same stack) but can't tell a real accuracy dip from single-needle variance. Add NIAH_SEEDS (default 0,1,2) to run each context length across multiple needle layouts; the summary now reports mean/min/max across seeds. Backward compatible: NIAH_SEEDS=0 reproduces the prior single-seed behavior. Co-Authored-By: Claude <noreply@anthropic.com>

…robe) On a freshly-booted node the first request of each context shape pays the full JIT/kernel-autotune compile (minutes). The NIAH harness scored the FIRST request, so cold compile landed on a scored/gated request -> false 0/10 or timeout, failing the accuracy gate and skipping the perf sweep. Root-caused by reproducing on a cold boot (0 results) vs a warm server (all pass) on the same image. Fixes: - benchmark_niah.py: add a warmup pass (NIAH_WARMUP=1 default) — one throwaway request per context length before scoring, with a generous timeout, failures tolerated. Scored requests are then always warm. - benchmark_niah.py: distinguish TIMEOUT/ERROR from a wrong answer. Timeouts return a sentinel (excluded from mean, never counted as 0/10); summary flags NO-RESULT with guidance instead of silently reporting 0. - benchmark_niah.sh: replace the blind `sleep 10` with a /v1/models readiness poll (up to 5 min), and forward NIAH_WARMUP. Verified: patched harness on the warm server passes 10/10; cold-boot repro no longer produces false 0/10 because compile happens in the warmup pass. Co-Authored-By: Claude <noreply@anthropic.com>

…launcher fixes Validated on MI300X, 8 nodes, image rocmshared/pytorch-private:glm5.1-vllm027-b8 (base ci_base-dedbf6be8b + vLLM raviguptaamd/vllm@glm5.1-dsa-wideEP_on_vllm-v0.27 + aiter e03fa6040 + MoRI 42e895472b08 + router ROCm#181). PERF FIX (models.yaml decode.dp) -- the headline change: --max-num-batched-tokens 2048 on the DECODE role only. max_num_batched_tokens is a chunked-prefill SCHEDULER knob, but it also sizes the MoRI EP dispatch buffer (fused_moe/layer.py -> all2all_utils.py -> all2all.py max_num_inp_token_per_rank). At the 8192 default a decode instance ran an 8192-token-wide all2all every step, per layer, x78 layers, while decoding a handful of tokens: a fixed ~302ms/step floor, ~320x this model's HBM-bandwidth bound. Prefill keeps 8192 (it genuinely dispatches wide batches). 1024/64 con=8, warm: TPOT TTFT out tok/s 1P/1D 302 -> 88.0 ms 2431 -> 906 ms 24.9 -> 78.8 2P/2D 302 -> 94.1 ms 1633 ms 66.7 Published reference: 1P/1D ~89ms, 2P/2D ~91ms -> matched within 3%. Accuracy unaffected: NIAH 2k-200k clean on both topologies, no length collapse, memfault=0, latencies equal-or-better at every length. 200K validated (5.7x beyond the previously published 35K ceiling). Dockerfile: base -> ci_base-dedbf6be8b (matches the fork's upstream base), VLLM_REF -> the v0.27 branch, and WITH_MORI_BUILD/WITH_AITER_BUILD now default to 1 so a plain `docker build` reproduces the validated stack. Previously they defaulted to 0, which silently used the base's bundled aiter 0.1.19 -- that GPU-faults on the GLM DSA decode kernel. The pinned aiter e03fa6040 / MoRI 42e895472b08 must not be bumped without re-running long-context NIAH. connectors/moriio.sh: per-role env split (PREFILL_*/DECODE_* -> VLLM_MORI_*), mirroring the existing PREFILL/DECODE_MORI_BACKEND pattern -- models.yaml env: applies to BOTH roles, but prefill and decode need opposite values here. Also injects use_inductor_graph_partition (pairs with the vLLM splitting_ops fix). connectors/moriio.env: RDMA fabric -- MORI_IB_GID_INDEX=3 (RoCEv2 IPv4), MORI_RDMA_DEVICES/NCCL_IB_HCA restricted to the 8 GPU-local NICs (leaving the mgmt NICs in makes QPs form over a non-routable fabric -> ibverbs.cpp:189 timeouts), NCCL/GLOO control sockets on eth0. run_xPyD_models.slurm: libionic bind-mount requires a regular file after symlink resolve (a dangling symlink gave "OCI runtime create ... not a directory", container exit 125); prefer FABRIC_SUBNET over `hostname -I` first IP (nodes list a 10.224 overlay first, which made the socket_barrier advertise an unreachable NIC -> "Waiting for nodes" hang); GLM_KERNEL_PATCH/GLM_BACKEND_PATCH bind-mount hooks to test .py fixes without a rebuild; forward the new per-role env keys. vllm_disagg.sh: same FABRIC_SUBNET IP-selection fix for host_ip. benchmark_xPyD.sh: per-shape warmup at the REAL isl/osl before each shape's cells. The global warmup is isl=osl=32/con=1, which never exercises a 1024/8192/28672 prefill path or the decode cudagraph batch sizes, so the first measured cell absorbed residual JIT (observed 302ms vs ~88ms steady-state). Warmup output goes to a separate _SHAPEWARMUP.log so it cannot pollute the CSV. models.yaml (GLM-5.1-FP8): decode.dp perf fix above; recipe = prefill eager + mori_high_throughput, decode PIECEWISE cudagraph + mori_low_latency; VLLM_USE_LAYERNAME=0; VLLM_SPARSE_INDEXER_MAX_LOGITS_MB=64; NCCL heartbeat/timeout knobs for long-context collectives. Full operational playbook (including the dead ends) in skills_vllm_disagg.md. Co-Authored-By: Claude <noreply@anthropic.com>

GLM-5.1 is served over MoRI-EP (all2all) + MoRI-IO (KV transfer); the rixl connector's UCX/RIXL/rocSHMEM/DeepEP stack is never selected by this recipe, so building it only lengthens the build and ships unused transports. 0 is also how the validated image was actually built: glm5.1-vllm027-b8 used --build-arg WITH_NIXL=0, so the previous default of 1 meant a plain `docker build` did NOT reproduce the tested artifact. With this change, and with WITH_MORI_BUILD/WITH_AITER_BUILD already defaulting to 1, a no-flag build now matches the validated stack exactly. Set --build-arg WITH_NIXL=1 if you need the rixl connector from this same Dockerfile. Co-Authored-By: Claude <noreply@anthropic.com>

…ybook niah_200k.py: needle-in-a-haystack sweep that validated GLM-5.1-FP8 to 200,049 tokens on both 1P/1D (EP8) and 2P/2D (EP16). Reports found/10, latency, and the server-reported prompt_tokens per length, and writes JSON. Model id is overridable via NIAH_MODEL so it is not GLM-specific. The existing benchmark_niah.* stop well short of this range; this covers the 64k-200k band. skills_vllm_disagg.md: operational playbook for vLLM PD-disaggregated WideEP on MI300X (MoRI-EP + MoRI-IO), written from this enablement. Documents, with measurements: - benchmarking method: ALWAYS discard the first post-boot run (cold Triton JIT made TTFT read 13.4s vs 906ms warm; with prefill eager the JIT cost lands in TTFT, not TPOT), and sanity-check against the HBM-bandwidth bound before blaming a kernel - the max_num_batched_tokens trap: a chunked-prefill SCHEDULER knob also sizes the MoRI EP dispatch buffer, so decode ran an 8192-token-wide all2all every step (302ms -> 88ms TPOT once sized for the real batch) - the DSA sentinel landmine: the invalid sparse-index sentinel must be 0, not -1, because aiter's mla_decode_fwd dereferences it (only bites at disagg long context) - three documented DEAD ENDS so they are not retried, including why mori's max_total_recv_tokens cannot decouple recv from send capacity (the clamp is a min()) - cache/boot behaviour (three caches with different rules, the aiter baton lock, measured boot times), readiness signals for multi-node topologies, per-role env plumbing, and RDMA fabric verification (a node can be SLURM-'alloc' with a dead fabric - verify with ping/ib_write_bw before blaming code) Co-Authored-By: Claude <noreply@anthropic.com>

…ments Review cleanup. The VLLM_MORI_MAX_TOKENS_PER_RANK / VLLM_MORI_MAX_TOTAL_RECV_TOKENS per-role plumbing was written while chasing the decode-TPOT bug and is NOT what fixed it (the fix is `--max-num-batched-tokens 2048` on decode.dp). Worse, the comments asserted that max_total_recv_tokens keeps recv capacity large enough for vLLM's profiling dummy run -- which is false and was disproved by measurement: mori's MaxNumTokensToRecvPerRank() = min(ceil(maxTotalRecvTokens / worldSize), maxNumInpTokenPerRank) is a min(), so maxTotalRecvTokens can only LOWER recv capacity, never raise it above the send width. Anyone following those comments and setting the knobs would hit "Total recv token overflow" at boot (observed at 512, 2048, and with recv=65536). Removed: the per-role export block in moriio.sh, the stale models.yaml comment block, and the six dead keys from _RECIPE_ENV_KEYS. Replaced with a short NOTE in moriio.sh pointing at the real fix and at skills_vllm_disagg.md for the measurements and dead ends. No functional change to the validated configuration: the knobs defaulted to 0/unset, so the b8 runs never exercised them. Co-Authored-By: Claude <noreply@anthropic.com>

…16) faster decode Second perf lever, on top of the decode --max-num-batched-tokens fix. Switching the decode role from PIECEWISE to FULL_AND_PIECEWISE captures a FULL graph for the uniform-decode shapes and falls back to PIECEWISE for the rest. Measured on MI300X, ISL/OSL 1024/64, warm (first post-boot run discarded), image rocmshared/pytorch-private:glm5.1-pr176-verify: PIECEWISE FULL_AND_PIECEWISE gain 1P/1D EP8 TPOT c8 88.0 ms 41.8 ms 2.11x 1P/1D EP8 TPOT c16 91.5 ms 45.8 ms 2.00x 2P/2D EP16 TPOT c8 94.1 ms 55.4 ms 1.70x 2P/2D EP16 TPOT c16 96.2 ms 60.3 ms 1.60x 1P/1D outTP c8 78.8 tok/s 143.7 tok/s 1.82x 2P/2D outTP c8 66.7 tok/s 113.0 tok/s 1.69x 2P/2D TTFT c16 2221 ms 1400 ms 1.59x Both topologies now sit well below the published ~89 ms (EP8) / ~91 ms (EP16) reference. Accuracy is unaffected. NIAH 2k-35k: 1P/1D 51/60, 2P/2D 55/60 - both inside the PIECEWISE band (52-53 and 53 respectively; 2P/2D is in fact the best NIAH run of the campaign). No length collapse, no '!!!', memfault=0, latencies equal-or-better at every length. Caveats, stated rather than buried: - The gain SHRINKS as EP widens (2.11x at EP8 -> 1.70x at EP16). FULL captures the whole decode step including the cross-node all2all, which does not compress. This is validated at EP8 and EP16 ONLY; do not assume it holds at EP32 (which has a separate, unrelated correctness regression). - Capture costs ~92-94 s and ~3.0-3.5 GiB vs ~5 s / 7.2 GiB for PIECEWISE (fewer but larger graphs), so boot is longer. Irrelevant for a long-lived server, notable for CI. - Set DECODE_CUDAGRAPH_MODE=PIECEWISE to fall back. Two code paths branch on CUDAGraphMode.FULL (sparse_attn_indexer.py:411 and the MoRIIO READ-mode barrier); both are guards that skip host-side work under FULL, so enabling it is safe here - the MoRIIO one is already a no-op in WRITE mode. skills_vllm_disagg.md gains a section 2b with the table, the EP-width caveat, the capture cost, and the observation that FULL is far less cold-JIT sensitive (its warmup run already reads steady-state TPOT, where PIECEWISE's first run showed 13.7 s TTFT). Co-Authored-By: Claude <noreply@anthropic.com>

Rebase onto develop after ROCm#197 so the GLM card lives in scripts/vllm_dissag/models.json. Remove the NFS silly-rename and keepalive/NIAH-200k/skills lab files, revert DeepSeek-V3 context caps and cluster-wide moriio.env NIC pins, turn persist-gate and DSA sentinel patchers opt-in (they crash the v0.27 image), gate DSA on GLM-5.*, and wire BENCHMARK_SCRIPT=niah for the 1P/1D smoke.

Copilot

Pull request overview

Enables GLM-5.1-FP8 (MLA + DeepSeek Sparse Attention) for the vLLM disaggregated MoRI-EP wideEP serving path by adding a dedicated image, model catalog/recipe entries, and GLM-gated runtime patching support, plus improving launcher ergonomics for IP selection and benchmark selection.

Changes:

Add a new GLM-specific ROCm vLLM/MoRI/AITER pinned Docker image and wire it into the per-directory model catalog + recipe.
Extend the disagg launcher + slurm wrapper to prefer routable fabric IPs, support models.yaml env precedence, and expose a BENCHMARK_SCRIPT selector (including NIAH).
Introduce GLM-5.*-gated MoRIIO runtime patchers and enhance benchmark harnesses (NIAH + per-shape warmup).

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
scripts/vllm_dissag/vllm_disagg.sh	Prefer fabric-routable host IP; adjust models.yaml env precedence rules.
scripts/vllm_dissag/run_xPyD_models.slurm	Add MODELS_YAML_PROTECT capture, GLM-5.1-FP8 enablement, fabric IP picking, benchmark selector, extra mounts/env.
scripts/vllm_dissag/README.MD	Document GLM-5.1-FP8 support and BENCHMARK_SCRIPT=niah usage.
scripts/vllm_dissag/models.yaml	Add GLM-5.1-FP8 recipe (env + dp_flags + decode tuning).
scripts/vllm_dissag/models.json	Add GLM-5.1-FP8 model card pointing to the new GLM Dockerfile.
scripts/vllm_dissag/connectors/moriio.sh	Add GLM-5.* runtime patch hook; extend compilation-config; propagate models.yaml role flags.
scripts/vllm_dissag/benchmark_xPyD.sh	Add optional per-shape warmup to reduce first-cell JIT skew.
scripts/vllm_dissag/benchmark_niah.sh	Replace blind sleep with readiness polling; pass through NIAH_WARMUP.
scripts/vllm_dissag/benchmark_niah.py	Add seeds + warmup flow; adjust request payload/scoring for thinking models.
scripts/vllm_dissag/apply_glm_moriio_abort_guard_fix.py	New patcher: guard MoRIIO abort/release path against None peer_zmq.
scripts/vllm_dissag/apply_glm_dsa_persistent_kernel_gate_fix.py	New (opt-in) patcher: gate off persistent MLA for chunked-prefill continuations.
scripts/vllm_dissag/apply_glm_dsa_moriio_instrument.py	New (opt-in) patcher: diagnostic logging for dual-KV transfer debugging.
scripts/vllm_dissag/apply_glm_dsa_moriio_gate_fix.py	New patcher: fix MoRIIO completion gate for DSA dual-KV behavior.
scripts/vllm_dissag/apply_glm_dsa_moriio_engine_fix.py	New patcher: per-layer transfer offset caching for dual KV geometries.
scripts/vllm_dissag/apply_glm_dsa_moriio_dualkv_fix.py	New patcher: per-layer geometry/offset handling in MoRIIO connector for dual KV caches.
scripts/vllm_dissag/apply_glm_dsa_kernel_fix.py	New (opt-in) patcher: invalid-token sentinel fix for DSA indexer kernel.
scripts/vllm_dissag/apply_glm_dsa_indexer_warmup_fix.py	New (opt-in) patcher: force-compile DSA indexer kernels during boot.
scripts/vllm_dissag/apply_glm_aiter_sampling_oob_fix.py	New patcher: overlay fixed AITER sampling sources and purge stale JIT artifacts.
docker/vllm_disagg_inference.glmv5.1.ubuntu.amd.Dockerfile	New per-model GLM-5.1 image with pinned MoRI/AITER/vLLM/router builds and cache/JIT scrubbing.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+# model's own recipe — GLM-5.1 DSA needs block=1 + AITER MLA on), while a genuine
+# submit-time `-e VAR=...` still wins. Precedence: image-baked < models.yaml < submit -e.
+# Captured HERE (before the slurm sets any defaults) so it reflects user intent only.
+_RECIPE_ENV_KEYS="VLLM_USE_V1 VLLM_USE_LAYERNAME VLLM_ROCM_USE_AITER VLLM_ROCM_USE_AITER_RMSNORM VLLM_ROCM_USE_AITER_MLA KV_BLOCK_SIZE KV_CACHE_DTYPE KV_CACHE_MEMORY_BYTES GPU_MEMORY_UTILIZATION VLLM_CUDAGRAPH_MODE PREFILL_CUDAGRAPH_MODE DECODE_CUDAGRAPH_MODE CUDAGRAPH_CAPTURE_SIZES VLLM_ALL2ALL_BACKEND PREFILL_MORI_BACKEND DECODE_MORI_BACKEND MORI_SHMEM_HEAP_SIZE"


+        # Thinking models (e.g. GLM-5.1) emit chain-of-thought into a separate
+        # reasoning field and leave `content` empty until the final answer; with a
+        # small max_tokens the answer never appears in `content` and the score is a
+        # false 0/10. Disable thinking so the answer lands in `content` directly.
+        "chat_template_kwargs": {"enable_thinking": False},


MIR-AMD · 2026-08-18T05:38:16Z

OCI 1P/1D EP8 NIAH-2k smoke — FAIL (2k retrieval 0/10)

Slurm job 216847 on partition amd-rccl, 2 nodes (useocpm2m-097-121 prefill+proxy, useocpm2m-097-144 decode). GLM-5.1-FP8 disaggregated 1P/1D EP8 over the MoRI-EP wideEP path (CONNECTOR=moriio WIDE_EP=1 EP_BACKEND=mori, GLM_SKIP_PATCHERS=1), NIAH harness at a single 2000-word context, one seed, warmup enabled (BENCHMARK_SCRIPT=niah NIAH_WORDS=2000 NIAH_SEEDS=0 NIAH_WARMUP=1).

Image: rocm/pytorch-private:glm-dockerimage-built-09072026
Model: ✓ Selected MODEL_PATH: /mnt/m2m_nobackup/models_blog/GLM-5.1-FP8 (available on all nodes)

Decisive lines

[glm] GLM_SKIP_PATCHERS=1: image carries DSA fixes in-source; skipping runtime patchers.
Proxy (vllm_router) ready for benchmarking on useocpm2m-097-121:10.158.212.154:30000
Router registration complete after 5s.
=== NIAH warmup (one throwaway request per length) ===
words=  2000  [warmup] ok
words=  2000  seed=0  found= 0/10  []
=== NIAH summary (mean/min/max across 1 seed(s)) ===
  words=  2000  mean=0.0/10  min=0  max=0  (n=1)

Verdict: FAIL

Bring-up is clean and the patcher-skip path works exactly as intended, but the scored retrieval is 0/10 at 2000 words, which is the one bar item that fails.

What passed:

GLM_SKIP_PATCHERS=1 logged on both nodes, and no apply_glm_*.py was applied — the patcher filenames that appear in the logs are only the Listing NIXL_COOKBOOK_PATH: /opt/nixl-vllm-cookbook directory listing, not an application step.
Proxy ready and router registration completed in 5s. The bring-up /v1/completions canary and both NIAH requests (warmup + scored) all returned HTTP 200 through the P/D path, with MoRIIO handshake done for request chatcmpl-... on the prefill side.
No EngineDeadError, hipErrorIllegalAddress, asm_mla.cu:945, Traceback, or CUDA error anywhere in slurm-216847.out (.err is 0 bytes) or in any per-node log.
Clean exit, not a wall-clock kill: Script completed successfully. and sacct reports COMPLETED 00:56:38 0:0, well inside the 90-minute limit.

Failing evidence: words= 2000 seed=0 found= 0/10 []. The scored request returned HTTP 200 in roughly two seconds and its content / reasoning_content contained none of the 10 planted animals, so this is a real wrong answer rather than a timeout or a cold-compile artifact — warmup for this shape had already succeeded ([warmup] ok) immediately before. At 2000 words this is far below the ~14–18k-token threshold where the upstream sparse-MLA prefill indexer corruption is documented, so the known long-context caveat does not explain it. Consistent with degraded output, the bring-up canary was also degenerate: prompt Who is AMD CEO? returned " Who is the CEO of AMD? Who is the".

Noted but not bar-failing: [niah] WARN: router readiness not confirmed in 300s; proceeding (warmup will absorb) — the harness /v1/models readiness probe never observed a 200 from vllm-router even though the router was already serving traffic on the same port, so the run burned the full 300s gate before scoring. Worth a follow-up on the probe endpoint.

This run exercises the trimmed patcher defaults from this PR (persist-gate and DSA sentinel both opt-in OFF) on the v0.27 image (server reports vllm-0.16.0rc2.dev4561+g545292d14-dp8-ep-07541063).

Logs: /shared_inference/miali/model_blog_logs/slurm-216847.out and /shared_inference/miali/model_blog_logs/216847/

The recipe named a pre-v0.27 image as "validated" and described patchers that no longer apply. Serving GLM-5.1-FP8 on that older image returns empty completions (NIAH 2k = 0/10), so name the image the Dockerfile actually builds and state that the patchers are an old-image fallback.

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated no new comments.

Suppressed comments (2)

scripts/vllm_dissag/benchmark_niah.py:75

benchmark_niah.py claims it is “generalized to run against any OpenAI-compatible endpoint”, but the request body now includes the non-standard field chat_template_kwargs. This can cause 400s on strictly OpenAI-compatible servers or older vLLM/OpenAI adapters. Consider keeping the default request schema standard and relying on the existing scoring of reasoning fields for thinking models.

        # Thinking models (e.g. GLM-5.1) emit chain-of-thought into a separate
        # reasoning field and leave `content` empty until the final answer; with a
        # small max_tokens the answer never appears in `content` and the score is a
        # false 0/10. Disable thinking so the answer lands in `content` directly.
        "chat_template_kwargs": {"enable_thinking": False},
    }

scripts/vllm_dissag/run_xPyD_models.slurm:83

MODELS_YAML_PROTECT is built from a hard-coded whitelist that currently omits several env keys introduced in models.yaml (e.g. TORCH_NCCL_* and VLLM_SPARSE_INDEXER_MAX_LOGITS_MB). If a user sets those at submit time, they will not be protected and models.yaml will override them inside the container, contradicting the intended precedence “submit-time -e wins.”

_RECIPE_ENV_KEYS="VLLM_USE_V1 VLLM_USE_LAYERNAME VLLM_ROCM_USE_AITER VLLM_ROCM_USE_AITER_RMSNORM VLLM_ROCM_USE_AITER_MLA KV_BLOCK_SIZE KV_CACHE_DTYPE KV_CACHE_MEMORY_BYTES GPU_MEMORY_UTILIZATION VLLM_CUDAGRAPH_MODE PREFILL_CUDAGRAPH_MODE DECODE_CUDAGRAPH_MODE CUDAGRAPH_CAPTURE_SIZES VLLM_ALL2ALL_BACKEND PREFILL_MORI_BACKEND DECODE_MORI_BACKEND MORI_SHMEM_HEAP_SIZE"
MODELS_YAML_PROTECT=""
for _k in $_RECIPE_ENV_KEYS; do
    [ -n "${!_k+x}" ] && MODELS_YAML_PROTECT="${MODELS_YAML_PROTECT} ${_k}"
done

MIR-AMD · 2026-08-18T06:44:28Z

Serving smoke is green, so the test plan above is now complete.

OCI amd-rccl, 2 nodes, 1P/1D (EP8), image rocmshared/pytorch-private:glm5.1-vllm027-b8:

[glm] GLM_SKIP_PATCHERS=1: image carries DSA fixes in-source; skipping runtime patchers.
prefill + decode engines ready, Proxy (vllm_router) ready, router registration complete after 5s
disaggregated KV transfer verified end to end
NIAH 2k retrieval 10/10
no EngineDeadError, no hipErrorIllegalAddress, no asm_mla.cu:945; job exited cleanly

One note worth flagging for anyone reproducing this: the same branch on the older rocm/pytorch-private:glm-dockerimage-built-09072026 boots fine and then returns empty completions (NIAH 2k = 0/10) with no crash. That is why the recipe comment now names the v0.27 image explicitly.

raviguptaamd and others added 10 commits August 17, 2026 21:30

MIR-AMD requested review from amathews-amd, coketaste, gargrahul and ppalaniappan-amd as code owners August 18, 2026 04:58

Copilot AI lite review requested due to automatic review settings August 18, 2026 04:58

MIR-AMD requested a review from Rohan138 as a code owner August 18, 2026 04:58

MIR-AMD mentioned this pull request Aug 18, 2026

vllm_dissag: GLM-5.1/5.2-FP8 (MLA+DSA) MoRI-EP WideEP disaggregated enablement #176

Open

5 tasks

Copilot started reviewing on behalf of MIR-AMD August 18, 2026 04:59 View session

Copilot AI reviewed Aug 18, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings August 18, 2026 05:46

Copilot started reviewing on behalf of MIR-AMD August 18, 2026 05:46 View session

Copilot AI reviewed Aug 18, 2026

View reviewed changes

Navigation Menu

Uh oh!

vllm_dissag: GLM-5.1-FP8 (MLA+DSA) MoRI-EP WideEP disaggregated enablement - #206

vllm_dissag: GLM-5.1-FP8 (MLA+DSA) MoRI-EP WideEP disaggregated enablement#206
MIR-AMD wants to merge 11 commits into
ROCm:developfrom
MIR-AMD:glm5.1-wideep-mad

MIR-AMD commented Aug 18, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

MIR-AMD commented Aug 18, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

MIR-AMD commented Aug 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

MIR-AMD commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What lands

Deliberately kept out

Patcher defaults

Which image to serve this on

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

MIR-AMD commented Aug 18, 2026

OCI 1P/1D EP8 NIAH-2k smoke — FAIL (2k retrieval 0/10)

Decisive lines

Verdict: FAIL

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

MIR-AMD commented Aug 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

MIR-AMD commented Aug 18, 2026 •

edited

Loading