vllm_disagg: GLM-5.2 (FP8 + MXFP4) on MI355X/gfx950 + AI NIC, 1P/1D EP8 - #3
Open
raviguptaamd wants to merge 13 commits into
Open
vllm_disagg: GLM-5.2 (FP8 + MXFP4) on MI355X/gfx950 + AI NIC, 1P/1D EP8#3raviguptaamd wants to merge 13 commits into
raviguptaamd wants to merge 13 commits into
Conversation
scripts/vllm_dissag/.nfs0000000016f44d2b00008188 was committed by accident. It is a stale byte-copy of run_xPyD_models.slurm (same #!/bin/bash, #SBATCH --job-name=vllm-pd, #SBATCH -N 2 header) left behind by NFS when the real file was rewritten while an open handle still referenced it. Nothing reads it and it will drift from the real launcher on every edit.
… a silent wrong-arch bug
Targets MI355X and the Pensando Ionic AI NIC:
GFX_COMPILATION_ARCH gfx942 -> gfx950
NIC_COMPILATION_ARCH cx7 -> ionic
MORI_GPU_ARCHS gfx942 -> gfx950
MORI_DEVICE_NIC (new) = ionic
MORI_DEVICE_NIC is pinned rather than left to auto-detect. Detection does resolve
correctly here (8x rocep*s0 -> driver readlink -> ionic, with
libionic.so.1.1.54.0-184 present), but pinning keeps the device-side IBGDA JIT
path deterministic across nodes.
Renames two build ARGs, which is the load-bearing part of this commit:
ARG PYTORCH_ROCM_ARCH -> ARG BUILD_ROCM_ARCH
ARG MAX_JOBS -> ARG BUILD_MAX_JOBS
(both assigned through to the original ENV names)
The base image (rocm/vllm-dev:ci_base-*) exports PYTORCH_ROCM_ARCH and MAX_JOBS as
ENV, and an inherited ENV overrides a same-named ARG in the child Dockerfile. So
`ARG PYTORCH_ROCM_ARCH="gfx950"` was being silently ignored and the build used the
base's `gfx90a;gfx942;gfx950` instead.
This surfaced only by luck: the base also sets MAX_JOBS to the empty string, so
setup.py::compute_num_jobs crashed on int(''). Had it been non-empty the build
would have "succeeded" while compiling for the wrong arch list, and every
downstream perf number would have been measured on an unintended binary with
nothing in the logs to say so. Confirmed fixed by the rebuild log:
"84 warnings generated when compiling for gfx950".
Also replaces the post-install verification heredoc with an equivalent
`python3 -c "$(printf ...)"` form, and reads the vLLM version via
importlib.metadata instead of `import vllm` (importing pulls torch -> amdsmi ->
libamd_smi.so, which is not loadable in the no-GPU build sandbox).
…, disable 2 harmful patchers
Three independent fixes to the MoRIIO connector wiring.
1. qp_per_transfer / num_workers / post_batch_size now go through
kv_connector_extra_config instead of VLLM_MORIIO_* env vars, which the
connector no longer reads. moriio_common.py:227-229 holds a rename map and
_warn_deprecated_env_vars() only warns; moriio_engine.py reads no env vars at
all; vllm/envs.py has no MORIIO entries. The values are consumed solely at
moriio_common.py:342-344 via extra_config.get(), defaulting to 1/1/-1.
So the env vars set in moriio.env were silently inert, and the live log read:
Using MoRIIO backend: RDMA (qp_per_transfer=1, post_batch_size=-1, num_workers=1)
i.e. one queue pair and one worker thread for the whole prefill->decode KV
handoff on a node with 8 RDMA rails. Grep that line in the decode log to
confirm the values now change.
Also passes host_ip explicitly. The connector otherwise derives it from
get_ip(), which on this cluster returns the non-routable public address, so
decode cannot notify prefill of block allocation and the two deadlock.
2. The DSA patcher gate goes from an exact "GLM-5.1-FP8" match to the whole GLM
DSA family (GLM-5.1-FP8 | GLM-5.2-FP8 | GLM-5.2-MXFP4). These are the same
architecture -- GlmMoeDsaForCausalLM, 78 layers, 256+1 experts, index_topk=2048,
verified by diffing config.json -- so they need identical patchers. Under the
old exact match GLM-5.2 silently skipped all of them.
3. Two patchers become opt-in (default OFF). Both are actively harmful on this
image; each retains a full rationale in-line, summarised here:
apply_glm_dsa_persistent_kernel_gate_fix.py (GLM_PERSIST_GATE=1 to restore)
Sets work_meta_data=None on chunked-prefill continuations, routing to a
non-persistent kernel that does not exist for this cell: gqa_ratio=64 +
fp8/fp8 hits AITER_CHECK(false) at asm_mla.cu:945 -> C++ abort, no Python
unwind, server dies. Explains the reproducible 2000-ok / 8000-dies boundary
(8192 = prefill max_num_batched_tokens). The image already carries aiter#3921
and vllm#47766, which superseded #47567 and whose diff is the literal inverse
of this patcher.
apply_glm_dsa_kernel_fix.py (GLM_DSA_SENTINEL_FIX=1 to restore)
Flips the DSA invalid-token sentinel 0 -> -1 per vllm#45324. This image ships
0 deliberately: aiter's decode-only mla_decode_fwd sparse kernel dereferences
paged_kv_indices, so -1 becomes kv_cache + (-1)*stride -> memory access fault,
and it only bites under disagg, which is what we run. Applying it killed
decode DP0 with hipErrorIllegalAddress on a 5-token warmup curl. With
index_topk=2048 a short prompt is the worst case, not the safest.
…r MI355X
Adds the two GLM-5.2 recipes and corrects two stale values inherited from the
GLM-5.1/MI300X recipe they were derived from.
New: GLM-5.2-FP8 (zai-org/GLM-5.2-FP8) and GLM-5.2-MXFP4 (amd/GLM-5.2-MXFP4),
1P/1D EP8 over MoRI-IO. Measured on 2x8 MI355X at ISL/OSL 28672/1024:
TPOT 33.7 / 36.4 / 40.2 ms at concurrency 16 / 32 / 64, against a 50 ms target.
GLM-5.2-MXFP4 is CONFIG-ONLY AND HAS NEVER BEEN BOOTED. It is a copy of the FP8
recipe with the two values that provably do not transfer re-derived. Treat its
first run as bring-up, not benchmark.
Two values carry the recipe; both are noted in full in-file.
DECODE_CUDAGRAPH_MODE: FULL_AND_PIECEWISE (was PIECEWISE)
The single change that fixed TPOT: 104 -> 34-40 ms. PIECEWISE splits the decode
graph on the 3 DSA ops x 78 layers, ~234 launch boundaries per step. The cost is
batch-INDEPENDENT, so it presents as a latency floor that does not move with
concurrency, batch size or fabric -- which is why it survived so much tuning
before being found. Co-requisite: use_inductor_graph_partition must stay ON and
no bare --enforce-eager, else boot and warmup both pass and the first real MLA
decode dies.
prefill.dp: --gpu-memory-utilization 0.72
Prefill only. The MLA chunked-prefill workspace is sized from a hardcoded
64k-token clamp (determine_chunked_prefill_workspace_size,
mla_attention.py:1935), NOT from max_num_batched_tokens, so it allocates
65536*64*(192+256)*2 = 3.50 GiB on top of the KV pool regardless of scheduler
batch size -- and lazily, on the first long prefill, i.e. after boot and warmup
have both reported healthy. At 0.80 this OOM'd at mla_attention.py:739.
Corrections to the shared GLM section:
VLLM_SPARSE_INDEXER_MAX_LOGITS_MB: 64 -> 2048
64MB was an MI300X crash workaround, and the crash cannot occur here: the
workaround kernel is behind a hard arch gate (rocm_aiter_mla_sparse.py:591
dispatches fp8_mqa_logits_gfx942 only if _ON_GFX942). Verified live in-image:
_ON_GFX942=False, arch=gfx950. The module self-describes as a "Temporary gfx942
fallback" for that part's 64 KiB LDS budget. Keeping 64MB cost 50 indexer
sub-chunks at ISL 28672 where 2048MB needs 2, each a full kernel chain over 78
layers. 2048 not 4096 deliberately: nearly all the reduction at half the peak
memory (2.00 GiB, against ~31 GiB/GPU idle).
KV_BLOCK_SIZE comment corrected (value unchanged at 1)
It claimed the DSA sparse indexer "REQUIRES" block-size 1. That inverts the real
constraint: rocm_aiter_mla_sparse.py:283 returns [1, 64] and indexer.py:137
returns `[1, 64] if current_platform.is_rocm() else [64]` -- ROCm is the
PERMISSIVE branch and it is non-ROCm that is locked to 64. Measured 1 vs 64:
<2% either way, so 1 stays as the validated default, but the comment would have
stopped anyone from trying 64.
The same section's "decode PIECEWISE" prose is updated to match the new value.
…/ionic env file
Everything needed to run this recipe on a second cluster without editing the
launcher. Previously several site facts were baked in as defaults from the MI300X
cluster it was written on.
FABRIC_SUBNET is the important one. It defaults to 10.158. (the MI300X fabric);
AAC is 10.2.80.. When it does not match, the awk picker falls through to the first
address on the line -- a 10.224.x overlay -- MASTER_ADDR is advertised as something
the peer cannot reach, and the run hangs at "Waiting for nodes" forever with no
error. So a fresh clone of this recipe would have hung on AAC.
Unlike every other key in the connector .env files, FABRIC_SUBNET is consumed by
the launcher on the HOST rather than inside the container, and the .env loader only
built `-e KEY=VAL` docker args. Setting it in the .env file would therefore have
been silently inert. The loader now has an explicit one-key host-export allow-list.
Deliberately an allow-list and not a blanket export: the other keys are HSA/RDMA
runtime settings that should not leak into srun and other host-side tooling.
Submit-time `export FABRIC_SUBNET=` still wins over the file, as before.
Also in the launcher:
- CONNECTOR_ENV_FILE is overridable, so a site can supply its own platform env
(e.g. connectors/moriio.env.aac) without touching the tracked default.
- CONTAINER_CLI auto-selects docker or podman by probing for a usable docker
daemon. Compute nodes here deny the daemon socket and only have podman. The
two differ in ways that matter: podman gets --group-add keep-groups (video
is not a valid group there) and no --shm-size (it does not accept it in this
configuration).
- MODEL_DIR defaults to /shared/data/amd_int/models when that exists.
- Optional site mounts (/shared/data, /shared/apps, /media/NVME, ...) are mounted
only when present, instead of failing on absence.
- The JIT cache host path is chosen from a candidate list keyed by image ID, and
falls back with a loud warning rather than silently paying a cold compile every
run.
- GLM-5.2-FP8 and GLM-5.2-MXFP4 added to the three model allow-lists.
- New niah / niah_perf benchmark cases.
connectors/moriio.env.aac (new) is the AAC MI355X + Pensando Ionic platform env:
8 ionic rails (rocep{9,25,105,121,137,153,233,249}s0) at MORI_IB_GID_INDEX=1,
control/bootstrap on the mgmt NIC via MORI_SOCKET_IFNAME=enp193s0f1np1, and
FABRIC_SUBNET=10.2.80..
The interface choice is not the obvious one and the file records why: the
public/default-route NIC answers ICMP but has TCP firewalled node-to-node, so
"the one that pings" picks the wrong link. Test the actual port -- measured
gpu-44 -> gpu-45, /dev/tcp/<public>/22 FAILS while /dev/tcp/10.2.80.16/22 succeeds.
MORI_SOCKET_IFNAME left undefined fell through to a hardcoded eth0 that does not
exist here, ShmemGetUniqueId failed, and the null shmem context then segfaulted
inside EpDispatchCombineHandle -> ShmemNumQpPerPe().
…er needle placement benchmark_niah.py printed a score table and always exited 0, so an accuracy regression could not fail a run -- it had to be spotted by eye. It now exits non-zero: 3 when a context length produced no usable result, 4 when mean retrieval is below NIAH_MIN_SCORE (default 8.0/10), 0 otherwise, with an explicit "NIAH VERDICT: PASS/FAIL" line and a per-row marker on the offending lengths. Needle placement was fully deterministic: every needle sat at exactly (i+1)*step, the same offsets for every seed, so the documented NIAH_SEEDS sweep was re-testing one identical layout and its mean/min/max spread across seeds could not show what it claimed to. Needles are now jittered within their slot (seed 0 keeps the exact old placement, so existing baselines stay comparable). Adds benchmark_niah_perf.sh: the same NIAH retrieval scoring driven at concurrency so accuracy is checked under load rather than only single-stream. Known gap, not addressed here: needles are placed at (i+1)*step for i in 0..len(ANIMALS)-1, so nothing is ever placed in the final step-width of the context. The last chunk -- often the interesting one for a long-context regression -- is never probed.
… backend and arch tags
Three reporting bugs. None affected the served workload, all affected the numbers
that get filed.
1. The first (lowest-concurrency) cell of every run was silently discarded. The
parse loop guarded its config lookup with `if i > 1`, which reads like a bounds
check but is not one: enumerate starts at 1, so i-1 is 0, a valid index --
sections[0] is the preamble that holds the first cell's [RUNNING] line. A 21-cell
sweep produced 20 rows. Regression-checked with a synthetic 3-cell log.
2. Rows with no parseable config or throughput were dropped silently, which made
this parser bug look like a benchmark-loop bug. They now print a warning naming
the result index.
3. Two hardcoded tags:
backend fell back to 'nixl'. The real selector is CONNECTOR={rixl|moriio},
resolved and exported by run_xPyD_models.slurm:188-237; RUN_MORI/RUN_DEEPEP are
the legacy flags mapped onto it there. So every run driven the current way --
including all the MoRIIO ones -- was filed as nixl. Now reads CONNECTOR,
defaulting to moriio, with the legacy flags still taking precedence.
gpu_architecture was hardcoded 'gfx942', so MI355X runs were filed as MI300X.
Now from GPU_ARCHITECTURE, defaulting to gfx950 (this recipe's target). It is
not introspected because this parser usually runs on the head node, which has
no GPU.
Adds GLM52_MI355X.md (invocation, measured numbers, the two load-bearing config values, the cluster-specific traps) and extends skills_vllm_disagg.md with the perf-tuning findings from this bring-up. Records the results honestly, including what is not solved: TPOT 33.7 / 36.4 / 40.2 ms at concurrency 16 / 32 / 64 -- target 50 ms, met. Prefill is 6,135-8,359 tok/s/rank against a 34,000 target, ~4-5x short, and is now the binding constraint. Profiled at 8192 tokens: MLA sparse attention ~82%, all GEMM 17.8%, of which MoE 2.9%. At 80K ISL the fitted single-request TTFT is ~13.7 s on one rank before any concurrency, against a <7 s SLO. TTFT at concurrency is drain time (458,752 tok / 39,340 tok/s = 11.66 s, an identity that reproduces the measurement), so more prefill nodes fix concurrent TTFT but do nothing for the single-request number. That needs tuned MLA sparse-attention kernels for gfx950 or intra-request sharding. Also records the levers closed BY MEASUREMENT so they are not re-litigated: MoE and AITER tuning (2.9% of prefill), indexer sub-chunking (<=2.5%), max_num_batched_tokens (<=4%), RDMA QP/worker count (-1.9%, inside +-0.4% noise), FP8BMM (~0.7%), block_size 1->64 (<2%). And why PCP is not the near-term answer: config/parallel.py:528 is DP-incompatible, rocm_aiter_mla_sparse.py has no PCP plumbing at all, and platforms/rocm.py:894 force-sets cudagraph_mode=PIECEWISE, which would silently undo the TPOT fix. Two constraints called out because both fail in a way that looks like success: use_inductor_graph_partition must stay ON and no bare --enforce-eager (boot and warmup pass, the first real MLA decode dies), and the MXFP4 recipe has never been booted.
Owner
Author
…oodput-gated) benchmark_xPyD.sh is a tuning sweep and cannot answer "do we meet the SLO". This adds a harness that can, and exits non-zero when we do not. Four differences from the sweep, each of which was producing a wrong answer: 1. --request-rate inf is a saturation test. It fires everything at once, so TTFT is just queue drain time -- at con=64/isl=28672 the measured TTFT is exactly 458,752 tok / 39,340 tok/s = 11.66 s, an identity rather than a property of the model. The customer's "<7 s avg TTFT" is about a *served* request, so this runs at a finite Poisson rate derived from Little's Law: rate = concurrency / SLO latency. That asks "can it serve the offered load AT the SLO", which is the actual question. 2. The sweep reports mean/max; the customer asked for p50/p95/p99. Now passes --percentile-metrics ttft,tpot,itl,e2el --metric-percentiles 50,95,99. 3. Pass/fail was a human reading a log. The SLO is now handed to vLLM directly via --goodput ttft:7000 tpot:50, so the fraction of traffic actually served acceptably is a number the server computes. A cell can have a passing mean TTFT while a third of its requests miss; goodput catches that and the mean does not. 4. --random-prefix-len 0 makes every request unique. The customer's use case is agentic and they flagged interest in long prefix caching -- an agent loop re-sends a large static context every turn. vLLM's RandomDataset builds the prefix once and shares it across requests, so PREFIX_FRAC (default 0.5) models that. 0.5 is a declared assumption, not a measurement: the customer gave no reuse ratio, so sweep it. Scenarios are the customer's own: 80K/1K (256K context) and 200K/1K (1M context), concurrency to 256/DP. The 200K concurrency list stops at 64 on purpose. GLM-5.2 MLA KV is 43.88 KiB/token (kv_lora_rank 512 + qk_rope 64, FP8, 78 layers), so 200K = 8.37 GiB/request against a ~930 GiB single-node pool -- concurrency 256 would need ~2,142 GiB and cannot fit on one decode node. Defaulting to a list that fits means a failure is a real failure and not a predictable OOM. The arithmetic is in the header so the ceiling can be recomputed rather than rediscovered. slo_report.py divides throughput by --dp-ranks because the customer's targets are per-rank and vLLM reports aggregate; not doing so overstates by the DP degree. It labels prefill tok/s as an end-to-end lower bound (total_input_tokens/duration, which includes decode time for the same requests) rather than passing it off as an isolated prefill-engine number.
…backend
The customer sheet gives an AVERAGE ISL and a context window. gen_workload.py turns
that into a lognormal whose mean is the average and whose p99 lands on the window --
but the realised mean of a single few-hundred-request draw from a right-skewed
distribution is a property of the seed, not of the workload:
CV = sqrt(exp(sigma^2)-1), SE(mean)/mean = CV/sqrt(n)
256K row: sigma 0.5833 -> CV 0.637, n=256 -> 4.0%
1M row: sigma 0.8778 -> CV 1.078, n=128 -> 9.5%
Measured over 12 seeds the realised means actually spanned 14.2% and 35.4%. Quoting
one run's mean would be quoting that seed.
RESAMPLE_PER_ITER=1 makes each iteration draw a FRESH sample (seed = SEED_BASE +
iter-1) so iterations POOL and the error falls as 1/sqrt(iters) -- to 1.3% and 3.0%
at ten. The previous behaviour (one sample replayed) is kept as the default because
it answers a different, also-valid question: server variance on a fixed workload.
Averaging one sample with itself cannot shrink its sampling error.
benchmark_avg_{80K,200K}_ten.sh are thin wrappers that set exactly that. All
measurement logic (Little's-Law rate, goodput thresholds, timeouts, warmup) stays in
benchmark_customer_slo.sh with one implementation; two copies would drift and the
drift would surface as an unexplained delta between the two rows.
pool_workload.py pools RAW LENGTHS and re-computes. Percentiles do not average: the
mean of ten p99s is not the p99 of the pool, and for a heavy tail it is materially
lower. It refuses to pool samples with different targets, because a pooled 80K+200K
sample describes no workload the customer asked about. Pooled results: 80K row
+0.6%, 200K row -1.4%, both p99 landing exactly on the stated window.
Stratified inverse-CDF placement was considered and rejected. It pins the mean to
1-2% with no pooling, but caps the sample at the (n-0.5)/n quantile, so p99 would
reach ~225K instead of 262,144. The window is the number the customer actually
stated; we do not trade it away to tidy a number they did not.
Also unhardcodes the driver so this runs against something other than vLLM:
BENCH_BACKEND (default vllm), BENCH_EXTRA_ARGS, and IGNORE_EOS. --ignore-eos is a
vLLM extension and is what makes OSL exact; dropping it means the model stops when
it wants to and the measured TPOT averages over a length we did not choose, so the
50 ms/token verdict becomes approximate. The header says so, because the risk is
that someone quotes it as the sheet's number anyway.
Two container defects fixed while wiring this up, both of which would have destroyed
a real run: the wrappers wrote results under /workspace, which is NOT a mounted path
(only -v ${LOG_PATH}:/run_logs is), so ten runs of results would have gone to the
container overlay and died with the container; and the pooler globs were hardcoded
to the default scenario labels, so overriding SCENARIOS would have matched nothing
and said so only at the very end of a ten-run job.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…orting Two defects in the existing NIAH sweep, both of which produce a misleading result rather than an error. NIAH_WORDS measures WORDS. The context numbers we are asked about are TOKENS, and for this filler the ratio is ~1.30 -- so "950,000" read as words is ~1.24M tokens, ABOVE a 1,048,576 window. vLLM REJECTS an over-length request with a 400 rather than truncating, and that arrives at the client as a transport error indistinguishable from a dead server. NIAH_TOKENS calibrates against the real tokenizer and approaches the target FROM BELOW: overshooting a window is a hard failure, undershooting is a slightly shorter test. Without transformers installed it falls back to NIAH_TOKENS_PER_WORD and prints an UNCALIBRATED warning, so the lengths are never silently passed off as exact. A bare score was uninterpretable. "7/10 at 950K" and "7/10 at 950K, all three misses past 80% depth" call for completely different investigations -- the second says the tail of the context is being dropped. Misses now report their depth, with a decile histogram. Exit codes 3 and 4 are deliberately distinct: 3 = some rung produced no usable result (dead server, timeout, rejection), 4 = every rung scored but a mean fell below NIAH_MIN_SCORE. "We could not measure it" and "we measured it and it is bad" are not the same finding. benchmark_niah_long.sh drives the ladder and is where the wall-clock budget lives. Prefill is ~QUADRATIC here: the sparse indexer scores each 8,192-token prefill chunk against all preceding keys (index_topk=2048 caps the DECODE key set, not the prefill scan). Scaling from a measured 4.80 s TTFT at 28,672 tokens the default ladder is ~3.3 h at one seed with no warmup, ~6.6 h with warmup, ~13.2 h at three seeds with warmup. The last TWO rungs are 74% of that, so dropping 950,000 alone buys back 44% -- the per-rung table is in the header so that trade can be made rather than rediscovered. Hence NIAH_SEEDS=0 and NIAH_WARMUP=0 here, unlike benchmark_niah.py. Timeouts scale per rung, quadratically, floored at 300 s. A single flat timeout sized for 950K means a dead server burns 4.4 h on the 32K rung and the job wall clock expires before a single line prints. Readiness is a real completion, not /v1/models: that endpoint is wrong in BOTH directions on this router -- 200 while every request 503s, and 503 while completions succeed -- because its listing path and its forwarding path consult different state. And the ladder invokes benchmark_niah.py directly rather than through benchmark_niah.sh, whose trailing `| tee` always exits 0, so a failing ladder would have reported success. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…t them Registers niah_long, avg_80k_ten and avg_200k_ten as BENCHMARK_SCRIPT values, adds the five new files to REQUIRED_FILES, and fixes two latent launcher defects found while wiring them up. The docker -e passthrough block is a hard gate: an env var not listed there simply does not exist inside the container. The ENTIRE customer_slo family -- SCENARIOS, SLO_ITERS, SLO_TTFT_MS, SLO_TPOT_MS, WORKLOAD_MODE, TAIL_FRAC, PREFIX_FRAC, MAX_PROMPTS, BURSTINESS, RESAMPLE_PER_ITER, SEED_BASE, DP_RANKS, TGT_* -- was never in it. That benchmark had only ever been runnable at its compiled-in defaults, and every override was accepted at the sbatch line and silently dropped. Adds those plus the four new NIAH token knobs. The selector's error message listed a stale set of valid values, omitting niah and niah_perf. A correct BENCHMARK_SCRIPT=niah_perf that tripped over an unrelated typo would be told its own value was illegal, which sends you looking in the wrong place. BENCHMARKS.md documents the three benchmarks for use against ANY OpenAI-compatible server, not just this launcher, since the question they answer is not specific to this cluster. It carries the dependency matrix (benchmark_niah.py, pool_workload.py and slo_report.py are stdlib-only and run anywhere; gen_workload.py needs transformers; only the SLO driver needs vllm bench serve), the runtime budget, the env-var tables, the exit-code contract, and the failure modes that present as good results -- chiefly that a rejected over-length request vanishes from the latency statistics and shows up only as a lower `completed` count. README.MD documented `export BENCHMARK_SCRIPT_FILE=benchmark_niah.sh`. That is the exact form the launcher's `case` discards: it assigns BENCHMARK_SCRIPT_FILE with a plain `=`, so an inherited value is overwritten and the default throughput sweep runs instead. Corrected to BENCHMARK_SCRIPT=niah, with a note, since a documented-but-inert knob costs a full job to diagnose. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
argv_assert.sh grepped the slurm for the literal
CONNECTOR_ENV_FILE="${SCRIPT_DIR}/connectors/${CONNECTOR}.env"
but 24b5c9c made that path overridable for site portability:
CONNECTOR_ENV_FILE="${CONNECTOR_ENV_FILE:-${SCRIPT_DIR}/connectors/${CONNECTOR}.env}"
so the suite has been red since, on code that is correct. A test that fails when the
product is right is worse than no test: it trains you to ignore the suite, and the
next real regression lands in a run that was already failing.
Matches the per-connector default instead of the exact assignment, which is the
behaviour actually under test. 25/25 pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on ROCm#176 (
glm5.1-wideep-clean), which is still open — base is that branch, notdevelop, so this diff stays at the 11 files that are actually new. Retarget todeveloponce ROCm#176 merges.
Enables GLM-5.2 (FP8 + MXFP4) on MI355X (gfx950) with AMD AI NIC (ionic) for 1P/1D EP8
disaggregated serving over MoRI-IO. Measured on AAC, 2 nodes x 8 x MI355X, ROCm 7.2.3.
Result
TPOT target met. ISL/OSL 28,672/1,024, FP8, EP8, 1P/1D:
Target is 50 ms avg. Before this change it was 104 ms.
TTFT target not met, and this PR does not fix it. Prefill measures 6,135–8,359
tok/s/rank against a 34,000 target (~4–5x short), and the fitted single-request TTFT at
80K ISL is ~13.7 s against a <7 s SLO.
GLM52_MI355X.md§5 records the profile behindthat, the levers already closed by measurement, and why PCP is not the near-term answer.
The two load-bearing values
DECODE_CUDAGRAPH_MODE: FULL_AND_PIECEWISE— the single change that moved TPOT104 ms -> 34–40 ms.
PIECEWISEsplits the decode graph on the 3 DSA ops x 78 layers,~234 launch boundaries per step. That cost is batch-independent, so it presents as a
latency floor that does not move with concurrency, batch size, or fabric — which is why it
survived so much tuning before being found. Co-requisite:
use_inductor_graph_partitionmust stay ON and no bare
--enforce-eager, or boot and warmup both pass and the first realMLA decode dies.
prefill.dp: --gpu-memory-utilization 0.72— the MLA chunked-prefill workspace issized from a hardcoded 64k-token clamp (
determine_chunked_prefill_workspace_size), notfrom
max_num_batched_tokens, so it allocates 3.50 GiB on top of the KV pool regardless ofscheduler batch size — lazily, on the first long prefill, i.e. after boot and warmup have
both reported healthy. 0.80 OOM'd.
A silent wrong-arch bug worth flagging
The base image
rocm/vllm-dev:ci_base-*exportsPYTORCH_ROCM_ARCHandMAX_JOBSasENV, which shadows the same-named
ARGin this child Dockerfile. SoARG PYTORCH_ROCM_ARCH="gfx950"was silently inert and the build produced gfx942 kernelswhile appearing to succeed. Fixed by renaming to
BUILD_ROCM_ARCH/BUILD_MAX_JOBSandassigning through. This was only caught because the base's
MAX_JOBShappened to be empty(
int('')crash) — otherwise it fails silently, so it is likely to affect other childDockerfiles in this tree.
Also in here
connectors/moriio.env.aac— AAC/ionic platform env: 8 railsrocep{9,25,105,121,137,153,233,249}s0,MORI_IB_GID_INDEX=1, mgmt NIC for bootstrap.FABRIC_SUBNETdefaulted to the MI300X cluster's10.158.prefix; on AAC (
10.2.80.) that advertises aMASTER_ADDRthe peer cannot reach and therun hangs at
Waiting for nodesforever with no error. Unlike every other key in the envfile this one is consumed on the host, so the
.envloader needed an explicitone-key host-export allow-list — a plain env entry would have been inert.
parse_to_csv.py— was silently dropping the first (lowest-concurrency) cell of everyrun (21
[RUNNING]cells -> 20 rows), tagging every MoRIIO run asnixl, and filing everyMI355X run under
gfx942. All three fixed; dropped rows now warn instead of vanishing.benchmark_niah.py— the accuracy check printed a score but always exited 0; now areal pass/fail gate, with jittered needle placement.
GLM_PERSIST_GATE,GLM_DSA_SENTINEL_FIX):both are correct on older images but here cause a C++ abort at
asm_mla.cu:945and ahipErrorIllegalAddressrespectively. The gate is by image, not by model..nfs0000000016f44d2b00008188, a stalebyte-copy of
run_xPyD_models.slurm).Testing status
FP8 is the measured path — NIAH accuracy plus the 28,672/1,024 sweep at concurrency
16/32/64 above.
GLM-5.2-MXFP4is config-only and has not been booted yet — bring-up is queued. Itsrecipe is the FP8 one with the two values that provably do not transfer re-derived. Both
models.yamlandGLM52_MI355X.mdflag it as untested; treat its first run as a bring-up,not a benchmark.