[TileRT] Add GLM-5.3 FP8 MI355X prefill/decode-disaggregated recipes (8k1k + AgentX) / 新增 GLM-5.3 FP8 MI355X 的 PD 分离配方（8k1k + AgentX） - #3330
Oseltamivir wants to merge 18 commits into
Conversation
…/ 新增 GLM-5.3 FP8 MI355X 的 PD 分离配方
Adds GLM-5.3 FP8 on MI355X as a prefill/decode-disaggregated TileRT recipe:
upstream vLLM ROCm prefill (TileRTConnector, kv_producer) + TileRT
decode_server + an OpenAI-compatible pd_router, framework=tilert, KV over
mooncake. Two scenarios, Single-turn 8k1k and AgentX agentic-coding, mirroring
the merged B200 glm5.1-fp8-b200-tilert entries on the amd_utils orchestration.
以 TileRT 的 prefill/decode 分离形态新增 MI355X 上的 GLM-5.3 FP8：上游 vLLM
ROCm 作 prefill + TileRT decode_server + 兼容 OpenAI 的 pd_router，KV 走
mooncake。两个场景：单轮 8k1k 与 AgentX 智能体编码，形态对齐已合并的 B200
glm5.1-fp8-b200-tilert，走 amd_utils 编排链。
IMAGES AND WHEEL / 镜像与轮子
decode ghcr.io/tile-ai/tilert-rocm-decode:0.1.6
prefill ghcr.io/tile-ai/tilert-rocm-prefill:0.1.6
(the upstream vllm/vllm-openai-rocm release image plus the mooncake
ROCm transfer engine built from source)
Neither image carries the wheel; setup_deps.sh installs tilert==0.1.6 from
PyPI into both ranks at container start -- full on decode, --no-deps connector
plugin on prefill -- so vLLM runs exactly as shipped and no file inside either
image is modified.
TOPOLOGY / 拓扑
1 prefill node (TP8) + 1 decode node (TP8), conc-list [1] because TileRT decode
serves one sequence at a time. bf16 KV cache on both roles -- the ROCm vLLM
sparse-MLA backend has no fp8_ds_mla dtype. max-model-len 202752. MTP via
spec-decoding=mtp at draft depth 3. Prefill runs --block-size 64: the ROCm
backend supports [1,64] and vLLM picks 1, which makes the connector's KI plane
copy fail and MLA address the wrong rows.
GOLDEN AL / 黄金接受率
Nothing hard-codes an acceptance length. server_tilert.sh reads the committed
curve at launch, selecting it exactly as infx/srt_slurm/synthetic_acceptance.py
does (<model-prefix>_<method>.yaml, then [model][thinking mode][draft tokens]),
applies it to agentic throughput only, suppresses it under RUN_EVAL/EVAL_ONLY
because simulated acceptance would score ~0 on GSM8K, and fails the run rather
than falling back to real acceptance when the curve or draft length is missing.
接受率不写死在任何配方或配置里，由 server_tilert.sh 启动时从提交的曲线读取，
选择规则与上游 srt-slurm 连接器一致；仅作用于 agentic 吞吐，eval 下自动关闭，
曲线缺失时直接失败而非静默回落。
NOTE FOR REVIEWERS: golden_al_distribution/glm5.3_mtp.yaml is GLM-5.2's curve,
copied. No SPEED-Bench run exists for GLM-5.3 and the two share a base and
layer topology. Only the K=3 cell is carried over. The file is labelled
PROVISIONAL. Flagged rather than left to read as a measured 5.3 curve.
LOCAL VALIDATION / 本地验证
Run on 2x8 MI350X with these images and wheel, through the official entry
points:
GSM8K (RUN_EVAL + EVAL_ONLY, 1319 items) strict 0.9750, flexible 0.9735
AgentX 3600 s, conc 1 submission_valid true, coverage
99.8%/99.9%, 532 requests
Single-turn 8k1k, conc 1 TTFT p50 1010 ms
MI350X is not MI355X: same gfx950 ISA and memory system, but 1000 W vs 1400 W
and 2200 vs 2400 MHz. These show the recipe runs and is accurate; they are not
leaderboard results, which the sweep produces on real mi355x runners.
本地数据跑在 MI350X 上，不是 MI355X，仅用于证明配方能跑且结果正确，
不作为打榜成绩。
Matrix UTs 335 passed. perf-changelog is a pure tail append; the 8457 lines
inherited from main are byte-for-byte unchanged.
|
Thanks for the contribution!
中文感谢你的贡献！
|
… / 修正 TileRT 配方的 Bash 规范与 MTP 草稿深度
Review follow-ups on the GLM-5.3 FP8 MI355X TileRT recipes.
- server_tilert.sh no longer invents defaults for caller-supplied
configuration. Every value now arrives from the recipe through job.slurm
and is validated with check_env_vars, mirroring server_atom.sh. The same
applies to the two recipes and to the tilert branch of setup_deps.sh.
- Draft depth is consistent. DECODE_MTP_SIZE=3 is the TileRT decode draft
depth, used by the weight converter and as the key the golden acceptance
curve is read at; PREFILL_SPEC_TOKENS=1 is the vLLM prefill rank's MTP
layer. Previously the converter and the curve used a hard-coded 3 while
the recipes exported an unused DECODE_MTP_SIZE=1.
- THINKING_MODE and MODEL_PREFIX are forwarded into the container and
validated, so the golden-acceptance selection no longer falls back to
built-in guesses.
- job.slurm drops MODEL_NAME="${MODEL_NAME:-None}", which masked a missing
model name for every engine, not just tilert.
- models_tilert.yaml keeps only model-owned engine environment; profile,
model_type and max_model_len moved to the recipes, which removes the
stale glm-5_2 model_type that the recipe already overrode.
- launch_mi355x-amds.sh and server.sh no longer carry unrelated comment and
blank-line deletions.
- MODELS.md/MODELS_zh.md and the two changelog entries point at the
upstream PR and carry the correct date.
以评审意见修正 GLM-5.3 FP8 MI355X 的 TileRT 配方：server_tilert.sh 不再为调用方
提供的配置自造默认值，全部改由配方经 job.slurm 传入并用 check_env_vars 校验，
与 server_atom.sh 保持一致；统一 MTP 草稿深度（decode 侧 DECODE_MTP_SIZE=3，
供权重转换与黄金接受率曲线取值，prefill 侧 PREFILL_SPEC_TOKENS=1）；将
THINKING_MODE 与 MODEL_PREFIX 传入容器并校验；移除 job.slurm 中会掩盖所有引擎
MODEL_NAME 缺失的兜底；models_tilert.yaml 只保留模型自有的引擎环境；撤销
launcher 与 server.sh 中与本次改动无关的删除；文档与 changelog 链接改为上游 PR。
|
Review sweep of the carried-over commit; follow-ups pushed in Fixed
Local checks
Still open for a CODEOWNER, unchanged from #3328:
AI model disclosure
中文对迁移过来的提交做了评审清扫，修复以
本地检查结果见上表。仍需 CODEOWNER 裁决： |
…完整 1M max-model-len was 202752 on both roles. That is GLM-5.1's context window, inherited from the B200 glm5.1-fp8-b200-tilert recipe this one mirrors (perf-changelog entry for that recipe records "max-seq-len 202752"), not GLM-5.3's. GLM-5.3 shares GLM-5.2's base, and every in-tree GLM-5.2 recipe -- TRT-LLM and SGLang alike -- runs at 1048576. Both roles now launch at 1048576, and the AgentX trace filter follows it through MAX_MODEL_LEN. 原配方两侧均为 202752，这是 GLM-5.1 的上下文窗口，沿用自本配方对标的 B200 glm5.1-fp8-b200-tilert，并非 GLM-5.3 的。GLM-5.3 与 GLM-5.2 同底座，而树内所有 GLM-5.2 配方（TRT-LLM 与 SGLang）均使用 1048576。现两侧均以 1048576 启动， AgentX 轨迹过滤经 MAX_MODEL_LEN 随之生效。
|
Context length corrected in 202752 is GLM-5.1's context window, inherited from the B200 Both roles now launch at Two things this changes for the sweep, worth watching in the logs: the prefill rank now sizes its KV allocation for a 1M window at The earlier sweep (35594261910) was superseded before its GPU jobs started; the restarted run is https://github.com/SemiAnalysisAI/InferenceX/actions/runs/35595701800. 中文
202752 是 GLM-5.1 的上下文窗口，沿用自本配方对标的 B200 现两侧均以 需在日志中关注的两点：prefill 侧现按 1M 窗口、 此前的 sweep（35594261910）在 GPU 作业开始前已被取代，重启后的运行为 35595701800。 |
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=35715072662 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=35715072662 |
…pool / 补齐 MI355X 上 TileRT 的 amd_utils 编排输入 The tilert arm of benchmarks/multi_node/runtime_settings.sh is the B200 srt-slurm profile. The MI355X recipes run through the shared amd_utils chain instead, whose submit.sh validates ROUTER_TYPE, PROXY_PING_PORT, HEADNODE_PORT and SERVER_PORT, and whose job.slurm and setup_deps.sh also require VLLM_ROUTER_IMAGE, SKIP_RDMA_CHECK, SKIP_GPU_SANITY, ENABLE_METRICS, PREFILL_ROUTER_POLICY, DECODE_ROUTER_POLICY, FLUSH_DRAIN_TIMEOUT, CLEAR_CACHE_BETWEEN_CONC, ROCM_PATH, UCX_HOME and RIXL_HOME. None of those were exported for FRAMEWORK=tilert, so submit.sh exited before sbatch and the recipe returned no job id. The AMD block is scoped by RUNNER_TYPE so the B200 TileRT lane is untouched. The fixed-seq-len recipe now sets KV_OFFLOADING=none itself: submit.sh and job.slurm validate it, and the matrix emits it only for agentic scenarios. launch_mi355x-amds.sh now fails when a recipe returns no job id. Its wait loop polled for slurm_job-.out while its liveness guard degenerated to `grep -q ""`, which matches any queued job of that user, so a failed submit span the loop until the job hit its time limit rather than reporting the error. benchmarks/multi_node/runtime_settings.sh 的 tilert 分支是 B200 的 srt-slurm 配置，而 MI355X 走的是共用 amd_utils 链：submit.sh 校验 ROUTER_TYPE、 PROXY_PING_PORT、HEADNODE_PORT、SERVER_PORT，job.slurm 与 setup_deps.sh 另需 VLLM_ROUTER_IMAGE、SKIP_RDMA_CHECK、SKIP_GPU_SANITY、ENABLE_METRICS、 PREFILL_ROUTER_POLICY、DECODE_ROUTER_POLICY、FLUSH_DRAIN_TIMEOUT、 CLEAR_CACHE_BETWEEN_CONC、ROCM_PATH、UCX_HOME、RIXL_HOME。FRAMEWORK=tilert 下这些一概未导出，导致 submit.sh 在 sbatch 之前退出、配方返回空 job id。AMD 专用 块以 RUNNER_TYPE 限定，不影响 B200。 定长配方自行设置 KV_OFFLOADING=none（矩阵仅在 agentic 场景下产出该字段）。 launch_mi355x-amds.sh 现在在配方未返回 job id 时直接失败：原等待循环会一直轮询 slurm_job-.out，而其存活判断退化为 `grep -q ""`，会匹配该用户任意排队作业，于是 提交失败不会报错，而是空转到作业超时。
|
Root cause of the stuck jobs, fixed in Why they hung. The Reproduced against the stub
Why it hung instead of failing. With Three fixes
Verified against a stub Sweep restarted: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/35602514701 中文
卡死原因： 为何是卡死而不是报错： 三处修复：① 已用桩 |
|
Sweep 35602514701: orchestration now works end to end, and the run failed on a cluster-data gap rather than on code.
Staging it now: No further code change is expected from this; the sweep will be re-dispatched against the same head once the weights are in place. 中文Sweep 35602514701：编排链已完全走通，本次失败源于集群数据缺失，而非代码。
正在补齐： 预计无需再改代码；权重就位后将以同一 head 重新调度 sweep。 |
Port the launcher fix from CrimsonDump/InferenceX commit 9e745f3. Set RDMAV_FORK_SAFE=1 before either role initializes verbs to avoid silent TCP fallback and degraded TTFT; TPOT is unaffected. Append performance changelog entries for 8k1k and AgentX. 在 prefill 和 decode 两侧初始化 verbs 前设置 RDMAV_FORK_SAFE=1，避免静默回退到 TCP 导致 TTFT 恶化；TPOT 不受影响。补充代码注释及 8k1k、AgentX 性能变更记录。
|
Applied the missing RDMA environment fix in commit 中文已在提交 |
…位后重新触发 sweep The three earlier sweeps failed at the model-availability check because /it-share/data/GLM-5.3 did not exist on the MI355X cluster. zai-org/GLM-5.3 is now staged there: 155 files, 141 shards, 704 GiB, every file byte-exact against the Hugging Face manifest, world-readable for the runner account. config.json confirms fp8 (e4m3, weight_block_size 128x128) and max_position_embeddings 1048576, matching the recipe's precision and context. No code change. Re-applying the sweep label produced label events but no workflow run, and run-sweep.yml has no workflow_dispatch trigger, so this empty commit provides the synchronize event. 此前三次 sweep 均因 MI355X 集群上不存在 /it-share/data/GLM-5.3 而在模型可用性 检查处失败。现已完成权重落盘：155 个文件、141 个分片、704 GiB，逐文件字节数与 Hugging Face manifest 完全一致，并已对 runner 账号开放读取权限。config.json 确认为 fp8（e4m3，weight_block_size 128x128）、max_position_embeddings 为 1048576，与配方的精度与上下文设置一致。 无代码改动。重新添加 sweep 标签只产生了 label 事件而未触发工作流，且 run-sweep.yml 没有 workflow_dispatch 触发器，故以此空提交产生 synchronize 事件。
…合入分支 perf-changelog.yaml was the only conflict: both sides appended to the tail (base 8457 lines; main +59, this branch +35). Resolved so main's 8516 lines stay byte-identical and this branch's four entries move to the physical end, as the append-only rule requires. 1002 entries, no other file conflicted. perf-changelog.yaml 是唯一冲突：双方均为尾部追加（基线 8457 行，main +59， 本分支 +35）。按 append-only 规则解决：保持 main 的 8516 行逐字节不变，本分支 的四条entries移至文件物理末尾。共 1002 条，其余文件无冲突。
…ileRT 路径不再校验 MoRI 与 SGLang 环境变量 Sweep 35625047683 reached the container, installed tilert 0.1.6 and found mooncake, then failed in env.sh on eleven variables it never reads: MORI_IO_SQ_BACKOFF_TIMEOUT_US, MORI_IO_QP_MAX_SEND_WR, MORI_IO_QP_MAX_CQE, MORI_IO_QP_MAX_SGE, MORI_IO_TC_DISABLE, UCX_IB_GID_INDEX, MORI_APP_LOG_LEVEL, SGLANG_ROUTER_STDOUT_LOGS, TORCH_NCCL_BLOCKING_WAIT, NCCL_BLOCKING_WAIT and SGLANG_OPT_USE_AITER_INDEXER. Those are MoRI-IO queue-pair tuning, the UCX RoCE GID index, SGLang router logging and the SGLang decode cuda-graph NCCL workaround. server_tilert.sh references none of them; KV moves over mooncake and no SGLang router starts. env.sh validated them before it had even validated ENGINE, so the list is now scoped to the engines that consume it. Supplying values instead would mean inventing MoRI tuning for a transport this recipe never uses. Verified that ENGINE=sglang-disagg still fails on the same list. Also export INFMAX_CONTAINER_WORKSPACE=/workspace for the agentic replay, which runs in this container rather than a separate client: benchmark_lib's install_agentic_deps and build_replay_cmd both require it, the SGLang path sets it in its client container's env file, and launch_mi355x-amds.sh exports it only on the DSv4.1-Flash single-node branch. Without it the agentic lane would have failed at the next step. sweep 35625047683 已进入容器、装好 tilert 0.1.6 并找到 mooncake，随后在 env.sh 因十一个它从不读取的变量而失败。这些是 MoRI-IO 队列对调优、UCX RoCE GID 索引、 SGLang router 日志以及 SGLang decode cuda-graph 的 NCCL 规避项；server_tilert.sh 一概不引用，KV 走 mooncake，也不启动 SGLang router。env.sh 甚至在校验 ENGINE 之前就校验了它们，现已将该列表限定到真正消费它们的引擎；若改为提供取值，等于为 本配方根本不用的传输层编造 MoRI 调优。已验证 ENGINE=sglang-disagg 仍会因同一 列表失败。 同时为 agentic 回放导出 INFMAX_CONTAINER_WORKSPACE=/workspace：回放在本容器内 运行而非独立客户端容器，benchmark_lib 的 install_agentic_deps 与 build_replay_cmd 均要求该变量，而 SGLang 路径是在其客户端容器的 env 文件中设置、launcher 仅在 DSv4.1-Flash 单节点分支导出。否则 agentic lane 会在下一步失败。
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| + [[ 1 -ne 0 ]] | ||
| + echo 'Failed to submit job' | ||
| Failed to submit job | ||
| + exit 1 |
There was a problem hiding this comment.
🟡 (optional) A stray debug shell trace was committed to the repo instead of being discarded, leaking local developer paths into history. It is a raw set -x trace from a failed submit.sh run on the author's machine (/Users/bshan/InferenceX) plus an unrelated agent scratchpad path (/tmp/claude-502/...), added whole in commit ad1c997. Fix: remove benchmarks/multi_node/amd_utils/neg.err from the PR; it carries no functional content and only pollutes the repo with another developer's local paths.
Why this was flagged
benchmarks/multi_node/amd_utils/neg.err is a new 93-line file added in commit ad1c997 (fix(tilert): supply the amd_utils orchestration inputs on the MI355X pool). It is a captured bash -x trace of a failed local submit.sh invocation, not source, config, or documentation. It embeds the author's home directory path /Users/bshan/InferenceX and a separate, unrelated agent session scratchpad path /tmp/claude-502/-Users-bshan-InferenceX/a3355d0f-16c4-46ef-b540-6e20f6eba79c/scratchpad/d3/models. On base, no such file exists; after merge it sits permanently in the repository as debug noise and exposes local filesystem layout with no reviewer benefit.
Verification: nit. The candidate's factual claims all check out. git diff --name-status shows A benchmarks/multi_node/amd_utils/neg.err — a brand-new 93-line file added whole in commit ad1c997 ("fix(tilert): supply the amd_utils orchestration inputs..."), with all 93 lines being insertions. The file contents are a raw bash -x (set -x) trace of a failed local submit.sh run, not… | nit.…
…p8 MLA KV 以容纳 1M 上下文
Sweep 35626519997 failed on both lanes for the same reason, on opposite roles:
decode (agentic, g14) torch.OutOfMemoryError: tried to allocate 99.06 GiB
with 95.94 GiB free; 184.17 GiB of weights already
resident on a 287.98 GiB card
prefill (8k1k, g09) ValueError: 91.71 GiB KV cache is needed, which is
larger than the available KV cache memory (85.25 GiB)
bf16 MLA KV does not fit at 1048576 tokens on either rank. The recipe chose
bf16 because the ROCm sparse-MLA backend has no fp8_ds_mla, which is true but
does not imply bf16: ROCM_AITER_MLA_SPARSE lists fp8 and fp8_e4m3 in
supported_kv_cache_dtypes, and TileRT's MlaNsaProfile.configure() maps
fp8_ds_mla, fp8 and fp8_e4m3 onto the same mla_fp8 layout. Only fp8_ds_mla is
CUDA-only. Plain fp8 therefore gives both ranks the matching layout that
TileRT's --kv-cache-dtype requires, and halves the cache:
KV_BYTES_FP8 = 528 B/token against KV_BYTES_BF16 = 1024 B/token. Decode drops
to roughly 51 GB alongside its weights, prefill to roughly 46 GiB inside its
85.25 GiB budget.
Also take the container-creation barrier timeout from CONTAINER_BARRIER_TIMEOUT
(5400s) rather than hardcoding 1800s, matching how server_sglang.sh sources it.
This recipe pulls two ~32 GB images, one per rank; on a node caching neither,
the rank that comes up first waits out the whole timeout while its peer pulls.
The observed cold decode pull took about 25 minutes of the old 30-minute budget.
sweep 35626519997 两条 lane 因同一原因在相对的两侧失败：decode 侧在 287.98 GiB
卡上已驻留 184.17 GiB 权重、再分配 99.06 GiB KV 时 OOM；prefill 侧因需要
91.71 GiB KV 而可用 85.25 GiB 被 vLLM 拒绝。1048576 上下文下 bf16 MLA KV 两侧
均无法容纳。配方原先选 bf16 是因为 ROCm 稀疏 MLA 没有 fp8_ds_mla，此事为真但并
不意味着只能用 bf16：ROCM_AITER_MLA_SPARSE 的 supported_kv_cache_dtypes 含 fp8
与 fp8_e4m3，而 TileRT 的 MlaNsaProfile.configure() 将 fp8_ds_mla/fp8/fp8_e4m3
映射到同一 mla_fp8 布局，仅 fp8_ds_mla 为 CUDA 独有。因此普通 fp8 即可让两侧获得
TileRT 所要求的一致布局，并将 KV 减半（每 token 528 字节对 1024 字节）。
屏障超时同时改为取自 CONTAINER_BARRIER_TIMEOUT（5400 秒），与
server_sglang.sh 的做法一致：本配方每 rank 各拉取约 32 GB 镜像，在均未缓存的
节点上，先起来的一侧会在对端拉取期间耗尽原有 30 分钟预算（实测冷拉取约 25 分钟）。
…合入分支 Two conflicts from main's twelve new commits. setup_deps.sh: #3349 deleted install_transformers_glm5 and its dispatch arm as retired runtime support, while this branch had added the tilert arm next to it. Took main's deletion and kept only the tilert helpers and elif arm, leaving no else branch. Safe for this recipe: both images already load glm_moe_dsa, as the last sweep showed -- decode loaded weights and prefill reached KV allocation. perf-changelog.yaml: this branch appends 43 lines at the tail. Main's side also carries mid-file insertions from #3207 and #2912, older branches that merged later; those bytes are preserved exactly and this branch's five entries stay at the physical end. 1009 entries, parses clean. main 的十二个新提交带来两处冲突。setup_deps.sh：#3349 作为「移除已退役运行时 支持」删除了 install_transformers_glm5 及其分派分支，而本分支曾在其旁新增 tilert 分支；现采用 main 的删除，仅保留 tilert 辅助函数与 elif 分支，不留 else。 对本配方安全：两个镜像均已支持 glm_moe_dsa（上轮 sweep 中 decode 已加载权重、 prefill 已进入 KV 分配阶段）。perf-changelog.yaml：本分支在尾部追加 43 行； main 侧另含 #3207、#2912 这两个较早分支较晚合入而产生的文件中部插入，其字节 原样保留，本分支五条条目仍位于文件物理末尾。共 1009 条，解析正常。
…kip-sweep] / 恢复 bf16 布局，1M 上下文取决于 DRAM 侧 PD 缓冲 Reverts the fp8 KV change from 19bea0b. It cannot run: vLLM 0.24.0's ROCM_AITER_MLA_SPARSE writes a flat 576 B/token fp8 row with a per-layer scale, and TileRT 0.1.6 infers the layout from the tensor stride and accepts only 1152 B (bf16) or 656 B (fp8_ds_mla), raising "unexpected MLA cache stride 576 B/token" at register_kv_caches on every prefill rank. The original bf16 choice was the only layout-correct one on ROCm. PREFILL_KV_DTYPE is now an explicit bfloat16 (auto resolves the same on this model; float16 would pass the stride check and be misread). Memory at 1048576 tokens, per rank, verified against the tilert and vLLM sources and the MI355X logs: decode : 90.72 GiB weights + 93.25 GiB engine cache window + 99.06 GiB PD receive buffer (~283 of 287.98 GiB) prefill: 90.45 GiB weights + 40.3 GiB profiling + 91.71 GiB vLLM KV + 99.06 GiB PD staging buffer outside vLLM's budget (~321 GiB) The two PD buffers are dense in max_seq_len. Both must move to pinned host memory for this context to start, which is a TileRT change; the recipe is set up for that release (GPU_MEM_UTIL 0.85 so vLLM's 222.5 GiB fits inside its budget) and keeps 1048576. Until the wheel and images are bumped the sweep cannot pass, hence [skip-sweep]. Also: drop the invented TILERT_CONVERT_DEVICE default; pass --num-mtp "$DECODE_MTP_SIZE" to decode_server so converter, golden key and engine depth agree by data flow; make the conversion-cache check read tilert_meta.json num_mtp so a stale non-MTP cache is re-converted instead of failing after a 90 GiB weight load; validate CONTAINER_BARRIER_TIMEOUT in job.slurm's tilert branch; correct the earlier changelog text (router deps are installed on decode only, models_tilert.yaml holds prefill env, not profile defaults) and replace the wrong halving arithmetic. 撤销 19bea0b 的 fp8 KV 改动：vLLM 0.24.0 的 ROCM_AITER_MLA_SPARSE 写出每 token 576 字节的扁平 fp8 行，而 TileRT 0.1.6 依步长推断布局，只接受 1152（bf16）或 656（fp8_ds_mla）字节，会在 register_kv_caches 报 "unexpected MLA cache stride 576 B/token"。原作者的 bf16 是 ROCm 上唯一正确的布局。 1048576 上下文下每 rank 的内存（已对照源码与日志核实）：decode 侧 90.72 GiB 权重 + 93.25 GiB 引擎缓存窗口 + 99.06 GiB PD 接收缓冲（约 283/287.98 GiB）；prefill 侧 90.45 GiB 权重 + 40.3 GiB 分析开销 + 91.71 GiB vLLM KV + 位于 vLLM 预算之外 的 99.06 GiB PD 暂存缓冲（约 321 GiB）。两个 PD 缓冲按 max_seq_len 密集分配， 须移入锁页主机内存才能在该上下文启动，此为 TileRT 侧改动；配方已为该版本就位 （GPU_MEM_UTIL 0.85）并保留 1048576。wheel 与镜像升级前 sweep 无法通过，故 [skip-sweep]。 另：移除 TILERT_CONVERT_DEVICE 的自造默认；向 decode_server 显式传递 --num-mtp； 转换缓存检查读取 tilert_meta.json 的 num_mtp；job.slurm 的 tilert 分支校验 CONTAINER_BARRIER_TIMEOUT；修正此前 changelog 中的错误表述与减半算术。
Verified root cause, and why 1M context is gated on a TileRT changeEvery claim below was checked against the exact code that ran (tilert 0.1.6 wheel; vLLM 0.24.0 from Decode rank (bf16 wire layout,
184.17 GiB resident (weights + engine window, matches the OOM message to 0.08%) + 99.06 requested = ~283 of 287.98 GiB. It OOM'd on g14 (6.29 GiB non-torch) and served on g15 (≤3.17 GiB non-torch): a node-marginal fit for one sequence. Prefill rank 0 (vLLM, Why the fp8 attempt (19bea0b) was wrong and is reverted. vLLM's Compared with the green SGLang arm (35403089860): same context setting (1,048,576, derived from the checkpoint), conc 16, Decision: full 1M context, PD buffers moved to DRAM. Branch state ( The sweep is gated on a TileRT release that places both PD buffers in DRAM. No such wheel exists on PyPI (latest 0.1.6). Below is a compile-checked patch against 0.1.6 that does it, for the TileRT team to take upstream — the repo forbids applying it in the recipe or image (docs/PR_REVIEW_CHECKLIST: no site-packages edits, no rebuilt wheels). It adds tilert-0.1.6-pd-buffers-in-dram.diff (5 files, py_compile clean)Only in new: __pycache__
diff -ru orig/decode_server.py new/decode_server.py
--- orig/decode_server.py 2026-09-22 10:06:55
+++ new/decode_server.py 2026-09-22 10:08:38
@@ -309,6 +309,7 @@
ap.add_argument('--num-mtp', type=int, choices=(3,), default=3, help='MTP draft depth for speculative decoding; needs --with-mtp')
ap.add_argument('--transport', choices=['mooncake', 'nixl'], default='mooncake', help="RDMA data-plane backend (must match prefill's tilert_transport)")
ap.add_argument('--kv-cache-dtype', default='fp8_ds_mla', help='MLA cache dtype (must match vLLM prefill); MLA-family profiles only')
+ ap.add_argument('--pd-buffer-device', choices=['cuda', 'cpu'], default='cuda', help="where the PD receive buffer lives; 'cpu' = pinned host memory registered with the transport (frees buffer_bytes(max_seq_len) of VRAM, adds one H2D copy per request)")
return ap
def main() -> None:
@@ -329,7 +330,7 @@
logger.info('loading TileRT engine (profile=%s, weights=%s)...', profile.name, args.model_weights_dir)
engine = profile.build_engine(model_weights_dir=args.model_weights_dir, max_seq_len=args.max_seq_len, with_mtp=args.with_mtp, ar_steps=8, num_mtp=num_mtp)
logger.info('TileRT engine ready (cache window %d, num_mtp %d)', engine.max_seq_len, num_mtp)
- server = ReceiveServer(profile, max_seq_len=args.max_seq_len, ctrl_port=args.ctrl_port, transport=args.transport)
+ server = ReceiveServer(profile, max_seq_len=args.max_seq_len, ctrl_port=args.ctrl_port, transport=args.transport, buffer_device='cpu' if args.pd_buffer_device == 'cpu' else 'cuda:0')
app = build_app(server, engine)
logger.info('decode server on :%d (profile=%s, engine=%s, ctrl=:%d)', args.http_port, profile.name, args.engine, args.ctrl_port)
sock = socket.socket(socket.AF_INET6, socket.SOCK_STREAM)
diff -ru orig/prefill_connector.py new/prefill_connector.py
--- orig/prefill_connector.py 2026-09-22 10:06:55
+++ new/prefill_connector.py 2026-09-22 10:07:21
@@ -49,6 +49,7 @@
self._max_seq = int(extra.get('tilert_max_seq_len', vllm_config.model_config.max_model_len))
self._profile = profiles.get_profile(extra.get('tilert_model', 'glm5'))
self._transport_name = extra.get('tilert_transport', 'mooncake')
+ self._pd_buffer_device = str(extra.get('tilert_pd_buffer_device', 'cuda'))
self._pending: dict[str, _Pending] = {}
self._kv_caches: dict = {}
self._reg = None
@@ -145,10 +146,18 @@
hostname = wire.local_ip()
total = self._profile.staging_bytes(self._reg, self._tp_rank, self._max_seq)
dev = torch.cuda.current_device()
- self._staging = torch.zeros(total, dtype=torch.uint8, device=f'cuda:{dev}')
+ # Dense in max_seq_len (99.06 GiB at 1M for glm5_2 bf16) and allocated
+ # outside vLLM's gpu-memory-utilization budget. In pinned host memory
+ # when tilert_pd_buffer_device == 'cpu'; extract() already fills it with
+ # copy_(), which performs the D2H transfer.
+ host = self._pd_buffer_device == 'cpu'
+ if host:
+ self._staging = torch.zeros(total, dtype=torch.uint8, pin_memory=True)
+ else:
+ self._staging = torch.zeros(total, dtype=torch.uint8, device=f'cuda:{dev}')
self._transport = make_transport(self._transport_name)
self._transport.init(hostname)
- self._transport.register(self._staging.data_ptr(), total, dev)
+ self._transport.register(self._staging.data_ptr(), total, dev, host=host)
self._sender_thread = threading.Thread(target=self._sender_loop, name='tilert-pd-sender', daemon=True)
self._sender_thread.start()
logger.info('worker ready: rank=%d transport=%s staging=%.1f MB profile=%s', self._tp_rank, self._transport.name, total / 1000000.0, self._profile.name)
Only in new/profiles: __pycache__
diff -ru orig/profiles/mla_nsa.py new/profiles/mla_nsa.py
--- orig/profiles/mla_nsa.py 2026-09-22 10:06:55
+++ new/profiles/mla_nsa.py 2026-09-22 10:07:21
@@ -125,14 +125,23 @@
for lid in range(self.num_layers):
ko = lid * max_seq_len * kv_bpt
kv_raw = buffer[ko:ko + seq * kv_bpt].view(seq, kv_bpt)
+ # A host-resident receive buffer is moved to the device plane by
+ # plane; the copy scales with the request's seq_len, not max_seq_len.
+ if not buffer.is_cuda:
+ kv_raw = kv_raw.to('cuda:0', non_blocking=True)
if self.mla_fp8:
kv = self._dequant_kv(kv_raw, seq)
else:
kv = kv_raw.view(torch.bfloat16).view(seq, KV_LORA_RANK).contiguous()
po = pe_base + lid * max_seq_len * PE_BPT
- pe = buffer[po:po + seq * PE_BPT].view(torch.bfloat16).view(seq, QK_ROPE_HEAD_DIM).contiguous()
+ pe = buffer[po:po + seq * PE_BPT]
+ if not buffer.is_cuda:
+ pe = pe.to('cuda:0', non_blocking=True)
+ pe = pe.view(torch.bfloat16).view(seq, QK_ROPE_HEAD_DIM).contiguous()
io = ki_base + lid * _max_pages(max_seq_len) * KI_PAGE_BYTES
ki_raw = buffer[io:io + npages * KI_PAGE_BYTES].view(npages, KI_PAGE_BYTES)
+ if not buffer.is_cuda:
+ ki_raw = ki_raw.to('cuda:0', non_blocking=True)
layers.append((self._dequant_ki(ki_raw, seq, self.ki_tiled), kv, pe))
torch.cuda.synchronize()
return ConvertedRequest(rid=received.rid, seq_len=seq, last_prompt_token=received.last_prompt_token, first_token_id=received.first_token_id, sampling=received.sampling, layers=layers)
diff -ru orig/receive_server.py new/receive_server.py
--- orig/receive_server.py 2026-09-22 10:06:55
+++ new/receive_server.py 2026-09-22 10:06:55
@@ -34,15 +34,24 @@
class ReceiveServer:
- def __init__(self, profile, max_seq_len: int, ctrl_port: int=5556, hostname: str | None=None, device: str='cuda:0', request_timeout: float=120.0, transport: str='mooncake'):
+ def __init__(self, profile, max_seq_len: int, ctrl_port: int=5556, hostname: str | None=None, device: str='cuda:0', request_timeout: float=120.0, transport: str='mooncake', buffer_device: str='cuda:0'):
self.profile = profile
self.max_seq_len = max_seq_len
self.ctrl_port = ctrl_port
self.device = device
self.request_timeout = request_timeout
total = profile.buffer_bytes(max_seq_len)
- logger.info('allocating receive buffer: %.2f GB on %s (profile=%s)', total / 1024 ** 3, device, profile.name)
- self.buffer = torch.zeros(total, dtype=torch.uint8, device=device)
+ # Dense in max_seq_len: 99.06 GiB at 1M tokens for glm5_2 bf16. Held in
+ # pinned host memory when buffer_device == 'cpu' so it does not compete
+ # with the weights and the engine cache window on the GPU; convert()
+ # then moves each request's planes to the device (H2D scales with the
+ # prompt, not with max_seq_len).
+ self.buffer_device = buffer_device
+ if buffer_device == 'cpu':
+ self.buffer = torch.zeros(total, dtype=torch.uint8, pin_memory=True)
+ else:
+ self.buffer = torch.zeros(total, dtype=torch.uint8, device=buffer_device)
+ logger.info('allocating receive buffer: %.2f GiB on %s (profile=%s)', total / 1024 ** 3, self.buffer.device, profile.name)
self.base_ptr = self.buffer.data_ptr()
self._hello_layout = profile.hello_layout(self.base_ptr, max_seq_len)
from tilert.pd_vllm.transport import make_transport
@@ -51,7 +60,7 @@
dev_id = torch.device(device).index or 0
self._transport = make_transport(transport)
self._transport.init(hostname)
- self._transport.register(self.base_ptr, total, dev_id)
+ self._transport.register(self.base_ptr, total, dev_id, host=(buffer_device == 'cpu'))
self._transport_meta = self._transport.local_meta()
logger.info('transport=%s ready, buffer registered (%.2f GB)', self._transport.name, total / 1024 ** 3)
self._lock = threading.Lock()
diff -ru orig/transport.py new/transport.py
--- orig/transport.py 2026-09-22 10:06:55
+++ new/transport.py 2026-09-22 10:08:13
@@ -8,7 +8,7 @@
def init(self, host: str) -> None:
...
- def register(self, ptr: int, nbytes: int, dev_id: int) -> None:
+ def register(self, ptr: int, nbytes: int, dev_id: int, host: bool=False) -> None:
...
def local_meta(self) -> dict:
@@ -28,7 +28,8 @@
raise RuntimeError(f'Mooncake engine init failed: {ret}')
self.session_id = f'{host}:{self.engine.get_rpc_port()}'
- def register(self, ptr: int, nbytes: int, dev_id: int) -> None:
+ def register(self, ptr: int, nbytes: int, dev_id: int, host: bool=False) -> None:
+ # Mooncake registers host (pinned) and device pointers through the same call.
ret = self.engine.batch_register_memory([ptr], [nbytes])
if ret != 0:
raise RuntimeError(f'Mooncake register failed: {ret}')
@@ -50,10 +51,12 @@
self._agent = nixl_agent(f'{host}:{os.getpid()}', nixl_agent_config(backends=['UCX']))
self._remotes: dict[bytes, str] = {}
self._dev = 0
+ self._mem_type = 'VRAM'
- def register(self, ptr: int, nbytes: int, dev_id: int) -> None:
- self._dev = dev_id
- self._agent.register_memory([(ptr, nbytes, dev_id, '')], 'VRAM')
+ def register(self, ptr: int, nbytes: int, dev_id: int, host: bool=False) -> None:
+ self._dev = 0 if host else dev_id
+ self._mem_type = 'DRAM' if host else 'VRAM'
+ self._agent.register_memory([(ptr, nbytes, self._dev, '')], self._mem_type)
def local_meta(self) -> dict:
return {'nixl_meta': base64.b64encode(self._agent.get_agent_metadata()).decode(), 'nixl_dev': self._dev}
@@ -65,8 +68,8 @@
rname = self._agent.add_remote_agent(meta_b)
self._remotes[meta_b] = rname
rdev = int(remote_meta.get('nixl_dev', 0))
- ld = self._agent.get_xfer_descs([(int(s), int(n), self._dev) for s, n in zip(srcs, lens)], 'VRAM')
- rd = self._agent.get_xfer_descs([(int(d), int(n), rdev) for d, n in zip(dsts, lens)], 'VRAM')
+ ld = self._agent.get_xfer_descs([(int(s), int(n), self._dev) for s, n in zip(srcs, lens)], self._mem_type)
+ rd = self._agent.get_xfer_descs([(int(d), int(n), rdev) for d, n in zip(dsts, lens)], self._mem_type)
h = self._agent.initialize_xfer('WRITE', ld, rd, rname)
try:
st = self._agent.transfer(h)中文已核实的根因。 decode 侧 bf16、1048576 上下文下有三块按 fp8 方案（19bea0bdf）错误，已撤销。 ROCm 的 vLLM 稀疏 MLA 后端没有 与 SGLang 绿色运行的对比。 同为 1M 上下文设置、并发 16，其 KV 为按可用显存划定的分页池（约 150 GiB，各会话分时共享），MoRI 分离亦复用同一池；TileRT 则按 决定：保留 1M，PD 缓冲移入 DRAM。 分支现为 bf16、1048576、 |
DRAM PD buffers: the ionic
|
| test | result |
|---|---|
| cumulative 1 GiB MRs, 4 KiB pages | 3 GiB ok, 4th → EINVAL |
| exact ceiling, 64 MiB steps | 3968 MiB = 1,015,808 × 4 KiB pages, then EINVAL |
| single MR, 4 KiB pages | 1, 2 GiB ok; ≥ 3 GiB → ENOMEM |
| rdma1 while rdma0 saturated | ok → the budget is per HCA |
THP-backed (MADV_HUGEPAGE, 100 % AnonHugePages verified) |
32 GiB cumulative ok; single 4/8/16/32 GiB MRs ok |
| 6 GiB THP region with one 2 MiB hole of 4 KiB pages | ENOMEM — the whole MR falls to 4 KiB granularity |
in the ROCm image: torch.zeros(8 GiB, pin_memory=True) |
already 2 MiB-backed, is_pinned=True, registers ok |
in the ROCm image: mmap+MADV_HUGEPAGE+hipHostRegister+torch.frombuffer (8 and 32 GiB) |
is_pinned=True, registers ok |
So the limit is not bytes but a ~2²⁰ page-table-entry budget per HCA: ~3.9 GiB at 4 KiB pages, ~1.9 TiB at 2 MiB pages — which is why device memory (large contiguous chunks) registers tens of GiB without trouble. The firmware rejects the admin command (dmesg: rdma0: opcode 3 error 5, exactly two lines for the two cumulative failures); oversized single MRs are refused earlier with ENOMEM. ibv_devinfo: page_size_cap 0x40201000 (4K/2M/1G), max_mr_size unlimited.
Consequence for the DRAM route: both PD buffers (99.06 GiB each at 1M) must be 2 MiB-backed end to end and that must be verified at allocation, not assumed — one 4 KiB-backed slice anywhere in the region and the MR fails. At 2 MiB, 99.06 GiB is ~50.7k entries, 5 % of the budget. These nodes run THP enabled=madvise, defrag=madvise, 3.17 TB RAM; hugetlbfs is not an option (11 GiB reserved, 0 free).
Patch updated (~/tilert-0.1.6-pd-buffers-in-dram.diff, 5 files, 231 lines, py_compile clean): transport.alloc_pinned_huge() allocates mmap+MADV_HUGEPAGE, populates every 2 MiB page, checks /proc/self/smaps reports AnonHugePages == size and raises otherwise, hipHostRegisters the region so the GPU can DMA to it, and returns a torch.frombuffer tensor (pinned). Both ReceiveServer (--pd-buffer-device cpu) and TileRTConnector (tilert_pd_buffer_device: cpu) use it. Full patch below.
tilert-0.1.6-pd-buffers-in-dram.diff v2
Only in new: __pycache__
diff -ru orig/decode_server.py new/decode_server.py
--- orig/decode_server.py 2026-09-22 10:06:55
+++ new/decode_server.py 2026-09-22 10:08:38
@@ -309,6 +309,7 @@
ap.add_argument('--num-mtp', type=int, choices=(3,), default=3, help='MTP draft depth for speculative decoding; needs --with-mtp')
ap.add_argument('--transport', choices=['mooncake', 'nixl'], default='mooncake', help="RDMA data-plane backend (must match prefill's tilert_transport)")
ap.add_argument('--kv-cache-dtype', default='fp8_ds_mla', help='MLA cache dtype (must match vLLM prefill); MLA-family profiles only')
+ ap.add_argument('--pd-buffer-device', choices=['cuda', 'cpu'], default='cuda', help="where the PD receive buffer lives; 'cpu' = pinned host memory registered with the transport (frees buffer_bytes(max_seq_len) of VRAM, adds one H2D copy per request)")
return ap
def main() -> None:
@@ -329,7 +330,7 @@
logger.info('loading TileRT engine (profile=%s, weights=%s)...', profile.name, args.model_weights_dir)
engine = profile.build_engine(model_weights_dir=args.model_weights_dir, max_seq_len=args.max_seq_len, with_mtp=args.with_mtp, ar_steps=8, num_mtp=num_mtp)
logger.info('TileRT engine ready (cache window %d, num_mtp %d)', engine.max_seq_len, num_mtp)
- server = ReceiveServer(profile, max_seq_len=args.max_seq_len, ctrl_port=args.ctrl_port, transport=args.transport)
+ server = ReceiveServer(profile, max_seq_len=args.max_seq_len, ctrl_port=args.ctrl_port, transport=args.transport, buffer_device='cpu' if args.pd_buffer_device == 'cpu' else 'cuda:0')
app = build_app(server, engine)
logger.info('decode server on :%d (profile=%s, engine=%s, ctrl=:%d)', args.http_port, profile.name, args.engine, args.ctrl_port)
sock = socket.socket(socket.AF_INET6, socket.SOCK_STREAM)
diff -ru orig/prefill_connector.py new/prefill_connector.py
--- orig/prefill_connector.py 2026-09-22 10:06:55
+++ new/prefill_connector.py 2026-09-22 11:54:44
@@ -49,6 +49,7 @@
self._max_seq = int(extra.get('tilert_max_seq_len', vllm_config.model_config.max_model_len))
self._profile = profiles.get_profile(extra.get('tilert_model', 'glm5'))
self._transport_name = extra.get('tilert_transport', 'mooncake')
+ self._pd_buffer_device = str(extra.get('tilert_pd_buffer_device', 'cuda'))
self._pending: dict[str, _Pending] = {}
self._kv_caches: dict = {}
self._reg = None
@@ -145,10 +146,19 @@
hostname = wire.local_ip()
total = self._profile.staging_bytes(self._reg, self._tp_rank, self._max_seq)
dev = torch.cuda.current_device()
- self._staging = torch.zeros(total, dtype=torch.uint8, device=f'cuda:{dev}')
+ # Dense in max_seq_len (99.06 GiB at 1M for glm5_2 bf16) and allocated
+ # outside vLLM's gpu-memory-utilization budget. In pinned host memory
+ # when tilert_pd_buffer_device == 'cpu'; extract() already fills it with
+ # copy_(), which performs the D2H transfer.
+ host = self._pd_buffer_device == 'cpu'
+ if host:
+ from tilert.pd_vllm.transport import alloc_pinned_huge
+ self._staging = alloc_pinned_huge(total, dev)
+ else:
+ self._staging = torch.zeros(total, dtype=torch.uint8, device=f'cuda:{dev}')
self._transport = make_transport(self._transport_name)
self._transport.init(hostname)
- self._transport.register(self._staging.data_ptr(), total, dev)
+ self._transport.register(self._staging.data_ptr(), total, dev, host=host)
self._sender_thread = threading.Thread(target=self._sender_loop, name='tilert-pd-sender', daemon=True)
self._sender_thread.start()
logger.info('worker ready: rank=%d transport=%s staging=%.1f MB profile=%s', self._tp_rank, self._transport.name, total / 1000000.0, self._profile.name)
Only in new/profiles: __pycache__
diff -ru orig/profiles/mla_nsa.py new/profiles/mla_nsa.py
--- orig/profiles/mla_nsa.py 2026-09-22 10:06:55
+++ new/profiles/mla_nsa.py 2026-09-22 10:07:21
@@ -125,14 +125,23 @@
for lid in range(self.num_layers):
ko = lid * max_seq_len * kv_bpt
kv_raw = buffer[ko:ko + seq * kv_bpt].view(seq, kv_bpt)
+ # A host-resident receive buffer is moved to the device plane by
+ # plane; the copy scales with the request's seq_len, not max_seq_len.
+ if not buffer.is_cuda:
+ kv_raw = kv_raw.to('cuda:0', non_blocking=True)
if self.mla_fp8:
kv = self._dequant_kv(kv_raw, seq)
else:
kv = kv_raw.view(torch.bfloat16).view(seq, KV_LORA_RANK).contiguous()
po = pe_base + lid * max_seq_len * PE_BPT
- pe = buffer[po:po + seq * PE_BPT].view(torch.bfloat16).view(seq, QK_ROPE_HEAD_DIM).contiguous()
+ pe = buffer[po:po + seq * PE_BPT]
+ if not buffer.is_cuda:
+ pe = pe.to('cuda:0', non_blocking=True)
+ pe = pe.view(torch.bfloat16).view(seq, QK_ROPE_HEAD_DIM).contiguous()
io = ki_base + lid * _max_pages(max_seq_len) * KI_PAGE_BYTES
ki_raw = buffer[io:io + npages * KI_PAGE_BYTES].view(npages, KI_PAGE_BYTES)
+ if not buffer.is_cuda:
+ ki_raw = ki_raw.to('cuda:0', non_blocking=True)
layers.append((self._dequant_ki(ki_raw, seq, self.ki_tiled), kv, pe))
torch.cuda.synchronize()
return ConvertedRequest(rid=received.rid, seq_len=seq, last_prompt_token=received.last_prompt_token, first_token_id=received.first_token_id, sampling=received.sampling, layers=layers)
diff -ru orig/receive_server.py new/receive_server.py
--- orig/receive_server.py 2026-09-22 10:06:55
+++ new/receive_server.py 2026-09-22 11:54:44
@@ -34,15 +34,25 @@
class ReceiveServer:
- def __init__(self, profile, max_seq_len: int, ctrl_port: int=5556, hostname: str | None=None, device: str='cuda:0', request_timeout: float=120.0, transport: str='mooncake'):
+ def __init__(self, profile, max_seq_len: int, ctrl_port: int=5556, hostname: str | None=None, device: str='cuda:0', request_timeout: float=120.0, transport: str='mooncake', buffer_device: str='cuda:0'):
self.profile = profile
self.max_seq_len = max_seq_len
self.ctrl_port = ctrl_port
self.device = device
self.request_timeout = request_timeout
total = profile.buffer_bytes(max_seq_len)
- logger.info('allocating receive buffer: %.2f GB on %s (profile=%s)', total / 1024 ** 3, device, profile.name)
- self.buffer = torch.zeros(total, dtype=torch.uint8, device=device)
+ # Dense in max_seq_len: 99.06 GiB at 1M tokens for glm5_2 bf16. Held in
+ # pinned host memory when buffer_device == 'cpu' so it does not compete
+ # with the weights and the engine cache window on the GPU; convert()
+ # then moves each request's planes to the device (H2D scales with the
+ # prompt, not with max_seq_len).
+ self.buffer_device = buffer_device
+ if buffer_device == 'cpu':
+ from tilert.pd_vllm.transport import alloc_pinned_huge
+ self.buffer = alloc_pinned_huge(total)
+ else:
+ self.buffer = torch.zeros(total, dtype=torch.uint8, device=buffer_device)
+ logger.info('allocating receive buffer: %.2f GiB on %s (profile=%s)', total / 1024 ** 3, self.buffer.device, profile.name)
self.base_ptr = self.buffer.data_ptr()
self._hello_layout = profile.hello_layout(self.base_ptr, max_seq_len)
from tilert.pd_vllm.transport import make_transport
@@ -51,7 +61,7 @@
dev_id = torch.device(device).index or 0
self._transport = make_transport(transport)
self._transport.init(hostname)
- self._transport.register(self.base_ptr, total, dev_id)
+ self._transport.register(self.base_ptr, total, dev_id, host=(buffer_device == 'cpu'))
self._transport_meta = self._transport.local_meta()
logger.info('transport=%s ready, buffer registered (%.2f GB)', self._transport.name, total / 1024 ** 3)
self._lock = threading.Lock()
diff -ru orig/transport.py new/transport.py
--- orig/transport.py 2026-09-22 10:06:55
+++ new/transport.py 2026-09-22 11:54:44
@@ -8,7 +8,7 @@
def init(self, host: str) -> None:
...
- def register(self, ptr: int, nbytes: int, dev_id: int) -> None:
+ def register(self, ptr: int, nbytes: int, dev_id: int, host: bool=False) -> None:
...
def local_meta(self) -> dict:
@@ -28,7 +28,8 @@
raise RuntimeError(f'Mooncake engine init failed: {ret}')
self.session_id = f'{host}:{self.engine.get_rpc_port()}'
- def register(self, ptr: int, nbytes: int, dev_id: int) -> None:
+ def register(self, ptr: int, nbytes: int, dev_id: int, host: bool=False) -> None:
+ # Mooncake registers host (pinned) and device pointers through the same call.
ret = self.engine.batch_register_memory([ptr], [nbytes])
if ret != 0:
raise RuntimeError(f'Mooncake register failed: {ret}')
@@ -50,10 +51,12 @@
self._agent = nixl_agent(f'{host}:{os.getpid()}', nixl_agent_config(backends=['UCX']))
self._remotes: dict[bytes, str] = {}
self._dev = 0
+ self._mem_type = 'VRAM'
- def register(self, ptr: int, nbytes: int, dev_id: int) -> None:
- self._dev = dev_id
- self._agent.register_memory([(ptr, nbytes, dev_id, '')], 'VRAM')
+ def register(self, ptr: int, nbytes: int, dev_id: int, host: bool=False) -> None:
+ self._dev = 0 if host else dev_id
+ self._mem_type = 'DRAM' if host else 'VRAM'
+ self._agent.register_memory([(ptr, nbytes, self._dev, '')], self._mem_type)
def local_meta(self) -> dict:
return {'nixl_meta': base64.b64encode(self._agent.get_agent_metadata()).decode(), 'nixl_dev': self._dev}
@@ -65,8 +68,8 @@
rname = self._agent.add_remote_agent(meta_b)
self._remotes[meta_b] = rname
rdev = int(remote_meta.get('nixl_dev', 0))
- ld = self._agent.get_xfer_descs([(int(s), int(n), self._dev) for s, n in zip(srcs, lens)], 'VRAM')
- rd = self._agent.get_xfer_descs([(int(d), int(n), rdev) for d, n in zip(dsts, lens)], 'VRAM')
+ ld = self._agent.get_xfer_descs([(int(s), int(n), self._dev) for s, n in zip(srcs, lens)], self._mem_type)
+ rd = self._agent.get_xfer_descs([(int(d), int(n), rdev) for d, n in zip(dsts, lens)], self._mem_type)
h = self._agent.initialize_xfer('WRITE', ld, rd, rname)
try:
st = self._agent.transfer(h)
@@ -86,4 +89,55 @@
key = (name or 'mooncake').lower()
if key not in _BACKENDS:
raise ValueError(f'unknown transport {name!r}; choices: {sorted(_BACKENDS)}')
- return _BACKENDS[key]()
\ No newline at end of file
+ return _BACKENDS[key]()
+
+def alloc_pinned_huge(total: int, device_index: int = 0):
+ """Host buffer for the PD data plane, backed end-to-end by 2 MiB pages.
+
+ The ionic RDMA VFs on the MI355X pool have a ~2**20 page-table-entry budget
+ per HCA: with 4 KiB pages ibv_reg_mr fails (EINVAL/ENOMEM) once ~3.9 GiB is
+ registered, while 2 MiB-backed regions register tens of GiB. One 4 KiB-backed
+ slice anywhere in the region drops the whole MR to 4 KiB granularity, so the
+ backing is verified from /proc/self/smaps before the memory is handed out.
+ The region is then hipHostRegister'ed so the GPU can DMA into it and torch
+ reports it pinned.
+ """
+ import ctypes, mmap, re
+ import torch
+ MiB = 1 << 20
+ if total % (2 * MiB):
+ total += 2 * MiB - total % (2 * MiB)
+ mm = mmap.mmap(-1, total, flags=mmap.MAP_PRIVATE | mmap.MAP_ANONYMOUS)
+ mm.madvise(mmap.MADV_HUGEPAGE)
+ for off in range(0, total, 2 * MiB):
+ mm[off] = 0
+ addr = ctypes.addressof(ctypes.c_char.from_buffer(mm))
+ huge = 0
+ for blk in re.split(r'\n(?=[0-9a-f]+-[0-9a-f]+ )', open('/proc/self/smaps').read()):
+ m = re.match(r'([0-9a-f]+)-([0-9a-f]+) ', blk)
+ if not m:
+ continue
+ lo, hi = int(m.group(1), 16), int(m.group(2), 16)
+ if hi <= addr or lo >= addr + total:
+ continue
+ h = re.search(r'AnonHugePages:\s+(\d+) kB', blk)
+ huge += int(h.group(1)) * 1024 if h else 0
+ if huge < total:
+ raise RuntimeError(f'PD host buffer is only {huge / 2**30:.2f} of {total / 2**30:.2f} GiB huge-page backed; '
+ 'the RDMA MR would fall back to 4 KiB pages and exceed the per-HCA entry budget '
+ '(check /sys/kernel/mm/transparent_hugepage/{enabled,defrag} and free memory)')
+ hip = None
+ for lib in ('/opt/rocm/lib/libamdhip64.so', 'libamdhip64.so', 'libcudart.so'):
+ try:
+ hip = ctypes.CDLL(lib)
+ break
+ except OSError:
+ continue
+ if hip is not None:
+ fn = hip.hipHostRegister if hasattr(hip, 'hipHostRegister') else hip.cudaHostRegister
+ rc = fn(ctypes.c_void_p(addr), ctypes.c_size_t(total), ctypes.c_uint(0))
+ if rc != 0:
+ raise RuntimeError(f'hipHostRegister failed rc={rc}')
+ buf = torch.frombuffer(mm, dtype=torch.uint8)
+ buf._pd_mmap = mm # keep the mapping alive with the tensor
+ return buf中文
TileRT 团队报告 ionic HCA 上注册的主机内存超过约 4 GiB 后 ibv_reg_mr 返回 EINVAL。已在 mia1-p01-g09 上以原生 libibverbs 复现：4 KiB 页时每 HCA 累计上限 3968 MiB = 1,015,808 页（EINVAL，固件拒绝，dmesg opcode 3 error 5），单个 ≥3 GiB 的 MR 报 ENOMEM，限制为每 HCA 独立；改用 2 MiB 大页（MADV_HUGEPAGE，smaps 确认 100% AnonHugePages）后 32 GiB 累计与单个 32 GiB MR 均注册成功。因此这是每 HCA 约 2²⁰ 条页表项的预算，而非字节上限；6 GiB 大页区域中哪怕一个 2 MiB 的 4 KiB 页空洞也会使整个 MR 失败。ROCm 容器内 torch.zeros(pin_memory=True) 已是 2 MiB 页支撑并可注册。补丁已更新为在分配时校验大页覆盖、hipHostRegister 并返回 pinned 张量。
DRAM PD buffers: cross-node mooncake transfer proven; chunking into <4 GiB MRs does not avoid the ionic limitCross-node proof (receiver
So every mechanism the DRAM patch relies on has now executed on this fabric: 2 MiB-backed host registration at full size, GPU mapping of that region, and RDMA into it across nodes. Populating 100 GiB of THP from a Python loop took 149 s; production code should use On "register memory regions one by one, each < 4 GiB": this does not work with 4 KiB pages. The ceiling is cumulative per HCA (~2²⁰ page-table entries), not per MR:
Each 4 KiB page costs one entry whichever MR it belongs to, so 25 × 4 GiB pieces still need ~26M entries; spreading over all 8 HCAs yields ~31 GiB, a third of one buffer. The only thing that lifts it is the entry size: 2 MiB backing registers 100 GiB as one MR. Chunking is harmless on top of huge pages but unnecessary. Remaining before the sweep can pass: the TileRT release carrying the DRAM buffers (patch v2 above), then the recipe bump ( 中文跨节点验证（接收端 g11、发送端 g09，与 TileRT 关于「逐块注册、每块小于 4 GiB」： 在 4 KiB 页下无效。上限是每 HCA 累计约 2²⁰ 条页表项，而非单个 MR：1 GiB 逐块注册到 3 GiB 后第 4 块 EINVAL；256 MiB 逐块到 3.75 GiB 后 EINVAL；64 MiB 逐块到 3968 MiB 后 EINVAL。每个 4 KiB 页占一条页表项，与所属 MR 无关；分成 25 个 4 GiB 块仍需约 2600 万条；分散到 8 个 HCA 也只有约 31 GiB。唯一有效的办法是改用 2 MiB 大页——100 GiB 可作为单个 MR 注册。大页之上再分块无害但没有必要。 sweep 通过前仍需： 含 DRAM 缓冲的 TileRT 版本、配方升级（版本、镜像、两个新开关），以及一次 |
…合入分支 perf-changelog.yaml only: main appended 15 lines (#3337), this branch appends at the tail; main's 8589 lines are byte-identical and this branch's entries follow them. configs/amd-master.yaml auto-merged. 仅 perf-changelog.yaml 冲突：main 追加 15 行（#3337），本分支尾部追加；main 的 8589 行逐字节不变，本分支条目位于其后。configs/amd-master.yaml 自动合并。
…ert 0.1.6 patch + waiver 3330 / PD 缓冲移入 2 MiB 大页锁页内存以支持 1M 上下文，附 tilert 0.1.6 补丁与豁免
TileRT 0.1.6 keeps its two PD buffers on the GPU, dense in max_seq_len:
99.06 GiB each at 1048576 tokens. With 90.72 GiB of weights and the 93.25 GiB
engine cache window the decode rank sits at ~283 of 287.98 GiB (node-marginal,
OOM on g14), and prefill rank 0 needs ~321 GiB (weights 90.45 + profiling
40.29 + vLLM KV 91.71 inside the budget + the 99.06 GiB staging buffer outside
it). No --kv-cache-dtype shrinks them on ROCm: TileRT accepts only the 1152 B
(bf16) or 656 B (fp8_ds_mla) stride and vLLM's ROCM_AITER_MLA_SPARSE has no
fp8_ds_mla.
setup_deps.sh now applies patches/tilert-0.1.6-pd-buffers-in-dram.patch to the
pip-installed tilert 0.1.6 on both ranks when TILERT_PD_BUFFER_DEVICE=cpu
(marker-gated, pinned to 0.1.6, checked before the wheel download). The
patch (5 pure-Python files under tilert/pd_vllm) adds decode_server
--pd-buffer-device and connector extra config tilert_pd_buffer_device, and
allocates both buffers with alloc_pinned_huge(): a 2 MiB-aligned mmap with
MADV_HUGEPAGE, populated via MADV_POPULATE_WRITE, verified from
/proc/self/smaps to be fully huge-page backed, hipHostRegister'ed, wrapped as
a pinned torch tensor; convert() moves each received plane to the device.
The verification matters: the Pensando ionic RDMA VFs cap 4 KiB-page
ibv_reg_mr at 3968 MiB per HCA (~2^20 entries), while 2 MiB-backed 100 GiB
regions register in one MR and took a cross-node mooncake write at 20.9 GiB/s
(measured on g09/g11). Registering in <4 GiB pieces does not help; the budget
is per HCA, not per MR.
server_tilert.sh validates TILERT_PD_BUFFER_DEVICE in {cpu,cuda} up front and
passes --pd-buffer-device only for cpu, so cuda runs the shipped wheel (which
has no such flag) unmodified. job.slurm forwards and validates the knob; both
recipes set cpu and GPU_MEM_UTIL 0.85 (222.5 GiB inside a 244.8 GiB budget).
vLLM is not patched. Engine-patch waiver: docs/waiver/3330.md; the upstream
TileRT link is pending.
TileRT 0.1.6 将两个按 max_seq_len 密集分配的 PD 缓冲放在 GPU 上，1048576 上下文下
各 99.06 GiB；连同 90.72 GiB 权重与 93.25 GiB 引擎缓存窗口，decode 侧约 283/287.98
GiB（节点边缘，g14 OOM），prefill rank 0 约需 321 GiB。ROCm 上无法用
--kv-cache-dtype 缩减：TileRT 只接受 1152（bf16）或 656（fp8_ds_mla）字节步长，而
vLLM 的 ROCM_AITER_MLA_SPARSE 没有 fp8_ds_mla。
现由 setup_deps.sh 在 TILERT_PD_BUFFER_DEVICE=cpu 时对两侧 pip 安装的 tilert 0.1.6
应用 patches/tilert-0.1.6-pd-buffers-in-dram.patch（有标记门控、固定 0.1.6，在下载
wheel 前检查）。补丁新增 decode_server --pd-buffer-device 与 connector 配置
tilert_pd_buffer_device，用 alloc_pinned_huge() 分配两个缓冲：2 MiB 对齐的 mmap +
MADV_HUGEPAGE，经 MADV_POPULATE_WRITE 预填充，依 /proc/self/smaps 校验全部为大页，
hipHostRegister 后包装为 pinned torch 张量；convert() 将接收到的各层数据搬到设备。
校验的必要性：ionic RDMA VF 对 4 KiB 页的 ibv_reg_mr 上限为每 HCA 3968 MiB（约 2^20
条页表项），而 2 MiB 大页的 100 GiB 区域可作单个 MR 注册，并以 20.9 GiB/s 接收跨节点
mooncake 写入（g09/g11 实测）；分成小于 4 GiB 的块无济于事，预算按 HCA 而非按 MR。
server_tilert.sh 预先校验 TILERT_PD_BUFFER_DEVICE 取值，仅在 cpu 时传递
--pd-buffer-device，cuda 下原样运行 wheel。job.slurm 转发并校验该变量；两份配方设为
cpu 并将 GPU_MEM_UTIL 设为 0.85。vLLM 未打补丁。引擎补丁豁免见 docs/waiver/3330.md，
上游 TileRT 链接待补。
Oseltamivir
added
the
engine-patch
…口 60 秒 sync.py --enable-port keeps the local port open for max(60, timeout // 2) seconds after the barrier passes, in the foreground, so a peer one poll behind still sees it. With the TileRT launcher's CONTAINER_BARRIER_TIMEOUT=5400 that became a 45-minute idle sleep on every rank: jobs 45373/45374 passed the barrier at 06:28 and sat in time.sleep until 07:13 with two nodes allocated (main thread in hrtimer_nanosleep, accept thread in inet_csk_accept). The old hardcoded 1800s hid a 15-minute version of the same sleep. sync.py gains --grace <seconds>; unset keeps the existing formula, so the SGLang/vLLM/ATOM arms are unchanged. server_tilert.sh passes --grace 60: both ranks pass within one 5 s poll of each other, and the prefill and decode stages have their own readiness waits (DECODE_WAIT, PREFILL_WAIT, ROUTER_WAIT). sync.py 的 --enable-port 在屏障通过后会在前台将本地端口保持 max(60, timeout // 2) 秒，以便晚一次轮询的对端仍能看到。TileRT 启动脚本的 CONTAINER_BARRIER_TIMEOUT=5400 使之变为每个 rank 空转 45 分钟：作业 45373/45374 于 06:28 通过屏障后一直 sleep 到 07:13，占用两个节点。原先硬编码的 1800 秒同样隐藏着 15 分钟的空转。 sync.py 新增 --grace <秒>；不传则沿用原公式，SGLang/vLLM/ATOM 各臂不受影响。 server_tilert.sh 传入 --grace 60：两侧在一次 5 秒轮询内相继通过，且后续 prefill/decode 阶段各有自己的就绪等待。
…经 torch.cuda.cudart 注册 PD 主机缓冲 Sweep 35693738597 job 45373: the decode rank died in alloc_pinned_huge with "neither libamdhip64 nor libcudart could be loaded". The patch dlopen'ed /opt/rocm/lib/libamdhip64.so or libamdhip64.so by name, which exists in the prefill image (ROCm 7.2.3) but not in tilert-rocm-decode:0.1.6, whose torch 2.12+rocm7.14 wheel ships only the versioned libamdhip64.so.7 and no /opt/rocm tree. alloc_pinned_huge now registers the region with torch.cuda.cudart() .cudaHostRegister(addr, total, 0), torch's own runtime binding, hipified on ROCm and present on CUDA builds; verified in both images (cudaError.success, tensor.is_pinned() True). The ctypes library search is gone. Waiver text updated to match. sweep 35693738597 的作业 45373：decode 侧在 alloc_pinned_huge 中因 "neither libamdhip64 nor libcudart could be loaded" 退出。补丁按名字 dlopen /opt/rocm/lib/libamdhip64.so 或 libamdhip64.so，prefill 镜像（ROCm 7.2.3）有，而 tilert-rocm-decode:0.1.6 的 torch 2.12+rocm7.14 wheel 只带版本化的 libamdhip64.so.7， 且无 /opt/rocm 目录。 现改为通过 torch.cuda.cudart().cudaHostRegister(addr, total, 0) 注册，即 torch 自身 的运行时绑定，ROCm 上已 hipify、CUDA 构建亦有；两镜像均验证通过 （cudaError.success，is_pinned 为 True）。移除 ctypes 库查找，豁免文本同步更新。
Add glm5.3* to the 1M-context trace-source pattern and append the required performance changelog entry. Explicit loader overrides remain unchanged. Validation: bash -n and git diff --check passed; 11 real-resolver behavioral cases passed with external dependency installation/download stubbed; changelog YAML, exact config key, and historical byte preservation checked. No GPU benchmark rerun. AI disclosure: GPT 6 Astra via Perplexity Computer implemented and validated this follow-up. No delegated agents. 中文：将 glm5.3* 加入 1M 上下文语料匹配分支，默认选择未截断的 062126 语料，并在性能变更记录末尾追加条目；显式 loader 覆盖保持不变。Shell 语法、差异检查、11 个真实解析函数行为用例及变更记录校验通过；未重新运行 GPU 基准。由 GPT 6 Astra 经 Perplexity Computer 实现和验证，未使用委派 agent。
…ic workspace before sourcing; merge origin/main / 结果目录与工作区分离、提前设置 agentic 工作区变量，并合入 origin/main Sweep 35704948491 brought both lanes up at 1048576 tokens with the PD buffers in DRAM (decode: "receive buffer ready: 99.06 GiB on cpu", "transport=mooncake ready", REQSTAT lines with 7252-token prompts converted and decoded; prefill vLLM ready at gpu-memory-utilization 0.85; router ready) and the 8k1k lane ran its benchmark to completion (16/16, median TTFT 888 ms, TPOT 1.89 ms). Both lanes then failed after the fact: - 8k1k: the launcher copied the result JSON into $GITHUB_WORKSPACE and its EXIT trap ran `sudo rm -rf "$BENCHMARK_LOGS_DIR"`. The tilert arm of benchmarks/multi_node/runtime_settings.sh is the B200 profile and sets BENCHMARK_LOGS_DIR="$GITHUB_WORKSPACE", so the trap deleted the checkout, results included. The AMD block now sets BENCHMARK_LOGS_DIR="$GITHUB_WORKSPACE/benchmark_logs", the convention runners/runtime_settings.sh uses for this pool, and launch_mi355x-amds.sh refuses a BENCHMARK_LOGS_DIR that is the checkout or contains it. - agentic: benchmark_lib.sh derives AGENTIC_DIR and AIPERF_DIR from INFMAX_CONTAINER_WORKSPACE when it is sourced; server_tilert.sh exported the variable later, inside run_agentic_replay, so the client bootstrap looked for /utils/agentic-benchmark/requirements.txt. The export now precedes the source line; resolve_trace_source and install_agentic_deps fail the replay instead of falling through (no errexit here); HF_HOME points at the node's /run_logs mount so the trace corpus is not re-downloaded per job. Merges origin/main (6c5eff0, #3360: priority classifier on Opus 5), no conflicts. sweep 35704948491 中两条 lane 均在 1048576 上下文下以 DRAM 中的 PD 缓冲成功启动， 8k1k 基准测试跑完（16/16，TTFT 中位数 888 ms，TPOT 1.89 ms），随后在事后失败： 8k1k 侧 launcher 将结果复制进 $GITHUB_WORKSPACE 后，EXIT trap 的 `rm -rf "$BENCHMARK_LOGS_DIR"` 因 tilert 分支沿用 B200 配置（值为 $GITHUB_WORKSPACE）而删除了整个检出目录；AMD 专用块现改为 $GITHUB_WORKSPACE/benchmark_logs，launcher 亦拒绝指向检出目录的取值。agentic 侧 benchmark_lib.sh 在 source 时由 INFMAX_CONTAINER_WORKSPACE 推导路径，而该变量此前在 之后才导出；现已提前，两个引导函数失败即返回，HF_HOME 指向节点 /run_logs 以免每次 重下语料。同时合入 origin/main（6c5eff0b8，#3360），无冲突。
Sweep 35704948491: both lanes served GLM-5.3 at 1M with DRAM PD buffers; 8k1k benchmark completed; two post-benchmark bugs fixedThe DRAM route works in production. Decode rank (job 45394, g15), from Prefill vLLM came up at First 8k1k numbers (conc 1, recovered from the node's
Why the lanes still failed — both after the benchmark, both fixed in
Also in 中文DRAM 方案已在生产运行中验证。 decode 侧日志： 首批 8k1k 数据（并发 1，从节点本地副本恢复）：16/16 完成，TTFT 中位数 888 ms，TPOT 中位数 1.89 ms，输出吞吐 351.7 tok/s，端到端中位数 2.68 s。 两条 lane 仍失败的原因均在基准测试之后，已在 |
Sweep 35715072662 on
|
| metric | value |
|---|---|
| successful requests | 16 / 16 |
| median / p99 TTFT | 886 ms / 1,189 ms |
| median / p99 TPOT | 1.97 ms / 2.15 ms |
| output / total token throughput | 344 / 3,114 tok/s |
| median E2E | 2.72 s |
Run-to-run: the aborted sweep's node-local copy showed 888 ms / 1.89 ms / 352 tok/s — same numbers within noise.
AgentX agentic-coding, conc 1, 3600 s (…_conc1.json; the workflow's agentic result guard passed):
| metric | value |
|---|---|
| requests | 234 successful / 245 total (11 warm-up, 1 error dropped) |
| TTFT median / p90 / p99 | 6.49 s / 9.28 s / 16.9 s |
| ITL median / p99 | 2.03 ms / 2.56 ms |
| output tok/s per user (interactivity) | 491 (p50 492) |
| request throughput | 0.06 req/s |
| ISL p50 seen by the client | ~337k tokens; individual requests up to 339k |
| decode acceptance | simulated at the golden 2.99 (K=3), per golden_al_distribution/glm5.3_mtp.yaml — output text is meaningless by design |
Those ~337k-token conversations are the point of the 1M context: at the inherited 202,752 they would have been filtered out. TTFT at that length is dominated by prefill plus the P→D copy of ~34 GB of bf16 KV per request (convert ≈ 200 ms, inject ≈ 1.1 s in the decode REQSTAT lines).
Still open before merge
- Correctness on this stack has not been proven. The sweep runs no GSM8K by default, and the agentic lane's text is meaningless under simulated acceptance. One
RUN_EVAL=truepass on MI355X (real MTP verification, bf16 KV through the DRAM path) is required; the author's 0.975 was on MI350X, pre-DRAM, at 202,752. docs/waiver/3330.mdneeds the upstream TileRT PR/issue link (currently "pending"); CODEOWNER sign-off checks for it.golden_al_distribution/glm5.3_mtp.yamlremains GLM-5.2's curve, labelled PROVISIONAL.- For merge via reuse, this is the validated full-sweep run: a maintainer can pin it with
/use 35715072662.
中文
sweep 35715072662（992c9c0a1）全绿：两条 lane、结果收集、对比、元数据上传全部成功，完整 1,048,576 上下文，PD 缓冲位于 DRAM。这是 MI355X 上首批完整的 GLM-5.3 结果。
单轮 8k1k、并发 1：16/16 成功，TTFT 中位数 886 ms（p99 1,189 ms），TPOT 中位数 1.97 ms，输出吞吐 344 tok/s，端到端中位数 2.72 s；与上一轮从节点恢复的数据（888 ms / 1.89 ms / 352 tok/s）一致。
AgentX 智能体编码、并发 1、3600 秒：234 成功 / 245 总计（11 预热、1 错误丢弃），TTFT 中位数 6.49 s（p99 16.9 s），ITL 中位数 2.03 ms，每用户输出 491 tok/s，请求吞吐 0.06 req/s，客户端所见 ISL 中位数约 33.7 万 token（单请求最高 33.9 万）；decode 侧按黄金曲线 2.99（K=3）模拟接受率，输出文本本身无意义。这些约 33.7 万 token 的会话正是 1M 上下文的意义所在：在沿用的 202,752 下它们会被过滤掉。
合并前仍待处理：① 该栈上的正确性尚未验证 —— sweep 默认不跑 GSM8K，需在 MI355X 上执行一次 RUN_EVAL=true；② docs/waiver/3330.md 缺上游 TileRT 链接；③ 黄金接受率曲线仍为 GLM-5.2 的临时副本；④ 通过复用合并时，可由维护者以 /use 35715072662 固定本次验证运行。
Why the run does not show on the site yet — and the app PR that fixes it
Fix: SemiAnalysisAI/InferenceX-app#1202 — Merge ordering: the same mapper runs at official ingest ( Also from the same pass: 中文
修复： InferenceX-app#1202。合并顺序： 正式入库使用同一 mapper，须先合并并部署 InferenceX-app#1202，再合并本 PR，否则首次入库会静默跳过 GLM-5.3 行。另： |
This branch has not been deployed
2 participants
Description
Adds GLM-5.3 FP8 on MI355X as a prefill/decode-disaggregated TileRT recipe: upstream vLLM ROCm prefill (TileRTConnector,
kv_producer) + TileRTdecode_server+ an OpenAI-compatiblepd_router,framework: tilert, KV over mooncake. Two scenarios — Single-turn 8k1k and AgentX agentic-coding — mirroring the merged B200glm5.1-fp8-b200-tilertentries on theamd_utilsorchestration.Artifacts. Two first-party images,
ghcr.io/tile-ai/tilert-rocm-decode:0.1.6(decode) andghcr.io/tile-ai/tilert-rocm-prefill:0.1.6(prefill, the upstreamvllm/vllm-openai-rocmrelease image plus the mooncake ROCm transfer engine). Neither carries the wheel:setup_deps.shinstallstilert==0.1.6from PyPI into both ranks at container start — full install on decode,--no-depsconnector plugin on prefill, so the image's torch/transformers/vLLM are untouched.Topology. 1 prefill node (TP8) + 1 decode node (TP8),
conc-list: [1]because TileRT decode serves one sequence at a time. bf16 KV cache on both roles — the ROCm vLLM sparse-MLA backend has nofp8_ds_mladtype.max-model-len1048576 (GLM-5.3's full context window), MTP at draft depth 3,gpu-memory-utilization0.85 on the prefill.Golden AL. Nothing hard-codes an acceptance length.
server_tilert.shreads the committed curve at launch, selecting it exactly asinfx/srt_slurm/synthetic_acceptance.pydoes, applies it to agentic throughput only, suppresses it underRUN_EVAL/EVAL_ONLY, and fails the run rather than silently falling back to real acceptance if the curve or draft length is missing.Two checklist items worth stating up front. Framework ordering: GLM-5.3 on AMD has no vLLM/SGLang entry, so this lands a TileRT arm first; the in-tree precedent is NVIDIA
glm5.1, also TileRT-only, merged in #2533 (2026-08-09) and #2650 (2026-08-23), both after the ordering rule was added in #1886 (2026-06-22). Engine patching, with waiver: vLLM runs exactly as shipped, but the TileRT PD transport layer is patched at container start (benchmarks/multi_node/amd_utils/patches/tilert-0.1.6-pd-buffers-in-dram.patch, 5 pure-Python files undertilert/pd_vllm, applied bysetup_deps.shto the pip-installedtilert==0.1.6on both ranks whenTILERT_PD_BUFFER_DEVICE=cpu; marker-gated, pinned to 0.1.6) so that both PD buffers live in 2 MiB-backed pinned host memory; at the full 1M context they do not fit on the GPU. Covered bydocs/waiver/3330.md— what is patched, why the unmodified stack cannot run this benchmark, upstream link (TileRT issue/PR pending), and the removal plan.AI model disclosure
claude-opus-5[1m]— Claude Opus 5 (1M context), via Claude Code. This is the identifier the original runtime exposed.d924515e5; the DRAM-buffer patch and waiver from432e65111):claude-opus-5[1m](Claude Opus 5, 1M context) throughfb663fa26, thenclaude-fable-5-1[1m](Claude Fable 5.1) from432e65111, both via Claude Code; the adversarial verification and review subagents ran onclaude-fable-5-1[1m]. Numbers quoted in those commits and comments were measured on the MI355X cluster or computed from the tilert 0.1.6 / vLLM 0.24.0 sources, as stated per item.GPT 6 Astra, via Perplexity Computer. Ported theRDMAV_FORK_SAFE=1fix from CrimsonDump's commit to both TileRT ranks, added explanatory comments and performance-changelog entries, and ran shell syntax and changelog/matrix validation. No delegated agents were used for this follow-up; no GPU/RDMA benchmark was rerun.Related Issue
Type of Change
Validation
Run on 2x8 MI350X with those images and wheel, through the official entry points:
RUN_EVAL=true EVAL_ONLY=true, 1319 items)submission_valid: true, coverage TTFT 99.8% / ITL 99.9%, 532 requests, 0 errorsMI350X is not MI355X. Same
gfx950ISA, 256 CU / 288 GB / 8 TB/s, but 1000 W vs 1400 W TBP and 2200 vs 2400 MHz peak. These numbers show the recipe runs and is accurate; they are not leaderboard results — the sweep produces those on realmi355xrunners.Checklist
docs/waiver/3330.md(TileRT PD transport only; vLLM unpatched)perf-changelog.yamland have not edited historical entriesOWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR中文
改动说明
以 TileRT 的 prefill/decode 分离形态新增 MI355X 上的 GLM-5.3 FP8：上游 vLLM ROCm 作 prefill（TileRTConnector、
kv_producer）+ TileRTdecode_server+ 兼容 OpenAI 的pd_router，framework: tilert，KV 走 mooncake。两个场景：单轮 8k1k 与 AgentX 智能体编码，形态对齐已合并的 B200glm5.1-fp8-b200-tilert，走amd_utils编排链。产物：两个第一方镜像 —— decode 用
ghcr.io/tile-ai/tilert-rocm-decode:0.1.6，prefill 用ghcr.io/tile-ai/tilert-rocm-prefill:0.1.6（上游vllm/vllm-openai-rocm发布镜像 + 源码构建的 mooncake ROCm 传输引擎）。两者都不含 wheel：setup_deps.sh在容器启动时从 PyPI 装tilert==0.1.6，decode 全装、prefill--no-deps只装 connector 插件，镜像内的 torch/transformers/vLLM 一概不动。拓扑：1 个 prefill 节点（TP8）+ 1 个 decode 节点（TP8），
conc-list: [1]，因为 TileRT decode 一次只服务一条序列。两侧均用 bf16 KV cache —— ROCm 上 vLLM 的稀疏 MLA 后端没有fp8_ds_mla。max-model-len1048576，MTP 草稿深度 3。黄金接受率：任何地方都不写死接受率。
server_tilert.sh在启动时读取提交的曲线，选择规则与infx/srt_slurm/synthetic_acceptance.py完全一致；仅作用于 agentic 吞吐，RUN_EVAL/EVAL_ONLY下自动关闭；曲线或草稿深度缺失时直接失败，而不是静默回落到真实接受率。两条主动说明的清单条款：① 框架顺序 —— AMD 侧 GLM-5.3 目前只有 TileRT；在树先例是 NVIDIA
glm5.1同样只有 TileRT（#2533、#2650，均在规则 #1886 之后合入）。② 引擎补丁与豁免 —— vLLM 原样运行；但 TileRT 的 PD 传输层在容器启动时被打补丁（patches/tilert-0.1.6-pd-buffers-in-dram.patch，由setup_deps.sh在TILERT_PD_BUFFER_DEVICE=cpu时应用），使两个 PD 缓冲位于 2 MiB 大页支撑的锁页主机内存中——完整 1M 上下文下它们无法放入 GPU。已按docs/waiver/3330.md提交豁免（补丁内容、未修改栈无法运行的原因、上游链接（TileRT PR 待开）、移除计划）。AI 模型使用说明
claude-opus-5[1m]—— Claude Opus 5（1M 上下文），经 Claude Code 使用。这是原始运行时暴露的标识。d924515e5起的评审修复、编排与内存分析；自432e65111起的 DRAM 缓冲补丁与豁免）：至fb663fa26为claude-opus-5[1m]，自432e65111起为claude-fable-5-1[1m]，均经 Claude Code；对抗性验证与评审子代理运行于claude-fable-5-1[1m]。所引数字均为 MI355X 集群实测或由 tilert 0.1.6 / vLLM 0.24.0 源码计算，逐项注明。GPT 6 Astra，经 Perplexity Computer 使用。将 CrimsonDump 提交中的RDMAV_FORK_SAFE=1修复移植到 TileRT 两侧，补充说明注释和性能变更记录，并执行 Shell 语法及变更记录/矩阵验证。本次后续修改未使用委派 agent，也未重新运行 GPU/RDMA 基准测试。关联 issue
无。
改动类型
新功能 + 配置改动。
验证结果
在 2×8 MI350X 上、用上述镜像与 wheel、走官方入口：GSM8K strict-match 0.9750、flexible-extract 0.9735；AgentX 3600 秒
submission_valid: true、覆盖率 99.8%/99.9%、532 请求、0 错误；单轮 8k1k TTFT p50 1010 ms、16/16 请求完成；矩阵逻辑单测 335 通过。gfx950ISA、同 256 CU / 288 GB / 8 TB/s，但 TBP 1000W vs 1400W、峰值频率 2200 vs 2400 MHz。这些数字用于证明配方能跑且结果正确，不是打榜成绩 —— 榜上成绩由 sweep 在真实mi355x机器上产出。检查清单
清单已在上方英文部分勾选，此处不重复。