refactor: migrate fixed-sequence recipes to SRT-Slurm / 将定长配方迁移至 SRT-Slurm

[adibarra]

Converts all 50 active Slurm single-node fixed-sequence configs to native SRT recipes: 35 SGLang, eight TRT-LLM, and seven ATOM, covering all 407 existing points. This includes DeepSeek-R1 and Qwen3.5 on H100/H200/B200/B300 and MI300X/MI325X/MI355X. Models, images, concurrency ranges, topology, and serving settings are preserved. Every converted entry exists in current main's active master configs; the deprecation cleanup in #3349 is included. The two Docker-only RTX PRO 6000 fixed-sequence configs are now retired into the existing NVIDIA archive, with original settings and scripts preserved.

Fixed-sequence jobs in the eleven Slurm pools now require an SRT recipe. There is no fallback to the former single-node Bash implementation; the 50 replaced scripts are removed. Active single-node fixed-sequence coverage is SRT-only: the two RTX Docker configs and their scripts are archived, and their unused runner mappings, launcher, and runtime settings are removed. There is no active fixed-sequence Bash/Docker fallback. Changelog planning preserves entries naming archived configs without scheduling them; unknown keys and retired append-only selections still fail. AgentX, existing multi-node paths, and explicitly selected SPEED-Bench collectors retain their current execution paths. AgentX migration will be a separate PR.

Recipes own serving and workload settings; pool profiles own scheduling, mounts, model locations, and image caches. The connector selects exactly one native variant, validates it against the matrix, and applies runtime inputs. Submission, allocation state, cancellation, result/power staging, and native provenance are checked explicitly. Native eval dispatch uses the existing InferenceX eval implementation and propagates eval or staging failures.

Concurrency-dependent settings use existing upstream SRT zip_override_* support. Lists within a group pair by index and override base; they are not a Cartesian product. For example, DeepSeek-R1 H200 TRT MTP uses MTP3 without DP attention at c4–32, then MTP1 with DP attention at c64–256, with corresponding batch/token budgets. Qwen3.5 H200 SGLang varies CUDA graph batch size with concurrency. InferenceX selects the indexed variant matching the job's concurrency and topology.

srt-slurm #26 is on hold and is not a dependency of this migration. TRT recipes use the existing native extra_args support to pass --served_model_name, with matching engine.served_model_name metadata for clients. The SRT submodule now points to our fork at c29ef7c, including the existing AMD → MoRI → ATOM stack (#1, #11, #7) and draft #27 for direct ATOM serving. Those dependencies remain unmerged and need review; #27 is stacked on #7. The goal remains returning to upstream once these capabilities land.

Validation:

• Local CPU suite at 8f3f0d551 after retiring the remaining Docker configs and fixing archived changelog selection: 1,991 passed, 1 skipped, and 187 subtests passed. Ruff, Bash syntax, changelog validation, and historical changelog byte preservation passed.

• All 407 points passed native schema, topology, submission selection, and expanded legacy/native command comparisons. Eval-only generation also passed for all 407 points. These checks do not establish hardware qualification.

• Qwen3.5 H200 c4, runtime a88a27a7cdf04d9f7b9e87492e6abdeadd38fc2c: 40/40 requests, 432.42 output tokens/s, about -0.2% versus the matching published baseline.

• Qwen3.5 H200 c16, same runtime: 160/160 requests, 1002.34 output tokens/s, about +0.6% versus the matching published baseline. Both runs use SGLang v0.5.14-cu130, FP8, TP8/EP8, 8k1k, and pass required eight-GPU power validation.

• Qwen3.5 H200 c64, runtime 2763e8d002efd1786d7c52e30c73410c0d0803c2: 640/640 requests, 1741.50 output tokens/s, about -1.5% versus the matching published baseline. Required eight-GPU power validation passed (maximum gap 1.183 s); Slurm completed with 0:0, and result collection succeeded.

• DeepSeek-R1 H200 c4, runtime 348f773eb6fba29cdb4216d41240cff122a4522e: 40/40 requests, 344.78 output tokens/s, about -0.04% versus the matching published baseline; required power validation passed.

• DeepSeek-R1 TRT MTP H200 c4, runtime b4724b08bb1488105ff6fba32a46ad37470d6417, SRT 2ac4eb1367dd2a78f597a72ca91afe4211d76b38: FP8, TRT-LLM 1.3.0rc14, TP8/EP8, MTP3, 8k1k, concurrency 4. Native E2E passed: 40/40 measured requests, 416.27 output tokens/s, correct served model name, eight-GPU power validation, result collection, and allocation cleanup. This exercises the existing recipe extra_args path without [NVIDIA] Add GB200 DSR1 FP4 TRT #26. A sequential legacy control at main 7e257acec661842d9b7f5651f92b4c29ea162fd7 completed 40/40 requests at 414.86 output tokens/s with identical input/output token totals: native throughput was +0.34%. The control workflow failed result discovery; its raw JSON was later included in the power-audit artifact. Offline validation of the unmodified telemetry in the verified cluster UTC timezone passed for all eight GPUs. That recovery does not change the failed workflow conclusion. This is one point on different H200 nodes, not full performance or accuracy qualification. Both allocations are released.

• DeepSeek-R1 TRT MTP H200 c64, runtime b4724b08, SRT 2ac4eb1: FP8, TRT-LLM 1.3.0rc14, TP8/EP8 with DP attention, MTP1, 8k1k, 640/640 measured requests, 1622.04 output tokens/s. E2E, eight-GPU power validation, result collection, and cleanup passed. This verifies the higher-concurrency recipe variant; no fresh c64 control has been completed.

• Qwen3.5 H200 eval integration, runtime a67435c0f8d2cb2e2d0c61c829ca4197a1368e64: SGLang v0.5.14-cu130, FP8, TP8/EP8, concurrency 4, GSM8K first 16 samples. All requests completed; strict and flexible exact match were 0.9375 (15/16). The workflow score gate failed against Qwen's 0.94 threshold. Native eval receipt was 0, metadata/artifacts staged, and Slurm completed 0:0. This proves the exercised execution/artifact path, not accuracy qualification; the limited-sample failure remains visible and the threshold is unchanged.

• MI300X Qwen3.5 c4 startup check, runtime 571fa51b3, SRT c29ef7c: FP8, SGLang v0.5.12-rocm720-mi30x, TP8/EP1, 8k1k. The allocation failed before server startup because the cluster preamble ended with a newline before the native && join; zero measured requests. The allocation was released. Commit 0bd3bf5f7 fixes both affected AMD preambles and adds controller-based terminal verification for pools without working accounting; native shell execution and negative status cases passed locally. The hardware retry at 8e279188e reached server readiness but failed before benchmark requests because pip --user is incompatible with the ROCm image virtual environment. Commit c430090a8 removes the forced user-site install. The AMD c4 hardware retry at 6f60a8bf2, SRT c29ef7c, passed E2E: 40/40 measured requests, 238.58 output tokens/s versus 241.17 in the matching 2026-05-17 InferenceX API baseline (-1.07%). Mean TTFT was 1.0813 s versus 0.9888 s (+9.35%); mean TPOT was 15.227 ms versus 15.143 ms (+0.56%). Required power passed on exactly eight GPUs (maximum sample gap 2 s); artifacts were collected and Slurm completed 0:0, with the allocation released. This historical comparison does not establish latency parity or accuracy qualification.

• DeepSeek-R1 B200 SGLang MTP c1, runtime c430090a8, SRT c29ef7c: FP4, SGLang v0.5.16-cu130, TP4/EP1, 8k1k. 10/10 requests completed, 297.44 output tokens/s versus 296.69 in the matching InferenceX API baseline dated 2026-08-06 (+0.25%); mean TTFT -3.59%, mean TPOT -0.02%. Slurm completed 0:0 and released the allocation. The workflow failed required power validation because the exclusive eight-GPU allocation exposed eight devices to the client instead of four; this is measured throughput evidence, not a green smoke. Commit 6f60a8bf2 limits native server/client steps to the validated serving GPU count while retaining node exclusivity. The verification run at 6f60a8bf2 passed E2E: 10/10 requests, 297.74 output tokens/s (+0.36% against the same API baseline), mean TTFT -3.79%, mean TPOT -0.11%. Power validation observed exactly four GPUs (maximum sample gap 1.071 s), artifacts were collected, and Slurm completed 0:0 with its allocation released. The five-recipe, 16-point representative batch is paced one allocation at a time. H200 c4 has now passed, completing initial low-concurrency coverage for all five recipes. The B200 Qwen TRT-MTP c4 run completed all three TP2/TP4/TP8 variants serially with required power validation. MI355X ATOM c4 now passes required power and cleanup after the sampler fix, as detailed below. The H200 c4 smoke passed at 77961ed42 with required power and no evals. MI300X c16 has since passed, as detailed below. H200 c64 has also passed, including its DP attention/MTP1 settings, as detailed below. B200 c32 has also passed with exactly four power-valid GPUs, as detailed below. H200 c256 and MI300X c64 have also passed, completing both recipes’ selected low/medium/high points. B200 SGLang c256 has now passed as well, completing its selected c1/c32/c256 points. B200 Qwen TRT c128 has also passed. MI355X ATOM c32 and c256 have now passed, completing the selected batch at 16/16. The final MI355X ATOM c256 point has passed. All 16 selected points across five recipes are complete: 11,290/11,290 measured requests, required power validation and cleanup passed, and all task-owned allocations are released. The batch always ran at most one task-owned GPU allocation. These no-eval smokes do not qualify all 50 converted configs or establish full performance or accuracy parity. The Qwen TP8 c4 historical delta remains unresolved: current exact CLI selection expands all three TP variants and rejects their shared experiment name as ambiguous, so no three-variant rerun was dispatched for a single follow-up point.

• Qwen3.5 B200 TRT-MTP c4, TP2, runtime 6f60a8bf2, SRT c29ef7c: FP4, TRT-LLM 1.3.0rc18, TP2/EP1, DP attention off, 8k1k. The benchmark job passed with 40/40 measured requests and 770.39 output tokens/s versus 769.23 in the matching 2026-06-24 API baseline (+0.15%). Mean TTFT +2.80%; mean TPOT -0.68%. Required power passed on exactly two GPUs (maximum sample gap 1.040 s); Slurm 16906 completed 0:0 and released its allocation. TP4 and TP8 also passed as recorded below; the full three-variant workflow completed successfully. No accuracy eval was requested.

• Qwen3.5 B200 TRT-MTP c4, TP8, runtime 6f60a8bf2, SRT c29ef7c: FP4, TRT-LLM 1.3.0rc18, TP8/EP8, DP attention off, 8k1k. The benchmark job passed with 40/40 measured requests and 1042.85 output tokens/s versus 1081.62 in the matching 2026-06-24 API baseline (-3.58%). Mean TTFT +3.55%; mean TPOT +2.17%. Required power passed on eight GPUs (maximum sample gap 1.001 s); Slurm 16907 completed 0:0 and released its allocation before TP4 started. This throughput decrease is flagged for a fresh comparison; a single historical-baseline smoke cannot identify its cause or establish performance parity.

• Qwen3.5 B200 TRT-MTP c4, TP4, runtime 6f60a8bf2, SRT c29ef7c: FP4, TRT-LLM 1.3.0rc18, TP4/EP4, DP attention off, 8k1k. 40/40 requests passed, 942.20 output tokens/s versus 917.70 in the matching 2026-06-24 API baseline (+2.67%); mean TTFT -6.64%, mean TPOT -1.68%. Required power passed on exactly four GPUs (maximum sample gap 1.001 s); Slurm 16908 completed 0:0 and released its allocation. All three c4 TP variants and result collection finished successfully.

• MI355X DSR1 ATOM-MTP c4 startup failure, runtime b3a5a7f39, SRT c29ef7c: FP4, ATOM rocm7.2.3_ubuntu24.04_py3.12_pytorch_release_2.10.0_atom20260511, TP8/EP1, 8k1k. Python 3.12 raised IndexError in importlib.cache_from_source while importing a PyTorch generated module from / with PYTHONPYCACHEPREFIX enabled. Zero benchmark requests; no throughput or accuracy result. Slurm 45548 failed 1:0 and released its allocation. The same import-path failure was reproduced locally. Commit 161aa243a restores the legacy container working-directory behavior through native srun_options.container-workdir=/infmax-workspace, using the existing repository mount and preserving caller overrides. All 69 focused CPU tests, native command generation, Ruff and changelog validation passed. The retry at 838607d8a completed 40/40 requests, confirming the startup fix; its separate power-validation failure is recorded below.

• MI355X DSR1 ATOM-MTP c4 measurement, runtime 838607d8a, SRT c29ef7c: FP4, TP8/EP1, DP attention off, 8k1k. 40/40 requests completed, 611.90 output tokens/s versus 605.22 in the matching 2026-05-20 API baseline (+1.10%); mean TTFT -2.56%, mean TPOT -0.35%. Slurm 45549 completed 0:0 and released its allocation. The workflow failed required power validation: all eight GPUs were observed, but CSV samples stopped 3.68–4.68 s before the benchmark ended (benchmark_window_not_bracketed). This is measured throughput, not a green smoke or accuracy qualification. The existing CSV filter was reproduced buffering live pipe input with mawk despite fflush(). Commit dd955705c replaces it with a shared Bash line reader, preserving validation and benchmark settings. All 237 focused CPU tests, a Linux streaming check, Ruff, shell syntax and append-only changelog validation passed. The c4 verification run at dd955705c passed E2E: 40/40 requests, 610.71 output tokens/s (+0.91%) against the same May 20 API baseline; mean TTFT -0.36%, mean TPOT -1.39%. Required power passed on all eight GPUs with a maximum sample gap of 2 s. Slurm 45550 completed 0:0 and released its allocation. No accuracy eval was requested. That MI355X allocation and the subsequent H200 c4 allocation are released.

H200 exact selection exposed stale single-digit entries in configs/runners.yaml: _7 matched no registered _07 runner. Commit 77961ed42 aligns both H200 labels with all 14 live registered two-digit names. Eight runner-filter CPU tests, exact one-point generation and append-only changelog validation passed. Benchmark settings are unchanged.

• H200 DSR1 TRT-MTP c4, current fork pin, runtime 77961ed42, SRT c29ef7c: FP8, TRT-LLM 1.3.0rc14, TP8/EP8, DP attention off, MTP3, 8k1k. E2E passed, 40/40 requests, 413.62 output tokens/s versus 413.01 in the matching 2026-05-18 API baseline (+0.15%); mean TTFT -1.97%, mean TPOT -0.36%. Required power passed on all eight GPUs (maximum sample gap 1.164 s); source revisions and artifacts were verified. Slurm 88762 completed 0:0 and released its allocation. No accuracy eval was requested.

• MI300X Qwen3.5 SGLang c16, runtime 77961ed42, SRT c29ef7c: FP8, SGLang v0.5.12-rocm720-mi30x, TP8/EP1, DP attention off, no speculation, 8k1k. E2E passed, 160/160 measured requests, 650.52 output tokens/s versus 652.14 in the matching 2026-05-17 API baseline (-0.25%); mean TTFT +0.89%, mean TPOT +0.21%. Required power passed on all eight GPUs (maximum sample gap 2 s). Source revisions and artifacts were verified. The native terminal-state gate and workflow cleanup passed; Slurm allocation 1159 was independently confirmed absent from the live queue. No accuracy eval; the historical comparison does not establish full performance parity.

• H200 DSR1 TRT-MTP c64, current fork pin, runtime 77961ed42, SRT c29ef7c: FP8, TRT-LLM 1.3.0rc14, TP8/EP8, DP attention enabled, MTP1, 8k1k. E2E passed, 640/640 measured requests, 1629.27 output tokens/s versus 1628.59 in the matching 2026-05-18 API baseline (+0.04%); mean TTFT -3.64%, mean TPOT +0.29%. Required power passed on all eight GPUs (maximum sample gap 1.154 s). The emitted TRT config confirms the concurrency-dependent settings. Source revisions, artifacts and cleanup were verified; Slurm 88777 completed 0:0 and released its allocation. No accuracy eval; the historical comparison does not establish full performance parity.

• B200 DSR1 SGLang-MTP c32, runtime 77961ed42, SRT c29ef7c: FP4, SGLang v0.5.16-cu130, TP4/EP1, DP attention off, 8k1k. E2E passed, 320/320 measured requests, 1699.06 output tokens/s versus 1717.36 in the matching 2026-08-06 API baseline (-1.07%); mean TTFT -2.17%, mean TPOT +1.20%. Required power passed on exactly four GPUs (maximum sample gap 1.072 s); native submission uses srun_options.gpus-per-node=4. Source revisions, artifacts and cleanup were verified; Slurm 16910 completed 0:0 and released its allocation. No accuracy eval; the historical comparison does not establish full performance parity.

• H200 DSR1 TRT-MTP c256, runtime 77961ed42, SRT c29ef7c: FP8, TRT-LLM 1.3.0rc14, TP8/EP8, DP attention enabled, MTP1, 8k1k. E2E passed, 2560/2560 measured requests, 2639.48 output tokens/s versus 2636.47 in the matching 2026-05-18 API baseline (+0.11%); mean TTFT -1.69%, mean TPOT -0.02%. Required power passed on all eight GPUs (maximum sample gap 1.169 s); emitted TRT settings include max_batch_size: 32. Source revisions, artifacts and cleanup were verified; Slurm 88778 completed 0:0 and released its allocation. The selected H200 c4/c64/c256 points have all passed; no accuracy eval or full performance qualification is claimed.

• MI300X Qwen3.5 SGLang c64, runtime 77961ed42, SRT c29ef7c: FP8, SGLang v0.5.12-rocm720-mi30x, TP8/EP1, DP attention off, no speculation, 8k1k. E2E passed, 640/640 measured requests, 1385.23 output tokens/s versus 1387.16 in the matching 2026-05-17 API baseline (-0.14%); mean TTFT +4.29% (2.0803 s versus 1.9947 s), mean TPOT -0.09%. Required power passed on all eight GPUs (maximum sample gap 2 s). Source revisions, artifacts and cleanup were verified; Slurm 1160 completed 0:0 and released its allocation. MI300X selected c4/c16/c64 are complete. No accuracy eval; the historical comparison does not establish full performance parity.

• B200 DSR1 SGLang-MTP c256, runtime 77961ed42, SRT c29ef7c: FP4, SGLang v0.5.16-cu130, TP4/EP4, DP attention enabled, 8k1k. E2E passed, 2560/2560 measured requests, 3843.97 output tokens/s versus 3904.54 in the matching 2026-08-06 API baseline (-1.55%); mean TTFT +1.89%, mean TPOT +1.68%. Server visibility and required power both covered exactly four GPUs (maximum sample gap 1.003 s). Source revisions, artifacts and cleanup were verified; Slurm 16911 completed 0:0 and released its allocation. Selected B200 SGLang c1/c32/c256 are complete, including the EP4/DP-attention transition. No accuracy eval; historical comparisons do not establish full performance parity.

• B200 Qwen3.5 TRT-MTP c128, runtime 77961ed42, SRT c29ef7c: FP4, TRT-LLM 1.3.0rc18, TP8/EP8, DP attention enabled, 8k1k. E2E passed, 1280/1280 measured requests, 8977.02 output tokens/s versus 8801.95 in the matching 2026-06-24 API baseline (+1.99%); mean TTFT -4.04%, mean TPOT +2.34%. Required power passed on all eight GPUs (maximum sample gap 1.140 s). Source revisions, artifacts and cleanup were verified; Slurm 16912 completed 0:0 and released its allocation. This covers the selected DP-attention transition; it does not resolve the separate TP8 c4 historical throughput decrease or qualify accuracy.

• MI355X DSR1 ATOM-MTP c32, runtime 77961ed42, SRT c29ef7c: FP4, ATOM rocm7.2.3_ubuntu24.04_py3.12_pytorch_release_2.10.0_atom20260511, TP8/EP1, DP attention off, 8k1k. E2E passed, 320/320 measured requests, 2002.39 output tokens/s versus 1978.91 in the matching 2026-05-20 API baseline (+1.19%); mean TTFT +1.28%, mean TPOT -1.14%. Required power passed on all eight GPUs (maximum sample gap 2 s). Source revisions, artifacts and cleanup were verified; Slurm 45584 completed 0:0 on mia1-p01-g14 and released its allocation. No accuracy eval; the historical comparison does not establish full performance parity.

• MI355X DSR1 ATOM-MTP c256, runtime 77961ed42, SRT c29ef7c: FP4, ATOM rocm7.2.3_ubuntu24.04_py3.12_pytorch_release_2.10.0_atom20260511, TP8/EP1, DP attention off, 8k1k. E2E passed, 2560/2560 measured requests, 3492.79 output tokens/s versus 3471.59 in the matching 2026-05-20 API baseline (+0.61%); mean TTFT +5.88% (2.8468 s), mean TPOT -0.91% (69.409 ms). Required power passed on all eight GPUs (maximum sample gap 2 s). Source revisions, artifacts and cleanup were verified; Slurm 45587 completed 0:0 on mia1-p01-g14 and released its allocation. This completes the 16-point representative batch. No accuracy eval; the historical comparison does not establish latency or full performance parity.

Keep draft. Seven configs have passed twenty-two hardware throughput points: six on earlier SRT pins plus sixteen current-pin H200, B200, MI300X and MI355X points above. The current 16-point representative batch has 16 passing functional smoke points across all 5 recipes. All selected low/medium/high points and topology variants are complete; all 16 points passed required power and cleanup, with no accuracy evals. B200 Qwen TRT TP8 is 3.58% below its historical API throughput baseline; this needs follow-up before claiming performance parity. Qwen comparisons use historical baselines; TRT MTP now has one fresh sequential comparison, with the control collection failure described above. Live qualification remains incomplete: broader TRT/MTP concurrency coverage, accuracy evals, other pools, and cancellation under load still need validation. Tests are paced one GPU allocation at a time. All active Slurm single-node fixed-sequence recipes are converted; broader AMD coverage and ATOM concurrency/accuracy qualification remain outstanding. Docker is excluded. The six passing NVIDIA points above predate the new fork pin and must not be treated as hardware validation of that pin; H200, B200, MI300X and MI355X now have passing hardware points for the current pin. Fork #27 passes the title/copyright checks; the stacked PR does not trigger the fork's main/master-only Linux CPU workflow. Its 3,102 local passing tests and 13 failures reproduced on the unchanged base are documented there. This replaces closed #3351.

中文

将全部 50 个活跃的 Slurm 单节点定长配置切换为原生 SRT 配方：35 个 SGLang、8 个 TRT-LLM、7 个 ATOM，覆盖全部 407 个现有测试点。包含 H100/H200/B200/B300 和 MI300X/MI325X/MI355X 上的 DeepSeek-R1 与 Qwen3.5。模型、镜像、并发范围、拓扑和服务设置保持一致。每个迁移条目均存在于当前 main 的活跃 master 配置中；已包含 #3349 的弃用清理。两个仅使用 Docker 的 RTX PRO 6000 定长配置现已退役至现有 NVIDIA 归档，原始设置与脚本均予保留。

十一个 Slurm 池的定长作业现在必须提供 SRT 配方，不再回退到旧版单节点 Bash 实现；50 个已替代脚本已删除。活跃的单节点定长覆盖仅使用 SRT：两个 RTX Docker 配置及其脚本已归档，不再使用的 runner 映射、启动器和运行时设置已移除，不保留活跃的定长 Bash/Docker 回退。Changelog 规划保留引用归档配置的历史条目，但不生成对应作业；未知键名及 append-only 选择已退役配置仍会报错。AgentX、现有多节点路径及显式选择的 SPEED-Bench 采集器继续使用当前执行路径。AgentX 迁移将单独提交 PR。

配方负责服务与工作负载设置；集群配置负责调度、挂载、模型位置及镜像缓存。连接器精确选择一个原生变体，与矩阵核对后再注入运行时参数。提交、分配状态、取消、结果与功耗产物及原生来源信息均有显式检查。原生 eval 调度复用现有 InferenceX 实现，并将评测或产物准备失败传递给作业。

随并发变化的设置使用 SRT 上游已有的 zip_override_* 功能。同一组内的列表按索引配对并覆盖 base，不会生成笛卡尔积。例如，DeepSeek-R1 H200 TRT MTP 在 c4–32 使用 MTP3、不启用 DP attention，在 c64–256 改用 MTP1 和 DP attention，并相应调整 batch/token 预算。Qwen3.5 H200 SGLang 则随并发调整 CUDA graph batch size。InferenceX 根据作业的并发与拓扑选择对应索引的变体。

srt-slurm #26 暂缓推进，本迁移不依赖该 PR。TRT 配方通过已有的原生 extra_args 支持传入 --served_model_name，并为客户端设置一致的 engine.served_model_name 元数据。SRT 子模块现固定到组织分叉的 c29ef7c，包含已有的 AMD → MoRI → ATOM 依赖链（#1、#11、#7），以及支持 ATOM 直接服务的草稿 #27。这些依赖均未合并，仍需评审；#27 基于 #7。长期目标仍是这些能力进入上游后回归上游。

验证：

• 8f3f0d551 退役剩余 Docker 配置并修复归档 Changelog 选择后的本地 CPU 测试：1,991 项通过、1 项跳过，另有 187 个子测试通过。Ruff、Bash 语法、changelog 校验及历史字节保留检查通过。

• 全部 407 个测试点通过原生 schema、拓扑、提交变体选择和展开后的旧版/原生命令对比；407 个 eval-only 配置生成检查也通过。这些检查不代表硬件验收完成。

• Qwen3.5 H200 c4，运行时 a88a27a7cdf04d9f7b9e87492e6abdeadd38fc2c：40/40 请求完成，输出吞吐 432.42 tokens/s，相对匹配的已发布基线约 -0.2%。

• Qwen3.5 H200 c16，相同运行时：160/160 请求完成，输出吞吐 1002.34 tokens/s，相对匹配的已发布基线约 +0.6%。两次运行均使用 SGLang v0.5.14-cu130、FP8、TP8/EP8、8k1k，八张 GPU 的强制功耗校验通过。

• Qwen3.5 H200 c64，运行时 2763e8d002efd1786d7c52e30c73410c0d0803c2：640/640 请求完成，输出吞吐 1741.50 tokens/s，相对匹配的已发布基线约 -1.5%。八张 GPU 的强制功耗校验通过（最大采样间隔 1.183 s）；Slurm 以 0:0 完成，结果汇总成功。

• DeepSeek-R1 H200 c4，运行时 348f773eb6fba29cdb4216d41240cff122a4522e：40/40 请求完成，输出吞吐 344.78 tokens/s，相对匹配的已发布基线约 -0.04%；强制功耗校验通过。

• DeepSeek-R1 TRT MTP H200 c4，运行时 b4724b08bb1488105ff6fba32a46ad37470d6417、SRT 2ac4eb1367dd2a78f597a72ca91afe4211d76b38：FP8、TRT-LLM 1.3.0rc14、TP8/EP8、MTP3、8k1k、并发 4。原生 E2E 通过：40/40 个计量请求完成，输出吞吐 416.27 tokens/s，服务模型名称正确，八张 GPU 的功耗校验、结果汇总和资源清理均通过。这验证了现有配方 extra_args 路径，无需 [NVIDIA] Add GB200 DSR1 FP4 TRT #26。随后串行运行的旧版对照使用 main 7e257acec661842d9b7f5651f92b4c29ea162fd7，40/40 请求完成，输出吞吐 414.86 tokens/s；输入和输出 token 总数完全一致，原生吞吐高 0.34%。对照工作流因未找到结果文件而失败，但原始 JSON 随后包含在上传的功耗审计产物中。按已核实的集群 UTC 时区离线校验未经修改的遥测，八张 GPU 均通过。恢复产物不会改变工作流失败的结论。这只是不同 H200 节点上的一个测试点，不代表完整性能或准确性验收。两个资源分配均已释放。

• DeepSeek-R1 TRT MTP H200 c64，运行时 b4724b08、SRT 2ac4eb1：FP8、TRT-LLM 1.3.0rc14、TP8/EP8、DP attention、MTP1、8k1k，640/640 个计量请求完成，输出吞吐 1622.04 tokens/s。E2E、八张 GPU 的功耗校验、结果收集及清理均通过。验证了高并发配方变体；尚未完成新的 c64 对照运行。

• Qwen3.5 H200 eval 集成检查，运行时 a67435c0f8d2cb2e2d0c61c829ca4197a1368e64：SGLang v0.5.14-cu130、FP8、TP8/EP8、并发 4，GSM8K 前 16 个样本。全部请求完成，strict/flexible exact match 均为 0.9375（15/16）。工作流分数门槛未通过，Qwen 要求 0.94。原生 eval 退出记录为 0，元数据及产物已准备，Slurm 以 0:0 完成。这只验证实际执行和产物路径，不代表准确性验收；保留小样本失败记录，阈值未变。

• MI300X Qwen3.5 c4 启动检查，运行时 571fa51b3、SRT c29ef7c：FP8、SGLang v0.5.12-rocm720-mi30x、TP8/EP1、8k1k。集群前置命令尾部的换行与原生 && 拼接冲突，资源分配在服务启动前失败；计量请求数为零。资源已释放。0bd3bf5f7 修复了两个受影响的 AMD 前置命令，并在 accounting 不可用的集群上通过控制器核验终态；原生 shell 实际执行和失败状态用例已在本地通过。基于 8e279188e 的实机重试已到达服务就绪阶段，但 pip --user 与 ROCm 镜像的虚拟环境不兼容，因而在发送基准请求前失败。c430090a8 移除了强制用户目录安装参数。基于 6f60a8bf2、SRT c29ef7c 的 AMD c4 实机重试 E2E 通过：40/40 个计量请求完成，输出吞吐 238.58 tokens/s；匹配的 2026-05-17 InferenceX API 基线为 241.17 tokens/s，变化 -1.07%。平均 TTFT 为 1.0813 s，基线为 0.9888 s（+9.35%）；平均 TPOT 为 15.227 ms，基线为 15.143 ms（+0.56%）。八张 GPU 的强制功耗校验通过，最大采样间隔 2 s；产物已收集，Slurm 以 0:0 完成并释放资源。历史基线比较不能证明延迟完全一致，也不代表准确性验收。

• DeepSeek-R1 B200 SGLang MTP c1，运行时 c430090a8、SRT c29ef7c：FP4、SGLang v0.5.16-cu130、TP4/EP1、8k1k。10/10 请求完成，输出吞吐 297.44 tokens/s；匹配的 InferenceX API 基线日期为 2026-08-06，吞吐为 296.69 tokens/s，变化 +0.25%；平均 TTFT -3.59%，平均 TPOT -0.02%。Slurm 以 0:0 完成并释放资源。节点独占分配向客户端暴露了八张 GPU，而预期为四张，因此工作流未通过必需的功耗校验；这是已测吞吐证据，不是通过的 smoke。6f60a8bf2 在保留节点独占的同时，将原生服务和客户端步骤限制为已验证的推理 GPU 数量。基于 6f60a8bf2 的验证运行 E2E 通过：10/10 请求完成，输出吞吐 297.74 tokens/s（相同 API 基线相比 +0.36%），平均 TTFT -3.79%，平均 TPOT -0.11%。功耗校验恰好观测到四张 GPU，最大采样间隔 1.071 s；产物已收集，Slurm 以 0:0 完成并释放资源。五个配方的 16 个代表点每次仅运行一个分配。H200 c4 现已通过，五个配方均已完成首轮低并发验证。B200 Qwen TRT-MTP c4 运行已串行完成 TP2/TP4/TP8，强制功耗校验均通过。修复采样器后，MI355X ATOM c4 已通过必需的功耗校验和资源清理，详情见下文。基于 77961ed42 的 H200 c4 smoke 已通过必需功耗校验，本次未运行 eval。MI300X c16 现已通过，详情见下文。H200 c64 也已通过，包括 DP attention/MTP1 配置，详情见下文。B200 c32 也已通过，功耗校验恰好覆盖四张 GPU，详情见下文。H200 c256 和 MI300X c64 均已通过，这两个配方选定的低、中、高并发测试点已全部完成。B200 SGLang c256 现也已通过，其选定的 c1/c32/c256 均已完成。B200 Qwen TRT c128 现也已通过。MI355X ATOM c32 和 c256 均已通过，选定批次已全部完成，达到 16/16。最后一个 MI355X ATOM c256 测试点已通过。五个配方的全部 16 个选定测试点均已完成：11,290/11,290 个计量请求完成，必需功耗校验及清理通过，本任务的全部资源分配均已释放。整个批次始终最多仅有一个 GPU 资源分配。这些未运行 eval 的 smoke 不代表全部 50 个已转换配置通过验收，也不能证明完整性能或准确性一致。Qwen TP8 c4 相对历史基线的差异仍未查明：当前 CLI 会展开三个 TP 变体，且因它们共享实验名称而拒绝精确名称选择，因此未为单个复测点重跑三个变体。

• Qwen3.5 B200 TRT-MTP c4、TP2，运行时 6f60a8bf2、SRT c29ef7c：FP4、TRT-LLM 1.3.0rc18、TP2/EP1、不启用 DP attention、8k1k。基准作业通过，40/40 个计量请求完成，输出吞吐 770.39 tokens/s；匹配的 2026-06-24 API 基线为 769.23 tokens/s，变化 +0.15%。平均 TTFT +2.80%，平均 TPOT -0.68%。功耗校验恰好观测到两张 GPU，最大采样间隔 1.040 s；Slurm 16906 以 0:0 完成并释放资源。TP4 和 TP8 也已通过，结果如下；三个变体的完整工作流已成功结束。本次未运行准确性评测。

• Qwen3.5 B200 TRT-MTP c4、TP8，运行时 6f60a8bf2、SRT c29ef7c：FP4、TRT-LLM 1.3.0rc18、TP8/EP8、不启用 DP attention、8k1k。基准作业通过，40/40 个计量请求完成，输出吞吐 1042.85 tokens/s；匹配的 2026-06-24 API 基线为 1081.62 tokens/s，变化 -3.58%。平均 TTFT +3.55%，平均 TPOT +2.17%。八张 GPU 的强制功耗校验通过，最大采样间隔 1.001 s；Slurm 16907 以 0:0 完成，并在 TP4 启动前释放资源。这一下降已列为后续新鲜对照的关注项；单次历史基线 smoke 不能确定原因或证明性能一致。

• Qwen3.5 B200 TRT-MTP c4、TP4，运行时 6f60a8bf2、SRT c29ef7c：FP4、TRT-LLM 1.3.0rc18、TP4/EP4、不启用 DP attention、8k1k。40/40 请求通过，输出吞吐 942.20 tokens/s；匹配的 2026-06-24 API 基线为 917.70 tokens/s，变化 +2.67%；平均 TTFT -6.64%，平均 TPOT -1.68%。四张 GPU 的强制功耗校验通过，最大采样间隔 1.001 s；Slurm 16908 以 0:0 完成并释放资源。三个 c4 TP 变体及结果收集均已成功结束。

• MI355X DSR1 ATOM-MTP c4 启动失败，运行时 b3a5a7f39、SRT c29ef7c：FP4、ATOM rocm7.2.3_ubuntu24.04_py3.12_pytorch_release_2.10.0_atom20260511、TP8/EP1、8k1k。启用 PYTHONPYCACHEPREFIX 且从 / 启动时，Python 3.12 在导入 PyTorch 动态生成模块的 importlib.cache_from_source 路径抛出 IndexError。基准请求数为零，无吞吐或准确性结果。 Slurm 45548 以 1:0 失败并释放资源。已在本地复现相同导入路径错误。161aa243a 通过原生 srun_options.container-workdir=/infmax-workspace 恢复旧版容器工作目录行为，使用现有仓库挂载并保留调用方覆盖。69 项定向 CPU 测试、原生命令生成、Ruff 和 changelog 校验均通过。基于 838607d8a 的重试完成了 40/40 个请求，确认启动问题已修复；另一个功耗校验失败记录如下。

• MI355X DSR1 ATOM-MTP c4 测量，运行时 838607d8a、SRT c29ef7c：FP4、TP8/EP1、不启用 DP attention、8k1k。40/40 请求完成，输出吞吐 611.90 tokens/s；匹配的 2026-05-20 API 基线为 605.22 tokens/s，变化 +1.10%；平均 TTFT -2.56%，平均 TPOT -0.35%。Slurm 45549 以 0:0 完成并释放资源。工作流未通过必需的功耗校验：八张 GPU 均有数据，但 CSV 样本在基准结束前 3.68–4.68 s 停止（benchmark_window_not_bracketed）。这是吞吐测量结果，不是通过的 smoke 或准确性验收。已复现现有 CSV 过滤器在 mawk 下即使调用 fflush() 仍缓冲管道输入的问题。dd955705c 将其替换为共享 Bash 逐行读取器，保留功耗校验和基准设置。237 项定向 CPU 测试、Linux 流式检查、Ruff、shell 语法及追加式 changelog 校验均通过。基于 dd955705c 的 c4 验证运行 E2E 通过：40/40 请求完成，输出吞吐 610.71 tokens/s（+0.91%），对比相同的 5 月 20 日 API 基线；平均 TTFT -0.36%，平均 TPOT -1.39%。八张 GPU 的必需功耗校验通过，最大采样间隔为 2 s。Slurm 45550 以 0:0 完成并释放资源。本次未运行准确性评测，该 MI355X 分配及后续 H200 c4 分配均已释放。

H200 精确选择暴露了 configs/runners.yaml 中过期的单数字条目：_7 无法匹配实际注册的 _07 runner。77961ed42 将两个 H200 标签清单与 14 个在线注册的两位数字名称对齐。8 项 runner 过滤 CPU 测试、精确单点生成及追加式 changelog 校验均通过，基准设置未变。

• H200 DSR1 TRT-MTP c4，当前分叉固定版本，运行时 77961ed42、SRT c29ef7c：FP8、TRT-LLM 1.3.0rc14、TP8/EP8、不启用 DP attention、MTP3、8k1k。E2E 通过，40/40 请求完成，输出吞吐 413.62 tokens/s；匹配的 2026-05-18 API 基线为 413.01 tokens/s，变化 +0.15%；平均 TTFT -1.97%，平均 TPOT -0.36%。八张 GPU 的必需功耗校验通过，最大采样间隔 1.164 s；源版本和产物均已核实。Slurm 88762 以 0:0 完成并释放资源，本次未运行准确性评测。

• MI300X Qwen3.5 SGLang c16，运行时 77961ed42、SRT c29ef7c：FP8、SGLang v0.5.12-rocm720-mi30x、TP8/EP1、不启用 DP attention、无推测解码、8k1k。E2E 通过，160/160 个计量请求完成，输出吞吐 650.52 tokens/s；匹配的 2026-05-17 API 基线为 652.14 tokens/s，变化 -0.25%；平均 TTFT +0.89%，平均 TPOT +0.21%。八张 GPU 的必需功耗校验通过，最大采样间隔 2 s。源版本及产物已核实；原生终态校验与工作流清理均通过，另行检查实时队列确认 Slurm 分配 1159 已释放。本次未运行准确性评测，历史基线比较不代表完整性能验收。

• H200 DSR1 TRT-MTP c64，当前分叉固定版本，运行时 77961ed42、SRT c29ef7c：FP8、TRT-LLM 1.3.0rc14、TP8/EP8、启用 DP attention、MTP1、8k1k。E2E 通过，640/640 个计量请求完成，输出吞吐 1629.27 tokens/s；匹配的 2026-05-18 API 基线为 1628.59 tokens/s，变化 +0.04%；平均 TTFT -3.64%，平均 TPOT +0.29%。八张 GPU 的必需功耗校验通过，最大采样间隔 1.154 s。实际生成的 TRT 配置确认了随并发变化的设置。源版本、产物与清理均已核实；Slurm 88777 以 0:0 完成并释放资源。本次未运行准确性评测，历史基线比较不代表完整性能验收。

• B200 DSR1 SGLang-MTP c32，运行时 77961ed42、SRT c29ef7c：FP4、SGLang v0.5.16-cu130、TP4/EP1、不启用 DP attention、8k1k。E2E 通过，320/320 个计量请求完成，输出吞吐 1699.06 tokens/s；匹配的 2026-08-06 API 基线为 1717.36 tokens/s，变化 -1.07%；平均 TTFT -2.17%，平均 TPOT +1.20%。必需功耗校验恰好覆盖四张 GPU，最大采样间隔 1.072 s；原生提交使用 srun_options.gpus-per-node=4。源版本、产物与清理均已核实；Slurm 16910 以 0:0 完成并释放资源。本次未运行准确性评测，历史基线比较不代表完整性能验收。

• H200 DSR1 TRT-MTP c256，运行时 77961ed42、SRT c29ef7c：FP8、TRT-LLM 1.3.0rc14、TP8/EP8、启用 DP attention、MTP1、8k1k。E2E 通过，2560/2560 个计量请求完成，输出吞吐 2639.48 tokens/s；匹配的 2026-05-18 API 基线为 2636.47 tokens/s，变化 +0.11%；平均 TTFT -1.69%，平均 TPOT -0.02%。八张 GPU 的必需功耗校验通过，最大采样间隔 1.169 s；实际 TRT 设置包含 max_batch_size: 32。源版本、产物与清理均已核实；Slurm 88778 以 0:0 完成并释放资源。选定的 H200 c4/c64/c256 均已通过；本次未运行准确性评测，也不代表完整性能验收。

• MI300X Qwen3.5 SGLang c64，运行时 77961ed42、SRT c29ef7c：FP8、SGLang v0.5.12-rocm720-mi30x、TP8/EP1、不启用 DP attention、无推测解码、8k1k。E2E 通过，640/640 个计量请求完成，输出吞吐 1385.23 tokens/s；匹配的 2026-05-17 API 基线为 1387.16 tokens/s，变化 -0.14%；平均 TTFT +4.29%（2.0803 s 对比 1.9947 s），平均 TPOT -0.09%。八张 GPU 的必需功耗校验通过，最大采样间隔 2 s。源版本、产物与清理均已核实；Slurm 1160 以 0:0 完成并释放资源。选定的 MI300X c4/c16/c64 已全部通过。本次未运行准确性评测，历史基线比较不代表完整性能验收。

• B200 DSR1 SGLang-MTP c256，运行时 77961ed42、SRT c29ef7c：FP4、SGLang v0.5.16-cu130、TP4/EP4、启用 DP attention、8k1k。E2E 通过，2560/2560 个计量请求完成，输出吞吐 3843.97 tokens/s；匹配的 2026-08-06 API 基线为 3904.54 tokens/s，变化 -1.55%；平均 TTFT +1.89%，平均 TPOT +1.68%。服务可见设备与必需功耗校验均恰好覆盖四张 GPU，最大采样间隔 1.003 s。源版本、产物与清理均已核实；Slurm 16911 以 0:0 完成并释放资源。选定的 B200 SGLang c1/c32/c256 已全部完成，覆盖 EP4/DP attention 配置切换。本次未运行准确性评测，历史基线比较不代表完整性能验收。

• B200 Qwen3.5 TRT-MTP c128，运行时 77961ed42、SRT c29ef7c：FP4、TRT-LLM 1.3.0rc18、TP8/EP8、启用 DP attention、8k1k。E2E 通过，1280/1280 个计量请求完成，输出吞吐 8977.02 tokens/s；匹配的 2026-06-24 API 基线为 8801.95 tokens/s，变化 +1.99%；平均 TTFT -4.04%，平均 TPOT +2.34%。八张 GPU 的必需功耗校验通过，最大采样间隔 1.140 s。源版本、产物与清理均已核实；Slurm 16912 以 0:0 完成并释放资源。本次覆盖选定的 DP attention 配置切换，但不能解释 TP8 c4 相对历史基线的吞吐下降，也不代表准确性验收。

• MI355X DSR1 ATOM-MTP c32，运行时 77961ed42、SRT c29ef7c：FP4、ATOM rocm7.2.3_ubuntu24.04_py3.12_pytorch_release_2.10.0_atom20260511、TP8/EP1、不启用 DP attention、8k1k。E2E 通过，320/320 个计量请求完成，输出吞吐 2002.39 tokens/s；匹配的 2026-05-20 API 基线为 1978.91 tokens/s，变化 +1.19%；平均 TTFT +1.28%，平均 TPOT -1.14%。八张 GPU 的必需功耗校验通过，最大采样间隔 2 s。源版本、产物与清理均已核实；Slurm 45584 在 mia1-p01-g14 上以 0:0 完成并释放资源。本次未运行准确性评测，历史基线比较不代表完整性能验收。

• MI355X DSR1 ATOM-MTP c256，运行时 77961ed42、SRT c29ef7c：FP4、ATOM rocm7.2.3_ubuntu24.04_py3.12_pytorch_release_2.10.0_atom20260511、TP8/EP1、不启用 DP attention、8k1k。E2E 通过，2560/2560 个计量请求完成，输出吞吐 3492.79 tokens/s；匹配的 2026-05-20 API 基线为 3471.59 tokens/s，变化 +0.61%；平均 TTFT +5.88%（2.8468 s），平均 TPOT -0.91%（69.409 ms）。八张 GPU 的必需功耗校验通过，最大采样间隔 2 s。源版本、产物与清理均已核实；Slurm 45587 在 mia1-p01-g14 上以 0:0 完成并释放资源。至此 16 个代表性测试点全部完成。本次未运行准确性评测，历史基线比较不能证明延迟或完整性能一致。

保持草稿。七个配置共通过二十二个实机吞吐点：六个使用较早的 SRT 固定版本，另十六个是上述当前固定版本上的 H200、B200、MI300X 和 MI355X 测试点。当前 16 个代表点已有 16 个通过功能 smoke，覆盖全部 5 个配方。选定的低、中、高并发测试点及拓扑变体均已完成；全部 16 点的必需功耗校验和清理通过，本批次未运行准确性评测。B200 Qwen TRT TP8 比历史 API 吞吐基线低 3.58%，在确认性能一致之前仍需进一步比较。Qwen 使用历史基线；TRT MTP 已完成一个新鲜的串行对照点，对照运行的结果收集失败如上所述。实机验收尚未完成：仍需验证更广的 TRT/MTP 并发范围、准确性评测、其他集群池及负载运行期间的取消行为。GPU 测试每次只运行一个分配。全部活跃的 Slurm 单节点定长配方已转换；更广的 AMD 覆盖及 ATOM 并发范围和准确性验收仍待完成，Docker 不在本次迁移范围内。上述六个通过的 NVIDIA 测试点早于新分叉固定版本，不能作为新版本的实机验证；H200、B200、MI300X 和 MI355X 已在当前固定版本上通过实机测试。分叉 #27 的标题及版权检查已通过；由于它基于另一 PR，不会触发仅面向 main/master 的 Linux CPU 工作流。该 PR 记录了 3,102 项本地通过测试及在未修改基线上复现的 13 项失败。本 PR 替代已关闭的 #3351。

[github-actions]

Thanks for the contribution!

• Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
• PR verification: Sweeps only run on labeled PRs. Add full-sweep-fail-fast (strongly recommended); use full-sweep-enabled only when matrix jobs should continue after a failure.
• After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.

中文

感谢你的贡献！

• **审阅：**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER，请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，再通过 Slack 联系核心维护者。必须严格遵循模板，并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have，才能触发签核验证。
• **PR 验证：**扫描仅在带有标签的 PR 上运行。强烈建议添加 full-sweep-fail-fast；仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。
• **合并后：**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复；参见重新运行失败任务的说明。

[cquil11]

this doesn't need to be in here it should be hard coded in benchmark-tmpl.yml I believe?

[cquil11]

shall this be standardized as well?