feature/mad-profiling - #207
Closed
mkuznet1 wants to merge 154 commits into
Closed
Conversation
unified_docker vllm --------- Co-authored-by: seungrokjung <seungrok.jung@amd.com> Co-authored-by: Aswin John Mathews <Aswin.Mathews@amd.com> Co-authored-by: seungrokj <144636725+seungrokj@users.noreply.github.com> Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>
Update default timeout to 4 hrs
Co-authored-by: Stephen Shao <ysha@amd.com>
* Added gitignore for project * Added .gitattributes --------- Co-authored-by: Stephen Shao <ysha@amd.com>
* Skip torchaudio to avoid torch reinstall * Remove data field for HF bert
* Updated the input args * Update arguments * Added multi models support * Fixed the typeError of tags * Fixed the logger object duplicated * Updated logger to handle multiple log files * Cleaned up the --model-name * Fixed the issue of logger init * Replaced model-name to tags --------- Co-authored-by: Stephen Shao <ysha@amd.com> Co-authored-by: Rahul Garg <rahul.garg@amd.com>
Co-authored-by: Stephen Shao <ysha@amd.com> Co-authored-by: Rahul Garg <rahul.garg@amd.com>
* Added parser of report generated by vllm * Added field of multiple_results in models of pyt_vllm of models.json * Added multiple results support * Added the parser_csv to vllm benchmark report * Added pandas to installation * Debug the output perf.csv * Debug the generation of perf.csv * Fixed the parsing error of perf and metric in run scripts of bert and gpt2. Updated the logger for file formatter. Updated the perf csv generation * Fixed the error * Fixed the parser of perf_model.csv * Fixed the model names in multiple results of pyt vllm models * Fixed the path error * Add data.json * Fixed the FileNotFound error of perf_model.csv * Fixed the error of NotFoundColumn mod in dataframe of perf * Fixed the error in model name column * Updated the perf result printout beautifully * Added columns as header when creating empty dataframe * Refactored the attributes in RunDetails * Added attributes git_commit and relative_change * Updated header of dataframe in parse csv * Remove workaround perf passing logic * Add dummy model to test multiple perf results * Rename dummy.ubuntu.amd.dockerfile to dummy.ubuntu.amd.Dockerfile changed the filename of Dockerfile * Fixed the issue of converting multiple results * Debugging the perf generation * Debug the perf generation * Removed assert for length of perf csv df * Fixed the issue of converting multiple results and cleaned up * Cleaned up logger --------- Co-authored-by: Stephen Shao <yu.shao@amd.com> Co-authored-by: Stephen Shao <ysha@amd.com>
Co-authored-by: Stephen Shao <ysha@amd.com>
Co-authored-by: Stephen Shao <u.shao@amd.com>
Co-authored-by: Younseo Roh <younseo.roh@amd.com>
* vllm rocm 6.3.1 release * vllm rocm 6.3.1 release * vllm rocm 6.3.1 release
Co-authored-by: Stephen Shao <ysha@amd.com>
* vllm 0.7.3 * vllm 0.7.3 * Update benchmark/vllm/README.md Co-authored-by: Peter Park <git@peterjunpark.com> * Update benchmark/vllm/README.md Co-authored-by: Peter Park <git@peterjunpark.com> * Update benchmark/vllm/README.md Co-authored-by: Peter Park <git@peterjunpark.com> * vllm 0.7.3 * disclaimer 2025 --------- Co-authored-by: Peter Park <git@peterjunpark.com>
* pyt training v25.4 update * Correct the dataset used for Huggingface PEFT * Add error output --------- Co-authored-by: alfuyao1986 <81382865+alfuyao1986@users.noreply.github.com> Co-authored-by: clairesonglee <claire.lee2@amd.com>
…OCm#28) * unified vllm docker with v0.7.3 Signed-off-by: seungrokjung <seungrok.jung@amd.com> * unified vllm docker with v0.7.3 Signed-off-by: seungrokjung <seungrok.jung@amd.com> --------- Signed-off-by: seungrokjung <seungrok.jung@amd.com> Co-authored-by: Rahul Garg <rahul.garg@amd.com>
* Update CODEOWNERS * Update CODEOWNERS
Co-authored-by: Rohan Potdar <66227218+Rohan138@users.noreply.github.com>
* Primus v26.3
Primus v26.3 release introduces the following new models and upgrades previously supported models
Qwen 3 30B BF16/FP8
Qwen 3 235B BF16/FP8
GPT OSS 20B BF16/FP8
GPT OSS 120B BF16/FP8
* Update README.md
added examples for multi-node training of mixtral 8x22B and llama3.1-405B. Also made some other changes.
* Update README.md
* Update README.md
* Update README.md
added multi-node training examples
* Update primus_megatron-lm_benchmark_setup.sh
accept Copilot suggestions
* Update primus_megatron-lm_benchmark_report.sh
* Update README.md with multinode and remove proxy models
* Update README.md torchtitan with multinode examples
* typo fix megatron README.md
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Update README.md
* Update primus_megatron-lm_benchmark_setup.sh
* Update primus_megatron-lm_benchmark_setup.sh
---------
Co-authored-by: Fuyuan Jing <167437074+amd-fuyuajin@users.noreply.github.com>
Co-authored-by: vidushi8 <vidgoyal@amd.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…m#161) * [sglang_disagg] Sync MoRI IO optimization from MAD-private PR #276 - Unified launcher with MoRI/Mooncake backend selection (KV_TRANSFER_BACKEND) - CX7 multi-rail NIC support, default to CX7 400G rail NICs - xP/yD multi-node support for DP_MODE=0 (TP-only) - DP_MODE=1 restricted to 1P1D (multi-node DP not yet supported) - Condensed RDMA/NCCL/Gloo env config in mori_ep_env.sh - Model flag catalog cleanup: dp settings only on DeepSeek-V3/R1 - Configurable benchmark combinations with random-range-ratio=1.0 - Dockerfile updated to rocm720 base image Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [sglang_disagg] Restrict to 1P1D only (xP=1, yD=1) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [sglang_disagg] Allow xP/yD multi-node, restrict to DP_MODE=0 only Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [sglang_disagg] Allow DP_MODE=1 for 1P1D, block multi-node DP Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Basem Barakat <basem.barakay@amd.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* enable wide ep sglang * update dockerfile with proper commits * update readme and remove unnecessary comments * update proper titles in readme
* Update pyt_chai1_inference.ubuntu.amd.Dockerfile To use main branch instead of v0.4.4 which is older * Update pyt_chai1_inference.ubuntu.amd.Dockerfile Updated
…m#168) The MoRI dispatch/combine shared-memory heap library default (4 GiB) is enough for EP<=16 but is exhausted by the larger EP all-to-all buffers at higher EP degrees (e.g. EP32 / 4 prefill nodes), where prefill warmup crashes with "Out of heap memory ... hip invalid argument" in dispatch_combine. Default to 16 GiB (still overridable via MORI_SHMEM_HEAP_SIZE).
Purpose is to add pytorch-xdit:v26.6 image to MAD workflows and to publish it.
Work here contains
Updated Dockerfile wrapping amdsiloai/pytorch-xdit:v26.6
which checked should be complete.
New update in the file to include the packages Few OS related packages needs to be installed
…arge_ep_benchmark (ROCm#167) * [madengine] Add 33 model cards for sglang_disagg, vllm_disagg, and large_ep Synced from MAD-private develop-basem-madengine-sglang-disagg branch: - 16 sglang_disagg cards (MoRI IO TP, MoRI DP, Mooncake) - 12 vllm_disagg cards (NIXL, MoRI, DeepEP) - 5 large EP benchmark cards (1N/2N/4N, MoRI only variants) - perf.csv reporting for madengine slurm_multi integration - vllm_disagg Dockerfile and script updates Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [madengine] Remove private docker image references from model cards Docker images are in a private registry and should not be published. DOCKER_IMAGE_NAME left empty to be supplied at runtime. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [madengine] Use placeholder for DOCKER_IMAGE_NAME in model cards madengine slurm_multi checks DOCKER_IMAGE_NAME before merging --additional-context env_vars. Empty string is treated as unset, causing "requires --registry or --use-image" error. Use placeholder "<supply-your-image>" so the check passes; users override at runtime. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Remove rdma-core v62.0 build/install workaround No longer needed — container image includes compatible rdma-core. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Remove RIXLE name from README SGLang row Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Basem Barakat <basem.barakay@amd.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Primus v26.4 Add llama3.1 8B mxfp4/mxfp8 examples. Update READMEs with torchtitan and Megatron. Move to Rock based pytorch docker images. Modified logic to detect device type for rock images * update git checkout to be from tag Signed-off-by: Gene Der Su <e870252314@gmail.com> --------- Signed-off-by: Gene Der Su <e870252314@gmail.com> Co-authored-by: Gene Der Su <e870252314@gmail.com>
…ROCm#171) * vllm_dissag: unify disaggregated launchers into one two-axis driver + MoRI-EP support Consolidate the three standalone vLLM disaggregated-inference launchers into a single launcher driven by two axes, and add the DeepSeek-V3 MoRI-EP recipe from MAD-private #324. Launcher - vllm_disagg.sh: one driver selecting behavior by CONNECTOR={rixl|moriio} x WIDE_EP={0=TP|1=wideEP}, with EP_BACKEND={mori|deepep} validated against the connector (moriio->mori, rixl->deepep; cross-pairs rejected). Legacy RUN_MORI / RUN_DEEPEP map on via a back-compat shim. - connectors/{rixl,moriio}.sh: per-connector hook implementations. rixl reproduces the legacy NixlConnector TP + DeepEP launchers byte-for-byte; moriio provides MoRIIO+TP (new) and MoRI-EP wideEP (from #324). parallelism.sh holds the TP-vs-wideEP helpers. - models.yaml: per-model flags + env catalog (replaces the inline declare -A maps). DeepSeek family is wideEP-only (TP rejected in the slurm gate). Deletes the 3 legacy launchers. Correctness fixes (live-validated on the ROCm 7.2.3 / mori121 image) - GPU-RDMA registration errno-14: ROCm 7.2.3 can't dmabuf-export HIP-VMM memory, so the KV cache must use expandable_segments:False (+ HSA_ENABLE_IPC_MODE_LEGACY=0). This platform env lives in per-connector connectors/<CONNECTOR>.env and is forwarded by the slurm via docker -e (must reach PID 1; PyTorch reads alloc-conf at import). - moriio+TP router: --intra-node-data-parallel-size = 1 for TP (was DP_PARALLEL_SIZE_LOCAL, which round-robined to nonexistent DP ranks -> data_parallel_rank out of range 500s). - Llama-70B TP: --disable-custom-all-reduce (avoids a hipIpcGetMemHandle HIP-IPC crash). - rixl connector honors ROUTER_BINARY (images without a baked vllm-router). No runtime patcher - The MoRIIO disagg fixes (#39276 notify, #41751 LL split, DP-rank hash-failsafe) are committed in-source in the vLLM the image builds from (public fork, Dockerfile VLLM_REF), so connector_runtime_patch is a no-op. Removed the runtime .py patchers (apply_39276_rebased.py, apply_moriio_2pd_patches.sh) and the SKIP_RUNTIME_PATCH gate — they were a drifting duplicate of fixes that already live upstream in the fork. Docker - docker/vllm_disagg_mori_ep_fullsource...Dockerfile: from-source image on the open rocm/vllm-dev ci_base — MoRI v1.2.1, AITER 0.1.16.post3, vLLM 06_29 wide-ep WRITE branch, vllm-router built from vllm-project/router PR#181 (DP-rank round-robin + 2P2D KV-notify dpfix). No public prebuilt image; DOCKER_IMAGE_NAME is BYO-build. Tests + docs - tests/: offline gates (gate_check combo allow/reject, argv_assert flag/env checks, parity_check golden argv diff vs the legacy launchers) + generic interactive drivers. - README / ARCHITECTURE / TEST_PLAN: combo matrix, per-model x combo enablement, diagrams. Validated live (ISL/OSL 512/512 and 1024/1024, plus 8192/1024 & 28000/1024 stress; 0 failed requests): moriio+TP8 (Llama-70B), moriio+wideEP DeepSeek-V3 1P/1D EP8 and 2P/2D EP16. rixl+TP and rixl+deepep assemble correct argv but need a NIXL/DeepEP-validated image (documented in the README image-capability table). Co-Authored-By: Claude <noreply@anthropic.com> * vllm_dissag: unify Docker image (all connectors) + persistent JIT cache Fold the MoRI-EP fullsource stack and the UCX/RIXL/rocSHMEM/DeepEP transport layer into the single vllm_disagg_inference Dockerfile, gated by WITH_NIXL (default 1 = all connectors). One image now serves moriio TP + MoRI-EP wideEP (incl. full DeepSeek-V3) and the rixl NIXL/DeepEP paths. Built + validated on ci_base; the moriio paths are live-proven (Llama-70B TP, DeepSeek-V3 wideEP EP8, 0 failed requests). rixl+TP builds and NIXL initializes but hits an image-level PyNCCL all-reduce error on this stack (cudagraph-independent) - tracked as a known limitation, not a launcher regression. Add a persistent, image-id-keyed JIT cache mount (host NVMe -> /opt/vllm_cache) to both the production launcher and the interactive test driver. The image points AITER_JIT_DIR/TRITON_CACHE_DIR/VLLM_CACHE_ROOT/COMGR_CACHE_DIR at /opt/vllm_cache, which was previously unmounted, so AITER CK kernels recompiled (~15 min) every run; now they compile once and warm-boot in ~1 min. Bump container shm to 256G. Co-Authored-By: Claude <noreply@anthropic.com> * vllm_dissag: genericize interactive test-driver header comment Drop the stale internal Slurm job number from run_interactive.sh's header; describe its role (driven by tests/drive_cell.sh via srun --overlap) instead. Co-Authored-By: Claude <noreply@anthropic.com> * vllm_dissag: strip runtime/recipe/fabric ENV from the image (clean artifact) The Docker image now ships only build + structural ENV (compile arch, source pins, lib paths, and the /opt/vllm_cache cache locations). Everything run-tunable is applied at launch, so the same image serves any model/cluster without a rebuild: - model-serving recipe (KV_BLOCK_SIZE, KV_CACHE_DTYPE, per-role *_CUDAGRAPH_MODE, *_MORI_BACKEND, GPU_MEMORY_UTILIZATION, KV_CACHE_MEMORY_BYTES, AITER_MLA, ...) -> scripts/vllm_dissag/models.yaml per-model env: (single source of truth). Added the per-role cudagraph defaults (prefill NONE, decode PIECEWISE) to the dense model entries, which previously inherited them from the image-level ENV. - MoRI/RDMA fabric tuning (MORI_RDMA_TC/SL, MORI_IB_GID_INDEX, MORI_NUM_QP_PER_PE, VLLM_MORIIO_*, HSA_*) -> connectors/moriio.env, alongside the platform env. These are cluster-specific (OCI MI300X RoCEv2 values), so baking them into a public image was wrong; overriding per-fabric now needs no rebuild. Baking recipe defaults globally also leaked DeepSeek-specific values (AITER_MLA=0, 20GB KV) onto dense models; per-model env: fixes that. Launcher ${VAR:-default} fallbacks match the previously-baked values, so the validated moriio TP and MoRI-EP paths are unchanged (verified: dense decode still emits cudagraph PIECEWISE from models.yaml; offline gate/argv/parity suites all pass). Co-Authored-By: Claude <noreply@anthropic.com> * vllm_dissag: drop redundant fullsource Dockerfile; single image for all The combined vllm_disagg_inference Dockerfile is a superset: WITH_NIXL=0 already produces exactly what vllm_disagg_mori_ep_fullsource did (MoRI-EP only: moriio TP/wideEP + deepep-from-base), and WITH_NIXL=1 (default) adds the rixl NIXL/DeepEP transport. Remove the duplicate fullsource Dockerfile and update the README + slurm error message + in-file build hint to reference the single Dockerfile. Co-Authored-By: Claude <noreply@anthropic.com> * vllm_dissag: remove golden argv fixtures + parity gate Drop tests/golden/ and tests/parity_check.sh: the byte-identical-vs-legacy gate served the migration (now complete) and can't be regenerated since the legacy launchers are deleted. Offline coverage stays via gate_check + argv_assert (run_all.sh). Updated README/TEST_PLAN/ARCHITECTURE and stale parity comments accordingly. * vllm_dissag: add NIAH long-context retrieval test Needle-in-a-haystack retrieval sweep (adapted from vllm-project/vllm#47042): plants animal names in filler text across growing context lengths and counts retrieval. niah_test.py hits the OpenAI-compatible endpoint; niah_bench.sh is the launcher-compatible wrapper (BENCHMARK_SCRIPT_FILE hook). * vllm_dissag: move NIAH into benchmark_* convention Relocate the NIAH long-context retrieval test from tests/niah_{test.py,bench.sh} to top-level benchmark_niah.{py,sh}, matching benchmark_xPyD.sh / benchmark_long_context.sh. Update internal ref, BENCHMARK_SCRIPT_FILE hook name, README benchmark section, and ARCHITECTURE file map. --------- Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Rohan Potdar <rohan.potdar@amd.com>
* Add vLLM Kimi K3 for MI355X Add pyt_vllm_kimi-k3 benchmarking moonshotai/Kimi-K3, a 2.8T-parameter MoE (16 of 896 experts active, 1M context) whose weights ship natively MXFP4 from quantization-aware training, so unlike the K2.6 pair there is no separate amd/ FP4 repo and only one model entry is needed. K3 requires vLLM >= 0.27.0, which is not yet in a tagged vllm-openai-rocm release. Add docker/pyt_vllm_kimi_k3 pinned to the model-specific :kimi-k3 image rather than retargeting docker/pyt_vllm, which would move ~35 other vLLM entries onto a pre-release base. The Dockerfile body is otherwise identical and should be folded back once K3 lands in a versioned ROCm image. Serving flags and env mirror the single-node TP8 gfx950 profile from recipes.vllm.ai/moonshotai/Kimi-K3/hw/mi355x.json. TP8 only: the ~1680 GB minimum footprint fits an 8x MI355X node (2304 GB) but not TP4, and not a single 8x MI300X node, hence skip_gpu_arch gfx942. The model is deliberately left out of the vllm_default sweep so a tag run does not pull the ~1.56 TB checkpoint and hold 8 GPUs; it is invocable explicitly by name. Two deviations from the recipe are intentional. MAD adds --no-enable-prefix-caching for benchmark hygiene, as it does for every model here. The gsm8k accuracy stage is disabled because lm_eval drives /v1/completions, where K3's always-on reasoning is returned inline and exhausts the max_gen_toks=2048 budget hardcoded in run_vllm.py, making the score meaningless. Verified statically only, no hardware run: the emitted server command was rendered through run_vllm.expand_configs with MAD_SYSTEM_GPU_ARCHITECTURE=gfx950 and diffed against the recipe argv with no missing flags, and the vllm_default sweep and the expansion counts for K2.6, DeepSeek-R1 and GLM-5.2 are unchanged. Note that the 1800s server-start timeout in run_vllm.py may be tight for a 1.56 TB load and is left for a follow-up. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Add SGLang Kimi K3 for MI355X Add pyt_sglang_kimi-k3 and pyt_sglang_kimi-k3_dspark benchmarking moonshotai/Kimi-K3 online serving, mirroring the vLLM K3 entry added in a40063d. Both track the AMD day-0 recipes in sgl-project/sglang#32548; K3 support itself is sgl-project/sglang#32541. Two entries rather than one because the two published recipes differ only in speculative decoding (DSPARK with the RadixArk/Kimi-K3-DSpark draft) and are reported as separate rows in the tracking issue, so both are worth tracking. They share a model repo, which is why the runner gains a --variant filter: the existing --model filter cannot select between them. Add scripts/sglang/run_sglang.py and configs/, a config-driven serving runner structured after scripts/vllm/run_vllm.py and emitting the same perf CSV schema, so multiple_results ingestion is shared. The existing scripts/sglang/run.sh is left untouched; it is bound to the offline getopts path and hard-exits on non-gfx94* GPUs. Add docker/pyt_sglang_kimi_k3 pinned to lmsysorg/sglang-rocm:rocm720-mi35x-k3-20260727, the day-0 image the issue's MI355X tables were measured against, rather than retargeting docker/pyt_sglang, which is still on lmsysorg/sglang:v0.4.5-rocm630 and would move the existing SGLang entry onto a pre-release base. A newer -20260728 tag exists but nothing published ties it to the recipe, so it is deliberately not adopted. Fold back once K3 lands in a versioned ROCm image. TP8 gfx950 only, hence skip_gpu_arch gfx942. Both are left out of any sweep tag so a tag run does not pull the checkpoint and hold 8 GPUs; they are invocable explicitly by name. Three deviations from the recipe are intentional. The runner emits --tp-size where the recipe writes --tp: there is no --tp server argument, tp_size declares the single alias --tensor-parallel-size, and --tp resolves only through argparse prefix matching, which any future --tp* option would silently break. The server-start timeout is raised to 5400s from run_vllm.py's 1800s, which that commit already flagged as tight for a checkpoint this size. There is no gsm8k accuracy stage, for the same reason as the vLLM entry: K3's always-on reasoning is returned inline over /v1/completions and exhausts the hardcoded 2048-token budget. The issue does not state its input and output lengths. They were recovered from its own tables, where (E2EL - TTFT) / TPOT + 1 lands on ~1024 output tokens on every row and concurrency x (inp + out) / E2EL reproduces the reported total throughput only at 8192 input tokens; keeping that shape is what makes these numbers comparable to the issue's. --random-range-ratio is pinned to 1.0 because it defaults to 0.0, which would randomize lengths down to a single token. Verified statically only, no hardware run. Config expansion yields 5 configs per variant and the variant filter separates them; an end-to-end dry run with subprocess stubbed produced the expected server and benchmark argv, both hf download calls, and correctly ignored a stale leading line in the appended JSONL output. All 14 server flags and 11 benchmark flags were validated against #32541's head branch rather than sglang main, which matters: the PR is still open and kimi_k3 is absent from main's reasoning DetectorMap, so the parser names and DSPARK are only valid inside the pinned image. Synthetic results were fed through madengine's handle_multiple_results, and filter_images_by_skip_gpu_arch skips both entries on gfx942 and runs them on gfx950 with no existing entry changed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * List Kimi-K3 in the SGLang blueprint row The Blueprints table in the root README is the entry point to each framework's landing page, so the SGLang row's Models column should name K3 now that both entries exist. The landing page itself, benchmark/sglang/README.md, was updated in ede3844. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Add ATom Kimi-K3 for MI355X Introduces ATom (ROCm/ATOM) integration to MAD, modeled after MAD-Private client-perf. Includes Dockerfile, run scripts, and Kimi-K3 config based on ROCm/ATom PR #1718 recipe (gfx950 TP8, MXFP4 w/ FlyDSL SiTUv2, FP8 KV cache). Co-Authored-By: Claude <noreply@anthropic.com> * Add atom inner models.json for standalone dev-mode runs Co-Authored-By: Claude <noreply@anthropic.com> * Update ATom Kimi-K3: correct image tag, env vars, batched tokens - Dockerfile: use rocm/atom-dev:rocm7.2.4_..._kimi_k3 image - Config: add all ATOM/AITER env vars from the MI355X recipe - Fix --max-num-batched-tokens to 10240 (recipe value) Co-Authored-By: Claude <noreply@anthropic.com> * Clean up ATom Kimi-K3: remove internal refs, dead profiling, dev leftovers Must-fix: - Remove internal references (MAD-private, client-perf-hub, PR #380) - Remove dead --profile flag and rocm-trace-lite clone from run.sh - Remove duplicate --kv_cache_dtype from perf.yaml, fix batched tokens to 10240 - Add missing env vars (ATOM_USE_UNIFIED_ATTN, ATOM_FORCE_ATTN_TRITON) - Simplify skip_gpu_arch to gfx942 (matches convention) - Remove atom_default tag (prevents accidental sweep of 1TB+ checkpoint) - Remove dev leftover scripts/atom/models.json Should-fix: - Guard HF_HUB_CACHE in run.sh (match vLLM pattern) Co-Authored-By: Claude <noreply@anthropic.com> * Fix ATom MAD_DATAHOME: use directly like vLLM/SGLang Use MAD_DATAHOME as the model path directly instead of joining with model name. Matches vLLM (run_vllm.py:436) and SGLang (run_sglang.py:317) behavior. Removes CHECK_LOCAL_DATA gate. Co-Authored-By: Claude <noreply@anthropic.com> * Rename ATom Dockerfile to pyt_atom_kimi_k3 to match convention vLLM and SGLang use model-specific Dockerfile names (pyt_vllm_kimi_k3, pyt_sglang_kimi_k3). ATom should too. Co-Authored-By: Claude <noreply@anthropic.com> * Add Kimi-K3 landing page: unified benchmark docs across vLLM, SGLang, ATom New benchmark/kimi_k3/README.md covering all three frameworks with: - Hardware requirements (8x MI350X/MI355X, TP8, ~1.56 TB checkpoint) - Quick start madengine commands for each framework - Framework comparison table and per-framework details - Standalone Docker run commands for each framework - Links to upstream recipes and model card Added as top entry in main README Blueprints table. Co-Authored-By: Claude <noreply@anthropic.com> * Clean up Kimi-K3 landing page: remove Notes sections and DSpark Co-Authored-By: Claude <noreply@anthropic.com> * Simplify hardware requirements section * Remove standalone serving section from Kimi-K3 landing page * Add benchmark results tables to Kimi-K3 landing page vLLM, SGLang, ATom serving throughput and latency numbers from MI350X 8xTP8 runs. Co-Authored-By: Claude <noreply@anthropic.com> * Add missing vLLM TTFT/TPOT latency numbers * Fix ATom inp=4096 mc=128 TTFT/TPOT numbers (verified against logs) * Clean up Kimi-K3 overview wording * Simplify Kimi-K3 overview to factual one-liner * Add standalone benchmarking section to Kimi-K3 landing page Step-by-step Docker run + serve + benchmark commands for each framework (vLLM, SGLang, ATom) without madengine. Co-Authored-By: Claude <noreply@anthropic.com> * Fix standalone commands to match MAD: --no-enable-prefix-caching, --host 127.0.0.1 Verified 1:1 against MAD configs and scripts: - vLLM: --enable-prefix-caching -> --no-enable-prefix-caching (run_vllm.py hardcodes this) - SGLang: --host 0.0.0.0 -> --host 127.0.0.1 (run_sglang.py HOST constant) - ATom: already correct Co-Authored-By: Claude <noreply@anthropic.com> * modify default parameters * Replace benchmark tables with links to framework blog posts * Fix ATom -> ATOM (AiTer Optimized Model, all caps) * Pin ATOM base image SHA (tag was re-pushed broken) --------- Co-authored-by: Stephen Shao <yu.shao@amd.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Tej Kiran <kiran.tej@amd.com> Co-authored-by: lcskrishna <lollachaitanya@gmail.com>
* Primus v26.5 * Use MAD_SECRETS_HFTOKEN instead of MAD_SECRET_HFTOKEN This wrapper previously only mapped MAD_SECRET_HFTOKEN -> HF_TOKEN, but the rest of the repo (and other workflows) commonly use MAD_SECRETS_HFTOKEN. Standardizing on MAD_SECRETS_HFTOKEN avoids confusing 'HF_TOKEN not set' failures when users set the existing variable name. Changes: - scripts/primus_train/run.sh: Check MAD_SECRETS_HFTOKEN instead of MAD_SECRET_HFTOKEN - benchmark/primus/README.md: Update documentation to use MAD_SECRETS_HFTOKEN Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com> * Remove models.json entries superseded by primus_train Drops the old pyt_megatron_lm_train_*, primus_pyt_megatron_lm_train_*, primus_pyt_train_*, and LLM-focused pyt_train_* entries now that primus_train consolidates them. Keeps the non-LLM pyt_train_* models (flux, stable-diffusion-xl, mochi-1, hunyuan-video, wan2_1-i2v, dlrm) that primus_train does not cover. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix primus_train run.sh arg handling and estimate MFU when logs omit it run.sh was forwarding the wrapper-only --config_path flag straight through to Primus' CLI (which rejects it) and let set -e abort the script on a non-zero training exit before perf extraction could run. extract_primus_perf.py now also estimates model_flops_utilization from tflops and AMD's published per-GPU peak FLOPS when a log format (e.g. Megatron 26.5+) doesn't print an mfu field directly. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: clairesonglee <claire.lee2@amd.com> Co-authored-by: Claude Sonnet 4 <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Updated the dockerfile as per the failures in ROCM-28890
…regated) and the mad-slurm-multinode skill (ROCm#194) * Add SGLang disaggregated P/D inference benchmark (DeepSeek-R1) - docker/sglang_disagg_inference_full_overlay.ubuntu.amd.Dockerfile: full-overlay inference image (RCCL + MoRI + NIXL/Mooncake KV-transfer in one build), with an optional ENABLE_RDMA62 build arg for hosts needing rdma-core v62 (OCI/CX7). MoRI is built with cmake<4.0 and --no-build-isolation: pip's build isolation resolves the newest cmake from PyPI, and cmake >= 4.0 changed the JSON emitted by gtest_discover_tests, which breaks the build on a GPU-less host. - scripts/sglang_disagg/: run.sh as the entrypoint for madengine's native sglang-disagg templated launcher, with in-container rank-ordered node-IP discovery (ip_rendezvous.py); routes both MoRI and Mooncake/NIXL backends through the single unified sglang_disagg_mori_io_ep.sh launcher. - parse_to_csv.py rewritten for full metric extraction into the madengine perf-CSV schema; the backend tag derives from RUN_MORI/KV_TRANSFER_BACKEND instead of being guessed from DP_MODE. - benchmark_xPyD.sh: resilient concurrency sweep (fail-fast plus per-point retries, honors RUN_LOG_DIR from run.sh instead of a hardcoded /run_logs). - models.json: register sglang-disagg-deepseek-r1-overlay. The name signals that the entry builds on the overlay Dockerfile and avoids overlap with future public-container SGLang DeepSeek-R1 entries. Validated on CX7/Mellanox-RoCE (OCI), 4 nodes / 32 GPUs, 2P2D: 56/56 config rows green across a con(8/16/32/64) x isl(1024/8192) sweep; and on AMD-AINIC/Pollara (gfx950), 2-node 1P1D with both MoRI and Mooncake KV-transfer: 56/56 green. Co-authored-by: Ilia Kosarev <Ilia.Kosarev@amd.com> Co-authored-by: Basem Barakat <Basem.Barakat@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com> * Add mad-slurm-multinode agent skill and cluster templates The skill packages the bootstrap for madengine's native multi-node SLURM path so the same launchers are reproducible on any of the three cluster archetypes we validated against (CX7/Mellanox-RoCE, AMD-AINIC/Pollara, Broadcom-Thor2-RoCE) without cluster-specific code baked into the scripts. - SKILL.md plus reference docs (cluster-types, deploy-bootstrap, gotchas, launch-and-results, manifests) and helper scripts (detect_cluster_env.sh, preflight.sh, validate_manifest.sh) that bootstrap madengine on a fresh node and drive the native launchers. - Cluster-agnostic mad.env templates and madengine manifest templates for CX7/Mellanox-RoCE, AMD-AINIC/Pollara and Broadcom-Thor2-RoCE. - Sanitized per-archetype walkthroughs (no real node names, queues or tokens). - .gitignore: keep the skill's *.template.json manifests tracked despite the broad *.json ignore. Used to run both workloads end-to-end (Primus Llama-3.1 scaleout and DeepSeek-R1 sglang-disagg) on CX7/Mellanox-RoCE (OCI) and AMD-AINIC/Pollara (gfx950) using only manifest-level configuration, with zero changes to madengine or MAD core scripts. Co-authored-by: Ilia Kosarev <Ilia.Kosarev@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com> * Add Primus Megatron-LM scaleout training benchmarks (Llama-3.1 8B/70B/405B) Primus v26.5 (ROCm#187) removed scripts/primus/ and replaced the per-model models.json entries with a single discovery-based primus_train entry. The scaleout harness this benchmark needs is not part of that consolidation, so it is carried over verbatim into scripts/primus_megatron-lm/ and kept self-contained (all cross-references inside the directory are relative). - scripts/primus_megatron-lm/: run.sh, primus_megatron-lm_benchmark_setup.sh and primus_megatron-lm_benchmark_report.{sh,py}, moved as-is from the removed scripts/primus/megatron-lm/. run.sh carries the Llama-3.1-405B model mapping and falls back to amd-smi, then to MAD_SYSTEM_GPU_ARCHITECTURE, when rocminfo cannot report the device. - primus_megatron-lm_benchmark_report.sh: global-batch-size renormalization for NUM_GPUS > 8, so the reported GBS matches the one actually used, plus run metadata for the perf CSV. - docker/primus_megatron_train_rccl_overlay.ubuntu.amd.Dockerfile: RCCL built from source over the Primus base, installed over every location torch resolves, with optional rdma-core for Broadcom Thor2 bnxt_re. Base moved to rocm/primus:v26.5. - models.json: register primus_pyt_megatron_lm_train_llama-3.1-{8b,70b,405b}_overlay. - mad-slurm-multinode skill: manifest templates and reference docs follow the new script path and base image. Validated on CX7/Mellanox-RoCE (OCI), 2 nodes / 16 GPUs: 8/8 configs green, no rendezvous or DistStoreError issues; scaleout training also confirmed on an AMD-AINIC/Pollara (gfx950) cluster. Co-authored-by: Ilia Kosarev <Ilia.Kosarev@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com> * Read Primus 26.5 throughput fields in Megatron benchmark postprocessing Primus 26.5 renamed the throughput fields on the Megatron iteration line, so both metric parsers silently matched nothing on a v26.5 base: the run trained fine and SLURM reported COMPLETED, but every per-node perf_primus-megatron-<model>.csv held only a header and the aggregated perf.csv came out empty. 26.5 prints "compute per GPU (TFLOP/s/GPU): X (avg Y)" and "tokens/s/GPU inst/harmonic mean: X/Y" where 26.4 printed "<name>): X/Y", and it still uses the 26.4 shape for the first iterations, before the running means exist. Both parsers now understand either shape: - benchmark_report.py consults the 26.5 shapes only when the existing patterns matched nothing, so 26.4 logs keep parsing exactly as before. - the STEP 12 patch reads each shape as a separate sed -e expression, so every iteration line yields exactly one <iteration> <value> pair regardless of which shape it used. The patch text also moves into a single variable shared by the --check probe and the apply, which previously were two hand-synchronised copies of the same 80-line heredoc. Verified against the captured stdout of a 2-node Llama-3.1-8B run on rocm/primus:v26.5 (BF16 9208.08 tok/s/GPU, 474.61 TFLOP/s/GPU; FP8 12471.15 / 642.80) and against a v26.4 log, whose parsed output is unchanged. * Address Copilot review nits in the Megatron-LM benchmark scripts benchmark_report.py imported pandas without ever using it, so the metrics export would fail with ModuleNotFoundError on any image that does not ship it. data was bound only inside the recognized-model branch but read unconditionally by the CSV writer, so a model outside that list raised NameError instead of writing an empty report. run.sh maps deepseek-v2 to DeepSeek-V2, which the list does not cover. Seeding data before the branch turns that path into a header-only CSV. run.sh repeated the GPT-OSS / Qwen-3 datatype branch twice back to back, leaving the second copy unreachable. Parsed output is unchanged for the registered Llama-3.1 models on both the 26.4 and the 26.5 log shapes. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Ilia Kosarev <Ilia.Kosarev@amd.com> Co-authored-by: Basem Barakat <Basem.Barakat@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com>
…nal (ROCm#200) aicomnet-305/sgl-dev-gfx950-rdma63 -> mad-rccl. One commit, one file: docker/sglang_disagg_inference_full_overlay.ubuntu.amd.Dockerfile. The problem None of the KV-transfer backends could run 2P/2D DeepSeek-R1 on the Broadcom Thor2 (bnxt_re) fabric, and each failed in its own way, which made it look like three unrelated bugs: mooncake logged 2604 Failed to create QP over a single run NIXL never started: UCX creates one interface queue pair while opening rc_verbs/bnxt_re1, and treats the failure as fatal MoRI built queue pairs that reached RTS, then moved zero bytes and aborted every request All three are the same defect. The bnxt_re provider in rdma-core 39.0/50.0 corrupts the verbs command buffer for work issued off worker threads, so the kernel answers ibv_create_qp with EFAULT (errno 14). Isolated on one node, in one container, on the same kernel, with 8 threads x 64 queue pairs and only the userspace changing: rdma-core queue pairs created 39.0 (distro) 33-66 of 512 63.0 (source) 512 of 512 What changed rdma-core. The stage that fixes this was already in this file as ENABLE_RDMA62. It worked, but it was off by default and its comment said only "hosts whose RDMA stack needs a newer libibverbs", so nothing connected it to Thor2 and it was never switched on. It is now RDMA_CORE_VERSION (63.0, on by default; empty keeps the base image's), documented against the measurement above, and it additionally: removes the distro rdma-core packages, so one libibverbs is left on disk instead of two of different vintages; drops the vendor bnxt_re provider that /etc/ld.so.conf.d puts on the loader path. It advertises kernel uABI 7-8, so on a host running the upstream driver (uABI 1) every device is rejected with Driver bnxt_re does not support the kernel ABI of 1, after which RCCL finds no IB plugin at all. Ordering is load-bearing and is called out in the file: the source build replaces the distro packages with dpkg -r --force-all, which leaves still-installed dependents (libucx0, openmpi, ...) with dangling deps, so every apt operation has to run before the removal. That is why the stage is last. Base image and stage gating. The default base becomes rocm/sgl-dev:v0.5.16-rocm720-mi35x-20260807, which already carries sglang, aiter, sgl-kernel, MoRI, RIXL and mooncake built for gfx950. Rebuilding those on top of it is both slow and a regression risk, so each component stage is now gated and defaults to off: the image is the base plus rdma-core, and each ENABLE_* arg turns one component into an A/B against what the base ships. Mooncake. Moves from the 0.3.6.post1 wheel to a source build of v0.3.12.post1 with ENABLE_MULTI_PROTOCOL=ON. Before upstream 45b84d3 (#2725) there is no cross-host guard, so a GPU address registered under both rdma and hip keeps hip's priority for any target: every cross-node KV chunk is pushed through the intra-node-only HIP transport and dies in hipIpcOpenMemHandle. Because the guard sits behind a CMake option that defaults to OFF, a build without it produces the same broken routing silently, so the stage greps the installed package for MC_DISABLE_HIP and fails the build if the guard did not land. Compatibility Existing users of this file get a different default image, so this is a behaviour change. The old configuration is still reachable in full: --build-arg BASE_DOCKER=lmsysorg/sglang:v0.5.12.post1-rocm720-mi30x \ --build-arg BUILD_GPU_TARGETS=gfx942 --build-arg MORI_GPU_ARCHS=gfx942 \ --build-arg ENABLE_RCCL_OVERLAY=1 --build-arg ENABLE_MORI_OVERLAY=1 \ --build-arg ENABLE_NIXL_OVERLAY=1 --build-arg ENABLE_MOONCAKE_OVERLAY=1 Hosts that do not want the new rdma-core can pass --build-arg RDMA_CORE_VERSION= to keep the base image's.
…pport Registers two MAD models for GPT-OSS-120B and wires them up for multi-node SLURM runs: - primus_pyt_megatron_lm_train_gpt-oss-120b_overlay: Megatron-LM pretraining on the RCCL overlay image, BF16 and FP8, at TP1 x PP2 x VP2 x EP8 (16 GPUs, so a 2-node run saturates 2x8). Upstream Primus ships only an MI355X config for this model, so gfx942 reuses it and gets the two values that actually differ as primus-cli overrides: micro_batch_size 2 (BF16) / 1 (FP8) and recompute_num_layers 9. The MI355X values (mbs 8, recompute 4) are tuned for 288GB HBM and OOM on 192GB, in the MoE grouped GEMM and in the last-stage fp32 cross-entropy logits buffer. - sglang-disagg-gpt-oss-120b-overlay: xPyD disaggregated serving with each prefill/decode node an independent TP=8 replica, no cross-node EP. The mxfp4 MoE layer is forced onto the Triton runner because the aiter CK MXFP4 GEMM is not compiled for gfx942, and a build-time patch teaches sglang's mxfp_supported() that gfx942 is MXFP4-capable (mirrors upstream PR 13929). Also adds run-manifest templates for both models to the mad-slurm-multinode skill. Verified on a 2-node gfx942 cluster with rocm/primus:v26.4: BF16 and FP8 both complete 10/10 iterations (203 TFLOP/s/GPU in BF16); sglang-disagg 2P2D across 4 nodes serves end to end. Co-authored-by: Cursor <cursoragent@cursor.com>
…a-3.1-70B and GPT-OSS on gfx950 Llama-3.1-70B-Instruct was servable by nothing here: the launcher aborted with "Model not found in models.yaml" and weight staging had no MODEL_NAME -> HF repo entry for it. Both are filled in, with the flags of the other 70B card (mem-fraction 0.65; the 0.73 of the smaller models does not fit). GPT-OSS on MI355X needed two kernel choices that only make sense as a pair. models.yaml already pinned --moe-runner-backend triton, but with aiter enabled sglang's mxfp4 layer builds an AiterMoeQuantInfo and hands it to the triton runner, so the first prefill forward died on a missing use_mxfp8 attribute (job 25796); SGLANG_USE_AITER=0 now sits next to the flag it belongs to, and an explicit value in the environment still wins. Attention moves to triton for the same class of reason: aiter's CK dispatch has no gfx950 kernel for the 18M-page KV pool a 288 GB card produces at page_size=1, and it died in prefill on MI355X while the same card ran on MI300X. The banner prints SGLANG_USE_AITER, since getting it wrong surfaces as an AttributeError deep in a traceback rather than as a config error. The gfx950 image stops rebuilding kernels. Measured on the base images, the mi30x sgl_kernel carries no gfx950 code at all while the same-version mi35x image carries gfx950 and no gfx942, so the rebuild was compensating for the wrong base; rocm/sgl-dev's mi35x tags already ship mori, nixl, mooncake and a librccl with gfx950 objects. What such an image still gets wrong is RDMA userspace: it puts Broadcom's out-of-tree bnxt_re provider on the loader path, which accepts only kernel uABI 7-8, so on an upstream-uABI host libibverbs rejects every device and RCCL dies in ncclCommInitRank. BNXT_RE_PROVIDER selects the upstream provider already present in the image. Profiling had no hook in these scripts, so a PD run could not be measured without editing them by hand. PROFILE_ENABLE=1 now drives the whole measurement configuration: benchmark_xPyD.sh runs one bench_serving profile point per role (sglang cannot profile both at once), writes to the shared /run_logs mount, keeps the profiled request stream out of the perf CSV, and runs after parse_to_csv.py so a profiler failure cannot cost the sweep. Decode drops HIP graphs, since a replayed graph dispatches one packet and hides every collective inside it -- rocprofv3 does not even survive the capture. Both roles get --disable-custom-all-reduce, because sglang's own all-reduce never enters torch.distributed and left a profile showing the startup barrier alone. Both flags live in PREFILL_MODEL_CONFIG / DECODE_MODEL_CONFIG rather than at a launch site: node 0 assembles its prefill command separately, so a flag added per launch site profiled three nodes out of four (job 25824). Servers are stopped with SIGINT and waited for, with a wider budget while profiling, because a bare kill let madengine tear the container down while rocprofv3 was still writing and left an empty output directory (job 25815). All of this costs performance, so throughput comes from a run without PROFILE_ENABLE and the communication profile from one with it. With PROFILE_ENABLE unset nothing new runs. The mad-slurm-multinode validator gains the two checks these runs taught it: "N/A (local image mode)" in built_images.dockerfile is a madengine mode rather than a missing file (and requires local_image: true), and a variable set only in deployment_config.env_vars never reaches the container, which is how job 25802 started with aiter on despite declaring SGLANG_USE_AITER=0. Co-authored-by: Cursor <cursoragent@cursor.com>
…-agnostic The post-processing side of the profiling effort was one 1264-line script with no tests, two hardcoded regen scripts, and engine knowledge in seven places. Four of those failed silently rather than loudly: report prose branched on `phase.dtype in SGLANG_ROLES`, so any engine that named its phases prefill and decode inherited sglang's claims about mooncake RDMA and --disable-custom-all-reduce -- output that was wrong and looked right. It becomes a skill next to mad-slurm-multinode, split along one line: scripts/collprof/core/ parses artifacts and composes reports and names no engine, while scripts/collprof/engines/ <engine>.py declares one EngineSpec -- log layout, phase source, log metrics, trace layout and resolution, sanity bounds, and the ReportNotes prose its reports print. Adding an engine is that module plus a registry line; a change that seems to need core/ means a missing spec field, added with a behaviour-preserving default. A test pins the invariant. Campaigns move from two shell scripts into a JSON catalog run by scripts/regen_reports.py, so a job is an entry rather than an edited script and one failing job does not stop the rest. Reports record the command and parser version that produced them, break every rejected record down by reason, and state when a sanity bound was hit and which flag raises it. 77 pytest cases on synthetic fixtures run in two seconds without a cluster. Three things the real artifacts settled. Trace-to-role mapping is read from the profile-point log's output_dir rather than from timestamps -- the container clock and the shared filesystem's differ by about 460 seconds here, and the hand-written scripts had been dropping the second trace directory of every role. Rank-local (nranks=1) communicators are records, not damage: they print `stream (nil)`, and rejecting them discarded 17200 per node per training phase and made the report's single-rank section unreachable. An empty capture directory is a named warning rather than a FileNotFoundError that costs a whole rebuild, because a role's idle replica captures nothing. The measurement side ships as manifest overlays under assets/manifest-overlay/ -- the context.tools block, NCCL_DEBUG_SUBSYS with COLL in both env blocks, trace directories and mounts, and the Primus experiment-YAML profiler keys -- applied to a mad-slurm-multinode manifest with jq and checked by that skill's validate_manifest.sh. SKILL.md pins the interpreter and installs openpyxl before parsing, since the scripts otherwise inherit whatever python3 is on PATH and a campaign once lost every workbook to that. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a new mad-profiling skill (docs + tooling + tests) to turn profiled MAD runs into collective-communication reports, and wires in supporting runtime/manifest/docker changes so SGLang disaggregated serving and Primus/Megatron-LM runs can produce measurable artifacts (RCCL logs, torch profiler traces, optional rocprofv3 stats). It also introduces GPT-OSS-120B-related templates/config updates to support profiling and reporting workflows across these engines.
Changes:
- Introduces the
mad-profilingskill with report-generation scripts (collective_report.py,regen_reports.py), engine specs (Primus + sglang-disagg), and a pytest suite to lock in parsing/report semantics. - Adds/updates profiling-oriented runtime behavior in SGLang disagg scripts (measurement flags, profiler point runner, graceful shutdown to preserve artifacts).
- Adds new/updated templates and documentation for manifests and docker overlays, including GPT-OSS-120B examples and cluster/runtime notes.
Reviewed changes
Copilot reviewed 52 out of 53 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/sglang_disagg/sglang_disagg_mori_io_ep.sh | Adds profiling measurement flags and graceful server shutdown helper to preserve profiler outputs. |
| scripts/sglang_disagg/run.sh | Adds GPT-OSS guardrails and additional model repo mappings; logs SGLANG_USE_AITER. |
| scripts/sglang_disagg/run_xPyD_models.slurm | Adds GPT-OSS-120B to the valid model lists. |
| scripts/sglang_disagg/models.yaml | Adds model presets for Llama-3.1-70B-Instruct and GPT-OSS-120B (with detailed ROCm/aiter notes). |
| scripts/sglang_disagg/benchmark_xPyD.sh | Adds optional torch-profiler capture flow and role-specific profiling points. |
| scripts/primus_megatron-lm/run.sh | Enables GPT-OSS-120B on MI300X/MI325X with datatype selection. |
| scripts/primus_megatron-lm/primus_megatron-lm_benchmark_report.sh | Adds MI355X-config reuse + memory overrides for GPT-OSS-120B on gfx942 devices. |
| docker/sglang_disagg_inference_sgl_dev.ubuntu.amd.Dockerfile | New “thin overlay” Dockerfile for sgl-dev images + bnxt_re provider selection. |
| docker/sglang_disagg_inference_full_overlay.ubuntu.amd.Dockerfile | Adds an MXFP4-on-gfx942 patch step for sglang detection logic. |
| .gitignore | Whitelists mad-profiling JSON assets; ignores Python cache directories. |
| .claude/skills/mad-slurm-multinode/scripts/validate_manifest.sh | Adds env-var placement warning and “local image mode” dockerfile validation logic. |
| .claude/skills/mad-slurm-multinode/references/manifests.md | Documents Primus + sglang-disagg GPT-OSS-120B manifest expectations and gotchas. |
| .claude/skills/mad-slurm-multinode/assets/manifests/sglang_disagg_gpt-oss-120b.template.json | New GPT-OSS-120B sglang-disagg manifest template with profiling-related env/mount guidance. |
| .claude/skills/mad-slurm-multinode/assets/manifests/primus_gpt-oss-120b.template.json | New Primus GPT-OSS-120B manifest template. |
| .claude/skills/mad-profiling/SKILL.md | New skill documentation: workflow, responsibilities, measurement setup, reporting and interpretation guidance. |
| .claude/skills/mad-profiling/scripts/tests/test_torch_trace.py | Tests for trace parsing + trace-to-phase mapping. |
| .claude/skills/mad-profiling/scripts/tests/test_runner.py | Tests for catalog runner behavior (regen_reports.py). |
| .claude/skills/mad-profiling/scripts/tests/test_report.py | End-to-end report generation tests with engine isolation guarantees. |
| .claude/skills/mad-profiling/scripts/tests/test_rccl_log.py | Tests for RCCL log discovery/parsing and damage detection. |
| .claude/skills/mad-profiling/scripts/tests/test_phase_and_cache.py | Tests for Phase aggregations and parse-cache behavior. |
| .claude/skills/mad-profiling/scripts/tests/test_engines.py | Tests for engine detection, ambiguity handling, and registry contract. |
| .claude/skills/mad-profiling/scripts/tests/conftest.py | Synthetic artifact fixtures for logs/traces and minimal “runs”. |
| .claude/skills/mad-profiling/scripts/setup.cfg | Adds flake8 + pytest configuration for the skill scripts. |
| .claude/skills/mad-profiling/scripts/regen_reports.py | New catalog-driven campaign rebuild script. |
| .claude/skills/mad-profiling/scripts/collprof/engines/sglang_disagg.py | Engine spec + trace resolution for sglang PD-disagg runs. |
| .claude/skills/mad-profiling/scripts/collprof/engines/primus.py | Engine spec + trace resolution for Primus/Megatron training runs. |
| .claude/skills/mad-profiling/scripts/collprof/engines/init.py | Engine registry and detection logic. |
| .claude/skills/mad-profiling/scripts/collprof/core/workbook.py | Workbook emitter (profile.xlsx) that composes markdown + CSVs. |
| .claude/skills/mad-profiling/scripts/collprof/core/units.py | Shared datatype/size utilities and formatting helpers. |
| .claude/skills/mad-profiling/scripts/collprof/core/torch_trace.py | Fast line-scan trace parser for collective sizes/process groups. |
| .claude/skills/mad-profiling/scripts/collprof/core/spec.py | EngineSpec contract between core and per-engine modules. |
| .claude/skills/mad-profiling/scripts/collprof/core/rocprof.py | rocprofv3 stats aggregation for durations. |
| .claude/skills/mad-profiling/scripts/collprof/core/report.py | Markdown/CSV report composition, normalization, and workbook integration. |
| .claude/skills/mad-profiling/scripts/collprof/core/rccl_log.py | RCCL log parsing with tear/damage detection and engine-metric harvesting. |
| .claude/skills/mad-profiling/scripts/collprof/core/phase.py | Phase data model and aggregations (per-rank normalization, totals, cache state). |
| .claude/skills/mad-profiling/scripts/collprof/core/cache.py | Parse cache keyed by file identity + parse version. |
| .claude/skills/mad-profiling/scripts/collprof/core/init.py | Core package init and invariants statement. |
| .claude/skills/mad-profiling/scripts/collprof/cli.py | CLI for one-job reporting, trace resolution, cache usage, and per-phase emit. |
| .claude/skills/mad-profiling/scripts/collprof/init.py | Package overview and layering boundary documentation. |
| .claude/skills/mad-profiling/scripts/collective_report.py | Entry point wrapper for the one-job report generator. |
| .claude/skills/mad-profiling/reports_template/run_findings.template.md | Template for campaign findings write-ups. |
| .claude/skills/mad-profiling/reports_template/README.md | Template usage documentation. |
| .claude/skills/mad-profiling/reports_template/model_comparison.template.md | Template for cross-run/model comparisons with required scope notes. |
| .claude/skills/mad-profiling/references/measurement-setup.md | Reference on how to make runs measurable and what artifacts mean. |
| .claude/skills/mad-profiling/references/interpretation.md | Reference on how to interpret reports and avoid invalid conclusions. |
| .claude/skills/mad-profiling/references/engines.md | Guide for adding new engines without touching core/. |
| .claude/skills/mad-profiling/references/data-quality.md | Torn-record detection rationale and how sanity bounds work. |
| .claude/skills/mad-profiling/assets/manifest-overlay/sglang-disagg.overlay.json | Overlay JSON to make sglang-disagg runs measurable. |
| .claude/skills/mad-profiling/assets/manifest-overlay/README.md | How to apply overlays and what each addition enables. |
| .claude/skills/mad-profiling/assets/manifest-overlay/primus-megatron.overlay.json | Overlay JSON to make Primus/Megatron runs measurable. |
| .claude/skills/mad-profiling/assets/manifest-overlay/primus-exp-yaml.profiler.yaml | Experiment YAML fragment enabling torch profiler for Primus/Megatron. |
| .claude/skills/mad-profiling/assets/jobs.example.json | Example campaign catalog for batch report regeneration. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+19
to
+21
| "dockerfile": "docker/primus_megatron_train_rccl_overlay.ubuntu.amd", | ||
| "dockercontext": "./docker", | ||
| "scripts": "scripts/primus/megatron-lm/run.sh", |
Comment on lines
+987
to
994
| export EXP=examples/megatron/configs/$GPT_OSS_CONFIG_DEVICE/gpt_oss_120B-$DATATYPE-pretrain.yaml | ||
| if [[ ! -f "$EXP" ]]; then | ||
| echo "Error: Config file not found: $EXP" | ||
| echo "Hint: add gpt_oss_120B-$DATATYPE-pretrain.yaml for $GPT_OSS_CONFIG_DEVICE in Primus configs." | ||
| else | ||
| MBS=$(grep -E '^\s*micro_batch_size:' $EXP | head -n1 | awk '{print $2}' | tr -d '\r') | ||
| GBS=$(grep -E '^\s*global_batch_size:' $EXP | head -n1 | awk '{print $2}' | tr -d '\r') | ||
| echo "[INFO] Extracted MBS=$MBS, GBS=$GBS from config: $EXP" |
Comment on lines
+199
to
+203
| lines = ["", "## Summary", "", | ||
| f"- Collectives parsed (nranks>1): **{view.grand_calls}** over {view.reps} ranks " | ||
| f"= {view.grand_calls // view.reps} per rank" | ||
| + (f", leaving out {view.idle} rank(s) that carried under " | ||
| f"{spec.limits.idle_rank_fraction:.0%} of the busiest rank" if view.idle else ""), |
Comment on lines
+48
to
+50
| merged = {**defaults, **entry} | ||
| name = entry.get("name") or entry["run_dir"] | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces the initial documentation, configuration templates, and example catalogs for the
mad-profilingskill. This skill provides comprehensive support for profiling collective communication in distributed training and inference workloads, focusing on configuring measurable runs and generating detailed communication reports. The changes include extensive documentation, manifest overlays for profiling, a sample experiment YAML for enabling the torch profiler, and a campaign catalog example.Documentation and Usage Instructions:
SKILL.mddescribing the scope, workflow, responsibilities, required inputs, and step-by-step instructions for using themad-profilingskill, including how to configure runs for profiling, generate reports, and interpret results.Configuration Templates and Overlays:
primus-megatron.overlay.json) specifying required environment variables, tools, and mount points to enable profiling and artifact collection.primus-exp-yaml.profiler.yaml) for enabling the torch profiler in Primus/Megatron-LM runs, with instructions and rationale for each field.Reporting Campaign Example:
jobs.example.json) demonstrating how to specify multiple profiling jobs, their directories, phases, and notes for batch report generation.