Sync mad-rccl with develop (resolves models.json migration conflict from #210) by i-kosarev · Pull Request #211 · ROCm/MAD

i-kosarev · 2026-08-18T15:04:45Z

Supersedes the conflicting merge in #210.

Why #210 conflicts

Both branches carry the same cherry-picked commits (#195, #196, #198, #201, #203) under different SHAs, so the only real divergence is:

develop: Migrated models from global to per directory #197 "Migrated models from global to per directory" — the root models.json was split into scripts/<dir>/models.json.
mad-rccl: Add multi-node scaleout benchmarks (Primus Megatron-LM, SGLang disaggregated) and the mad-slurm-multinode skill #194 (multi-node scaleout benchmarks + mad-slurm-multinode skill) and sglang-disagg: fix queue-pair creation on Broadcom Thor2 (rdma-core 63.0), and make every overlay stage optional #200 (sglang-disagg QP fix / optional overlay stages), which kept adding entries to the root models.json.

Hence a single conflicted file: models.json.

How it is resolved

Resolution follows develop's migration rather than reverting it:

root models.json takes develop's post-migration content — only primus_train, which must stay at the root because scripts/primus_train/ has a get_models_json.py (per the discovery rule documented in tools/migrate_to_dir_models.py);
the four entries that existed only on mad-rccl were migrated into per-directory files using that tool's own translation rules (dockerfile → ../../docker/..., scripts → path relative to the script dir), so they match the convention of the already-migrated entries:
- scripts/primus_megatron-lm/models.json (new file) — primus_pyt_megatron_lm_train_llama-3.1-{8b,70b,405b}_overlay;
- scripts/sglang_disagg/models.json — sglang-disagg-deepseek-r1-overlay appended to develop's 16 entries.

Verification

Union of model names across all models.json files is exactly the union of both branches (137) — nothing dropped from either side (checked programmatically both ways).
Every models.json parses; no conflict markers remain.
scripts paths of the migrated entries resolve to existing files; both overlay Dockerfiles exist (docker/primus_megatron_train_rccl_overlay.ubuntu.amd.Dockerfile, docker/sglang_disagg_inference_full_overlay.ubuntu.amd.Dockerfile).

Test plan

madengine discovers the three primus_pyt_megatron_lm_train_llama-3.1-*_overlay models from scripts/primus_megatron-lm/models.json
madengine discovers sglang-disagg-deepseek-r1-overlay from scripts/sglang_disagg/models.json
existing develop-migrated models still resolve unchanged

Made with Cursor

Signed-off-by: Rohan138 <rohanpotdar138@gmail.com>

* pytorch-xdit:v26.7 release Purpose is to add pytorch-xdit:v26.7 image to MAD workflows and to publish it. Work here contains Update base image to pytorch-xdit:v26.7 Replace ltx_2 with ltx_2_3 which checked should be complete. * update docker

* Fix Minimax gfx942 Signed-off-by: Rohan138 <rohanpotdar138@gmail.com> * Fix Minimax gfx942 again; add breakable cudagraph for DSV4 Signed-off-by: Rohan138 <rohanpotdar138@gmail.com> --------- Signed-off-by: Rohan138 <rohanpotdar138@gmail.com>

Signed-off-by: Andy Friedrich <afriedri@amd.com>

Signed-off-by: Rohan138 <rohanpotdar138@gmail.com> Co-authored-by: Rohan138 <rohanpotdar138@gmail.com>

* Migrated models from global to per directory This migration moves model definitions from the root models.json file to per-directory models.json files within each script directory. Changes: - Migrated 132 models to 20 per-directory models.json files - Kept 1 model in root models.json (primus_train has get_models_json.py) - Added tools/migrate_to_dir_models.py migration script Path translations applied: - dockerfile: docker/X -> ../../docker/X - scripts: scripts/<dir>/file.sh -> file.sh - dockercontext: kept as-is (project-root-relative) * add back recent changes to models.json that were missed --------- Co-authored-by: Lewis <leconcio@amd.com>

develop (ROCm#197) migrated the root models.json into per-directory scripts/<dir>/models.json files, while mad-rccl kept adding entries to the root file (ROCm#194 multi-node scaleout, ROCm#200 sglang-disagg overlay), so the two histories conflicted on models.json. Resolution follows develop's migration: - root models.json takes develop's post-migration content (only primus_train, which must stay because scripts/primus_train has get_models_json.py); - the four entries that existed solely on mad-rccl are migrated into per-directory files using tools/migrate_to_dir_models.py's own translation rules, so paths match the convention used by the migrated entries: * scripts/primus_megatron-lm/models.json (new) — the three primus_pyt_megatron_lm_train_llama-3.1-{8b,70b,405b}_overlay entries; * scripts/sglang_disagg/models.json — sglang-disagg-deepseek-r1-overlay. Verified: the union of model names across all models.json files is exactly the union of both branches (137), nothing dropped from either side; every file parses; scripts paths resolve and both overlay Dockerfiles exist. Co-authored-by: Cursor <cursoragent@cursor.com>

Copilot

Pull request overview

This PR resolves the models.json merge conflict between mad-rccl and develop by adopting develop’s per-directory models.json migration and migrating the remaining mad-rccl-only model entries into the appropriate scripts/<dir>/models.json files (keeping only primus_train at the repo root due to get_models_json.py discovery).

Changes:

Added/updated tooling (tools/migrate_to_dir_models.py) to migrate root models.json entries into scripts/<dir>/models.json with path translation rules.
Moved model definitions from the root models.json into per-script models.json files across many scripts/* directories.
Preserved primus_train as the only entry in the root models.json to avoid conflicting with scripts/primus_train/get_models_json.py.

Reviewed changes

Copilot reviewed 23 out of 23 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tools/migrate_to_dir_models.py	Adds a migration utility to split root `models.json` into per-directory manifests with path translation.
models.json	Shrinks root manifest to only `primus_train` post-migration.
scripts/atom/models.json	Adds per-directory model manifest for the `atom` workload.
scripts/dummy/models.json	Adds per-directory model manifest for dummy workload(s).
scripts/huggingface_bert/models.json	Adds per-directory model manifest for HuggingFace BERT script.
scripts/huggingface_gpt2/models.json	Adds per-directory model manifest for HuggingFace GPT-2 script.
scripts/jax-maxtext/models.json	Adds per-directory model manifest for JAX MaxText workloads.
scripts/large-ep-benchmark/models.json	Adds per-directory model manifest for large-EP benchmark workloads.
scripts/mochi/models.json	Adds per-directory model manifest for Mochi inference workload.
scripts/ncf/models.json	Adds per-directory model manifest for NCF training workload.
scripts/primus_megatron-lm/models.json	Adds per-directory model manifest for Primus Megatron-LM scaleout training overlay entries.
scripts/pyt_chai1_inference/models.json	Adds per-directory model manifest for Chai-1 inference workload.
scripts/pyt_clip_inference/models.json	Adds per-directory model manifest for CLIP inference workload.
scripts/pyt_huggingface_diffusers/models.json	Adds per-directory model manifest for HuggingFace Diffusers finetuning workload.
scripts/pyt_hy_video/models.json	Adds per-directory model manifest for Hunyuan video workload.
scripts/pyt_janus_pro_inference/models.json	Adds per-directory model manifest for Janus Pro inference workload.
scripts/pyt_wan2.1_inference/models.json	Adds per-directory model manifest for Wan 2.1 inference workload.
scripts/pyt_xdit/models.json	Adds per-directory model manifest for xDiT workloads.
scripts/pytorch_train/models.json	Adds per-directory model manifest for PyTorch training workloads.
scripts/sglang/models.json	Adds per-directory model manifest for SGLang workloads.
scripts/sglang_disagg/models.json	Adds per-directory model manifest for SGLang disaggregated workloads and appends the overlay entry.
scripts/vllm/models.json	Adds per-directory model manifest for vLLM workloads (large list migrated).
scripts/vllm_dissag/models.json	Adds per-directory model manifest for vLLM disaggregated P/D workloads.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

-    "training_precision": "",
-    "tags": [
-      "pyt",
-      "large_ep",
-      "deepep",
-      "mori_ep",
-      "benchmark"
-    ],
-    "timeout": -1,
-    "distributed": {
-      "launcher": "slurm_multi"
-    },
-    "env_vars": {
-      "DOCKER_IMAGE": "rocm/pytorch-private:large-ep-benchmark-rocm720-mori-42e89547-20260605",
-      "DOCKER_IMAGE_NAME": "<supply-your-image>",
-      "IBDEVICES": "mlx5_0",
-      "SKIP_DEEPEP": "0"
-    },
-    "args": "-N 2 -n 2"
-  },
-  {
-    "name": "pyt_large_ep_bench_2n_mori_only",
-    "url": "",
-    "dockerfile": "docker/large_ep_benchmark",
-    "scripts": "scripts/large-ep-benchmark/run_benchmark.sbatch",
-    "data": "",
-    "n_gpus": "-1",
-    "owner": "mad.support@amd.com",
-    "training_precision": "",
-    "tags": [
-      "pyt",
-      "large_ep",
-      "large_ep_scaling",
-      "mori_ep",
-      "benchmark"
-    ],
-    "timeout": -1,
-    "distributed": {
-      "launcher": "slurm_multi"
-    },
-    "env_vars": {
-      "DOCKER_IMAGE": "rocm/pytorch-private:large-ep-benchmark-rocm720-mori-42e89547-20260605",
-      "DOCKER_IMAGE_NAME": "<supply-your-image>",
-      "IBDEVICES": "mlx5_0",
-      "SKIP_DEEPEP": "1"
-    },
-    "args": "-N 2 -n 2"
-  },
-  {
-    "name": "pyt_large_ep_bench_4n_mori_only",
-    "url": "",
-    "dockerfile": "docker/large_ep_benchmark",
-    "scripts": "scripts/large-ep-benchmark/run_benchmark.sbatch",
-    "data": "",
-    "n_gpus": "-1",
-    "owner": "mad.support@amd.com",
-    "training_precision": "",
-    "tags": [
-      "pyt",
-      "large_ep",
-      "large_ep_scaling",
-      "mori_ep",
-      "benchmark"
-    ],
-    "timeout": -1,
-    "distributed": {
-      "launcher": "slurm_multi"
-    },
-    "env_vars": {
-      "DOCKER_IMAGE": "rocm/pytorch-private:large-ep-benchmark-rocm720-mori-42e89547-20260605",
-      "DOCKER_IMAGE_NAME": "<supply-your-image>",
-      "IBDEVICES": "mlx5_0",
-      "SKIP_DEEPEP": "1"
-    },
-    "args": "-N 4 -n 4"
-  },
-  {
-    "name": "primus_train",
-    "dockerfile": "docker/primus",
-    "dockercontext": ".",
-    "scripts": "scripts/primus_train",
-    "n_gpus": "-1",
-    "owner": "mad.support@amd.com",
-    "training_precision": "",
-    "tags": [
-      "training",
-      "primus",
-      "megatron",
-      "pretrain"
-    ],
-    "args": "",
-    "env": {
-      "HSA_NO_SCRATCH_RECLAIM": "1",
-      "PRIMUS_TURBO_ATTN_V3_ATOMIC_FP32": "1",
-      "NVTE_CK_IS_V3_ATOMIC_FP32": "1"
+    {
+        "name": "primus_train",
+        "dockerfile": "docker/primus",
+        "dockercontext": ".",
+        "scripts": "scripts/primus_train",


+        "timeout": -1,
+        "args": "--model_repo microsoft/phi-4 --test_option latency --num_gpu 1 --datatype float16"


mkuznet1

lgtm

Sync mad-rccl with develop (resolves models.json migration conflict).

Rohan138 and others added 7 commits August 6, 2026 14:30

Fix Minimax gfx942 (ROCm#196)

672f362

Signed-off-by: Rohan138 <rohanpotdar138@gmail.com>

pytorch-xdit:v26.7 release (ROCm#195)

3efcb2d

* pytorch-xdit:v26.7 release Purpose is to add pytorch-xdit:v26.7 image to MAD workflows and to publish it. Work here contains Update base image to pytorch-xdit:v26.7 Replace ltx_2 with ltx_2_3 which checked should be complete. * update docker

Fix Minimax M3 gfx942 (reland) (ROCm#198)

34fbced

* Fix Minimax gfx942 Signed-off-by: Rohan138 <rohanpotdar138@gmail.com> * Fix Minimax gfx942 again; add breakable cudagraph for DSV4 Signed-off-by: Rohan138 <rohanpotdar138@gmail.com> --------- Signed-off-by: Rohan138 <rohanpotdar138@gmail.com>

MI300 Minimax TP constraint (ROCm#201)

bb9ff32

Signed-off-by: Andy Friedrich <afriedri@amd.com>

drop tp8 from minimax m2.7 (ROCm#203)

c38612f

Signed-off-by: Rohan138 <rohanpotdar138@gmail.com> Co-authored-by: Rohan138 <rohanpotdar138@gmail.com>

Copilot AI lite review requested due to automatic review settings August 18, 2026 15:04

i-kosarev requested review from Rohan138, amathews-amd, coketaste, gargrahul and ppalaniappan-amd as code owners August 18, 2026 15:04

Copilot started reviewing on behalf of i-kosarev August 18, 2026 15:05 View session

Copilot AI reviewed Aug 18, 2026

View reviewed changes

mkuznet1 approved these changes Aug 18, 2026

View reviewed changes

i-kosarev merged commit 57a8d7a into ROCm:mad-rccl Aug 18, 2026

i-kosarev deleted the sync/mad-rccl-develop-resolve branch August 18, 2026 15:57

gargrahul pushed a commit that referenced this pull request Aug 18, 2026

Merge pull request #211 from i-kosarev/sync/mad-rccl-develop-resolve

c117e9c

Sync mad-rccl with develop (resolves models.json migration conflict).

Navigation Menu

Uh oh!

Sync mad-rccl with develop (resolves models.json migration conflict from #210) - #211

Sync mad-rccl with develop (resolves models.json migration conflict from #210)#211
i-kosarev merged 7 commits into
ROCm:mad-rcclfrom
i-kosarev:sync/mad-rccl-develop-resolve

i-kosarev commented Aug 18, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

mkuznet1 left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

		"timeout": -1,
		"args": "--model_repo microsoft/phi-4 --test_option latency --num_gpu 1 --datatype float16"

Uh oh!

Conversation

i-kosarev commented Aug 18, 2026

Why #210 conflicts

How it is resolved

Verification

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

mkuznet1 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants