Use configure_pytorch_release_matrix.py to drive the CI job matrix by ScottTodd · Pull Request #6082 · ROCm/TheRock

ScottTodd · 2026-06-24T00:19:14Z

Motivation

Fixes #6030. On #5744 we added gfx1250 support to ROCm but support is not yet there in PyTorch across all supported release branches, so we filtered it from the GPU targets list for release workflows. We did not have a way to filter for CI workflows, so PRs that used the ci:run-all-archs label like bump PRs (example: #6020) were failing to build pytorch.

Now we'll get the same filtering across CI and releases.

Technical Details

This is like #5942 which added build_tools/github_actions/configure_rocm_python_test_matrix.py. Dynamically generating a job matrix using a script will give us greater flexibility (finer-grained inclusions/exclusions, opt-in to expand the matrix, etc.), at the cost of some code complexity and indirection.

Note

CI workflows use workflow_call to run pytorch jobs while release workflows use workflow_dispatch. This generates a new matrix for CI but continues to use the same workflow_dispatch inputs for releases, to keep the workflows easy to dispatch directly (otherwise you'd need to type out a JSON matrix to trigger). There is potential there for the generated/used matrix to be inconsistent across that boundary.

Test Plan

updated unit tests
multi_arch_ci.yml run using prebuilt artifacts: https://github.com/ROCm/TheRock/actions/runs/28060884840/job/83076620803
multi_arch_ci_asan.yml run using prebuilt artifacts: https://github.com/ROCm/TheRock/actions/runs/28202142368
multi_arch_release.yml run: https://github.com/ROCm/TheRock/actions/runs/28187486907 --> triggered https://github.com/ROCm/TheRock/actions/runs/28196069145
multi_arch_ci.yml on this PR with the opt-in gfx125x label: https://github.com/ROCm/TheRock/actions/runs/28200714865?pr=6082

Test Result

gfx125X-dcgpu / gfx125x is filtered from pytorch builds except for the release/2.11 branch
a "build-pytorch" section is added to the "Multi-Arch CI Configuration" summary, listing which matrix jobs are going to be added to the build graph

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

Move PyTorch version, Python version, and PyTorch-specific family filtering into configure_pytorch_release_matrix.py, then thread the resulting matrix through multi-arch CI and release workflows. Tests: D:/projects/TheRock/.venv/Scripts/python.exe -m pytest github_actions/tests/configure_pytorch_release_matrix_test.py github_actions/tests/configure_multi_arch_ci_test.py Tests: pre-commit run --files <changed files> Assisted-by: Codex

Replace the per-ref dict configuration with explicit release/CI ref lists and an unsupported-family map. Remove the unused matrix wrapper so callers use a single entry point with defaulted optional overrides. Testing: - D:/projects/TheRock/.venv/Scripts/python.exe -m pytest github_actions/tests/configure_pytorch_release_matrix_test.py github_actions/tests/configure_multi_arch_ci_test.py - pre-commit run --files build_tools/github_actions/configure_pytorch_release_matrix.py build_tools/github_actions/configure_multi_arch_ci.py build_tools/github_actions/tests/configure_pytorch_release_matrix_test.py Assisted-by: Codex

…orch-shared-matrix

therock-pr-bot · 2026-06-24T00:19:52Z

❌ PR Check — Action Required

Check	Status	Details
🌿 Branch Name	✅ Pass	—
📝 PR Title/Description	❌ Fail	Error: Title does not follow Conventional Commits style. Expected: start with a valid type (feat, fix, docs, …). Desired format: `type(optional-scope): short description` ─── Error: PR description must reference a JIRA ID or ISSUE ID. Expected: include a `JIRA ID` or `ISSUE ID` line. The separator may be `:` or `-` (or omitted), and the value can be a JIRA key, a number (with or without `#`), or a link. Accepted examples: • `JIRA ID : TESTAUTO-6039` • `JIRA ID -` #330 • `JIRA ID` #330 • `ISSUE ID : TESTUTO-3334` • `ISSUE ID` #3334 • `ISSUE ID - TESTAUTO-3433` • `ISSUE ID : https://github.com/<org_name>/<repo_name>/issues/1234` Current: no valid JIRA/ISSUE reference found
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	—
🔎 pre-commit	⏳ Pending	⏳ Still running…
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

⚠️ 1 policy check(s) failed. Please address the issues above before this PR can be Reviewed.

🚫 Please fix the failed policies

❌ PR Title/Description

The Not ready to Review label was added to this PR. Once all policies pass, the label is removed automatically.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

therock-pr-bot · 2026-06-24T00:19:54Z

🚫 Please fix the failed policies before requesting reviews.

The following policy checks failed:

❌ PR Title/Description

The Not ready to Review label has been added to this PR.
Once all policies pass, the label will be removed automatically.

Pass the PyTorch build toggle and release Python version into the multi-arch setup workflow so setup-generated build configs match the release workflows that consume them. Also let manually triggered multi_arch_ci.yml runs skip PyTorch builds, and keep ASAN setup callers opted out explicitly. Assisted-by: Codex

…orch-shared-matrix

Avoid depending on the current real GPU-family support matrix when testing that an empty generated PyTorch matrix disables PyTorch builds. Testing: - D:/projects/TheRock/.venv/Scripts/python.exe -m pytest github_actions/tests/configure_multi_arch_ci_test.py Assisted-by: Codex

…orch-shared-matrix

ScottTodd · 2026-06-26T15:56:01Z

      linux_test_labels: ${{ inputs.linux_test_labels || '' }}
      prebuilt_stages: ${{ inputs.prebuilt_stages || '' }}
      baseline_run_id: ${{ inputs.baseline_run_id || '' }}
+      build_pytorch: false


Note how build_pytorch is now an input to setup_multi_arch.yml and how configure_multi_arch_ci.py no longer has build_pytorch=(suffix != "asan"),

ScottTodd · 2026-06-26T15:56:48Z

+      build_pytorch:
+        type: boolean
+        default: true
+        description: "Build PyTorch wheels"


Note that CI https://github.com/ROCm/TheRock/actions/workflows/multi_arch_ci.yml now has this input, just like the release workflow:

I'll also add "build_jax" in #6117

ScottTodd · 2026-06-29T21:24:28Z

I have a few follow-ups that are blocked on this, would like reviews soon.

rahulc-gh

LGTM

rahulc-gh · 2026-06-30T00:21:52Z

+def _append_build_pytorch(lines: list[str], outputs: CIOutputs) -> None:
+    lines.append("| Platform | Python | PyTorch ref | Families |")
+    lines.append("|----------|--------|-------------|----------|")
+
+    rows = 0
+    for platform, config in [
+        ("Linux", outputs.builds.linux),
+        ("Windows", outputs.builds.windows),
+    ]:
+        if config is None:
+            continue
+        for row in config.pytorch_build_matrix:
+            families = ", ".join(
+                f"`{family}`" for family in row["amdgpu_families"].split(";")
+            )
+            lines.append(
+                f"| {platform} | `{row['python_version']}` | "
+                f"`{row['pytorch_git_ref']}` | {families} |"
+            )
+            rows += 1
+
+    if rows == 0:
+        lines.append("| — | — | — | — |")
+    lines.append("")
+
+


This is nice, having it in summary

rahulc-gh · 2026-06-30T00:27:27Z

thank you for cleaning it up , looks good to me.

## Motivation Progress on #5634 and #6218. This builds on #6117 to provide the JAX equivalent to #6082. Now the dynamic matrix of JAX builds to run will be generated at the _start_ of CI and release pipelines during "setup" / "configure CI" and a table will be included in the summary: <img width="750" height="683" alt="image" src="https://github.com/user-attachments/assets/37b21bd3-1906-4d2f-9732-0602fab9776a" /> ## Technical Details The new `build_jax` option now has these settings: Workflow | `build_jax` value | notes -- | -- | -- `multi_arch_ci.yml` | `false` | Matching previous behavior<br>can later be enabled (opt-in, automatic based on files edited, etc.) `multi_arch_ci_asan.yml` | `false` | New behavior due to #6218 `multi_arch_release.yml` | `inputs.build_jax`<br>(default `true`) | Matching previous behavior `multi_arch_release_asan.yml` | `false` | New behavior due to #6218 ## Test Plan * Dev release triggered to observe CI configuration: https://github.com/ROCm/TheRock/actions/runs/28470132517 ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

ScottTodd added 3 commits June 22, 2026 13:40

Merge remote-tracking branch 'upstream/main' into users/scotttodd/pyt…

5b77b92

…orch-shared-matrix

ScottTodd added the gfx125x Issue/PR relates to gfx125x family label Jun 24, 2026

therock-pr-bot Bot added the Not ready to Review PR has unresolved policy failures — reviews blocked label Jun 24, 2026

ScottTodd mentioned this pull request Jun 25, 2026

fix(ci): apply per-ref amdgpu exclusions in multi-arch PyTorch CI #6134

Closed

4 tasks

ScottTodd added 2 commits June 25, 2026 10:04

Cleanup tests and input ordering

8c549ca

Move inputs down in release_asan.yml

011d84b

rahulc-gh mentioned this pull request Jun 25, 2026

feat(pytorch): enable gfx125x for PyTorch release/2.11 builds #6122

Merged

2 tasks

ScottTodd added 3 commits June 25, 2026 13:58

Merge remote-tracking branch 'upstream/main' into users/scotttodd/pyt…

2bed74e

…orch-shared-matrix

Merge remote-tracking branch 'upstream/main' into users/scotttodd/pyt…

7067605

…orch-shared-matrix

ScottTodd marked this pull request as ready for review June 26, 2026 15:54

ScottTodd requested review from geomin12 and rahulc-gh June 26, 2026 15:54

ScottTodd commented Jun 26, 2026

View reviewed changes

This was referenced Jun 29, 2026

fix(jax): skip JAX wheel builds/tests on ASan builds #6213

Closed

[ASAN]: Skip building python packages and frameworks in asan pipelines until supported #6218

Open

ScottTodd requested a review from marbre June 29, 2026 21:24

ScottTodd removed the Not ready to Review PR has unresolved policy failures — reviews blocked label Jun 29, 2026

rahulc-gh mentioned this pull request Jun 29, 2026

[ci] Add MI455 (gfx125X) bringup to multi-arch CI ROCm/rocm-systems#7983

Merged

rahulc-gh approved these changes Jun 30, 2026

View reviewed changes

ScottTodd merged commit 554b425 into main Jun 30, 2026
148 of 197 checks passed

ScottTodd deleted the users/scotttodd/pytorch-shared-matrix branch June 30, 2026 16:42

ScottTodd mentioned this pull request Jun 30, 2026

Thread JAX build matrix through setup #6270

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Use configure_pytorch_release_matrix.py to drive the CI job matrix#6082

Use configure_pytorch_release_matrix.py to drive the CI job matrix#6082
ScottTodd merged 9 commits into
mainfrom
users/scotttodd/pytorch-shared-matrix

ScottTodd commented Jun 24, 2026 •

edited

Loading

Uh oh!

therock-pr-bot Bot commented Jun 24, 2026 •

edited

Loading

Uh oh!

therock-pr-bot Bot commented Jun 24, 2026

Uh oh!

ScottTodd Jun 26, 2026

Uh oh!

ScottTodd Jun 26, 2026

Uh oh!

ScottTodd commented Jun 29, 2026

Uh oh!

rahulc-gh left a comment

Uh oh!

rahulc-gh Jun 30, 2026

Uh oh!

rahulc-gh Jun 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

ScottTodd commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

Test Plan

Test Result

Submission Checklist

Uh oh!

therock-pr-bot Bot commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ PR Check — Action Required

Uh oh!

therock-pr-bot Bot commented Jun 24, 2026

Uh oh!

ScottTodd Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

ScottTodd Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

ScottTodd commented Jun 29, 2026

Uh oh!

rahulc-gh left a comment

Choose a reason for hiding this comment

Uh oh!

rahulc-gh Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

rahulc-gh Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ScottTodd commented Jun 24, 2026 •

edited

Loading

therock-pr-bot Bot commented Jun 24, 2026 •

edited

Loading