Skip to content

Update CoreML model testing in CI #12808

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 8 commits into from
Jul 28, 2025
Merged

Update CoreML model testing in CI #12808

merged 8 commits into from
Jul 28, 2025
+163 −60

Conversation

metascroy
Copy link
Contributor

This PR updates the alpha/beta CoreML model testing in a few ways:

  • All models are now testing with to_edge_lower_and_transform, the method we recommend in docs
  • We run pybindings on exported models to ensure 1) they run, and 2) the output is similar to pytorch eager
  • We add more model tests to CI

@metascroy metascroy requested a review from cccclai as a code owner July 24, 2025 05:09
@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Jul 24, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/12808

Note: Links to docs will display an error until the docs builds have been completed.

⏳ No Failures, 19 Pending

As of commit 4d7045f with merge base 2c84f70 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 24, 2025
@metascroy metascroy requested review from digantdesai and removed request for cccclai July 24, 2025 05:09
@github-actions GitHub Actions
Copy link

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@metascroy metascroy requested a review from shoumikhin as a code owner July 24, 2025 18:12
@metascroy metascroy requested a review from cccclai July 24, 2025 21:02
@metascroy
Copy link
Contributor Author

@digantdesai @cccclai can I get a stamp?

mergennachin
mergennachin reviewed Jul 27, 2025
View reviewed changes
.github/workflows/trunk.yml Outdated
@@ -590,12 +590,20 @@ jobs:
echo "Finishing installing coreml."

# Build and test coreml model
MODELS=(mv3 ic4 resnet50 edsr mobilebert w2l)
for MODEL_NAME in "${MODELS[@]}"; do
for MODEL_NAME in dl3 edsr emformer_join ic3 ic4 mobilebert mv2 mv3 resnet50 vit w2l; do
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@metascroy have you seen this PR?

https://github.com/pytorch/executorch/pull/12371/files

What are your thoughts on consolidating? I can drop my PR but maybe you can take a look and see if there are things worth incorporating?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mergennachin I missed that PR, but I can incorporate the following changes from your PR here:

  • Split CoreML/MPS into two separate jobs
  • Merge the model lists from the two PRs

For the runtime checks, I'll proceed with the pybind testing I'm adding here (vs. the C++ runner checking in your PR) because with pybinding we also check model accuracy.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @metascroy

Changes look great

digantdesai
digantdesai reviewed Jul 28, 2025
View reviewed changes
@metascroy metascroy force-pushed the update-coreml-model-testing branch from 1a9f07b to 4d7045f Compare July 28, 2025 17:47
@metascroy metascroy merged commit 37e3003 into main Jul 28, 2025
325 checks passed
@metascroy metascroy deleted the update-coreml-model-testing branch July 28, 2025 18:25
@mergennachin mergennachin mentioned this pull request Jul 28, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@digantdesai digantdesai digantdesai left review comments

@mergennachin mergennachin mergennachin left review comments

@cccclai cccclai cccclai approved these changes

@shoumikhin shoumikhin Awaiting requested review from shoumikhin shoumikhin is a code owner

Assignees
No one assigned
Labels
ciflow/trunk CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@metascroy @digantdesai @mergennachin @cccclai @facebook-github-bot