Skip to content

[build] Monkeypatch gen_rst to call inside subprocess #3355

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
May 15, 2025
Merged

Conversation

clee2000
Copy link
Contributor

@clee2000 clee2000 commented May 7, 2025

Replace sphinx gallery's generate_file_rst call with a wrapper that calls that function but in a subprocess, and remove some things that we added in order to handle files changing global state

I tried using multiprocess.pool and apply async but ran into pickling problems. I don't understand why this current implementation doesn't have that issue

parallelism got added in some version of sphinx gallery, but it uses joblib, which seems to result in errors (conflicts with functions in subprocess library? im not sure). Additionally, it has no effect if you set parallel = 1 (it will not put each file run into its own process and run singly) so you need parallel >= 2, but I'm not sure if there are any tutorials that require being run on their own (ex for memory profiling)

Afaict (although I am not familiar with joblib) the way I do it here is similar to how sphinx gallery does it with job lib but they can wrap the actual call instead of needing to replace the function
https://github.com/sphinx-gallery/sphinx-gallery/blob/dd092a09513ea1d0616ac9e59b8d76d5a8217e4a/sphinx_gallery/gen_rst.py#L594-L608

I also tried to call sphinx once per file in #3351 but there's an over head of ~5 min per file due to sphinx doing extra stuff (generating the html for all the other files?) resulting in taking 2x longer

Copy link

pytorch-bot bot commented May 7, 2025

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/tutorials/3355

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 919d7ed with merge base b2e614d (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@clee2000 clee2000 marked this pull request as draft May 7, 2025 19:00
@clee2000 clee2000 changed the title [test] memformat [test] monkeypatch gen_rst to call inside subprocess May 15, 2025
Copy link
Contributor

@malfet malfet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just remove reset_seeds function completely

@clee2000 clee2000 changed the title [test] monkeypatch gen_rst to call inside subprocess [build] Monkeypatch gen_rst to call inside subprocess May 15, 2025
@clee2000 clee2000 marked this pull request as ready for review May 15, 2025 17:47
@clee2000 clee2000 merged commit 78933b1 into main May 15, 2025
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants