-
Notifications
You must be signed in to change notification settings - Fork 4.2k
[build] Monkeypatch gen_rst to call inside subprocess #3355
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/tutorials/3355
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 919d7ed with merge base b2e614d ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, just remove reset_seeds
function completely
e54c7e8
to
a9972bc
Compare
Replace sphinx gallery's generate_file_rst call with a wrapper that calls that function but in a subprocess, and remove some things that we added in order to handle files changing global state
I tried using multiprocess.pool and apply async but ran into pickling problems. I don't understand why this current implementation doesn't have that issue
parallelism got added in some version of sphinx gallery, but it uses joblib, which seems to result in errors (conflicts with functions in subprocess library? im not sure). Additionally, it has no effect if you set parallel = 1 (it will not put each file run into its own process and run singly) so you need parallel >= 2, but I'm not sure if there are any tutorials that require being run on their own (ex for memory profiling)
Afaict (although I am not familiar with joblib) the way I do it here is similar to how sphinx gallery does it with job lib but they can wrap the actual call instead of needing to replace the function
https://github.com/sphinx-gallery/sphinx-gallery/blob/dd092a09513ea1d0616ac9e59b8d76d5a8217e4a/sphinx_gallery/gen_rst.py#L594-L608
I also tried to call sphinx once per file in #3351 but there's an over head of ~5 min per file due to sphinx doing extra stuff (generating the html for all the other files?) resulting in taking 2x longer