-
-
Notifications
You must be signed in to change notification settings - Fork 31.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
gh-131656: Solved the cross-interpreter support of multiprocessing
module
#131657
base: main
Are you sure you want to change the base?
Conversation
Cross-interpreter support for multiprocessing module
Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool. If this change has little impact on Python users, wait for a maintainer to apply the |
multiprocessing
module
for path in data['sys_path']: | ||
if path in sys.path: | ||
sys_path.remove(path) | ||
sys.path.extend(sys_path) # For cross-interpreter support |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this should not be needed with the user supplied sys_path manipulation function in the parent. that can already know what the entire sys.path of the set_executable interpreter should be and pre-populate the value being sent to the child. no in child filtering or extend needed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, I'll do it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@gpshead But how can the parent process detect the appropriate sys.path
of the child since the child may has its own configs (e.g.HKEY_CURRENT_USER\SOFTWARE\Python\PythonCore\3.12\PythonPath
on Windows) and its own implementation of how to read these configs that differs from the parent.
Firstly, I've decided to simply replace the base path. For instance, D:\Python313\Lib
is simply replaced with D:\Python312\Lib
, and D:\Python313\DLLs
is replaced with D:\Python312\DLLs
,
but if the Python 3.12 is installed inside Cygwin and its sys.path
is D:\cygwin\usr\local\lib\python3.12
, there is no way to detect it.
So, I suppose that merging the paths is a better and more elegant solution:
# In the parent process
sys_path=[] # The paths being sent to the child
for path in sys.path:
if sys.base_prefix not in os.path.abspath(path): # Filter out the path that is not related to the parent (e.g. current working directory) by simply using `sys.base_prefix`
sys_path.append(path)
...
In the child:
for path in data['sys_path']:
if path in sys.path:
sys_path.remove(path) # remove duplicate paths
sys.path.extend(sys_path)
However, my code using sys.base_prefix
may be not a elegant way since you cannot use not in
in Linux. Could you suggest a better way, please?
A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated. Once you have made the requested changes, please leave a comment on this pull request containing the phrase |
The parent could run the child interpreter once when setting all of this up to get it's sys.path. |
That it a good idea while it has a slight performance overhead, I'll do it. |
The issue #131656 could be solved by slightly modifying the
prepare
andget_preparation_data
functions frommultiprocessing.spawn
module, which is responsible for exchanging Python runtime states includingsys.path
.After modifying, my example code in #131656 successfully runs:
multiprocessing
module #131656