Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-131656: Solved the cross-interpreter support of multiprocessing module #131657

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

pypy66
Copy link

@pypy66 pypy66 commented Mar 24, 2025

The issue #131656 could be solved by slightly modifying the prepare and get_preparation_data functions from multiprocessing.spawn module, which is responsible for exchanging Python runtime states including sys.path.
After modifying, my example code in #131656 successfully runs:

console screen

pypy66 added 2 commits March 24, 2025 19:57
Cross-interpreter support for multiprocessing module
@pypy66 pypy66 requested a review from gpshead as a code owner March 24, 2025 12:26
Copy link

cpython-cla-bot bot commented Mar 24, 2025

All commit authors signed the Contributor License Agreement.
CLA signed

@bedevere-app
Copy link

bedevere-app bot commented Mar 24, 2025

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

@pypy66 pypy66 changed the title gh-131656: Solved the cross-interpreter issue of subprocess module gh-131656: Solved the cross-interpreter support of subprocess module Mar 24, 2025
@pypy66 pypy66 changed the title gh-131656: Solved the cross-interpreter support of subprocess module gh-131656: Solved the cross-interpreter support of multiprocessing module Mar 24, 2025
for path in data['sys_path']:
if path in sys.path:
sys_path.remove(path)
sys.path.extend(sys_path) # For cross-interpreter support
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should not be needed with the user supplied sys_path manipulation function in the parent. that can already know what the entire sys.path of the set_executable interpreter should be and pre-populate the value being sent to the child. no in child filtering or extend needed.

Copy link
Author

@pypy66 pypy66 Mar 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I'll do it.

Copy link
Author

@pypy66 pypy66 Mar 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gpshead But how can the parent process detect the appropriate sys.path of the child since the child may has its own configs (e.g.HKEY_CURRENT_USER\SOFTWARE\Python\PythonCore\3.12\PythonPath on Windows) and its own implementation of how to read these configs that differs from the parent.
Firstly, I've decided to simply replace the base path. For instance, D:\Python313\Lib is simply replaced with D:\Python312\Lib, and D:\Python313\DLLs is replaced with D:\Python312\DLLs,
but if the Python 3.12 is installed inside Cygwin and its sys.path is D:\cygwin\usr\local\lib\python3.12, there is no way to detect it.
So, I suppose that merging the paths is a better and more elegant solution:

# In the parent process
sys_path=[] # The paths being sent to the child
for path in sys.path:
    if sys.base_prefix not in os.path.abspath(path): # Filter out the path that is not related to the parent (e.g. current working directory) by simply using `sys.base_prefix`
        sys_path.append(path)
...

In the child:

for path in data['sys_path']:
    if path in sys.path:
        sys_path.remove(path) # remove duplicate paths
sys.path.extend(sys_path)

However, my code using sys.base_prefix may be not a elegant way since you cannot use not in in Linux. Could you suggest a better way, please?

@bedevere-app
Copy link

bedevere-app bot commented Mar 25, 2025

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

@gpshead
Copy link
Member

gpshead commented Mar 27, 2025

But how can the parent process detect the appropriate sys.path of the child since the child may has its own configs

The parent could run the child interpreter once when setting all of this up to get it's sys.path.

@pypy66
Copy link
Author

pypy66 commented Mar 29, 2025

But how can the parent process detect the appropriate sys.path of the child since the child may has its own configs

The parent could run the child interpreter once when setting all of this up to get it's sys.path.

That it a good idea while it has a slight performance overhead, I'll do it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants