gh-64192: Make imap()/imap_unordered() in multiprocessing.pool actually lazy
#136871
Conversation
|
All commit authors signed the Contributor License Agreement. |
|
Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool. If this change has little impact on Python users, wait for a maintainer to apply the |
obaltian
force-pushed
the
feature/add-buffersize-to-multiprocessing
branch
from
3cf40e3 to
c1f8081
Compare
|
Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool. If this change has little impact on Python users, wait for a maintainer to apply the |
Using `threading.Semaphore` makes it easier to cap the number of concurrently ran tasks. It also makes it possible to remove busy wait in child thread by waiting for semaphore. Also I've updated code to use the backpressure pattern - the new tasks are scheduled as soon as the user consumes the old ones.
obaltian
force-pushed
the
feature/add-buffersize-to-multiprocessing
branch
from
c1f8081 to
ec37be8
Compare
|
Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool. If this change has little impact on Python users, wait for a maintainer to apply the |
|
Please open an issue first for this one (I don't know if it's something that is discussed in the sprint). |
|
Previous discussion for executor: #74028 |
|
@picnixz thanks. The issue already exists, I will add more detailed description here soon, just wanted to run all tests for now. |
picnixz
changed the title
imap()/imap_unordered() in multiprocessing.Pool actually lazyimap()/imap_unordered() in multiprocessing.Pool actually lazy
|
Thanks! I'm actually adding the GH issue inthe title so that we can backref it easily. |
|
I'm adding a skipnews just for the CI otherwise you'll get a notification saying that the labels are incorrect. |
obaltian
changed the title
imap()/imap_unordered() in multiprocessing.Pool actually lazyimap()/imap_unordered() in multiprocessing.pool actually lazy
This new behavior allow smaller real concurrency number than number of running processes. Previously, it was not allowed since we implicitly incremented buffersize by `self._processes`.
obaltian
force-pushed
the
feature/add-buffersize-to-multiprocessing
branch
from
9114142 to
5b8db62
Compare
Uh oh!
There was an error while loading. Please reload this page.