Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

multiprocessing Queue and concurrent.futures.ProcessPoolExecutor #214

Open
zpz opened this issue May 27, 2023 · 0 comments
Open

multiprocessing Queue and concurrent.futures.ProcessPoolExecutor #214

zpz opened this issue May 27, 2023 · 0 comments

Comments

@zpz
Copy link
Owner

zpz commented May 27, 2023

if I use a multiprocessing Queue with concurrent.futures.ProcessPoolExecutor, I'd get

RuntimeError: Queue objects should only be shared between processes through inheritance

such as with this message

Traceback (most recent call last):
  File "/home/docker-user/mpservice/tests/experiments/test.py", line 19, in <module>
    asyncio.run(main())
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/home/docker-user/mpservice/tests/experiments/test.py", line 14, in main
    t.result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/lib/python3.10/multiprocessing/queues.py", line 244, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/usr/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
  File "/usr/lib/python3.10/multiprocessing/queues.py", line 58, in __getstate__
    context.assert_spawning(self)
  File "/usr/lib/python3.10/multiprocessing/context.py", line 373, in assert_spawning
    raise RuntimeError(
RuntimeError: Queue objects should only be shared between processes through inheritance

Try to understand this behavior

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant