Multiprocessing Pool Crash

[jadczak]

Attempting to profile programs with multiprocessing yields the following error:

Error in program being profiled:
 Can't pickle <class 'scalene.replacement_sem_lock.replacement_semlock.ReplacementSemLock'>: attribute lookup replacement_semlock.ReplacementSemLock on scalene.replacement_sem_lock failed

I was able to reproduce the error using the example from #809 (comment)

import multiprocessing as mp

def fun(args):
    x = 0
    for i in range(100):
        x += 1
    return args

if __name__ == "__main__":
    with mp.Pool(mp.cpu_count()) as p:
        r = p.map(fun, range(1000000))
    print(sum(x for x in r))
    print(((1000000-1) * 1000000) / 2)

Expected behavior
No crash?

Screenshots

Desktop (please complete the following information):

• OS: Windows 11
• Browser Firefox
• Version 129
• Python: CPython 3.12.3
• Scalene: Repository version

Error Text

Error in program being profiled:
 Can't pickle <class 'scalene.replacement_sem_lock.replacement_semlock.ReplacementSemLock'>: attribute lookup replacement_semlock.ReplacementSemLock on scalene.replacement_sem_lock failed
Traceback (most recent call last):
  File "C:\Users\||||||||\scratch\py312\Lib\site-packages\scalene\scalene_profiler.py", line 1791, in profile_code
    exec(code, the_globals, the_locals)
  File "C:\Users\||||||||\scratch\scalene_mp.py", line 10, in <module>
    with mp.Pool(mp.cpu_count()) as p:
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 215, in __init__
    self._repopulate_pool()
  File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 306, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 329, in _repopulate_pool_static
    w.start()
  File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\context.py", line 337, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\popen_spawn_win32.py", line 95, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <class 'scalene.replacement_sem_lock.replacement_semlock.ReplacementSemLock'>: attribute lookup replacement_semlock.ReplacementSemLock on scalene.replacement_sem_lock failed
Scalene: The specified code did not run for long enough to profile.
By default, Scalene only profiles code in the file executed and its subdirectories.
To track the time spent in all files, use the `--profile-all` option.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\spawn.py", line 113, in spawn_main
    new_handle = reduction.duplicate(pipe_handle,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\reduction.py", line 79, in duplicate
    return _winapi.DuplicateHandle(
           ^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [WinError 6] The handle is invalid

[emeryberger]

• Works fine on Mac OS X, repository version of Scalene.
• Found an occasional race condition on exit on Ubuntu 20, but not the issue mentioned above (perhaps it's a WSL thing)

[jadczak]

This was on my windows machine not running in WSL. My wild shot in the dark is maybe a fork vs spawn thing since windows doesn't support fork 🤷

Once I get back to my windows machine I'll give WSL a try and see if I run into similar problems. It doesn't look like python 3.12.4 fixed anything relevant to multiprocessing, but I'll give a few other versions a try to see if there is something funky going on with my particular version of python.

[jadczak]

Tested the following python versions on Windows:
3.12.5 - OSError every time
3.11.9 - OSError every time

Tested the following python version on Clear Linux
3.12.4 - Worked every time tested

Test the following python version WSL
3.11.2 - Frequently hangs. No error when it finishes.