scalene: Scalene process cannot be used in place of multiprocessing process
Describe the bug I think Scalene tries to replace multiprocessing processes with its own. However, in doing so, it makes the project unable to pickle the functions
To Reproduce Steps to reproduce the behavior:
- use this in a python file
from multiprocessing import Pool
def f(x):
return x * x
if __name__ == '__main__':
with Pool(processes=3) as pool:
result = pool.imap_unordered(f, range(20))
- run
scalene <filename>.py - raises error
Can't pickle <class 'scalene.replacement_mp_lock.replacement_mp_semlock.ReplacementSemLock'>: attribute lookup replacement_mp_semlock.ReplacementSemLock on scalene.replacement_mp_lock failed
Desktop (please complete the following information):
- OS:
Windows 10 - Version
scalene==1.5.26,cloudpickle==2.2.1
If you have not yet tried with the repository version (python3 -m pip install git+https://github.com/plasma-umass/scalene), please try that before reporting.
Additional context Full error:
Error in program being profiled:
Can't pickle <class 'scalene.replacement_mp_lock.replacement_mp_semlock.ReplacementSemLock'>: attribute lookup replacement_mp_semlock.ReplacementSemLock on scalene.replacement_mp_lock failed
Traceback (most recent call last):
File "C:\Users\USER\.venv\Lib\site-packages\scalene\scalene_profiler.py", line 1857, in profile_code
exec(code, the_globals, the_locals)
File "C:\Users\USER\<filename>.py", line 9, in <module>
with Pool(processes=3) as pool:
^^^^^^^^^^^^^^^^^
File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\context.py", line 119, in Pool
return Pool(processes, initializer, initargs, maxtasksperchild,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\pool.py", line 215,
in __init__
self._repopulate_pool()
File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\pool.py", line 306,
in _repopulate_pool
return self._repopulate_pool_static(self._ctx, self.Process,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\pool.py", line 329,
in _repopulate_pool_static
w.start()
File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
^^^^^^^^^^^^^^^^^
File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\context.py", line 336, in _Popen
return Popen(process_obj)
^^^^^^^^^^^^^^^^^^
File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\popen_spawn_win32.py", line 94, in __init__
reduction.dump(process_obj, to_child)
File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\reduction.py", line
60, in dump
ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <class 'scalene.replacement_mp_lock.replacement_mp_semlock.ReplacementSemLock'>: attribute lookup replacement_mp_semlock.ReplacementSemLock on scalene.replacement_mp_lock failed
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\spawn.py", line 111, in spawn_main
new_handle = reduction.duplicate(pipe_handle,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\reduction.py", line
79, in duplicate
return _winapi.DuplicateHandle(
^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [WinError 6] The handle is invalid/EOFError: Ran out of input
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 20 (8 by maintainers)
I can confirm it works. Thanks for your help.
I’ve moved the class out of the function; can you see if that helps? Thanks!
python3 -m pip install git+https://github.com/plasma-umass/scalene@issue_651Can't pickle <class 'scalene.replacement_mp_lock.replacement_mp_semlock.ReplacementSemLock'>: attribute lookup replacement_mp_semlock.ReplacementSemLock on scalene.replacement_mp_lock failed