scalene: Scalene process cannot be used in place of multiprocessing process

Describe the bug I think Scalene tries to replace multiprocessing processes with its own. However, in doing so, it makes the project unable to pickle the functions

To Reproduce Steps to reproduce the behavior:

  1. use this in a python file
from multiprocessing import Pool


def f(x):
    return x * x


if __name__ == '__main__':
    with Pool(processes=3) as pool:
        result = pool.imap_unordered(f, range(20))
  1. run scalene <filename>.py
  2. raises error Can't pickle <class 'scalene.replacement_mp_lock.replacement_mp_semlock.ReplacementSemLock'>: attribute lookup replacement_mp_semlock.ReplacementSemLock on scalene.replacement_mp_lock failed

Desktop (please complete the following information):

  • OS: Windows 10
  • Version scalene==1.5.26, cloudpickle==2.2.1

If you have not yet tried with the repository version (python3 -m pip install git+https://github.com/plasma-umass/scalene), please try that before reporting.

Additional context Full error:

Error in program being profiled:
 Can't pickle <class 'scalene.replacement_mp_lock.replacement_mp_semlock.ReplacementSemLock'>: attribute lookup replacement_mp_semlock.ReplacementSemLock on scalene.replacement_mp_lock failed
Traceback (most recent call last):
  File "C:\Users\USER\.venv\Lib\site-packages\scalene\scalene_profiler.py", line 1857, in profile_code
    exec(code, the_globals, the_locals)
  File "C:\Users\USER\<filename>.py", line 9, in <module>
    with Pool(processes=3) as pool:
         ^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\pool.py", line 215, 
in __init__
    self._repopulate_pool()
  File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\pool.py", line 306, 
in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\pool.py", line 329, 
in _repopulate_pool_static
    w.start()
  File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\context.py", line 336, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\popen_spawn_win32.py", line 94, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\reduction.py", line 
60, in dump
    ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <class 'scalene.replacement_mp_lock.replacement_mp_semlock.ReplacementSemLock'>: attribute lookup replacement_mp_semlock.ReplacementSemLock on scalene.replacement_mp_lock failed
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\spawn.py", line 111, in spawn_main
    new_handle = reduction.duplicate(pipe_handle,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\reduction.py", line 
79, in duplicate
    return _winapi.DuplicateHandle(
           ^^^^^^^^^^^^^^^^^^^^^^^^

OSError: [WinError 6] The handle is invalid/EOFError: Ran out of input

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 20 (8 by maintainers)

Most upvoted comments

I can confirm it works. Thanks for your help.

I’ve moved the class out of the function; can you see if that helps? Thanks!

python3 -m pip install git+https://github.com/plasma-umass/scalene@issue_651

Can't pickle <class 'scalene.replacement_mp_lock.replacement_mp_semlock.ReplacementSemLock'>: attribute lookup replacement_mp_semlock.ReplacementSemLock on scalene.replacement_mp_lock failed