scalene: Unpickling error when using `--profile-all` and multiprocessing.

When the code to profile is using a multiprocessing Pool, scalene throws an unpickling related error.

Scalene failed to initialize.
Traceback (most recent call last):
  File "/Users/myUser/.virtualenvs/myVirtualEnv/lib/python3.9/site-packages/scalene/scalene_profiler.py", line 2059, in run_profiler
    exit_status = profiler.profile_code(
  File "/Users/myUser/.virtualenvs/myVirtualEnv/lib/python3.9/site-packages/scalene/scalene_profiler.py", line 1861, in profile_code
    did_output = Scalene.output_profile()
  File "/Users/myUser/.virtualenvs/myVirtualEnv/lib/python3.9/site-packages/scalene/scalene_profiler.py", line 933, in output_profile
    json_output = Scalene.__json.output_profiles(
  File "/Users/myUser/.virtualenvs/myVirtualEnv/lib/python3.9/site-packages/scalene/scalene_json.py", line 226, in output_profiles
    stats.merge_stats(python_alias_dir)
  File "/Users/myUser/.virtualenvs/myVirtualEnv/lib/python3.9/site-packages/scalene/scalene_statistics.py", line 405, in merge_stats
    value = unpickler.load()
  File "/Users/myUser/.virtualenvs/myVirtualEnv/lib/python3.9/site-packages/scalene/hashablelist.py", line 56, in extend
    self.__hash__()
  File "/Users/myUser/.virtualenvs/myVirtualEnv/lib/python3.9/site-packages/scalene/hashablelist.py", line 30, in __hash__
    if not self._hash:
AttributeError: 'HashableList' object has no attribute '_hash'

To reproduce, try running scalene --profile-all example.py with the following code:

import multiprocessing

if __name__ == "__main__":
    with multiprocessing.Pool() as pool:
        pass

I have solved this, by modifying HashableList to inherit from collections.UserList instead of the built-in list. I can create a PR if someone can reproduce this issue.

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Reactions: 1
  • Comments: 15 (8 by maintainers)

Commits related to this issue

Most upvoted comments

Great! Thanks @emeryberger

@emeryberger It works in my local machine. Also, that’s a much cleaner way to solve it, thanks!