scalene: Unpickling error when using `--profile-all` and multiprocessing.
When the code to profile is using a multiprocessing Pool, scalene throws an unpickling related error.
Scalene failed to initialize.
Traceback (most recent call last):
File "/Users/myUser/.virtualenvs/myVirtualEnv/lib/python3.9/site-packages/scalene/scalene_profiler.py", line 2059, in run_profiler
exit_status = profiler.profile_code(
File "/Users/myUser/.virtualenvs/myVirtualEnv/lib/python3.9/site-packages/scalene/scalene_profiler.py", line 1861, in profile_code
did_output = Scalene.output_profile()
File "/Users/myUser/.virtualenvs/myVirtualEnv/lib/python3.9/site-packages/scalene/scalene_profiler.py", line 933, in output_profile
json_output = Scalene.__json.output_profiles(
File "/Users/myUser/.virtualenvs/myVirtualEnv/lib/python3.9/site-packages/scalene/scalene_json.py", line 226, in output_profiles
stats.merge_stats(python_alias_dir)
File "/Users/myUser/.virtualenvs/myVirtualEnv/lib/python3.9/site-packages/scalene/scalene_statistics.py", line 405, in merge_stats
value = unpickler.load()
File "/Users/myUser/.virtualenvs/myVirtualEnv/lib/python3.9/site-packages/scalene/hashablelist.py", line 56, in extend
self.__hash__()
File "/Users/myUser/.virtualenvs/myVirtualEnv/lib/python3.9/site-packages/scalene/hashablelist.py", line 30, in __hash__
if not self._hash:
AttributeError: 'HashableList' object has no attribute '_hash'
To reproduce, try running scalene --profile-all example.py with the following code:
import multiprocessing
if __name__ == "__main__":
with multiprocessing.Pool() as pool:
pass
I have solved this, by modifying HashableList to inherit from collections.UserList instead of the built-in list. I can create a PR if someone can reproduce this issue.
About this issue
- Original URL
- State: closed
- Created a year ago
- Reactions: 1
- Comments: 15 (8 by maintainers)
Commits related to this issue
- Moved to inherit from collections.UserList, hopefully fixing https://github.com/plasma-umass/scalene/issues/647. — committed to plasma-umass/scalene by emeryberger 10 months ago
Great! Thanks @emeryberger
@emeryberger It works in my local machine. Also, that’s a much cleaner way to solve it, thanks!