spaCy: multi-processing.py example code returns a joblib error (macOS, Spacy2.0.3)
I’m running the example multi-processing script referenced in the Spacy docs at https://spacy.io/usage/processing-pipelines#multithreading. At github https://github.com/explosion/spacy/blob/master/examples/pipeline/multi_processing.py
I get the following output:
> python3 multi_processing.py '/Users/me/Documents/devl/quick_entity/data/spacy_multi_example', 'en_core_web_sm'
Loaded model 'en_core_web_sm'
Loading IMDB data...
Processing texts...
Traceback (most recent call last):
File "multi_processing.py", line 79, in <module>
plac.call(main)
File "/Users/me/Documents/devl/quick_entity/myenv/lib/python3.6/site-packages/plac_core.py", line 328, in call
cmd, result = parser.consume(arglist)
File "/Users/me/Documents/devl/quick_entity/myenv/lib/python3.6/site-packages/plac_core.py", line 207, in consume
return cmd, self.func(*(args + varargs + extraopts), **kwargs)
File "multi_processing.py", line 43, in main
executor(tasks)
File "/Users/me/Documents/devl/quick_entity/myenv/lib/python3.6/site-packages/joblib/parallel.py", line 789, in __call__
self.retrieve()
File "/Users/me/Documents/devl/quick_entity/myenv/lib/python3.6/site-packages/joblib/parallel.py", line 699, in retrieve
self._output.extend(job.get(timeout=self.timeout))
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/pool.py", line 644, in get
raise self._value
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/pool.py", line 424, in _handle_tasks
put(task)
File "/Users/me/Documents/devl/quick_entity/myenv/lib/python3.6/site-packages/joblib/pool.py", line 371, in send
CustomizablePickler(buffer, self._reducers).dump(obj)
AttributeError: Can't pickle local object 'FeatureExtracter.<locals>.feature_extracter_fwd'
I’m getting the same error on my own code when I try to enable multiprocessing so any suggestions about how to get this working (or possible workarounds) would be appreciated.
Info about spaCy
- spaCy version: 2.0.3
- Platform: Darwin-16.7.0-x86_64-i386-64bit
- Python version: 3.6.2
- Models: en, en_core_web_lg
pip freeze
aniso8601==1.3.0 certifi==2017.11.5 chardet==3.0.4 click==6.7 cymem==1.31.2 cytoolz==0.8.2 dill==0.2.7.1 en-core-web-sm==2.0.0 Flask==0.12.2 Flask-RESTful==0.3.6 ftfy==4.4.3 html5lib==0.999999999 idna==2.6 itsdangerous==0.24 Jinja2==2.10 joblib==0.11 MarkupSafe==1.0 more-itertools==3.2.0 msgpack-numpy==0.4.1 msgpack-python==0.4.8 murmurhash==0.28.0 numpy==1.13.3 pathlib==1.0.1 plac==0.9.6 preshed==1.0.0 python-dateutil==2.6.1 pytz==2017.3 regex==2017.4.5 requests==2.18.4 six==1.11.0 spacy==2.0.3 termcolor==1.1.0 thinc==6.10.1 toolz==0.8.2 tqdm==4.19.4 ujson==1.35 urllib3==1.22 wcwidth==0.1.7 webencodings==0.5.1 Werkzeug==0.12.2 wrapt==1.10.11
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Reactions: 7
- Comments: 18 (1 by maintainers)
Still facing issues