keras: Multiprocessing using fit_generator(pickle_safe=True) fails
I’m trying to use fit_generator to seperate data loader from trainer.
model.fit_generator(data_gen(), samples_per_epoch=10000, nb_epoch=1, pickle_safe=True, verbose=0)
Excuting this code produces error like below:
Traceback (most recent call last):
File "main_generator.py", line 138, in <module>
model.fit_generator(data_gen(), samples_per_epoch=10000, nb_epoch=1, pickle_safe=True, verbose=0)
File "C:\dev\WinPython-64bit-3.5.2.3Qt5\python-3.5.2.amd64\lib\site-packages\keras\models.py", line 935, in fit_generator
initial_epoch=initial_epoch)
File "C:\dev\WinPython-64bit-3.5.2.3Qt5\python-3.5.2.amd64\lib\site-packages\keras\engine\training.py", line 1470, in fit_generator
pickle_safe=pickle_safe)
File "C:\dev\WinPython-64bit-3.5.2.3Qt5\python-3.5.2.amd64\lib\site-packages\keras\engine\training.py", line 436, in generator_queue
thread.start()
File "C:\dev\WinPython-64bit-3.5.2.3Qt5\python-3.5.2.amd64\lib\multiprocessing\process.py", line 105, in start
self._popen = self._Popen(self)
File "C:\dev\WinPython-64bit-3.5.2.3Qt5\python-3.5.2.amd64\lib\multiprocessing\context.py", line 212, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\dev\WinPython-64bit-3.5.2.3Qt5\python-3.5.2.amd64\lib\multiprocessing\context.py", line 313, in _Popen
return Popen(process_obj)
File "C:\dev\WinPython-64bit-3.5.2.3Qt5\python-3.5.2.amd64\lib\multiprocessing\popen_spawn_win32.py", line 66, in __init__
reduction.dump(process_obj, to_child)
File "C:\dev\WinPython-64bit-3.5.2.3Qt5\python-3.5.2.amd64\lib\multiprocessing\reduction.py", line 59, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'generator_queue.<locals>.data_generator_task'
I also tried to test keras/tests/keras/test_multiprocessing.py, but it failed.
Here is output for test_multiprocessing.py: test_multiprocessing.faillog.txt
Is it bug of Keras itself? Any fixes available?
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 28
Commits related to this issue
- Fix keras call back test (Pickle related) — committed to raymondxyang/tensorflow by deleted user 7 years ago
This is still an issue for python 3.5, windows 7.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.
Today I’ve tryed the “multiprocessing.py” test on a Mac, with the same configuration of my: python 2.7.13, keras 1.0.8 and works fine. So the problem is only for windows, now we have the evidence