keras: Multiprocessing using fit_generator(pickle_safe=True) fails

I’m trying to use fit_generator to seperate data loader from trainer.

model.fit_generator(data_gen(), samples_per_epoch=10000, nb_epoch=1, pickle_safe=True, verbose=0)

Excuting this code produces error like below:

Traceback (most recent call last):
  File "main_generator.py", line 138, in <module>
    model.fit_generator(data_gen(), samples_per_epoch=10000, nb_epoch=1, pickle_safe=True, verbose=0)
  File "C:\dev\WinPython-64bit-3.5.2.3Qt5\python-3.5.2.amd64\lib\site-packages\keras\models.py", line 935, in fit_generator
    initial_epoch=initial_epoch)
  File "C:\dev\WinPython-64bit-3.5.2.3Qt5\python-3.5.2.amd64\lib\site-packages\keras\engine\training.py", line 1470, in fit_generator
    pickle_safe=pickle_safe)
  File "C:\dev\WinPython-64bit-3.5.2.3Qt5\python-3.5.2.amd64\lib\site-packages\keras\engine\training.py", line 436, in generator_queue
    thread.start()
  File "C:\dev\WinPython-64bit-3.5.2.3Qt5\python-3.5.2.amd64\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "C:\dev\WinPython-64bit-3.5.2.3Qt5\python-3.5.2.amd64\lib\multiprocessing\context.py", line 212, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\dev\WinPython-64bit-3.5.2.3Qt5\python-3.5.2.amd64\lib\multiprocessing\context.py", line 313, in _Popen
    return Popen(process_obj)
  File "C:\dev\WinPython-64bit-3.5.2.3Qt5\python-3.5.2.amd64\lib\multiprocessing\popen_spawn_win32.py", line 66, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\dev\WinPython-64bit-3.5.2.3Qt5\python-3.5.2.amd64\lib\multiprocessing\reduction.py", line 59, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'generator_queue.<locals>.data_generator_task'

I also tried to test keras/tests/keras/test_multiprocessing.py, but it failed.

Here is output for test_multiprocessing.py: test_multiprocessing.faillog.txt

Is it bug of Keras itself? Any fixes available?

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 28

Commits related to this issue

Most upvoted comments

This is still an issue for python 3.5, windows 7.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.

Today I’ve tryed the “multiprocessing.py” test on a Mac, with the same configuration of my: python 2.7.13, keras 1.0.8 and works fine. So the problem is only for windows, now we have the evidence