keras: fit_generator using use_multiprocessing=True does not work on Windows 8.1 x64, python 3.5
Dear Keras community
I have been using keras succesfully for many tasks.
After implementing a custom data generator using the keras Sequence class, I tried using the use_multiprocessing=True of the fit_generator function, with more than 1 worker (so data can be fed to my GPU).
Unfortunately, after testing this setup in 3 different machines, the code seems to work only on Linux (even having a different GPU).
- Behaviour using my windows 8.1 machine with python 3.5 installed using conda: Program freezes and do not start loading or training.
- Behaviour using my archlinux laptop, same software setup as the windows machine using conda: Program does work, feeds the data from disk, allows multiple threads and uses GPU for training.
Is this the expected behaviour on a windows machine?
Kind regards,
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 4
- Comments: 25 (7 by maintainers)
This still seems to be an issue. When use_multithreading=True, it is just hanging and literally nothing is happening. I am running it on Windows 10.
Setting workers to a number that is bigger than 1 seems to improve the speed even if use_multithereading=False. Why is this setup improving it?
Additionally, I also wonder if one needs to make its class generator (with
Sequence) thread safe? Since I am not able to set use_multithreading to True, I am then wondering if I need to make my generator thread safe? In that case, I am also wondering if the thread safe version will give the desired performance improvement…?I even asked a question related to this topic (regarding how things should be working in wondows 10) on Stackoverflow. https://stackoverflow.com/questions/52932406/is-the-class-generator-inheriting-sequence-thread-safe-in-keras-tensorflow But noone has replied so far…