keras: fit_generator crashes though keras.utils.data_utils.Sequence was used
When model.fit_generator is used with workers=0 and subclasses of keras.utils.data_utils.Sequence for both training and validation data, API of Sequence is not recognized inside evaluate_generator, it raises:
File ".../keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File ".../keras/engine/training.py", line 1415, in fit_generator
initial_epoch=initial_epoch)
File ".../keras/engine/training_generator.py", line 230, in fit_generator
validation_steps,
File ".../keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File ".../keras/engine/training.py", line 1469, in evaluate_generator
verbose=verbose)
File ".../keras/engine/training_generator.py", line 298, in evaluate_generator
else:
ValueError: `steps=None` is only valid for a generator based on the `keras.utils.Sequence` class. Please specify `steps` or use the `keras.utils.Sequence` class.
Example code:
from keras import Sequential
from keras.layers import Dense
from keras.utils.data_utils import Sequence
import numpy as np
class Dataset(Sequence):
def __getitem__(self, index):
return np.random.uniform(size=(16, 8)), np.random.uniform(size=(16, 1))
def __len__(self):
return 128
model = Sequential([Dense(4, activation='relu', input_shape=(8,)),
Dense(1, activation='sigmoid')])
model.compile(loss='mse', optimizer='adam')
model.fit_generator(generator=Dataset(), validation_data=Dataset(),
workers=0)
Issue can be fixed here by replacing:
if isinstance(val_data, Sequence):
val_enqueuer_gen = iter(val_data)
with
if isinstance(val_data, Sequence):
val_enqueuer_gen = iter(val_data)
validation_steps = len(val_data)
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 3
- Comments: 40 (11 by maintainers)
Commits related to this issue
- Fix `fit_generator` for `workers=0` (#11285) ### Summary This PR provides fix for `engine.training_generator.fit_generator` when `workers=0`. It used train data for validation and ignored `len(val_... — committed to keras-team/keras by arquolo 6 years ago
- Avoided the error (ValueError: `steps_per_epoch=None` is only valid for ...) that occurs with Keras 2.2.4 reported at https://github.com/keras-team/keras/issues/10855#issuecomment-437571885 Avoided th... — committed to Minyus/Keras-PyTorch-AvP-transfer-learning by Minyus 5 years ago
The solution which I got is
steps_per_epoch=len(x_train)/batch_sizeor whatever you want the number of steps for each epoch by settingsteps_per_epochinfit_generatorHi, i was struggling with the exact same error message today on Windows, Tensorflow 1.12.0. For me it turned out that I used tensorflow import (import tensorflow.keras.utils.Sequence as sequence) for the sequence generator, but for the model I was training I used the keras import (from keras.layers import… and from keras.models import…etc.) changing to tensorflow.keras.layers, tensorflow.keras.models…etc worked for me.
It’s works. Thanks!
Same issue here.
So, it’s not Keras issue, but Tensorflow one. That’s why it’s still closed. By the way,
tensorflow.kerasisn’t always updated with latest changes inkeras, so it’s better to use either onlykerasimports or onlytensorflow.kerasimports, never both.Using datagen.sample_per_validation//BS still falls short of 56 samples for batch size 64. Any solution to fill the last batch?
change steps_per_epoch=None to steps_per_epoch=nb_train_samples // batch_size