keras: fit_generator crashes though keras.utils.data_utils.Sequence was used
When model.fit_generator
is used with workers=0
and subclasses of keras.utils.data_utils.Sequence
for both training and validation data, API of Sequence
is not recognized inside evaluate_generator
, it raises:
File ".../keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File ".../keras/engine/training.py", line 1415, in fit_generator
initial_epoch=initial_epoch)
File ".../keras/engine/training_generator.py", line 230, in fit_generator
validation_steps,
File ".../keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File ".../keras/engine/training.py", line 1469, in evaluate_generator
verbose=verbose)
File ".../keras/engine/training_generator.py", line 298, in evaluate_generator
else:
ValueError: `steps=None` is only valid for a generator based on the `keras.utils.Sequence` class. Please specify `steps` or use the `keras.utils.Sequence` class.
Example code:
from keras import Sequential
from keras.layers import Dense
from keras.utils.data_utils import Sequence
import numpy as np
class Dataset(Sequence):
def __getitem__(self, index):
return np.random.uniform(size=(16, 8)), np.random.uniform(size=(16, 1))
def __len__(self):
return 128
model = Sequential([Dense(4, activation='relu', input_shape=(8,)),
Dense(1, activation='sigmoid')])
model.compile(loss='mse', optimizer='adam')
model.fit_generator(generator=Dataset(), validation_data=Dataset(),
workers=0)
Issue can be fixed here by replacing:
if isinstance(val_data, Sequence):
val_enqueuer_gen = iter(val_data)
with
if isinstance(val_data, Sequence):
val_enqueuer_gen = iter(val_data)
validation_steps = len(val_data)
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 3
- Comments: 40 (11 by maintainers)
Commits related to this issue
- Fix `fit_generator` for `workers=0` (#11285) ### Summary This PR provides fix for `engine.training_generator.fit_generator` when `workers=0`. It used train data for validation and ignored `len(val_... — committed to keras-team/keras by arquolo 6 years ago
- Avoided the error (ValueError: `steps_per_epoch=None` is only valid for ...) that occurs with Keras 2.2.4 reported at https://github.com/keras-team/keras/issues/10855#issuecomment-437571885 Avoided th... — committed to Minyus/Keras-PyTorch-AvP-transfer-learning by Minyus 5 years ago
The solution which I got is
steps_per_epoch=len(x_train)/batch_size
or whatever you want the number of steps for each epoch by settingsteps_per_epoch
infit_generator
Hi, i was struggling with the exact same error message today on Windows, Tensorflow 1.12.0. For me it turned out that I used tensorflow import (import tensorflow.keras.utils.Sequence as sequence) for the sequence generator, but for the model I was training I used the keras import (from keras.layers import… and from keras.models import…etc.) changing to tensorflow.keras.layers, tensorflow.keras.models…etc worked for me.
It’s works. Thanks!
Same issue here.
So, it’s not Keras issue, but Tensorflow one. That’s why it’s still closed. By the way,
tensorflow.keras
isn’t always updated with latest changes inkeras
, so it’s better to use either onlykeras
imports or onlytensorflow.keras
imports, never both.Using datagen.sample_per_validation//BS still falls short of 56 samples for batch size 64. Any solution to fill the last batch?
change steps_per_epoch=None to steps_per_epoch=nb_train_samples // batch_size