keras: CNN-LSTM with video frame sequence: InvalidArgumentError: You must feed a value for placeholder tensor
I’m building a CNN-LSTM network in Keras (v2.02) + Tensorflow (v1.0.1) using video frames as input. I’m setting up the network as shown below:
import tensorflow as tf
import keras
import cv2
video = keras.layers.Input(shape=(None, 299,299,3),name='video_input')
cnn = keras.applications.InceptionV3(weights='imagenet',
include_top='False',
pooling='avg')
cnn.trainable = False
encoded_frame = keras.layers.TimeDistributed(cnn)(video)
encoded_vid = keras.layers.LSTM(256)(encoded_frame)
outputs = keras.layers.Dense(128, activation='relu')(encoded_vid)
Some of the tensor properties are below:
video
<tf.Tensor 'video_input:0' shape=(?, ?, 299, 299, 3) dtype=float32>
cnn.input
<tf.Tensor 'input_1:0' shape=(?, 299, 299, 3) dtype=float32>
cnn.output
<tf.Tensor 'predictions/Softmax:0' shape=(?, 1000) dtype=float32>
encoded_frame
<tf.Tensor 'time_distributed_1/Reshape_1:0' shape=(?, ?, 1000) dtype=float32>
encoded_vid
<tf.Tensor 'lstm_1/TensorArrayReadV3:0' shape=(?, 256) dtype=float32>
outputs
<tf.Tensor 'dense_1/Relu:0' shape=(?, 128) dtype=float32>
Now I build the model and fit the data:
model = keras.models.Model(inputs=[video],outputs=outputs)
model.compile(optimizer='adam',
loss='mean_squared_logarithmic_error')
# Generate random targets
y = np.random.random(size=(128,))
y = np.reshape(y,(-1,128))
model.fit(x=frame_sequence, y=y, validation_split=0.0,shuffle=False, batch_size=1)
where frame_sequence
is a sequence of video frames from one video:
frame_sequence.shape
(1, 48, 299, 299, 3)
All seems well up to the training step model.fit
, where I get an error attributed to the input_1
placeholder in the InceptionV3 model input:
InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'input_1' with dtype float
[[Node: input_1 = Placeholder[dtype=DT_FLOAT, shape=[], _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
Training works without error if I build my CNN from scratch instead of loading InceptionV3. For example, replacing InceptionV3 with:
cnn = Sequential()
cnn.add(Conv2D(64, (3, 3), activation='relu', padding='same', input_shape=(229, 229, 3)))
cnn.add(Conv2D(64, (3, 3), activation='relu'))
cnn.add(MaxPooling2D((2, 2)))
cnn.add(Conv2D(128, (3, 3), activation='relu', padding='same'))
cnn.add(Conv2D(128, (3, 3), activation='relu'))
cnn.add(MaxPooling2D((2, 2)))
cnn.add(Conv2D(256, (3, 3), activation='relu', padding='same'))
cnn.add(Conv2D(256, (3, 3), activation='relu'))
cnn.add(Conv2D(256, (3, 3), activation='relu'))
cnn.add(MaxPooling2D((2, 2)))
cnn.add(Flatten())
Here is some minimal code to reproduce the issue.
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Reactions: 7
- Comments: 22 (6 by maintainers)
@Feynman27 Potential workaround: set learning phase to 0 and pass
Lambda(lambda x: cnn(x))
toTimeDistributed
. Here’s the full example: https://gist.github.com/alfiya400/9d3bf303966f87a3c2aa92a0a0a54662I also checked that output from TimeDistributed and
cnn.predict
match each other.The drawback of this approach: you can’t use Dropouts and there might be other restrictions when learning_phase is set to 0.