keras: CNN-LSTM with video frame sequence: InvalidArgumentError: You must feed a value for placeholder tensor

I’m building a CNN-LSTM network in Keras (v2.02) + Tensorflow (v1.0.1) using video frames as input. I’m setting up the network as shown below:

import tensorflow as tf
import keras
import cv2

video = keras.layers.Input(shape=(None, 299,299,3),name='video_input')

cnn = keras.applications.InceptionV3(weights='imagenet',
                                 include_top='False',
                                 pooling='avg')

cnn.trainable = False
encoded_frame = keras.layers.TimeDistributed(cnn)(video)
encoded_vid = keras.layers.LSTM(256)(encoded_frame)
outputs = keras.layers.Dense(128, activation='relu')(encoded_vid)

Some of the tensor properties are below:

video
<tf.Tensor 'video_input:0' shape=(?, ?, 299, 299, 3) dtype=float32>

cnn.input
<tf.Tensor 'input_1:0' shape=(?, 299, 299, 3) dtype=float32>

cnn.output
<tf.Tensor 'predictions/Softmax:0' shape=(?, 1000) dtype=float32>    

encoded_frame
<tf.Tensor 'time_distributed_1/Reshape_1:0' shape=(?, ?, 1000) dtype=float32>

encoded_vid
<tf.Tensor 'lstm_1/TensorArrayReadV3:0' shape=(?, 256) dtype=float32>

outputs
<tf.Tensor 'dense_1/Relu:0' shape=(?, 128) dtype=float32>

Now I build the model and fit the data:

model = keras.models.Model(inputs=[video],outputs=outputs)
model.compile(optimizer='adam',
          loss='mean_squared_logarithmic_error')
# Generate random targets
y = np.random.random(size=(128,)) 
y = np.reshape(y,(-1,128))
model.fit(x=frame_sequence, y=y, validation_split=0.0,shuffle=False, batch_size=1)

where frame_sequence is a sequence of video frames from one video:

frame_sequence.shape
(1, 48, 299, 299, 3)

All seems well up to the training step model.fit, where I get an error attributed to the input_1 placeholder in the InceptionV3 model input:

InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'input_1' with dtype float
 [[Node: input_1 = Placeholder[dtype=DT_FLOAT, shape=[], _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

Training works without error if I build my CNN from scratch instead of loading InceptionV3. For example, replacing InceptionV3 with:

cnn = Sequential()
cnn.add(Conv2D(64, (3, 3), activation='relu', padding='same', input_shape=(229, 229, 3)))
cnn.add(Conv2D(64, (3, 3), activation='relu'))
cnn.add(MaxPooling2D((2, 2)))
cnn.add(Conv2D(128, (3, 3), activation='relu', padding='same'))
cnn.add(Conv2D(128, (3, 3), activation='relu'))
cnn.add(MaxPooling2D((2, 2)))
cnn.add(Conv2D(256, (3, 3), activation='relu', padding='same'))
cnn.add(Conv2D(256, (3, 3), activation='relu'))
cnn.add(Conv2D(256, (3, 3), activation='relu'))
cnn.add(MaxPooling2D((2, 2)))
cnn.add(Flatten())

Here is some minimal code to reproduce the issue.

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Reactions: 7
  • Comments: 22 (6 by maintainers)

Most upvoted comments

@Feynman27 Potential workaround: set learning phase to 0 and pass Lambda(lambda x: cnn(x)) to TimeDistributed. Here’s the full example: https://gist.github.com/alfiya400/9d3bf303966f87a3c2aa92a0a0a54662

I also checked that output from TimeDistributed and cnn.predict match each other.

The drawback of this approach: you can’t use Dropouts and there might be other restrictions when learning_phase is set to 0.