keras: LSTM (return_sequences=True) followed by TimeDistributed(Dense) doesn't work!

Hello, everyone, Recently I wanted to build a network as following Input --> sequence padding and masking 0 --> LSTM (return_sequence=True) --> Timedistributed(Dense(outputnumber, softmax)), but an error of mismatched dimension was reported.

  # Model definition
  model = Sequential()
  model.add(Masking(mask_value=0., input_shape=(None,100)))
  model.add(LSTM(args.dimension, dropout_W=0.2, dropout_U=0.2, return_sequences=True)) 
  model.add(TimeDistributed(Dense(97, activation='softmax')))

The error is as : ValueError: Error when checking model target: expected timedistributed_1 to have 3 dimensions, but got array with shape (3000, 97)

The dataset has 3000 sequences as input.

Thanks.

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 1
Comments: 16 (3 by maintainers)

Most upvoted comments

Thanks again, I have figured out why. Since I thought the softmax layer would automatically deal with the timedistributed stuff, I didn’t explicitly set expand the labels to 3-dimension. Now it works.

wsstriving on Feb 1, 2017

Thanks again, I have figured out why. Since I thought the softmax layer would automatically deal with the timedistributed stuff, I didn’t explicitly set expand the labels to 3-dimension. Now it works.

Can you please share your workaround solution? I faced the same problem!

un-lock-me on Jun 5, 2019

Would be great if somebody could explain! Thank you!

skeydan on Apr 22, 2017

Here is the code


model.add(LSTM(num_neurons,
                      #  input_dim = lstm_num_features,
                       batch_input_shape=(batch_size, X_train.shape[1], X_train.shape[2]), 
                       return_sequences = True))
            model.add(TimeDistributed(Dense(lstm_num_predictions)))
            model.compile(loss='mean_squared_error', optimizer='adam')

And the error happens when I try to fit the model:

model.fit(X_train, y_train, nb_epoch = num_epochs, batch_size = batch_size)

The error is

Exception: Error when checking model target: expected timedistributed_14 to have 3 dimensions, but got array with shape (95, 1)

skeydan on Apr 22, 2017

Thanks for reply, and the output of model.summary() (The input feature dimension is now 39): ` screenshot from 2017-02-01 10 35 54

I tried to change the input shape to (None,None,39) but it will report errors of dimension mismatch. ` By the way, if I set the return_sequences=False, and no timedistributed stuff used, everything works fine. I just thought maybe it was the wrong usage of TimeDistributed

wsstriving on Feb 1, 2017

The input to the LSTM should also be a 3D array. Make sure that is the case.

Your masking input_shape seems weird and should probably have 3 dims.

Make sure to mention when the error occurs. When you build the model, when you compile, when you run?

Try running model.summary(). If the error happens when you start passing data, check the shape of your data.

bstriner on Jan 29, 2017