tensorflow: Error converting NMT sequence to sequence example to .tflite model

I am following this example: https://www.tensorflow.org/tutorials/text/nmt_with_attention

It is working as it should be and saving checkpoints. I want to now convert this to a TF Lite model following this example: https://www.tensorflow.org/lite/convert/python_api#converting_a_savedmodel_ or https://www.tensorflow.org/lite/convert/python_api#converting_a_concrete_function_

Here is what I am running to save and them convert:

tflite_input_tensor = tf.constant(1., shape=[64, 39])
tflite_target_tensor = tf.constant(1., shape=[64, 7])
tflite_enc_hidden_tensor = tf.constant(1., shape=[64, 1024])
export_dir = "saved_models"
checkpoint.f = train_step
to_save = checkpoint.f.get_concrete_function(tflite_input_tensor, tflite_target_tensor, tflite_enc_hidden_tensor)
tf.saved_model.save(checkpoint, export_dir, to_save)

converter = tf.lite.TFLiteConverter.from_concrete_functions([to_save])
tflite_model = converter.convert()

But I am getting this error:

~/anaconda3/envs/main/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py in __call__(self, inputs, *args, **kwargs)
    845                     outputs = base_layer_utils.mark_as_return(outputs, acd)
    846                 else:
--> 847                   outputs = call_fn(cast_inputs, *args, **kwargs)
    848 
    849             except errors.OperatorNotAllowedInGraphError as e:

~/anaconda3/envs/main/lib/python3.6/site-packages/tensorflow_core/python/autograph/impl/api.py in wrapper(*args, **kwargs)
    290   def wrapper(*args, **kwargs):
    291     with ag_ctx.ControlStatusCtx(status=ag_ctx.Status.DISABLED):
--> 292       return func(*args, **kwargs)
    293 
    294   if inspect.isfunction(func) or inspect.ismethod(func):

TypeError: call() missing 2 required positional arguments: 'hidden' and 'enc_output'

Trained with:

@tf.function
def train_step(inp, targ, enc_hidden):
    loss = 0

    with tf.GradientTape() as tape:
        enc_output, enc_hidden = encoder(inp, enc_hidden)

        dec_hidden = enc_hidden

        dec_input = tf.expand_dims([targ_lang.word_index['<start>']] * BATCH_SIZE, 1)

        # Teacher forcing - feeding the target as the next input
        for t in range(1, targ.shape[1]):
            # passing enc_output to the decoder
            predictions, dec_hidden, _ = decoder(dec_input, dec_hidden, enc_output)

            loss += loss_function(targ[:, t], predictions)
            
            # using teacher forcing
            dec_input = tf.expand_dims(targ[:, t], 1)

    batch_loss = (loss / int(targ.shape[1]))

    variables = encoder.trainable_variables + decoder.trainable_variables

    gradients = tape.gradient(loss, variables)

    optimizer.apply_gradients(zip(gradients, variables))

    return batch_loss

EPOCHS = 3

for epoch in range(EPOCHS):
    start = time.time()

    enc_hidden = encoder.initialize_hidden_state()
    total_loss = 0

    for (batch, (inp, targ)) in enumerate(dataset.take(steps_per_epoch)):
        batch_loss = train_step(inp, targ, enc_hidden)
        total_loss += batch_loss

        if batch % 100 == 0:
            print('Epoch {} Batch {} Loss {:.4f}'.format(epoch + 1,
                                                         batch,
                                                         batch_loss.numpy()))
    # saving (checkpoint) the model every 2 epochs
    if (epoch + 1) % 1 == 0:
        checkpoint.save(file_prefix = checkpoint_prefix)

    print('Epoch {} Loss {:.4f}'.format(epoch + 1, total_loss / steps_per_epoch))
    print('Time taken for 1 epoch {} sec\n'.format(time.time() - start))

Somehow the parameters for the Decoder call is not being passed in?

class Decoder(tf.keras.Model):
    def __init__(self, vocab_size, embedding_dim, dec_units, batch_sz):
        super(Decoder, self).__init__()
        self.batch_sz = batch_sz
        self.dec_units = dec_units
        self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)
        self.rnn = tf.keras.layers.GRU(self.dec_units,
                                       return_sequences=True,
                                       return_state=True,
                                       recurrent_initializer='glorot_uniform',
                                       unroll=True)
        
        self.fc = tf.keras.layers.Dense(vocab_size)

        # used for attention
        self.attention = BahdanauAttention(self.dec_units)

    def call(self, x, hidden, enc_output):
        # enc_output shape == (batch_size, max_length, hidden_size)
        context_vector, attention_weights = self.attention(hidden, enc_output)

        # x shape after passing through embedding == (batch_size, 1, embedding_dim)
        x = self.embedding(x)

        # x shape after concatenation == (batch_size, 1, embedding_dim + hidden_size)
        x = tf.concat([tf.expand_dims(context_vector, 1), x], axis=-1)

        # passing the concatenated vector to the GRU
        output, state = self.rnn(x)

        # output shape == (batch_size * 1, hidden_size)
        output = tf.reshape(output, (-1, output.shape[2]))

        # output shape == (batch_size, vocab)
        x = self.fc(output)

        return x, state, attention_weights

I understand there may be some trouble converting the GRU layers, but I will tackle that next. This seems to blow up before it even can check if GRU is able to be converted.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 19 (5 by maintainers)

Most upvoted comments

Glad to hear that it’s working for you.

I think with the new converter coming out soon, it’s probably easier to combine the encoder/decoder into the single graph and also wrap loop inside of the model. The new converter will be great at handling the control flow ops. Please be patient about it and stay tuned. Thanks!

For anyone following, I found a way to get this working by splitting up the encoder and decoder into two tflite models for the time being and then managing the loop that decodes the sequence externally.

@tf.function
def eval_step_enc(enc_input):
    enc_out, enc_hidden = encoder(enc_input, [tf.zeros((1, units))])
    
    return enc_out, enc_hidden

@tf.function
def eval_step_dec(dec_input, enc_out, dec_hidden):
    predictions, dec_hidden, _ = decoder([dec_input,dec_hidden,enc_out])
    scores = tf.exp(predictions) / tf.reduce_sum(tf.exp(predictions), axis=1)
    dec_input = tf.expand_dims(tf.argmax(predictions, axis=1, output_type=tf.int32), 1)
    
    return dec_input, enc_out, dec_hidden, scores

# ...standard TFLite conversion code

I am guessing this is due to the fact certain control flow operations are not supported in TFLite that would allow a concrete function with a loop to convert properly. This will do for now. I am getting great accuracy and have this running in a native Android app on sample data.

Thank you for the help and looking forward to the new updated as we round out the year!