tensorflow: TF2.0: Translation model: Error when restoring the saved model: Unresolved object in checkpoint (root).optimizer.iter: attributes

Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): MacOs Mojave 10.14.5
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: NA
TensorFlow installed from (source or binary):
TensorFlow version (use command below):’ 2.0.0-beta1’
Python version: 3.7
Bazel version (if compiling from source): NA
GCC/Compiler version (if compiling from source): NA
CUDA/cuDNN version: NA
GPU model and memory: NA

You can collect some of this information using our environment capture script You can also obtain the TensorFlow version with: 1. TF 1.0: python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)" 2. TF 2.0: python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"

Describe the current behavior

I am trying to restore the checkpoints and predict on different sentences NMT Attention Model. While restoring the checkpoints and predicting, I am getting gibberish results with warning below:

Unresolved object in checkpoint (root).optimizer.iter: attributes { name: "VARIABLE_VALUE" full_name: "Adam/iter" checkpoint_key: "optimizer/iter/.ATTRIBUTES/VARIABLE_VALUE" }

Below is the additional warnings that I am getting and the results:

WARNING: Logging before flag parsing goes to stderr.
W1008 09:57:52.766877 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer.iter
W1008 09:57:52.767037 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer.beta_1
W1008 09:57:52.767082 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer.beta_2
W1008 09:57:52.767120 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer.decay
W1008 09:57:52.767155 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer.learning_rate
W1008 09:57:52.767194 4594230720 util.py:244] Unresolved object in checkpoint: (root).decoder.embedding.embeddings
W1008 09:57:52.767228 4594230720 util.py:244] Unresolved object in checkpoint: (root).decoder.gru.state_spec
W1008 09:57:52.767262 4594230720 util.py:244] Unresolved object in checkpoint: (root).decoder.fc.kernel
W1008 09:57:52.767296 4594230720 util.py:244] Unresolved object in checkpoint: (root).decoder.fc.bias
W1008 09:57:52.767329 4594230720 util.py:244] Unresolved object in checkpoint: (root).encoder.embedding.embeddings
W1008 09:57:52.767364 4594230720 util.py:244] Unresolved object in checkpoint: (root).encoder.gru.state_spec
W1008 09:57:52.767396 4594230720 util.py:244] Unresolved object in checkpoint: (root).decoder.gru.cell.kernel
W1008 09:57:52.767429 4594230720 util.py:244] Unresolved object in checkpoint: (root).decoder.gru.cell.recurrent_kernel
W1008 09:57:52.767461 4594230720 util.py:244] Unresolved object in checkpoint: (root).decoder.gru.cell.bias
W1008 09:57:52.767493 4594230720 util.py:244] Unresolved object in checkpoint: (root).decoder.attention.W1.kernel
W1008 09:57:52.767526 4594230720 util.py:244] Unresolved object in checkpoint: (root).decoder.attention.W1.bias
W1008 09:57:52.767558 4594230720 util.py:244] Unresolved object in checkpoint: (root).decoder.attention.W2.kernel
W1008 09:57:52.767590 4594230720 util.py:244] Unresolved object in checkpoint: (root).decoder.attention.W2.bias
W1008 09:57:52.767623 4594230720 util.py:244] Unresolved object in checkpoint: (root).decoder.attention.V.kernel
W1008 09:57:52.767657 4594230720 util.py:244] Unresolved object in checkpoint: (root).decoder.attention.V.bias
W1008 09:57:52.767688 4594230720 util.py:244] Unresolved object in checkpoint: (root).encoder.gru.cell.kernel
W1008 09:57:52.767721 4594230720 util.py:244] Unresolved object in checkpoint: (root).encoder.gru.cell.recurrent_kernel
W1008 09:57:52.767755 4594230720 util.py:244] Unresolved object in checkpoint: (root).encoder.gru.cell.bias
W1008 09:57:52.767786 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).decoder.embedding.embeddings
W1008 09:57:52.767818 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).decoder.fc.kernel
W1008 09:57:52.767851 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).decoder.fc.bias
W1008 09:57:52.767884 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).encoder.embedding.embeddings
W1008 09:57:52.767915 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).decoder.gru.cell.kernel
W1008 09:57:52.767949 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).decoder.gru.cell.recurrent_kernel
W1008 09:57:52.767981 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).decoder.gru.cell.bias
W1008 09:57:52.768013 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).decoder.attention.W1.kernel
W1008 09:57:52.768044 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).decoder.attention.W1.bias
W1008 09:57:52.768077 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).decoder.attention.W2.kernel
W1008 09:57:52.768109 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).decoder.attention.W2.bias
W1008 09:57:52.768143 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).decoder.attention.V.kernel
W1008 09:57:52.768175 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).decoder.attention.V.bias
W1008 09:57:52.768207 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).encoder.gru.cell.kernel
W1008 09:57:52.768239 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).encoder.gru.cell.recurrent_kernel
W1008 09:57:52.768271 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).encoder.gru.cell.bias
W1008 09:57:52.768303 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).decoder.embedding.embeddings
W1008 09:57:52.768335 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).decoder.fc.kernel
W1008 09:57:52.768367 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).decoder.fc.bias
W1008 09:57:52.768399 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).encoder.embedding.embeddings
W1008 09:57:52.768431 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).decoder.gru.cell.kernel
W1008 09:57:52.768463 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).decoder.gru.cell.recurrent_kernel
W1008 09:57:52.768495 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).decoder.gru.cell.bias
W1008 09:57:52.768527 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).decoder.attention.W1.kernel
W1008 09:57:52.768559 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).decoder.attention.W1.bias
W1008 09:57:52.768591 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).decoder.attention.W2.kernel
W1008 09:57:52.768623 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).decoder.attention.W2.bias
W1008 09:57:52.768654 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).decoder.attention.V.kernel
W1008 09:57:52.768686 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).decoder.attention.V.bias
W1008 09:57:52.768718 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).encoder.gru.cell.kernel
W1008 09:57:52.768750 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).encoder.gru.cell.recurrent_kernel
W1008 09:57:52.768782 4594230720 util.py:244] Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).encoder.gru.cell.bias
W1008 09:57:52.768816 4594230720 util.py:252] A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/alpha/guide/checkpoints#loading_mechanics for details.
Input: <start> hola <end>
Predicted translation: ? attack now relax hello

The warning at the very end says:

‘A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used…’ what does it mean? Describe the expected behavior Checkpoints should be restored properly Code to reproduce the issue Provide a reproducible test case that is the bare minimum necessary to generate the problem.

checkpoint_dir = './training_checkpoints'
checkpoint_prefix = os.path.join(checkpoint_dir, "ckpt")
checkpoint = tf.train.Checkpoint(optimizer=optimizer,
                                 encoder=encoder,
                                 decoder=decoder)
checkpoint.restore(tf.train.latest_checkpoint(checkpoint_dir))

Other info / logs Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 15
Comments: 32 (1 by maintainers)

Most upvoted comments

Closing this issue as it has been inactive for more than 15 days. Please add additional comments and we can open this issue again. Thanks!

+117

gowthamkpr on Oct 28, 2019

The error even shows up in the guide https://www.tensorflow.org/tutorials/keras/save_and_load

Train on 1000 samples Epoch 1/5 WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.iter WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.beta_1 WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.beta_2 WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.decay WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.learning_rate WARNING:tensorflow:A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(…).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/guide/checkpoint#loading_mechanics for details. WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.iter

+24

kirahman2 on Jun 4, 2020

Hopefully the simplest example that can reproduce the issue

# Tested with TF 2.0.0, Linux Ubuntu 18.04, Python 3.7.3, TF installed from binary
import tensorflow as tf

# create model and optimizer and checkpoint
model = tf.keras.models.Sequential([tf.keras.layers.Dense(5)])
opt = tf.keras.optimizers.Adam(0.1)
checkpoint_dir = 'ckpts'
ckpt = tf.train.Checkpoint(opt=opt, model=model)
manager = tf.train.CheckpointManager(ckpt, checkpoint_dir, max_to_keep=3)

# train with one example
example_x = tf.constant([[1.]])
example_y = tf.constant([[1.,2.,3.,4.,5.]])
model.compile(loss="mean_squared_error", optimizer=opt)
model.fit(example_x, example_y, epochs=1)

save_path = manager.save()
print("Saved checkpoint: {}".format(save_path))

# ========== restart from scratch but restore from checkpoint
model = tf.keras.models.Sequential([tf.keras.layers.Dense(5)])
opt = tf.keras.optimizers.Adam(0.1)

ckpt = tf.train.Checkpoint(opt=opt, model=model)
manager = tf.train.CheckpointManager(ckpt, checkpoint_dir, max_to_keep=3)
print('restoring...')
status = ckpt.restore(manager.latest_checkpoint)
# assert_consumed() fails with:
    # AssertionError: Unresolved object in checkpoint (root).opt.iter: attributes {
    #   name: "VARIABLE_VALUE"
    #   full_name: "Adam/iter"
    #   checkpoint_key: "opt/iter/.ATTRIBUTES/VARIABLE_VALUE"
status.assert_consumed()

+21

thierryherrmann on Nov 2, 2019

This is my take: The issue is that the new model object does not have the variable iter which is created when the optimiser is called on training samples. If you add these lines after you created the new model:

model.compile(loss="mean_squared_error", optimizer=opt)
model.fit(example_x, example_y, epochs=1)

Then the issue is resolved. Or by using status.assert_existing_objects_matched(), you can avoid the error as well, as it only ensures variables in the new model gets restored.

+17

jiahuei on Jan 15, 2020

Finally found out a cause and solution for this problem. Basically, it’s because Adam is initializing its attributes like beta_1, beta_2, etc, as a Python float object, which are not tracked by tensorflow. By converting them into tf.Variable, we can let them tracked by tensorflow and thus we can load weights without issues.

https://gist.github.com/yoshihikoueno/4ff0694339f88d579bb3d9b07e609122

+12

yoshihikoueno on Jul 17, 2020

Please take a look at the similar issue and let me know if it helps!!. Also try to test it with Tensorflow 2.0 and let me know if the issue still persists.

+11

gowthamkpr on Oct 8, 2019

I have a similar issue, and couldn’t resolve it. Error log:

Epoch 00004: saving model to checkpoints/yolov3_train_4.tf
9/9 [==============================] - 112s 12s/step - loss: 337.0419 - yolo_output_0_loss: 1.6753 - yolo_output_1_loss: 27.6452 - yolo_output_2_loss: 296.4273- val_loss: 1033934.1250 - val_yolo_output_0_loss: 1.5464 - val_yolo_output_1_loss: 17083.2285 - val_yolo_output_2_loss: 1016838.5000
Epoch 00004: early stopping
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer-8
W0424 06:25:47.220819 140280049981248 util.py:144] Unresolved object in checkpoint: (root).layer-8
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer-9
W0424 06:25:47.221170 140280049981248 util.py:144] Unresolved object in checkpoint: (root).layer-9
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer-10
W0424 06:25:47.221260 140280049981248 util.py:144] Unresolved object in checkpoint: (root).layer-10
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer-11
W0424 06:25:47.221323 140280049981248 util.py:144] Unresolved object in checkpoint: (root).layer-11
WARNING:tensorflow:A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/guide/checkpoint#loading_mechanics for details.
W0424 06:25:47.221471 140280049981248 util.py:152] A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/guide/checkpoint#loading_mechanics for details.

+10

saikrishnadas on Apr 24, 2020

I got the same error message and warning message in tensorflow-gpu 2.0

checkpoint = tf.train.Checkpoint(model=model, optimizer=optimizer)
checkpoint.save(checkpoint_prefix)

The optimizer is Adam

  lr_scheduler = tf.keras.optimizers.schedules.ExponentialDecay(
    initial_learning_rate=0.001,
    decay_steps=1000000,
    decay_rate=math.exp(-1),
    staircase=False)
  optimizer = tf.keras.optimizers.Adam(learning_rate=lr_scheduler)

The model is a subclassed tf.keras.Model

cockroachzl on Dec 19, 2019

@gowthamkpr I had upgraded to TF2.0 and tested it, but still has the same error. From https://github.com/tensorflow/tensorflow/issues/27937, I tried following the steps:

Saved the checkpoint
Re-initialized the encoder and decoder
Reloaded/restored the checkpoints to predict
The results are gibberish

Is it possible that the tokenizer is assigning random ID s to tokens when reloading the checkpoint?

nlpds on Oct 9, 2019

I also encountered this issue. However, most of my warnings were for training parameters like optimizers state. When looking the issue up, I came across an answer that said this occurs because I was loading weights from a checkpoint and then only using it for inference/prediction and not for training. Therefore the training parameters (like optimizer state) in the checkpoint were being unused and hence the warnings.

It would be great if someone could let me know how accurate an explanation this is. (Have attached the link below if someone wants to check the answer out)

https://stackoverflow.com/a/59582698

arvindas on Jul 15, 2020

For custom tensorflow models, I found that I needed to feed in input into them before loading the weights/checkpoints. That’s because some model weight shapes do not get properly configured before any input is fed into the network.

Wingtail on Feb 21, 2021

Same problem, please reopen this issue

KacperKubara on Jul 13, 2020

I have also the same issue, could you please reopen this issue?

pgagliar on May 22, 2020

I have the same issue, can this please be reopened?

JulianFerry on May 16, 2020

I have a similar issue, and couldn’t resolve it. Error log:

Epoch 00004: saving model to checkpoints/yolov3_train_4.tf
9/9 [==============================] - 112s 12s/step - loss: 337.0419 - yolo_output_0_loss: 1.6753 - yolo_output_1_loss: 27.6452 - yolo_output_2_loss: 296.4273- val_loss: 1033934.1250 - val_yolo_output_0_loss: 1.5464 - val_yolo_output_1_loss: 17083.2285 - val_yolo_output_2_loss: 1016838.5000
Epoch 00004: early stopping
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer-8
W0424 06:25:47.220819 140280049981248 util.py:144] Unresolved object in checkpoint: (root).layer-8
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer-9
W0424 06:25:47.221170 140280049981248 util.py:144] Unresolved object in checkpoint: (root).layer-9
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer-10
W0424 06:25:47.221260 140280049981248 util.py:144] Unresolved object in checkpoint: (root).layer-10
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer-11
W0424 06:25:47.221323 140280049981248 util.py:144] Unresolved object in checkpoint: (root).layer-11
WARNING:tensorflow:A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/guide/checkpoint#loading_mechanics for details.
W0424 06:25:47.221471 140280049981248 util.py:152] A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/guide/checkpoint#loading_mechanics for details.

anyone could solve it? i’m using efficientDet

y9luiz on Jul 29, 2020

I also found that many people won’t even notice this issue because they are not using assert_consumed method, which will postpone the checks. The conversion into tf.Variable will take place once the optimizer actually gets called, then the weights (beta_1 for example) will be loaded silently.

yoshihikoueno on Jul 17, 2020