tensorflow: WARNING:tensorflow:Gradients do not exist for variables

While training BERT on TPU i am getting these warnings and my precision and recall is zero while accuracy is 100

Train for 45205 steps, validate for 206 steps
WARNING:tensorflow:Gradients do not exist for variables ['tf_bert_model_1/bert/pooler/dense/kernel:0', 'tf_bert_model_1/bert/pooler/dense/bias:0'] when minimizing the loss.
WARNING:tensorflow:Gradients do not exist for variables ['tf_bert_model_1/bert/pooler/dense/kernel:0', 'tf_bert_model_1/bert/pooler/dense/bias:0'] when minimizing the loss.
WARNING:tensorflow:Gradients do not exist for variables ['tf_bert_model_1/bert/pooler/dense/kernel:0', 'tf_bert_model_1/bert/pooler/dense/bias:0'] when minimizing the loss.
WARNING:tensorflow:Gradients do not exist for variables ['tf_bert_model_1/bert/pooler/dense/kernel:0', 'tf_bert_model_1/bert/pooler/dense/bias:0'] when minimizing the loss.

Here is google colab file i am using https://colab.research.google.com/drive/1l0Eoram6vJRK5xQBmnCx-eU1e46higlo

updated colab file https://colab.research.google.com/drive/1D-eKgddRHROG39R2iyqE3qR7w0N0OHAB

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 21 (3 by maintainers)

Most upvoted comments

but the warning is still there. WARNING:tensorflow:Gradients do not exist for variables ['tf_bert_model/bert/pooler/dense/kernel:0', 'tf_bert_model/bert/pooler/dense/bias:0'] when minimizing the loss. I am thinking it should be an error instead of warning. becuase if gradient donot exist, how it will be updated?

+14

talhaanwarch on Mar 13, 2020

I have a similar warning message using TF2.2 on Colab GPU: WARNING:tensorflow:Gradients do not exist for variables [‘Variable:0’] when minimizing the loss.

I figured though that the Variables are still being updated. Is this warning ignorable?

acse-ncv19 on Jun 10, 2020

@talhaanwarch The warning occurs on GPU too. Its not specific to TPU. Please find the the gist here. As mentioned in this issue comment, this is not an bug. It means that these variables are not updated during training.

If you have more questions, please post the issue here or in stackoverflow. Thanks!

gowthamkpr on Jun 17, 2020

https://stackoverflow.com/questions/57144586/tensorflow-gradienttape-gradients-does-not-exist-for-variables-intermittently

optimizer.apply_gradients(
    (grad, var) 
    for (grad, var) in zip(gradients, model.trainable_variables) 
    if grad is not None
)

zccyman on Feb 24, 2021