tensorflow: ERROR message when using tf.SyncReplicasOptimizer

I’m running distributed tensorflow with estimators, in order to it in sync mode I’m using tf.SyncReplicasOptimizer, but casually (specially after evaluation) I see the following error on the master:

ERROR:tensorflow:==================================
Object was never used (type <class 'tensorflow.python.framework.ops.Tensor'>):
<tf.Tensor 'report_uninitialized_variables/boolean_mask/Gather:0' shape=(?,) dtype=string>

['File "cifar10_main.py", line 538, in <module>\n    tf.app.run()', 'File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 48, in run\n    _sys.exit(main(_sys.argv[:1] + flags_passthrough))', 'File "cifar10_main.py", line 518, in main\n    hooks), run_config=config)', 'File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/learn_runner.py", line 210, in run\n    return _execute_schedule(experiment, schedule)', 'File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/learn_runner.py", line 47, in _execute_schedule\n    return task()', 'File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 501, in train_and_evaluate\n    hooks=self._eval_hooks)', 'File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 681, in _call_evaluate\n    hooks=hooks)', 'File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 292, in evaluate\n    name=name)', 'File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 638, in _evaluate_model\n    features, labels, model_fn_lib.ModeKeys.EVAL)', 'File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/estimator/estimator.py", line 545, in _call_model_fn\n    features=features, labels=labels, **kwargs)', 'File "cifar10_main.py", line 331, in _resnet_model_fn\n    gradvars, global_step=tf.train.get_global_step())', 'File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/sync_replicas_optimizer.py", line 252, in apply_gradients\n    variables.global_variables())', 'File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/tf_should_use.py", line 170, in wrapped\n    return _add_should_use_warning(fn(*args, **kwargs))', 'File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/tf_should_use.py", line 139, in _add_should_use_warning\n    wrapped = TFShouldUseWarningWrapper(x)', 'File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/tf_should_use.py", line 96, in __init__\n    stack = [s.strip() for s in traceback.format_stack()]']
==================================

Code available at:https://github.com/tensorflow/models/blob/master/tutorials/image/cifar10_estimator/cifar10_main.py

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 29 (13 by maintainers)

Most upvoted comments

Can anyone suggest a workaround of this case? What would be the best replace of SyncReplicasOptimizer? I’m running inception from research repo and I’m facing the same issue when running distributed tensorflow.

I also encountered a similar error when attempting to use SyncReplicasOptimizer for MNIST, and was wondering if any progress has been made on this issue. Thanks.