tensorflow: Repeated "Already exists: Resource" errors when I increase my model size

System information

  • Custom model built by creating convolutional network tf graph with SGD optimizer and running
  • Windows 10
  • TensorFlow installed from pip
  • TensorFlow version (use command below): 1.12
  • Python version: 3.6
  • Running on cpu

I’ve created a convolutional network by making a custom graph for feed forward and using a tensorflow optimizer to train it (I’ve tried with SGD and Adam).

`sess = tf.Session()
optimizer = tf.train.GradientDescentOptimizer(learning_rate=.01)

input_size = 16
input_matrix = tf.placeholder(tf.float32, shape=[1,input_size,input_size])
label_ph = tf.placeholder(tf.float32, shape=[28*28])
model_out = Model.feed_forward(input_matrix)

init = tf.global_variables_initializer()
sess.run(init)

loss = tf.losses.mean_squared_error([tf.convert_to_tensor(label_ph, tf.float32)], [model_out])
train = optimizer.minimize(loss)
sess.graph.finalize()`

This is my graph definition. I know Model.feed_forward is opaque but it’s basically a series of functions that get variable weights and feed input through layers. Just to give you an idea here’s an example of a convolutional feed forward:

`def feed_input(input_tensors, filters, biases, stride, zero_pad=True):
        if zero_pad:
            pad_size = int((int(filters[0].shape[0]) + 1)/2 - 1)
            input_tensors = tf.map_fn(lambda inp:
                tf.pad(inp, [[pad_size, pad_size], [pad_size, pad_size]]), input_tensors)

        _, shape_x, _ = input_tensors.shape

        unflattened_output = tf.map_fn(lambda input_tensor: tf.convert_to_tensor([[[tf.add(tf.tensordot(curr_filter,
            ConvLayer.__get_input_slice__(input_tensor, x, y, tf.shape(curr_filter)[0]), 2), biases[filter_ind])
         for x in range(0, shape_x - curr_filter.shape[0] + 1, stride)]
         for y in range(0, shape_x - curr_filter.shape[0] + 1, stride)]
         for filter_ind, curr_filter in enumerate(filters)], dtype=tf.float32), input_tensors)

        shape_1, shape_2, shape_3, shape_4 = unflattened_output.shape

        return tf.reshape(unflattened_output, [shape_1 * shape_2, shape_3, shape_4])`

After defining my graph I am running a loop that gets some input and runs sess.run(train, feed_dict={input_matrix:img, label_ph:label_image}). If I make input_size smaller (e.g. 4) or if I reduce the number of layers in my network, everything runs fine and it does exactly what I expect it to do. However, when my network grows in the ways I just mentioned it consistently spits out these errors during sess.run(train, feed_dict={input_matrix:img, label_ph:label_image}):

2018-11-15 00:46:31.069913: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.069914: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.069914: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.069927: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.070138: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.078467: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.080434: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.081575: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.083125: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.085370: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.089281: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.090932: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.091080: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.091398: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.091916: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.092735: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.098503: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.094920: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.099036: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.101236: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.101988: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.101966: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.103264: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.102945: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.106092: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.106858: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.108494: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.109495: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.109987: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.112326: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.112525: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.113595: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.113831: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.115342: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.115418: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.120685: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.122569: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.122708: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.123904: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.124832: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.126926: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.127336: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.129575: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.128498: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.115477: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.130180: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.130601: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.131469: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.135938: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.132834: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.139111: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.139728: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.140827: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.141429: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.142202: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.140004: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.142621: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.144039: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.147757: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.145043: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.149502: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.146723: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.148971: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.145861: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.149850: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.151642: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.152947: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.154227: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.155666: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.157249: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.157877: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.159295: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.160204: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.163481: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.163750: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.164518: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.166533: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.172400: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.174768: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar 2018-11-15 00:46:31.174793: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar Traceback (most recent call last): File “C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py”, line 1334, in _do_call return fn(*args) File “C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py”, line 1319, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File “C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py”, line 1407, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.AlreadyExistsError: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar [[{{node gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var}} = TemporaryVariabledtype=DT_FLOAT, shape=[5,5], var_name=“gradients/…dd/tmp_var”, _device=“/job:localhost/replica:0/task:0/device:CPU:0”]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File “C:/Users/Aidan/Documents/PycharmProjects/SketchIt/src/network/FeedImages.py”, line 57, in <module> sess.run(train, feed_dict={input_matrix:img, label_ph:label_image}) File “C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py”, line 929, in run run_metadata_ptr) File “C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py”, line 1152, in _run feed_dict_tensor, options, run_metadata) File “C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py”, line 1328, in _do_run run_metadata) File “C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py”, line 1348, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.AlreadyExistsError: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar [[{{node gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var}} = TemporaryVariabledtype=DT_FLOAT, shape=[5,5], var_name=“gradients/…dd/tmp_var”, _device=“/job:localhost/replica:0/task:0/device:CPU:0”]]

Anyone know what might be the cause? Maybe some concurrency issue since its telling me things that shouldn’t exist have already been created? Help would be much appreciated, I’m close to making this thing work! Thanks for your time.

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 40 (16 by maintainers)

Most upvoted comments

I’ll try to work on a fix, I’ll let you know when it will be in master, so you could test it.

Here is the corrected code example:

from tensorflow.core.protobuf import rewriter_config_pb2
config = tf.ConfigProto()
off = rewriter_config_pb2.RewriterConfig.OFF
config.graph_options.rewrite_options.memory_optimization  = off
return tf.Session(config=config)

If you are using the estimators API, try something like this:

import tensorflow as tf
from tensorflow.core.protobuf import rewriter_config_pb2
session_config = tf.ConfigProto()
off = rewriter_config_pb2.RewriterConfig.OFF
session_config.graph_options.rewrite_options.arithmetic_optimization = off
run_config = tf.estimator.RunConfig(session_config=session_config)
...
estimator = tf.estimator.Estimator(model_fn, model_dir, run_config)

Note to developers: this issue is not fixed. Theses are just workarounds as far as I can understand. Could you please re-open the issue?

This is definitely an issue in Tensorflow V2.6.0. I’m using python 3.9. Tensorflow installed using binaries(pip). Code runs on a CPU.

Like the comments above suggest, turning off arithmetic optimization was also the ~fix~ workaround for me. But their code is not compatible with V2, so I’ll post a snippet with what worked. I used the docs on graph optimization for reference: https://www.tensorflow.org/guide/graph_optimization.

import contextlib
@contextlib.contextmanager
def options(options):
  old_opts = tf.config.optimizer.get_experimental_options()
  tf.config.optimizer.set_experimental_options(options)
  try:
    yield
  finally:
    tf.config.optimizer.set_experimental_options(old_opts)

def test_function(body):
  with options({'arithmetic_optimization': False}):
          tf_tensor = tf.constant([body])
          if model:
              tensor_object = model(tf_tensor)
.....

I’m not sure if this is an issue because I’m running this on a CPU but it seems like if the code isn’t able to detect a GPU then these should be sensibly turned off.

I have this exact same issue in tf 2.0 beta0. Is there a workaround for that?

I tried:

tf.config.optimizer.set_experimental_options({'arithmetic_optimization': False})

and

tf.config.optimizer.set_experimental_options({'arithmetic_optimization': rewriter_config_pb2.RewriterConfig.OFF})

but it didn’t help.

E: For me issue happens only with AdamOptimizer. RMSProp is fine

Same problem here. Workaround is not a fix. Reopen it.

Just remove “google3” from the example and it should work.

You can disable all optimizations in arithmetic optimizer with something like:

    from google3.third_party.tensorflow.core.protobuf import rewriter_config_pb2
    
    config_proto = tf.ConfigProto()

    off = rewriter_config_pb2.RewriterConfig.OFF
    config_proto.graph_options.rewrite_options.arithmetic_optimization = off
    
    session = tf.Session(config=config_proto)

another example in https://github.com/tensorflow/tensorflow/blob/b17d53c0cd07bbae46a55dc9c12d3f48a1c19604/tensorflow/python/debug/lib/debug_graph_reconstruction_test.py#L45-L51

But I suspect that the real problem is in memory optimizer, you can disable it with this option: https://github.com/tensorflow/tensorflow/blob/4e19402174593095a0837d6e7a8ca3f686b67543/tensorflow/core/protobuf/rewriter_config.proto#L122

Can you try if disabling one of them helps.

I’m having the same issue and I can’t get the workaround to work. Do I do something with the tf.Session instance after creating it or will it automatically be used?

For now I just try different sequence lengths in my model, some work, some don’t.

EDIT: Figured it out: To make keras use the model call tensorflow.keras.backend.set_session with the model instance.

I also meet the same issue with tf.nn.conv1d, by using

    config_proto = tf.ConfigProto(log_device_placement=False, allow_soft_placement=True)
    off = rewriter_config_pb2.RewriterConfig.OFF
    config_proto.graph_options.rewrite_options.arithmetic_optimization = off

    sess = tf.Session(config=config_proto)

the issue is fixed. So i think maybe tf1.13 has some inner bug.

I have the same problem on TF 1.12 + Ubuntu 16.04 and my custom code. Weirdly enough, even though I share the vact_scope code in different networks, the problem does not happen for all of them (It might be due to calling tf.nn.dynamic_rnn in the same scope twice with scope.reuse_variables() between the calls)

@ezhulenev, any news on the problem? Any workarounds? Maybe there’s a way to disable the ArithmeticOptimizer? (or whole Grappler? What is the expected performance hit then?)

2019-02-07 17:26:05.004296: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.030579: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.031806: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.048779: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.052690: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.054059: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.056958: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.062997: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.074335: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.093691: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.106734: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.128811: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.145847: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.147453: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.164530: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.168118: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.174878: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.181530: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.242049: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.287269: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.375503: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.737421: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.737421: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.756781: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.757632: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.915909: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE

Weirdly, I see strange nested scopes (__per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/ N10tensorflow19TemporaryVariableOp6TmpVarE), which I do not create in the code (the GLOBAL_VARIABLES collection does not contain any variables with vact_iteration/vact_iteration in the name)