tensorflow: Upgrade from r11 to r12 prodeuces "Variables not defined" when using any optimizer but GradientDescentOptimizer

To clarify, we just need to put a scope around the model-construction part.

with tf.variable_scope(tf.get_variable_scope()) as scope:
  for i in xrange(2):
    ... code as before until ...reuse_varables() ....

grads = list_grads[0] + list_grads[1]
... rest of code as before ...

Hope that helps!

Sure, let me try to clarify.

When you do tf.get_variable_scope().reuse_variables() you set the current scope to reuse variables. If you call the optimizer in such scope, it’s trying to reuse slot variables, which it cannot find, so it throws an error. If you put a scope around, the tf.get_variable_scope().reuse_variables() only affects that scope, so when you exit it, you’re back in the non-reusing mode, the one you want.

Hope that helps, let me know if I should clarify more.

Sorry sherry – the current behaviour is correct. Your code is leaking reuse – it just wasn’t checked before. It could cause all other troubles, and I think we should correct the leaky reuse cases, not revert the slot change. I’ll write more on the test cases, closing this.

@lukaszkaiser Hello, I found that your workaround to put a variable_scope which wraps the outermost num_gpus loop, but I am still confused why it does eliminate the error.

with tf.variable_scope(tf.get_variable_scope()) as vscope:
  for i in xrange(FLAGS.num_gpus):
    with tf.device('/gpu:%d' % i):
      with tf.name_scope('%s_%d' % (cifar10.TOWER_NAME, i)) as scope:
        loss = tower_loss(scope)
        tf.get_variable_scope().reuse_variables()     # HERE

Is it just because that the tf.get_variable_scope() (which is identical to vscope) is explicitly created than the implicit default? Then, what do these two VariableScope objects differ in?

What do you mean by “leaky reuse”? Could you please clarify me? /cc @cesc-park

i am a student,i am not very familiar with tensorflow, i just follow @lukaszkaiser and use with ’ tf.variable_scope(tf.get_variable_scope(),reuse=tf.AUTO_REUSE) as scope:’ and delete the ‘tf.get_variable_scope().reuse_variables()’ my code is work . i am runing the code of ROLO.

@Huayra007 if you remove the

images_for_tensorboard = generator(z_placeholder,batch_size,z_dimension)
tf.summary.image('Generated_images',images_for_tensorboard,5)

should be able to run it. You are calling 2 time your generator. So or you remove the snippet or you add reuse to your generator code as such:

def generator(z,batch_size,z_dim,reuse=False):
    if (reuse):
        tf.get_variable_scope().reuse_variables()

    g_w1 = tf.get_variable('g_w1',[z_dim,56*56],dtype=tf.float32,initializer=tf.truncated_normal_initializer(stddev=0.02))
    g_b1 = tf.get_variable('g_b1',[56*56],dtype=tf.float32,initializ

and when you call it for the tensorboard as such:

with tf.variable_scope(tf.get_variable_scope()) as scope:
    images_for_tensorboard = generator(z_placeholder, batch_size, z_dimensions,reuse=True)
    tf.summary.image('Generated_images', images_for_tensorboard, 5)

I hope this helps. Good luck with your GANs 😉

Ah, great. Your explanation is clear and helpful. Thanks!

To sum, a thing to remember is that where the (Adam-like) optimizer acts, i.e. opt.apply_gradients(...) (which is where the error is thrown from) should lie in the scope with reuse=False in order to properly create the slot variables.