tensorflow: ExponentialMovingAverage.average duplicates the current scope name

Using tensorflow nightly.

import tensorflow as tf
with tf.name_scope('scope'):
    x = tf.Variable(42, dtype=tf.float32)
    ema = tf.train.ExponentialMovingAverage(decay=0.9)
    apply_op = ema.apply([x])
    average = ema.average(x)
    print average.name   # 'scope/scope/Variable/ExponentialMovingAverage:0'
    print ema.average_name(x)  # 'scope/Variable/ExponentialMovingAverage'

About this issue

Original URL
State: closed
Created 8 years ago
Comments: 21 (17 by maintainers)

Most upvoted comments

Isn’t it true that ExponentialMovingAverage is used currently almost always outside any scopes? It shouldn’t break (many) existing use cases. If I remember correctly, the class simply doesn’t work well if the scope name is duplicated and users have to do manual hacking to fix the variable names.

The workaround that I’m using currently is to wrap the class with

with tf.name_scope(None):
   ema = tf.train.ExponentialMovingAverage(...)
   # [rest of the code]

rafaljozefowicz on Jun 28, 2016

@joel-shor Oh. You can change ‘W’ to ‘W2’ and move it outside of the scope, it still throws error. I’m only trying to point out that EMA cannot work inside reuse=True. By the way I don’t think I’m calling EMA on the same tensors, because calling add twice creates two tf.Operation.

ppwwyyxx on Dec 1, 2016

The recently-introduced new variables in EMA also brings error when using with reuse=True. The example below seems like a common pattern in batch normalization. It works before, but now:

def f(v):
    ema = tf.train.ExponentialMovingAverage(0.9)
    vema = ema.apply([v])
    return vema

with tf.variable_scope('s'):
    v1 = tf.get_variable('W', shape=[])
    v1 = v1 + 1
    f(v1)
with tf.variable_scope('s', reuse=True):
    v2 = tf.get_variable('W', shape=[])
    v2 = v2 + 1
    f(v2)

ValueError: Variable s/s_1/s_1/add/ExponentialMovingAverage/biased does not exist, or was not created with tf.get_variable(). Did you mean to set reuse=None in VarScope?

ppwwyyxx on Nov 30, 2016

@ppwwyyxx I’m not sure I follow. How would we be able to safely break backwards compatibility in the future? An advance warning that models will break is not a sufficient fix.

girving on Jun 29, 2016

It’s not just ExponentialMovingAverage, but also the accumulators used by optimizers that have this behavior. Anyone who’s using either of these inside a scope of any kind would suffer backwards-incompatible checkpoints due to the naming difference. Especially given there is an easy workaround (i.e. do not create / use ExponentialMovingAverage or Optimizers inside a name scope), the pain that fixing this would cause outweighs the benefits.

strategist333 on Jun 28, 2016