tensorflow: Tensorflow v2 Variable name uniquification for Keras Layers in eager is inconsistent

Tensorflow v2.0a

When creating e.g. keras models I would assume, that when I run make_generator_model twice in eager mode that the trainable_variable names are identical.

Why would I assume this? Because the tf.train.Checkpoint and Checkpointable api makes you believe that variables are coupled with their corresponding object/class and uniquification of variables would be no longer necessary. And indeed, this is the case when creating a variable with the same name twice (as can be seen at the end of the code)

What do I get instead? In the below example the variables of the second make_generator_model() call will be uniquified.

# First call
['dense/kernel:0', 'batch_normalization_v2/gamma:0', 'batch_normalization_v2/beta:0', 'conv2d_transpose/kernel:0', 'batch_normalization_v2_1/gamma:0', 'batch_normalization_v2_1/beta:0', 'conv2d_transpose_1/kernel:0', 'batch_normalization_v2_2/gamma:0', 'batch_normalization_v2_2/beta:0', 'conv2d_transpose_2/kernel:0']

# Second
['dense_1/kernel:0', 'batch_normalization_v2_3/gamma:0', 'batch_normalization_v2_3/beta:0', 'conv2d_transpose_3/kernel:0', 'batch_normalization_v2_4/gamma:0', 'batch_normalization_v2_4/beta:0', 'conv2d_transpose_4/kernel:0', 'batch_normalization_v2_5/gamma:0', 'batch_normalization_v2_5/beta:0', 'conv2d_transpose_5/kernel:0']

# Third
['dense/kernel:0', 'batch_normalization_v2/gamma:0', 'batch_normalization_v2/beta:0', 'conv2d_transpose/kernel:0', 'batch_normalization_v2_1/gamma:0', 'batch_normalization_v2_1/beta:0', 'conv2d_transpose_1/kernel:0', 'batch_normalization_v2_2/gamma:0', 'batch_normalization_v2_2/beta:0', 'conv2d_transpose_2/kernel:0']

# Fourth
['dense/kernel:0', 'batch_normalization_v2/gamma:0', 'batch_normalization_v2/beta:0', 'conv2d_transpose/kernel:0', 'batch_normalization_v2_1/gamma:0', 'batch_normalization_v2_1/beta:0', 'conv2d_transpose_1/kernel:0', 'batch_normalization_v2_2/gamma:0', 'batch_normalization_v2_2/beta:0', 'conv2d_transpose_2/kernel:0']

# Manual Creation
<tf.Variable 'test:0' shape=() dtype=int32, numpy=1>
<tf.Variable 'test:0' shape=() dtype=int32, numpy=1>
import tensorflow as tf
from tensorflow.keras import layers

def make_generator_model():
    model = tf.keras.Sequential()
    model.add(layers.Dense(7*7*256, use_bias=False, input_shape=(100,)))
    model.add(layers.BatchNormalization())
    model.add(layers.LeakyReLU())

    model.add(layers.Reshape((7, 7, 256)))
    assert model.output_shape == (None, 7, 7, 256) # Note: None is the batch size

    model.add(layers.Conv2DTranspose(128, (5, 5), strides=(1, 1), padding='same', use_bias=False))
    assert model.output_shape == (None, 7, 7, 128)
    model.add(layers.BatchNormalization())
    model.add(layers.LeakyReLU())

    model.add(layers.Conv2DTranspose(64, (5, 5), strides=(2, 2), padding='same', use_bias=False))
    assert model.output_shape == (None, 14, 14, 64)
    model.add(layers.BatchNormalization())
    model.add(layers.LeakyReLU())

    model.add(layers.Conv2DTranspose(1, (5, 5), strides=(2, 2), padding='same', use_bias=False, activation='tanh'))
    assert model.output_shape == (None, 28, 28, 1)

    return model


m1 = make_generator_model()
noise = tf.random.normal([1, 100])
generated_image = m1(noise, training=False)
print([v.name for v in m1.trainable_variables])

m2 = make_generator_model()
noise = tf.random.normal([1, 100])
generated_image = m2(noise, training=False)
print([v.name for v in m2.trainable_variables])

with tf.Graph().as_default():
    m1 = make_generator_model()
    noise = tf.random.normal([1, 100])
    generated_image = m1(noise, training=False)
    print([v.name for v in m1.trainable_variables])

with tf.Graph().as_default():
    m2 = make_generator_model()
    noise = tf.random.normal([1, 100])
    generated_image = m2(noise, training=False)
    print([v.name for v in m2.trainable_variables])

a = tf.Variable(1, name='test')
b = tf.Variable(1, name='test')
print(a)
print(b)

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 26 (21 by maintainers)

Most upvoted comments

The command layer = Dense(10) only invokes __init__ of the class Dense.

However, the variables for Dense are created in Dense.build https://github.com/tensorflow/tensorflow/blob/e1c98eeb8ff8a25d2ecdcf1961974d6c9c1e3df4/tensorflow/python/keras/layers/core.py#L998 which is automatically called when we call layer(inputs) within the scope self._name_scope() https://github.com/tensorflow/tensorflow/blob/e1c98eeb8ff8a25d2ecdcf1961974d6c9c1e3df4/tensorflow/python/keras/engine/base_layer.py#L610-L614

As for your example

with tf.name_scope('scoped'):
    dense2(tf.ones([10, 20]))

since all related variables are created during the first call layer(inputs), any subsequent calls do not invoke dense2.build to create new variables and the scope scoped does not change the name of already created variables.