tensorflow: Tensorflow v2 Variable name uniquification for Keras Layers in eager is inconsistent
Tensorflow v2.0a
When creating e.g. keras models I would assume, that when I run make_generator_model twice in eager mode that the trainable_variable names are identical.
Why would I assume this?
Because the tf.train.Checkpoint and Checkpointable api makes you believe that variables are coupled with their corresponding object/class and uniquification of variables would be no longer necessary. And indeed, this is the case when creating a variable with the same name twice (as can be seen at the end of the code)
What do I get instead?
In the below example the variables of the second make_generator_model() call will be uniquified.
# First call
['dense/kernel:0', 'batch_normalization_v2/gamma:0', 'batch_normalization_v2/beta:0', 'conv2d_transpose/kernel:0', 'batch_normalization_v2_1/gamma:0', 'batch_normalization_v2_1/beta:0', 'conv2d_transpose_1/kernel:0', 'batch_normalization_v2_2/gamma:0', 'batch_normalization_v2_2/beta:0', 'conv2d_transpose_2/kernel:0']
# Second
['dense_1/kernel:0', 'batch_normalization_v2_3/gamma:0', 'batch_normalization_v2_3/beta:0', 'conv2d_transpose_3/kernel:0', 'batch_normalization_v2_4/gamma:0', 'batch_normalization_v2_4/beta:0', 'conv2d_transpose_4/kernel:0', 'batch_normalization_v2_5/gamma:0', 'batch_normalization_v2_5/beta:0', 'conv2d_transpose_5/kernel:0']
# Third
['dense/kernel:0', 'batch_normalization_v2/gamma:0', 'batch_normalization_v2/beta:0', 'conv2d_transpose/kernel:0', 'batch_normalization_v2_1/gamma:0', 'batch_normalization_v2_1/beta:0', 'conv2d_transpose_1/kernel:0', 'batch_normalization_v2_2/gamma:0', 'batch_normalization_v2_2/beta:0', 'conv2d_transpose_2/kernel:0']
# Fourth
['dense/kernel:0', 'batch_normalization_v2/gamma:0', 'batch_normalization_v2/beta:0', 'conv2d_transpose/kernel:0', 'batch_normalization_v2_1/gamma:0', 'batch_normalization_v2_1/beta:0', 'conv2d_transpose_1/kernel:0', 'batch_normalization_v2_2/gamma:0', 'batch_normalization_v2_2/beta:0', 'conv2d_transpose_2/kernel:0']
# Manual Creation
<tf.Variable 'test:0' shape=() dtype=int32, numpy=1>
<tf.Variable 'test:0' shape=() dtype=int32, numpy=1>
import tensorflow as tf
from tensorflow.keras import layers
def make_generator_model():
model = tf.keras.Sequential()
model.add(layers.Dense(7*7*256, use_bias=False, input_shape=(100,)))
model.add(layers.BatchNormalization())
model.add(layers.LeakyReLU())
model.add(layers.Reshape((7, 7, 256)))
assert model.output_shape == (None, 7, 7, 256) # Note: None is the batch size
model.add(layers.Conv2DTranspose(128, (5, 5), strides=(1, 1), padding='same', use_bias=False))
assert model.output_shape == (None, 7, 7, 128)
model.add(layers.BatchNormalization())
model.add(layers.LeakyReLU())
model.add(layers.Conv2DTranspose(64, (5, 5), strides=(2, 2), padding='same', use_bias=False))
assert model.output_shape == (None, 14, 14, 64)
model.add(layers.BatchNormalization())
model.add(layers.LeakyReLU())
model.add(layers.Conv2DTranspose(1, (5, 5), strides=(2, 2), padding='same', use_bias=False, activation='tanh'))
assert model.output_shape == (None, 28, 28, 1)
return model
m1 = make_generator_model()
noise = tf.random.normal([1, 100])
generated_image = m1(noise, training=False)
print([v.name for v in m1.trainable_variables])
m2 = make_generator_model()
noise = tf.random.normal([1, 100])
generated_image = m2(noise, training=False)
print([v.name for v in m2.trainable_variables])
with tf.Graph().as_default():
m1 = make_generator_model()
noise = tf.random.normal([1, 100])
generated_image = m1(noise, training=False)
print([v.name for v in m1.trainable_variables])
with tf.Graph().as_default():
m2 = make_generator_model()
noise = tf.random.normal([1, 100])
generated_image = m2(noise, training=False)
print([v.name for v in m2.trainable_variables])
a = tf.Variable(1, name='test')
b = tf.Variable(1, name='test')
print(a)
print(b)
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 26 (21 by maintainers)
The command
layer = Dense(10)only invokes__init__of the classDense.However, the variables for
Denseare created inDense.buildhttps://github.com/tensorflow/tensorflow/blob/e1c98eeb8ff8a25d2ecdcf1961974d6c9c1e3df4/tensorflow/python/keras/layers/core.py#L998 which is automatically called when we calllayer(inputs)within the scopeself._name_scope()https://github.com/tensorflow/tensorflow/blob/e1c98eeb8ff8a25d2ecdcf1961974d6c9c1e3df4/tensorflow/python/keras/engine/base_layer.py#L610-L614As for your example
with tf.name_scope('scoped'): dense2(tf.ones([10, 20]))since all related variables are created during the first call
layer(inputs), any subsequent calls do not invokedense2.buildto create new variables and the scopescopeddoes not change the name of already created variables.