tensorflow: tf.layers.Layer.set_scope() problem might cause unexpected duplicate name ValueError

• OS Platform and Distribution: Ubuntu 16.04
• TensorFlow installed from (source or binary): binary
• TensorFlow version: 1.4.1
• Python version: 3.5.2
• CUDA/cuDNN version: CUDA 8.0, CUDNN 7.0.5
• GPU model and memory: GTX1080 (8G)
• Exact command to reproduce: python3 test.py
• Have I written custom code: True
• Bazel version: N/A
• GCC version: N/A

Repoduce code:

import tensorflow as tf

_BATCH_NORM_DECAY = 0.997
_BATCH_NORM_EPSILON = 1e-5

def two_batchnorm(inputs):
    with tf.variable_scope('two_batchnorm'):
        inputs = tf.layers.batch_normalization(
            inputs=inputs,
            axis=3,
            momentum=_BATCH_NORM_DECAY,
            epsilon=_BATCH_NORM_EPSILON,
            center=True,
            scale=True,
            training=True,
            fused=True)
        inputs = tf.layers.batch_normalization(
            inputs=inputs,
            axis=3,
            momentum=_BATCH_NORM_DECAY,
            epsilon=_BATCH_NORM_EPSILON,
            center=True,
            scale=True,
            training=True,
            fused=True)
    return inputs

inputs = tf.placeholder(tf.float32, [1, 5, 5, 3])
x = inputs
x = two_batchnorm(x)
x = two_batchnorm(x)

It’ll trigger an unexpected ValueError as following:

ValueError: Variable two_batchnorm/batch_normalization/gamma already exists, disallowed.

Removing the variable scope with tf.variable_scope('two_batchnorm') in two_batchnorm will work as expected.

All variables defined in the graph should be (in creation sequense):

two_batchnorm/batch_normalization/gamma:0
two_batchnorm/batch_normalization/beta:0
two_batchnorm/batch_normalization/moving_mean:0
two_batchnorm/batch_normalization/moving_variance:0
two_batchnorm/batch_normalization_1/gamma:0
two_batchnorm/batch_normalization_1/beta:0
two_batchnorm/batch_normalization_1/moving_mean:0
two_batchnorm/batch_normalization_1/moving_variance:0
two_batchnorm/batch_normalization_2/gamma:0
two_batchnorm/batch_normalization_2/beta:0
two_batchnorm/batch_normalization_2/moving_mean:0
two_batchnorm/batch_normalization_2/moving_variance:0
two_batchnorm/batch_normalization_3/gamma:0
two_batchnorm/batch_normalization_3/beta:0
two_batchnorm/batch_normalization_3/moving_mean:0
two_batchnorm/batch_normalization_3/moving_variance:0

However, with tf.layers.Layer’s add_variable logics, it’ll result in an unexpected value name as following (in creation sequence):

two_batchnorm/batch_normalization/gamma:0
two_batchnorm/batch_normalization/beta:0
two_batchnorm/batch_normalization/moving_mean:0
two_batchnorm/batch_normalization/moving_variance:0
two_batchnorm/batch_normalization_1/gamma:0
two_batchnorm/batch_normalization_1/beta:0
two_batchnorm/batch_normalization_1/moving_mean:0
two_batchnorm/batch_normalization_1/moving_variance:0
two_batchnorm/batch_normalization/gamma:0                        <-- ValueError raised here.
two_batchnorm/batch_normalization/beta:0
two_batchnorm/batch_normalization/moving_mean:0
two_batchnorm/batch_normalization/moving_variance:0
two_batchnorm/batch_normalization_1/gamma:0
two_batchnorm/batch_normalization_1/beta:0
two_batchnorm/batch_normalization_1/moving_mean:0
two_batchnorm/batch_normalization_1/moving_variance:0

One solution might be using self._name to setup Layer’s scope, not self._base_name.