tensorflow: tf.keras.layers (in TF 1.13.1): variable_scope does not work
Run the following code:
import tensorflow as tf
for index in range(2):
with tf.variable_scope(name_or_scope=tf.get_variable_scope(), reuse=True if index > 0 else None):
outputs = tf.keras.layers.Dense(10).apply(tf.ones([10, 10]))
print(tf.trainable_variables())
There will be 4 variables:
[<tf.Variable 'dense/kernel:0' shape=(10, 10) dtype=float32>,
<tf.Variable 'dense/bias:0' shape=(10,) dtype=float32>,
<tf.Variable 'dense_1/kernel:0' shape=(10, 10) dtype=float32>,
<tf.Variable 'dense_1/bias:0' shape=(10,) dtype=float32>]
But if you change tf.keras.layers to tf.layers:
import tensorflow as tf
for index in range(2):
with tf.variable_scope(name_or_scope=tf.get_variable_scope(), reuse=True if index > 0 else None):
outputs = tf.layers.Dense(10).apply(tf.ones([10, 10]))
print(tf.trainable_variables())
There will be only 2 variables (the correct result):
[<tf.Variable 'dense/kernel:0' shape=(10, 10) dtype=float32_ref>,
<tf.Variable 'dense/bias:0' shape=(10,) dtype=float32_ref>]
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 22 (2 by maintainers)
When will you deprecate keras? When will you deprecate Tensorflow? When will you deprecate Google?
Maybe we could have a warning when instantiating tf.keras layers inside a reuse scope? That would have saved me a lot of debugging.
graphandvariable_scopeused to be your basic mechanism, and now you are depreciating it.tf.layers.xxxused to be your major selling point, and now you are depreciating it.Just tell me: is there anything that you will not depreciate?
And: When will you deprecate keras? (it is only a third-party package) When will you deprecate Tensorflow? When will you deprecate Google?
I am very respectful, logical, and not aggressive. Thank you.
Yes, you are very correct: all things evolve and change. So I really want to know:
When will you deprecate keras? When will you deprecate Tensorflow? When will you deprecate Google?
Knowing this in advance is very important to our users, but why do you deem this as “aggressive”?
Anyway, closing this issue is the best way if you cannot answer these questions.
Personally, I consider it very aggressive to keep repeating the same questions and spamming over several issues. Please understand that all APIs evolve and change. There was a time to discuss these API changes but it has been many months ago. Now the ship has sailed and these will stay deprecated unless valid reasoning asks for their reintroduction.
Closing, as https://github.com/tensorflow/tensorflow/issues/27016#issuecomment-480621768 suggests that this is not a real issue.
@mihaimaruseac I think it would be more clear in the deprecating msg to state explicitly something like
do not try to replace tf.layers with tf.keras.layers until you are on TF version > 2.0.I also think using simpler functional-style (like
dense()) makes a sense (actually I like that as well as OOP styles) when it’s more handy. My point was that the keras layers should work well being compatible with variable_scope. Please refer to the discussion in #20842 as well.And please be respectful, logical, but not aggressive. Nobody will listen to you if you say like this.
according to the confusing warning message, i spent a whole day to modify the funtional tf.layers to the OOP tf.keras.layers, but finally found that they do not support variable scope…shit
If this issue is not solved in TF 1.X, please do not mislead users to replace tf.layers to tf.keras.layers By the way, I think the change after TF 1.12 is a failure.
@mihaimaruseac They need to be updated for sure, but there are a whole bunch of GAN models implemented in tensorflow ,either official tutorials or just someone trying to recreate a research paper that follow the convention used in the official tensorflow TPU GAN below. Basically, create a generator/discriminator by stacking a bunch of layers under a variable scope with reuse =tf.auto_reuse. Now, when we just keep everything else the same and replace tf.layers with the tf.keras class, everything runs as expected except that the layers are not getting reused and there are typically no warnings/errors, nothing to indicate that something is wrong except that the loss diverges. https://github.com/tensorflow/tpu/tree/master/models/experimental/dcgan What changes would you recommend to transition to the keras API if creating a Keras Model and using its methods are not suitable(TPU distribute strategy does not support Keras GAN models currently as far as I can tell) Thanks
When you use
tf.keras.layers.Dense(), you actually do not need to bother using variable scope. The layer object itself allows you to reuse variables, as those variables-to-be-shared are created in the constructor.However, I also agree that the expected behavior is just what is described in the issue (on 1.X), as it sounds more consistent. I know variable_scope will be depcreated, but since that was also the way in 1.x Keras layers should respect variable scopes as well and this should be fixed.