tensorflow: tf.keras: regularizer does not add to total losses

I try to add l2 regularizer for tf.keras.layers.CuDNNLSTM, here is my code for creating layers:

   regulizer = tf.keras.regularizers.l2(l=1) 
    net = tf.keras.layers.CuDNNLSTM(lstm_units,
                                    kernel_initializer=tf.keras.initializers.glorot_normal(),
                                    unit_forget_bias=True,
                                    recurrent_initializer=tf.keras.initializers.glorot_normal(),
                                    kernel_regularizer=regulizer,
                                    return_sequences=True)(input_data)

after I create my graph, I gather all the losses with tf.get_collection(tf.GraphKeys.LOSSES)

but 1. only mse error is in the return list. 2. the initial losses is same as without regularizer even if I set regularizer co-efficient very big.

Does someone know how to apply regularizer to keras layer in tensorflow?

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 17 (9 by maintainers)

Most upvoted comments

@scotthuang1989 As far as I understand, Tensorflow will depreciate the use of global variables. I think the better way to handle this is then:

total_loss = loss + tf.add_n(model.losses)

Where the second term contains weights and activations regulirization losses

jrabary on Oct 9, 2018

I just encountered the same issue. Apparently layers in native tf and keras are different in their ways of dealing with regularization loss.

I found an example of dealing with such regularization loss on tf official example: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/eager/python/examples/densenet/densenet_graph_test.py

They use tf.keras.model.losses to collect such loss. In general I think it is nowadays a good practice to subclass tf.keras.model API for building tensorflow models. If you do have mixed sources of regularization losses, you may consider collect them using tf.losses.get_regularization_loss and tf.keras.model.losses respectively.

SayHiRay on Nov 19, 2018

@scotthuang1989 I think you are right. tf’s add_loss() adds regularization loss to GraphKeys.REGULARIZATION_LOSSES, but keras’ add_loss() doesn’t. So tf.losses.get_regularization_loss() works for tf layer but not keras layer. For keras layer, you should call layer._losses or layer.get_losses_for().

I also see @fchollet’s comment that GraphKeys.REGULARIZATION_LOSSES is deprecated, so the correct way is to go through each and every layer/variable and get loss out manually. You can also use tf instead of keras, it works, for now.

@fchollet There is really a mismatch in tf and keras behaviors. And regarding your comment on deprecation, please consider either update the code or the documentation to keep consistency and avoid confusion. Thanks.

tranhungnghiep on Nov 14, 2018