keras: clipnorm doesn't work with Embedding
I’m getting a Traceback every time “clipnorm” is used in NN with Embedding layer. Here is a simple script where the problem is obvious:
import numpy as np
from keras.layers import Input, Embedding
from keras.optimizers import Adam
from keras.models import Model
input_layer = Input(shape = (1,) )
embedding = Embedding(input_dim = 1,
output_dim = 1)(input_layer)
model = Model(input = input_layer, output = embedding)
model.compile(optimizer = Adam(clipnorm = 1.0), loss = 'mse')
X = np.array([[1]])
Y = np.array([[[0.5]]])
model.fit(X, Y, nb_epoch = 1)
Failure:
I tensorflow/core/common_runtime/gpu/gpu_device.cc:867] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:01:00.0)
I tensorflow/core/common_runtime/gpu/gpu_device.cc:867] Creating TensorFlow device (/gpu:1) -> (device: 1, name: GeForce GTX TITAN X, pci bus id: 0000:02:00.0)
Traceback (most recent call last):
File "./clipnorm-bug.py", line 20, in <module>
model.fit(X, Y, nb_epoch = 1)
File "/usr/local/lib/python3.5/dist-packages/keras/engine/training.py", line 1079, in fit
self._make_train_function()
File "/usr/local/lib/python3.5/dist-packages/keras/engine/training.py", line 696, in _make_train_function
self.total_loss)
File "/usr/local/lib/python3.5/dist-packages/keras/optimizers.py", line 379, in get_updates
grads = self.get_gradients(loss, params)
File "/usr/local/lib/python3.5/dist-packages/keras/optimizers.py", line 71, in get_gradients
grads = [clip_norm(g, self.clipnorm, norm) for g in grads]
File "/usr/local/lib/python3.5/dist-packages/keras/optimizers.py", line 71, in <listcomp>
grads = [clip_norm(g, self.clipnorm, norm) for g in grads]
File "/usr/local/lib/python3.5/dist-packages/keras/optimizers.py", line 9, in clip_norm
g = K.switch(n >= c, g * c / n, g)
TypeError: unsupported operand type(s) for *: 'IndexedSlices' and 'float'
Keras version is 1.1.0, TensorFlow is 0.10rc
clipvalue on the other hand works fine.
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Reactions: 9
- Comments: 15 (10 by maintainers)
Commits related to this issue
- Fixed clipvalue-->clipnorm after fix in Keras for https://github.com/fchollet/keras/issues/3859 — committed to 0xnurl/keras_character_based_ner by 0xnurl 7 years ago
Hi, When will this patch be included in the main Keras branch and the pip release? Thanks a lot.
@kracwarlock Sorry about the delay, I didn’t have a free GPU for testing. The latest one works for me as well.
+1, I also got bit by this as well. @kracwarlock i’ll try out your patch in a bit and let you know if it works on my setup, thanks for contributing.
EDIT: it works for me, thanks!
I think I switched back to keras/tf versions where this was working my project does not need this anymore but will try to write a fix if I get time
@kracwarlock Did you manage to find a solution?
@fchollet - This seems like a serious issue regarding clipnorm as it’s very useful for training RNN’s, which often employs an Embedding layer. Could you give some guidance on how to fix this issue?
I too just ran into this issue with
clipnorm
and embeddings. I also confirm thatclipvalue
does work.The way tensorflow handles scalar multiplication with
IndexedSlices
is: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/math_ops.py#L458-L467@fchollet Should we convert the scalar to tensor
clip_norm
to the backend from https://github.com/fchollet/keras/blob/master/keras/optimizers.py and convert it thereI can make a PR on the desired behavior