keras: clipnorm doesn't work with Embedding

I’m getting a Traceback every time “clipnorm” is used in NN with Embedding layer. Here is a simple script where the problem is obvious:

import numpy as np
from keras.layers import Input, Embedding
from keras.optimizers import Adam
from keras.models import Model

input_layer = Input(shape = (1,) )

embedding = Embedding(input_dim = 1,
                      output_dim = 1)(input_layer)

model = Model(input = input_layer, output = embedding)

model.compile(optimizer = Adam(clipnorm = 1.0), loss = 'mse')

X = np.array([[1]])
Y = np.array([[[0.5]]])
model.fit(X, Y, nb_epoch = 1)

Failure:

I tensorflow/core/common_runtime/gpu/gpu_device.cc:867] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:01:00.0)
I tensorflow/core/common_runtime/gpu/gpu_device.cc:867] Creating TensorFlow device (/gpu:1) -> (device: 1, name: GeForce GTX TITAN X, pci bus id: 0000:02:00.0)
Traceback (most recent call last):
  File "./clipnorm-bug.py", line 20, in <module>
    model.fit(X, Y, nb_epoch = 1)
  File "/usr/local/lib/python3.5/dist-packages/keras/engine/training.py", line 1079, in fit
    self._make_train_function()
  File "/usr/local/lib/python3.5/dist-packages/keras/engine/training.py", line 696, in _make_train_function
    self.total_loss)
  File "/usr/local/lib/python3.5/dist-packages/keras/optimizers.py", line 379, in get_updates
    grads = self.get_gradients(loss, params)
  File "/usr/local/lib/python3.5/dist-packages/keras/optimizers.py", line 71, in get_gradients
    grads = [clip_norm(g, self.clipnorm, norm) for g in grads]
  File "/usr/local/lib/python3.5/dist-packages/keras/optimizers.py", line 71, in <listcomp>
    grads = [clip_norm(g, self.clipnorm, norm) for g in grads]
  File "/usr/local/lib/python3.5/dist-packages/keras/optimizers.py", line 9, in clip_norm
    g = K.switch(n >= c, g * c / n, g)
TypeError: unsupported operand type(s) for *: 'IndexedSlices' and 'float'

Keras version is 1.1.0, TensorFlow is 0.10rc

clipvalue on the other hand works fine.

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Reactions: 9
  • Comments: 15 (10 by maintainers)

Commits related to this issue

Most upvoted comments

Hi, When will this patch be included in the main Keras branch and the pip release? Thanks a lot.

@kracwarlock Sorry about the delay, I didn’t have a free GPU for testing. The latest one works for me as well.

+1, I also got bit by this as well. @kracwarlock i’ll try out your patch in a bit and let you know if it works on my setup, thanks for contributing.

EDIT: it works for me, thanks!

I think I switched back to keras/tf versions where this was working my project does not need this anymore but will try to write a fix if I get time

@kracwarlock Did you manage to find a solution?

@fchollet - This seems like a serious issue regarding clipnorm as it’s very useful for training RNN’s, which often employs an Embedding layer. Could you give some guidance on how to fix this issue?

I too just ran into this issue with clipnorm and embeddings. I also confirm that clipvalue does work.

The way tensorflow handles scalar multiplication with IndexedSlices is: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/math_ops.py#L458-L467

@fchollet Should we convert the scalar to tensor

I can make a PR on the desired behavior