tensorflow: Out-of-memory error in eager mode loop

Following code gives an out-of-memory error when running in tensorflow-2.5. The code itself does not do anything useful it is just meant to highlight what I believe is a problem.

For the small problem (i.e. b, a, c = 1, 1000, 5 # small problem) the program completes. For the large problem (i.e. b, a, c = 1, 10000, 5 # large problem) the program runs out of memory.

My expectation would have been that in eager mode allocated tensors have the lifetime of one iteration, while it seems more memory is being allocated with each iteration. Similar code with identical parameters runs in Pytorch.

Am I wrong in my expectation or is this a bug? If I am wrong how could I do this in Tensorflow?

import tensorflow as tf

from tensorflow.keras.losses import Loss

class EagerOom(Loss):

    def __init__(self):
        super(EagerOom, self).__init__()

    def __call__(self, targets, logits):
        logits = tf.reshape(logits, [-1])
        targets = tf.reshape(targets, [-1])
        return self.inner(targets, logits)


    @tf.custom_gradient
    def inner(self, targets, logits):

        g_logits = tf.zeros_like(logits)

        for idx in range(len(targets)):
            print(f"idx: {idx}")
            g_logits_upd = logits - logits[idx]
            g_logits += g_logits_upd
        
        loss = tf.reduce_mean(targets * logits)

        def grad_fn(upstream):
            g_targets = tf.zeros_like(targets)
            return g_targets, g_logits

        return loss, grad_fn



def main():
    tf.random.set_seed(0)
    # b, a, c = 1, 1000, 5 # small problem
    b, a, c = 1, 10000, 5 # large problem
    logits = tf.random.normal((b, a, c))
    targets = tf.random.uniform((b, a), 0, c + 1, dtype=tf.int32)
    targets = tf.one_hot(targets, depth=c, axis=-1)
    loss_fn = EagerOom()
    with tf.GradientTape(persistent=True) as tape:
        tape.watch(logits)
        loss = loss_fn(targets, logits)
    print("Done.")


if __name__ == '__main__':
    main()

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 15 (7 by maintainers)

Most upvoted comments

Hi, jvishnuvardhan@,

Thanks for trying this out again. I think the principal issue still applies –

As surprising as it sounds, even if you do a custom_gradient(), the operations are still recorded on the tape by TensorFlow. This is not documented in custom_gradient()'s docstring. A change is easy to implement (and I poked around with one or two weeks ago) – but our current consensus is that we unlikely have to freedom to change the behavior of custom_gradient due to TF’s backward compatibility promises (e.g. some may rely on automated high order gradients).

As a workaround, you can probably stop the recording by calling tf.stop_gtradient to all arguments of the function with custom_gradient:

    @tf.custom_gradient
    def inner(self, targets, logits):
        targets, logits = tf.stop_gradient([targets, logitcs])
        ...
        return loss, grad_fn