tensorflow: Out-of-memory error in eager mode loop
Following code gives an out-of-memory error when running in tensorflow-2.5. The code itself does not do anything useful it is just meant to highlight what I believe is a problem.
For the small problem (i.e. b, a, c = 1, 1000, 5 # small problem) the program completes. For the large problem (i.e. b, a, c = 1, 10000, 5 # large problem) the program runs out of memory.
My expectation would have been that in eager mode allocated tensors have the lifetime of one iteration, while it seems more memory is being allocated with each iteration. Similar code with identical parameters runs in Pytorch.
Am I wrong in my expectation or is this a bug? If I am wrong how could I do this in Tensorflow?
import tensorflow as tf
from tensorflow.keras.losses import Loss
class EagerOom(Loss):
def __init__(self):
super(EagerOom, self).__init__()
def __call__(self, targets, logits):
logits = tf.reshape(logits, [-1])
targets = tf.reshape(targets, [-1])
return self.inner(targets, logits)
@tf.custom_gradient
def inner(self, targets, logits):
g_logits = tf.zeros_like(logits)
for idx in range(len(targets)):
print(f"idx: {idx}")
g_logits_upd = logits - logits[idx]
g_logits += g_logits_upd
loss = tf.reduce_mean(targets * logits)
def grad_fn(upstream):
g_targets = tf.zeros_like(targets)
return g_targets, g_logits
return loss, grad_fn
def main():
tf.random.set_seed(0)
# b, a, c = 1, 1000, 5 # small problem
b, a, c = 1, 10000, 5 # large problem
logits = tf.random.normal((b, a, c))
targets = tf.random.uniform((b, a), 0, c + 1, dtype=tf.int32)
targets = tf.one_hot(targets, depth=c, axis=-1)
loss_fn = EagerOom()
with tf.GradientTape(persistent=True) as tape:
tape.watch(logits)
loss = loss_fn(targets, logits)
print("Done.")
if __name__ == '__main__':
main()
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 15 (7 by maintainers)
Hi, jvishnuvardhan@,
Thanks for trying this out again. I think the principal issue still applies –
As surprising as it sounds, even if you do a custom_gradient(), the operations are still recorded on the tape by TensorFlow. This is not documented in custom_gradient()'s docstring. A change is easy to implement (and I poked around with one or two weeks ago) – but our current consensus is that we unlikely have to freedom to change the behavior of custom_gradient due to TF’s backward compatibility promises (e.g. some may rely on automated high order gradients).
As a workaround, you can probably stop the recording by calling tf.stop_gtradient to all arguments of the function with custom_gradient: