TensorFlowASR: OOM when training with train_conformer or train_subword_conformer

Hi, I’m trying to train conformer for the Lithuanian language.

I’m training on NVIDIA Tesla P100 and I keep getting OOM after some time. Using all default values from examples/conformer/config.yml.

Using a custom audio dataset with wav audio files up to 10 seconds. At first tried with up to 30, then up to 15.

What could I do differently?

[Train] [Epoch 1/20] |▏                   | 1316/181580 [23:54<42:25:21,  1.18batch/s, transducer_loss=110.57875]
[Train] [Epoch 1/20] |▏                   | 1316/181580 [23:54<42:25:21,  1.18batch/s, transducer_loss=110.32569]2020-12-05 14:53:17.261450: W tensorflow/core/common_runtime/bfc_allocator.cc:431] Allocator (GPU_0_bfc) ran out of memory trying to allocate 4.42GiB (rounded to 4746649600)requested by op StatefulPartitionedCall/gradient_tape/conformer/conformer_joint/conformer_joint_vocab/Tensordot/MatMul/MatMul
Current allocation summary follows.
2020-12-05 14:53:17.263848: W tensorflow/core/common_runtime/bfc_allocator.cc:439] *****************************************************************************************__*****____
2020-12-05 14:53:17.263895: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at matmul_op.cc:481 : Resource exhausted: OOM when allocating tensor with shape[3708320,320] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
  File "examples/conformer/train_conformer.py", line 140, in <module>
    conformer_trainer.fit(train_dataset, eval_dataset, train_bs=args.tbs, eval_bs=args.ebs)
  File "/root/.cache/pypoetry/virtualenvs/tf-asr-FRX1YIGf-py3.6/lib/python3.6/site-packages/tensorflow_asr/runners/base_runners.py", line 312, in fit
    self.run()
  File "/root/.cache/pypoetry/virtualenvs/tf-asr-FRX1YIGf-py3.6/lib/python3.6/site-packages/tensorflow_asr/runners/base_runners.py", line 192, in run
    self._train_epoch()
  File "/root/.cache/pypoetry/virtualenvs/tf-asr-FRX1YIGf-py3.6/lib/python3.6/site-packages/tensorflow_asr/runners/base_runners.py", line 213, in _train_epoch
    raise e
  File "/root/.cache/pypoetry/virtualenvs/tf-asr-FRX1YIGf-py3.6/lib/python3.6/site-packages/tensorflow_asr/runners/base_runners.py", line 207, in _train_epoch
    self._train_function(train_iterator)  # Run train step
  File "/root/.cache/pypoetry/virtualenvs/tf-asr-FRX1YIGf-py3.6/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 780, in __call__
    result = self._call(*args, **kwds)
  File "/root/.cache/pypoetry/virtualenvs/tf-asr-FRX1YIGf-py3.6/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 814, in _call
    results = self._stateful_fn(*args, **kwds)
  File "/root/.cache/pypoetry/virtualenvs/tf-asr-FRX1YIGf-py3.6/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2829, in __call__
    return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
  File "/root/.cache/pypoetry/virtualenvs/tf-asr-FRX1YIGf-py3.6/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1848, in _filtered_call
    cancellation_manager=cancellation_manager)
  File "/root/.cache/pypoetry/virtualenvs/tf-asr-FRX1YIGf-py3.6/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1924, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/root/.cache/pypoetry/virtualenvs/tf-asr-FRX1YIGf-py3.6/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 550, in call
    ctx=ctx)
  File "/root/.cache/pypoetry/virtualenvs/tf-asr-FRX1YIGf-py3.6/lib/python3.6/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.ResourceExhaustedError:  OOM when allocating tensor with shape[3708320,320] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
	 [[{{node StatefulPartitionedCall/gradient_tape/conformer/conformer_joint/conformer_joint_vocab/Tensordot/MatMul/MatMul}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
 [Op:__inference__train_function_69904]

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 17

Most upvoted comments

With gradient accumulation, it worked fine.

Reduced BS to 2. So far so good. Since I have just a single GPU it will take forever 😃