TensorFlowASR: OOM when training with train_conformer or train_subword_conformer
Hi, I’m trying to train conformer for the Lithuanian language.
I’m training on NVIDIA Tesla P100 and I keep getting OOM after some time.
Using all default values from examples/conformer/config.yml.
Using a custom audio dataset with wav audio files up to 10 seconds. At first tried with up to 30, then up to 15.
What could I do differently?
[Train] [Epoch 1/20] |[32m▏ [39m| 1316/181580 [23:54<42:25:21, 1.18batch/s, transducer_loss=110.57875][0m[0;31m
[Train] [Epoch 1/20] |[32m▏ [39m| 1316/181580 [23:54<42:25:21, 1.18batch/s, transducer_loss=110.32569][0m[0;31m2020-12-05 14:53:17.261450: W tensorflow/core/common_runtime/bfc_allocator.cc:431] Allocator (GPU_0_bfc) ran out of memory trying to allocate 4.42GiB (rounded to 4746649600)requested by op StatefulPartitionedCall/gradient_tape/conformer/conformer_joint/conformer_joint_vocab/Tensordot/MatMul/MatMul
Current allocation summary follows.
[0m[0;31m2020-12-05 14:53:17.263848: W tensorflow/core/common_runtime/bfc_allocator.cc:439] *****************************************************************************************__*****____
2020-12-05 14:53:17.263895: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at matmul_op.cc:481 : Resource exhausted: OOM when allocating tensor with shape[3708320,320] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[0m[0;31mTraceback (most recent call last):
File "examples/conformer/train_conformer.py", line 140, in <module>
[0m[0;31m [0m[0;31mconformer_trainer.fit(train_dataset, eval_dataset, train_bs=args.tbs, eval_bs=args.ebs)
[0m[0;31m File "/root/.cache/pypoetry/virtualenvs/tf-asr-FRX1YIGf-py3.6/lib/python3.6/site-packages/tensorflow_asr/runners/base_runners.py", line 312, in fit
[0m[0;31m self.run()
File "/root/.cache/pypoetry/virtualenvs/tf-asr-FRX1YIGf-py3.6/lib/python3.6/site-packages/tensorflow_asr/runners/base_runners.py", line 192, in run
[0m[0;31m self._train_epoch()
File "/root/.cache/pypoetry/virtualenvs/tf-asr-FRX1YIGf-py3.6/lib/python3.6/site-packages/tensorflow_asr/runners/base_runners.py", line 213, in _train_epoch
[0m[0;31m raise e
[0m[0;31m File "/root/.cache/pypoetry/virtualenvs/tf-asr-FRX1YIGf-py3.6/lib/python3.6/site-packages/tensorflow_asr/runners/base_runners.py", line 207, in _train_epoch
[0m[0;31m self._train_function(train_iterator) # Run train step
File "/root/.cache/pypoetry/virtualenvs/tf-asr-FRX1YIGf-py3.6/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 780, in __call__
[0m[0;31m [0m[0;31mresult = self._call(*args, **kwds)[0m[0;31m
[0m[0;31m File "/root/.cache/pypoetry/virtualenvs/tf-asr-FRX1YIGf-py3.6/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 814, in _call
[0m[0;31m results = self._stateful_fn(*args, **kwds)
File "/root/.cache/pypoetry/virtualenvs/tf-asr-FRX1YIGf-py3.6/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2829, in __call__
[0m[0;31m return graph_function._filtered_call(args, kwargs) # pylint: disable=protected-access[0m[0;31m
[0m[0;31m File "/root/.cache/pypoetry/virtualenvs/tf-asr-FRX1YIGf-py3.6/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1848, in _filtered_call
[0m[0;31m [0m[0;31mcancellation_manager=cancellation_manager)[0m[0;31m
[0m[0;31m File "/root/.cache/pypoetry/virtualenvs/tf-asr-FRX1YIGf-py3.6/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1924, in _call_flat
[0m[0;31m ctx, args, cancellation_manager=cancellation_manager))
File "/root/.cache/pypoetry/virtualenvs/tf-asr-FRX1YIGf-py3.6/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 550, in call
[0m[0;31m [0m[0;31mctx=ctx)
[0m[0;31m File "/root/.cache/pypoetry/virtualenvs/tf-asr-FRX1YIGf-py3.6/lib/python3.6/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
[0m[0;31m [0m[0;31minputs, attrs, num_outputs)[0m[0;31m
[0m[0;31mtensorflow.python.framework.errors_impl.ResourceExhaustedError[0m[0;31m: OOM when allocating tensor with shape[3708320,320] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node StatefulPartitionedCall/gradient_tape/conformer/conformer_joint/conformer_joint_vocab/Tensordot/MatMul/MatMul}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[Op:__inference__train_function_69904]
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 17
With gradient accumulation, it worked fine.
Reduced BS to 2. So far so good. Since I have just a single GPU it will take forever 😃