wordcab-transcribe: Error in diarization: CUDA out of memory
Running on Quadro RTX5000
Full error message below:
Traceback (most recent call last):
File \"/app/wordcab_transcribe/services/asr_service.py\", line 294, in runner
results = await asyncio.get_event_loop().run_in_executor(
File \"/usr/lib/python3.8/concurrent/futures/thread.py\", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File \"/app/wordcab_transcribe/logging.py\", line 89, in sync_wrapper
result = func(*args, **kwargs)
File \"/app/wordcab_transcribe/services/asr_service.py\", line 345, in process_diarization
utterances = self.services[\"diarization\"](task[\"input\"])
File \"/app/wordcab_transcribe/logging.py\", line 89, in sync_wrapper
result = func(*args, **kwargs)
File \"/app/wordcab_transcribe/services/diarize_service.py\", line 73, in __call__
self.model.diarize()
File \"/opt/pysetup/.venv/lib/python3.8/site-packages/torch/utils/_contextlib.py\", line 115, in decorate_context
return func(*args, **kwargs)
File \"/opt/pysetup/.venv/lib/python3.8/site-packages/nemo/collections/asr/models/msdd_models.py\", line 1180, in diarize
self.clustering_embedding.prepare_cluster_embs_infer()
File \"/opt/pysetup/.venv/lib/python3.8/site-packages/nemo/collections/asr/models/msdd_models.py\", line 699, in prepare_cluster_embs_infer
self.emb_sess_test_dict, self.emb_seq_test, self.clus_test_label_dict, _ = self.run_clustering_diarizer(
File \"/opt/pysetup/.venv/lib/python3.8/site-packages/nemo/collections/asr/models/msdd_models.py\", line 866, in run_clustering_diarizer
scores = self.clus_diar_model.diarize(batch_size=self.cfg_diar_infer.batch_size)
File \"/opt/pysetup/.venv/lib/python3.8/site-packages/nemo/collections/asr/models/clustering_diarizer.py\", line 456, in diarize
all_reference, all_hypothesis = perform_clustering(
File \"/opt/pysetup/.venv/lib/python3.8/site-packages/nemo/collections/asr/parts/utils/speaker_utils.py\", line 486, in perform_clustering
cluster_labels = speaker_clustering.forward_infer(
File \"/opt/pysetup/.venv/lib/python3.8/site-packages/nemo/collections/asr/parts/utils/offline_clustering.py\", line 1288, in forward_infer
mat = getMultiScaleCosAffinityMatrix(
File \"/opt/pysetup/.venv/lib/python3.8/site-packages/nemo/collections/asr/parts/utils/offline_clustering.py\", line 525, in getMultiScaleCosAffinityMatrix
score_mat_torch = getCosAffinityMatrix(emb_t)
File \"/opt/pysetup/.venv/lib/python3.8/site-packages/nemo/collections/asr/parts/utils/offline_clustering.py\", line 439, in getCosAffinityMatrix
sim_d = ScalerMinMax(sim_d)
File \"/opt/pysetup/.venv/lib/python3.8/site-packages/nemo/collections/asr/parts/utils/offline_clustering.py\", line 78, in ScalerMinMax
v_norm = (X - v_min) / (v_max - v_min)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.60 GiB (GPU 0; 15.75 GiB total capacity; 8.37 GiB already allocated; 1.46 GiB free; 9.30 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
I am not sure how much of this is related to this library as opposed to some general CUDA jazz. Have you guys come across this before?
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 17 (9 by maintainers)
The problem has been isolated and understood with PR #180.
I will work on a fix to allow larger audio files in the
Clusteringstep of the diarization process.It looks like one step inside the complete NeMo process requires VRAM proportionally to the audio duration. This is not the case for previous steps where audio is cut in chunks to avoid CUDA problems, as logged by NeMo.