wordcab-transcribe: Error in diarization: CUDA out of memory

Running on Quadro RTX5000

Full error message below:


Traceback (most recent call last): 

File \"/app/wordcab_transcribe/services/asr_service.py\", line 294, in runner
    results = await asyncio.get_event_loop().run_in_executor(
  File \"/usr/lib/python3.8/concurrent/futures/thread.py\", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
 File \"/app/wordcab_transcribe/logging.py\", line 89, in sync_wrapper
    result = func(*args, **kwargs)
  File \"/app/wordcab_transcribe/services/asr_service.py\", line 345, in process_diarization
    utterances = self.services[\"diarization\"](task[\"input\"])
  File \"/app/wordcab_transcribe/logging.py\", line 89, in sync_wrapper
    result = func(*args, **kwargs)
  File \"/app/wordcab_transcribe/services/diarize_service.py\", line 73, in __call__
    self.model.diarize()
  File \"/opt/pysetup/.venv/lib/python3.8/site-packages/torch/utils/_contextlib.py\", line 115, in decorate_context
    return func(*args, **kwargs)
  File \"/opt/pysetup/.venv/lib/python3.8/site-packages/nemo/collections/asr/models/msdd_models.py\", line 1180, in diarize
    self.clustering_embedding.prepare_cluster_embs_infer()
 File \"/opt/pysetup/.venv/lib/python3.8/site-packages/nemo/collections/asr/models/msdd_models.py\", line 699, in prepare_cluster_embs_infer
    self.emb_sess_test_dict, self.emb_seq_test, self.clus_test_label_dict, _ = self.run_clustering_diarizer(
  File \"/opt/pysetup/.venv/lib/python3.8/site-packages/nemo/collections/asr/models/msdd_models.py\", line 866, in run_clustering_diarizer
    scores = self.clus_diar_model.diarize(batch_size=self.cfg_diar_infer.batch_size)
  File \"/opt/pysetup/.venv/lib/python3.8/site-packages/nemo/collections/asr/models/clustering_diarizer.py\", line 456, in diarize
    all_reference, all_hypothesis = perform_clustering(
  File \"/opt/pysetup/.venv/lib/python3.8/site-packages/nemo/collections/asr/parts/utils/speaker_utils.py\", line 486, in perform_clustering
    cluster_labels = speaker_clustering.forward_infer(
  File \"/opt/pysetup/.venv/lib/python3.8/site-packages/nemo/collections/asr/parts/utils/offline_clustering.py\", line 1288, in forward_infer
    mat = getMultiScaleCosAffinityMatrix(
  File \"/opt/pysetup/.venv/lib/python3.8/site-packages/nemo/collections/asr/parts/utils/offline_clustering.py\", line 525, in getMultiScaleCosAffinityMatrix
    score_mat_torch = getCosAffinityMatrix(emb_t)
  File \"/opt/pysetup/.venv/lib/python3.8/site-packages/nemo/collections/asr/parts/utils/offline_clustering.py\", line 439, in getCosAffinityMatrix
    sim_d = ScalerMinMax(sim_d)
  File \"/opt/pysetup/.venv/lib/python3.8/site-packages/nemo/collections/asr/parts/utils/offline_clustering.py\", line 78, in ScalerMinMax
    v_norm = (X - v_min) / (v_max - v_min)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.60 GiB (GPU 0; 15.75 GiB total capacity; 8.37 GiB already allocated; 1.46 GiB free; 9.30 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

I am not sure how much of this is related to this library as opposed to some general CUDA jazz. Have you guys come across this before?

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 17 (9 by maintainers)

Most upvoted comments

The problem has been isolated and understood with PR #180.

I will work on a fix to allow larger audio files in the Clustering step of the diarization process.

It looks like one step inside the complete NeMo process requires VRAM proportionally to the audio duration. This is not the case for previous steps where audio is cut in chunks to avoid CUDA problems, as logged by NeMo.