faster-whisper: (WSL2) RuntimeError: CUDA failed with error out of memory

I have seen some other talk of memory leaks (#390), but I’m having a more sporadic, shorter term issue.

I’ve experienced this on both an RTX 4070 with 12GB VRAM and an RTX 3090 with 24GB VRAM.

Traceback (most recent call last):
  File "/home/coder/.local/bin/whisper-ctranslate2", line 8, in <module>
    sys.exit(main())
  File "/home/coder/.local/lib/python3.8/site-packages/src/whisper_ctranslate2/whisper_ctranslate2.py", line 527, in main
    result = transcribe.inference(
  File "/home/coder/.local/lib/python3.8/site-packages/src/whisper_ctranslate2/transcribe.py", line 164, in inference
    for segment in segments:
  File "/home/coder/.local/lib/python3.8/site-packages/faster_whisper/transcribe.py", line 384, in generate_segments
    ) = self.generate_with_fallback(encoder_output, prompt, tokenizer, options)
  File "/home/coder/.local/lib/python3.8/site-packages/faster_whisper/transcribe.py", line 584, in generate_with_fallback
    result = self.model.generate(
RuntimeError: CUDA failed with error out of memory

Sometimes, while transcribing, I will be using well under half of the GPU’s VRAM, and whisper-ctranslate2 will suddenly fail with this error message. I suppose it is attempting to make a single, super large allocation, but it is unclear why.

The traceback is pointing at faster_whisper, which is why I opened the issue here, and not on whisper-ctranslate2, but I suppose the issue could be over there somehow.

I have used Whisper a lot since it was released, and I have used it via the reference implementation, this implementation, and whisper.cpp. This implementation blows the other two out of the water in terms of performance, but it is frustrating to see it fail during transcription for no reason at all.

Neither of the other two Whisper implementations that I’ve used have ever had this problem even a single time, as best as I can recall.

For additional context, I’ve been doing all of this under WSL2 under Windows 11. The RAM usage of even the large-v2 model is nowhere near enough to justify running out of VRAM even on a 12GB card, especially when Windows shows that the VRAM never went above like 5GB.

This issue occurs sporadically. I can sometimes get lucky and process many hours of audio without this happening. Other times, a particular file will trip it up nearly continuously.

Today, I was transcribing this video since I like to have better subtitles when watching long form technical content. On one particularly bad attempt, the medium.en model crashed after transcribing the first 2 minutes of the video. On other attempts, I’ve seen the large-v2 model get through maybe half of the video before crashing. It’s not consistent, and I’m not doing anything else of any note on this computer when these issues occur. I was able to use faster_whisper on the small.en model on this video fine. After a few attempts, medium.en made it through without crashing.

As previously stated, I can use the GPU to transcribe this kind of content with any other implementation, and I don’t run into the issue. I’ve also used plenty of other large ML models on this computer just fine, including llama2, SDXL, and more. There’s nothing obviously wrong with my computer or software.

I haven’t tested any other faster-whisper applications yet, but I’ve been intending to try whisperX, which looks promising. Unfortunately, this issue is sporadic, as I’ve said, so unless whisperX happened to crash immediately under testing, it would be very hard to know for sure whether the issue is absent there.

I really don’t understand why this is happening, and why no one else is talking about this (which means they may not be encountering it, I suppose). There could be something wrong with my computer, but I have no clue how that would manifest in this one specific way.

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Comments: 22 (7 by maintainers)

Most upvoted comments

I have the same problem running it with wsl2 but it works if i use the whisper-standalone-win (windows exe) on the same machine. So seems a WSL2 problem as some of you already said.

in my case:

Traceback (most recent call last):
  File "/root/infer.py", line 6, in <module>
    for segment in segments:
  File "/usr/local/lib/python3.10/dist-packages/faster_whisper/transcribe.py", line 1189, in restore_speech_timestamps
    for segment in segments:
  File "/usr/local/lib/python3.10/dist-packages/faster_whisper/transcribe.py", line 594, in generate_segments
    ) = self.generate_with_fallback(encoder_output, prompt, tokenizer, options)
  File "/usr/local/lib/python3.10/dist-packages/faster_whisper/transcribe.py", line 884, in generate_with_fallback
    result = self.model.generate(
RuntimeError: CUDA failed with error out of memory

I also got the same problem by using the https://github.com/fedirz/faster-whisper-server docker on WSL2 with various models and different options

I’m seeing this issue, too. I think it’s a WSL2 problem, so I posted it on their repo: https://github.com/microsoft/WSL/issues/8447#issuecomment-2334035165