faster-whisper: major slowdown with batching commit - cpu only
Hey there, looking for advice on how to debug.
Trialling the latest commit (#856) on my existing script, I can see a roughly 8x slowdown compared to the previous commit.
Not using new BatchedPipeline, just original transcribe generator, same parameters, same audio, same cpu (cpu only, no gpu). I load the model once, then run transcribe 30s chunks in a numpy ndarray, as audio comes from a livestream.
Any pointers on how to further pinpoint the slowdown to help debug?
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 22 (6 by maintainers)
This toxic behaviour is not getting you anything good. If you have few cores and that PR slowed you down, discuss it with the developers involved. Calm down and realize you’re on this planet with other people which are not offending you full throttle. You’re free to switch to adult behaviour and discuss politely, or to open a fork.
Okay, I see that CTranslate2 has an unorthodox way of setting the threads. Looking at their codebase, they seem to be setting a default value of 4 (or less looking at the actual number of CPU threads) if
OMP_NUM_THREADSenvironment value is not set and thecpu_threadsis set to 0. Other tools like torch and numpy would always try to use the number of CPU threads available in the machine unless you setMKL_NUM_THREADS = OMP_NUM_THREADS = Nin your environment.But its unclear, whether they limit the number of threads set, if it’s manually passed as above, e.g. 16.
Doing a test now on machine with 4 physical cores and 8 threads, on an audio file of 30 seconds, VAD is disabled, verifying core usage using
htop:Results:
Conclusion:
So yes, this is substantially slowing down the transcription on machines that does not have 16 threads. The usual rule of thumb here is to let the toolkit decide the INTER_THREAD parallelism but unfortunately CTranslate2 does not seem to do it but instead rely on a sane default of 4. I then recommend using the
multiprocessing.cpu_count() // 2logic as the default in faster-whisper which is guaranteed to give the best speed for everybody out there. ML pipelines usually never benefits from virtual threads (aka hyper-threading), in this case its actually quite slower than the number of physical cores.If you all agree, I can do a PR to set the default this way.
Any PR as well as any release can have any number of bugs. The authors of the PR are looking into it, meanwhile you can use 1.0.2 or whatever version works well for your system. Bugs and regressions happen everyday, everywhere on github and the wise response is to be helpful to authors, not to offend them. You want faster-whisper to works well on 1-4 cores? Provide feedback, test stuff, be helpful.
I never claimed it was invalid to have 3-4 cores. The Faster-whisper PR conducted several evaluations that confirmed a significant speed-up in general. Batching was a highly requested feature for the Faster-whisper project. You are encouraged to open a PR with alternative solutions. Targeting specific contributors is disrespectful and against our community guidelines.
This is really a shitty answer and your PR really messed up the whisper. It is really valid scenario to use whisper model instances on 3-4 cores, your bullshit PR make it slow down instead of make it faster.
Thanks @ozancaglayan , I think you’ve nailed this particular issue, setting cpu_threads or making that change sees the post-batch version pretty much on par in my tests. Haven’t had time to test other changes but that’s a good start.
Also I don’t cosign the unhinged, rude comments above, very strange.
I created a PR for this now
You still do not get it and you think too simple to relate that to testing and bug fixing. This is normal as you do not see my overall analysis of the PR.
https://github.com/SYSTRAN/faster-whisper/issues/937
Based on the design and implementation nature of the PR, any bug fix will not correct the issues.
No this is not toxic behaviour, this is a strong response to avoid the destruction of the project as the issue is not only the performance of faster-whisper on low-core systems. Faster Whisper is not just a batch-oriented and offline project and this PR and these developers transform the project to it without knowing/respecting project origins and community usage.
How you can claim that is the requested feature for faster-whisper? And is this viable to solve the batching problem like this? Currently batching can be handled by alternative solutions.
What is the general scenarios you mention that speeds up the faster whisper? are you sure you are mapping to the all used scenarios ? how can you speak in your limited context?
You need to revert the commit to make any alternative solution to be done on top of it. Regarding PR and the things done are bullshit.
Getting faster-whisper’s flexibility and agility and making it something bloated, is not what the faster-whisper community should demand.
Let’s revert the code, and make all efforts to make this PR separate and agile. Please note that making an effort does not make you the owner of the repo.