vllm: [Performance]: bitsandbytes quantization slow
Proposal to improve performance
Improve bitsandbytes quantization inference speed
Report of performance regression
I’m testing llama-3.2-1b on a toy dataset. For offline inference using the LLM class, the original model from Huggingface took 45 seconds but the 4-bit model (both inflight quantized and unsloth quantized) took 71 seconds.
I wonder if I’m not serving the quantized models properly or is it expected that bnb quantization leads to very slow inference speed due to being under optimized at this point.
Misc discussion on performance
No response
Your current environment (if you think it is necessary)
The output of `python collect_env.py`
Can't run this on my work machine.
A100-80GB vllm==0.6.3.post1
Before submitting a new issue…
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 15 (3 by maintainers)
We also conducted some experiments to test BNB performance (without
model loadingandprofile_run):Hi @matthewdouglas Here are the benchmark results
Inflight quantizaion(4 bit)
8 bit quantization
All tests were conducted using f93bf, and the improvements are indeed as @matthewdouglas mentioned. It’s worth upgrading to
0.45.0.@lance0108 There is no plan at the moment, optimizing for pure throughput with weight-only quantization is a losing battle. The best kernels we have for this are Marlin or Machete, which work with GPTQ/AWQ quantized checkpoints.