vllm: [Performance]: bitsandbytes quantization slow

Proposal to improve performance

Improve bitsandbytes quantization inference speed

Report of performance regression

I’m testing llama-3.2-1b on a toy dataset. For offline inference using the LLM class, the original model from Huggingface took 45 seconds but the 4-bit model (both inflight quantized and unsloth quantized) took 71 seconds.

I wonder if I’m not serving the quantized models properly or is it expected that bnb quantization leads to very slow inference speed due to being under optimized at this point.

Misc discussion on performance

No response

Your current environment (if you think it is necessary)

The output of `python collect_env.py`
Can't run this on my work machine.

A100-80GB vllm==0.6.3.post1

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 15 (3 by maintainers)

Most upvoted comments

We also conducted some experiments to test BNB performance (without model loading and profile_run): image

Hi @matthewdouglas Here are the benchmark results

Inflight quantizaion(4 bit)

lQDPJx6RI_xFIv3NAQrNBYywWdF1EPTSWJ0HQwWFVBxkAA_1420_266

8 bit quantization

lQDPKGjc3M8g5r3NAWvNBaCwjBkBJl5E02sHQwxdlM02AA_1440_363

All tests were conducted using f93bf, and the improvements are indeed as @matthewdouglas mentioned. It’s worth upgrading to 0.45.0.

@lance0108 There is no plan at the moment, optimizing for pure throughput with weight-only quantization is a losing battle. The best kernels we have for this are Marlin or Machete, which work with GPTQ/AWQ quantized checkpoints.

llm = LLM(
    model="<local_dir>/Llama-3.2-1B-Instruct",
    dtype=torch.bfloat16,
    quantization="bitsandbytes",
    load_format="bitsandbytes"
)

prompts = ["question1", "question2", ...]
sampling_params = SamplingParams(
    temperature=0,
    max_tokens=256
)
outputs = llm.generate(prompts, samping_params)