vllm: [Bug]: Can't serve mistral3.1-AWQ on 24G GPU

Your current environment

vllm 0.8.1

πŸ› Describe the bug

Can I serve mistral3.1-awq (OPEA/Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-awq-sym )model on A10 GPU (24G)?

I get OOM for the following command:

  1. command
HF_TOKEN=XX VLLM_USE_V1=0 vllm serve OPEA/Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-awq-sym  --dtype half  \
--tensor-parallel-size 1  --max-model-len 4096 --gpu-memory-utilization 0.95   --distributed-executor-backend mp \  
--served-model-name mistral --tokenizer-mode mistral --config-format mistral --load_format mistral  \
--hf-config-path mistralai/Mistral-Small-3.1-24B-Instruct-2503   --tokenizer mistralai/Mistral-Small-3.1-24B-Instruct-2503  

1.1. error:

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 320.00 MiB. GPU 0 has a total capacity of 21.99 GiB of which 19.38 MiB is free. Including non-PyTorch memory, this process has 21.96 GiB memory in use. Of the allocated memory 21.67 GiB is allocated by PyTorch, and 5.59 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

I tried to pass awq to the quantization argument --quantization awq, but I get this error:

 quant_config = get_quant_config(model_config, load_config)
  File "/lib/python3.9/site-packages/vllm/model_executor/model_loader/weight_utils.py", line 196, in get_quant_config
    raise ValueError(
ValueError: Cannot find the config file for awq

Do I miss anything here?

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 32 (14 by maintainers)

Most upvoted comments

Not sure why. Can you comment on #15664?

Support for HF format Mistral3 was only released in vLLM v0.8.3

How to load it in quantization form using vllm command?

VLLM_USE_V1=0 vllm serve OPEA/Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-awq-sym \
  --max-model-len 128 \
  --gpu-memory-utilization 0.95 \
  --served-model-name mistral