vllm: [Bug]: Can't serve mistral3.1-AWQ on 24G GPU
Your current environment
vllm 0.8.1
π Describe the bug
Can I serve mistral3.1-awq (OPEA/Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-awq-sym )model on A10 GPU (24G)?
I get OOM for the following command:
- command
HF_TOKEN=XX VLLM_USE_V1=0 vllm serve OPEA/Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-awq-sym --dtype half \
--tensor-parallel-size 1 --max-model-len 4096 --gpu-memory-utilization 0.95 --distributed-executor-backend mp \
--served-model-name mistral --tokenizer-mode mistral --config-format mistral --load_format mistral \
--hf-config-path mistralai/Mistral-Small-3.1-24B-Instruct-2503 --tokenizer mistralai/Mistral-Small-3.1-24B-Instruct-2503
1.1. error:
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 320.00 MiB. GPU 0 has a total capacity of 21.99 GiB of which 19.38 MiB is free. Including non-PyTorch memory, this process has 21.96 GiB memory in use. Of the allocated memory 21.67 GiB is allocated by PyTorch, and 5.59 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
I tried to pass awq to the quantization argument --quantization awq, but I get this error:
quant_config = get_quant_config(model_config, load_config)
File "/lib/python3.9/site-packages/vllm/model_executor/model_loader/weight_utils.py", line 196, in get_quant_config
raise ValueError(
ValueError: Cannot find the config file for awq
Do I miss anything here?
Before submitting a new issueβ¦
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 32 (14 by maintainers)
Not sure why. Can you comment on #15664?
Support for HF format Mistral3 was only released in vLLM v0.8.3