- docker image
vllm/vllm-openai:v0.2.7, startup command:
docker run -it -d -p 5003:8000 \
--name vllm-api \
-v $(pwd)/huggingface:/root/.cache/huggingface \
-v $(pwd)/Qwen-14B-Chat-Int4:/workspace/qwen \
--runtime nvidia --gpus all \
--ipc=host \
vllm/vllm-openai:v0.2.7 \
--trust-remote-code \
--dtype auto \
--tensor-parallel-size 2 \
--quantization gptq \
--model qwen
- The graphics card is
NVIDIA GeForce RTX 4090 24G*2
- Use
Qwen-7B-Chat-Int4works properly, UseQwen-14B-Chat-Int4error:
INFO 02-01 01:41:59 api_server.py:727] args: Namespace(host=None, port=8000, allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], served_model_name=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, model='qwen', tokenizer=None, revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=True, download_dir=None, load_format='auto', dtype='auto', max_model_len=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=2, max_parallel_loading_workers=None, block_size=16, seed=0, swap_space=4, gpu_memory_utilization=0.9, max_num_batched_tokens=None, max_num_seqs=256, max_paddings=256, disable_log_stats=False, quantization='gptq', enforce_eager=False, max_context_len_to_capture=8192, engine_use_ray=False, disable_log_requests=False, max_log_len=None)
WARNING 02-01 01:41:59 config.py:175] gptq quantization is not fully optimized yet. The speed can be slower than non-quantized models.
2024-02-01 01:42:02,376 INFO worker.py:1724 -- Started a local Ray instance.
INFO 02-01 01:42:04 llm_engine.py:70] Initializing an LLM engine with config: model='qwen', tokenizer='qwen', tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=2048, download_dir=None, load_format=auto, tensor_parallel_size=2, quantization=gptq, enforce_eager=False, seed=0)
WARNING 02-01 01:42:05 tokenizer.py:62] Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/workspace/vllm/entrypoints/openai/api_server.py", line 737, in <module>
engine = AsyncLLMEngine.from_engine_args(engine_args)
File "/workspace/vllm/engine/async_llm_engine.py", line 500, in from_engine_args
engine = cls(parallel_config.worker_use_ray,
File "/workspace/vllm/engine/async_llm_engine.py", line 273, in __init__
self.engine = self._init_engine(*args, **kwargs)
File "/workspace/vllm/engine/async_llm_engine.py", line 318, in _init_engine
return engine_class(*args, **kwargs)
File "/workspace/vllm/engine/llm_engine.py", line 109, in __init__
self._init_workers_ray(placement_group)
File "/workspace/vllm/engine/llm_engine.py", line 249, in _init_workers_ray
self._run_workers(
File "/workspace/vllm/engine/llm_engine.py", line 795, in _run_workers
driver_worker_output = getattr(self.driver_worker,
File "/workspace/vllm/worker/worker.py", line 81, in load_model
self.model_runner.load_model()
File "/workspace/vllm/worker/model_runner.py", line 64, in load_model
self.model = get_model(self.model_config)
File "/workspace/vllm/model_executor/model_loader.py", line 65, in get_model
model = model_class(model_config.hf_config, linear_method)
File "/workspace/vllm/model_executor/models/qwen.py", line 231, in __init__
self.transformer = QWenModel(config, linear_method)
File "/workspace/vllm/model_executor/models/qwen.py", line 193, in __init__
self.h = nn.ModuleList([
File "/workspace/vllm/model_executor/models/qwen.py", line 194, in <listcomp>
QWenBlock(config, linear_method)
File "/workspace/vllm/model_executor/models/qwen.py", line 147, in __init__
self.mlp = QWenMLP(config.hidden_size,
File "/workspace/vllm/model_executor/models/qwen.py", line 49, in __init__
self.c_proj = RowParallelLinear(intermediate_size,
File "/workspace/vllm/model_executor/layers/linear.py", line 495, in __init__
self.linear_weights = self.linear_method.create_weights(
File "/workspace/vllm/model_executor/layers/quantization/gptq.py", line 100, in create_weights
raise ValueError(
ValueError: The input size is not aligned with the quantized weight shape. This can be caused by too large tensor parallel size.
input_size_per_partitionvalue is6848, self.quant_config.group_sizevalue is128
- Using a GPU
--tensor-parallel-size 1works properly,but the answer is wrong:
request:
{
"max_tokens": 200,
"model": "qwen",
"stream": false,
"stop": ["<|endoftext|>","<|im_end|>","<|im_start|>"],
"messages": [
{
"role": "user",
"content": "hello"
}
]
}
response:
{
"id": "cmpl-5d4e4ffb826c4ec6972e72453d9698f9",
"object": "chat.completion",
"created": 599739,
"model": "qwen",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "The answer is 1210."
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 9,
"total_tokens": 19,
"completion_tokens": 10
}
}
- Qwen-14B-Chat-Int4 is not supported, or I am using it in a wrong way? Thank you for all your assistance.
the member of QWEN group tell me that the kernel of vllm requires intermediate_size (13696) to be an integer multiple of quantized group_size (128) * TP(tensor-parallel-size), so currently TP can only take 1. In addition, the intermediate_size of Qwen1.5-72B-Chat-GPTQ-Int4 is 24576 and 7B is 11008,u can find this in config.json
Same problem. one GPU ,work fine! two GPU report this error!
This happened to me, too, Same problem for Qwen1.5-14B-Chat-AWQ