vllm: Run local llm qwen-72b failed
When I run local llm qwen-72b-chat, it can`t work normal, it will be stuck like this:
Command
python -m vllm.entrypoints.openai.api_server --model /mnt/bynas/llms_store/Qwen-72B-Chat --trust-remote-code --tensor-parallel-size 4 --host 0.0.0.0 --port 5000 \
--served-model-name Qwen/Qwen-72B-Chat --gpu-memory-utilization 0.5
Environment 4 * Nvidia A6000 48G cuda 12.3 python 3.11
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 15 (3 by maintainers)
Have you fixed this problem? I have same problem on 8*A100 80G after initializing llm engine and it stucks. @thugbobby