vllm: Run local llm qwen-72b failed

When I run local llm qwen-72b-chat, it can`t work normal, it will be stuck like this: image Command

python -m vllm.entrypoints.openai.api_server --model /mnt/bynas/llms_store/Qwen-72B-Chat --trust-remote-code --tensor-parallel-size 4 --host 0.0.0.0 --port 5000 \
--served-model-name Qwen/Qwen-72B-Chat --gpu-memory-utilization 0.5

Environment 4 * Nvidia A6000 48G cuda 12.3 python 3.11

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 15 (3 by maintainers)

Most upvoted comments

When I run local llm qwen-72b-chat, it can`t work normal, it will be stuck like this: image Command

python -m vllm.entrypoints.openai.api_server --model /mnt/bynas/llms_store/Qwen-72B-Chat --trust-remote-code --tensor-parallel-size 4 --host 0.0.0.0 --port 5000 \
--served-model-name Qwen/Qwen-72B-Chat --gpu-memory-utilization 0.5

Environment 4 * Nvidia A6000 48G cuda 12.3 python 3.11

Have you fixed this problem? I have same problem on 8*A100 80G after initializing llm engine and it stucks. @thugbobby