vllm: [Bug]: Can't serve on ray cluster although passing VLLM_HOST_IP
Your current environment
vLLM API server version 0.7.2
π Describe the bug
I create a cluster with two instance each with 1 GPU.
- head
RAY_num_heartbeats_timeout=600 ray start --head --node-ip-address HEAD-IP \
--port 6379 \
--ray-client-server-port 10001 \
--object-manager-port=8076 \
--node-manager-port=8077
--------------------
Ray runtime started.
--------------------
Next steps
To add another node to this Ray cluster, run
ray start --address='HEAD-IP:6379'
- worker
ray start --object-manager-port=8076 \
--address='HEAD-IP:6379' \
--node-manager-port=8077
-
serve model with two option, first only send
--tensor-parallel-size 1 --pipeline-parallel-size 2and second time with--tensor-parallel-size 2and with both I have the following error: -
Error:
2025-02-19 07:41:17,494 INFO worker.py:1654 -- Connecting to existing Ray cluster at address: HEAD-IP:6379...
2025-02-19 07:41:17,507 INFO worker.py:1832 -- Connected to Ray cluster. View the dashboard at 127.0.0.1:8265
(autoscaler +18s) Tip: use `ray status` to view detailed cluster status. To disable these messages, set RAY_SCHEDULER_EVENTS=0.
(autoscaler +18s) Error: No available node types can fulfill resource request {'node:HEAD-IP:6379': 0.001, 'GPU': 1.0}. Add suitable node types to this cluster to resolve this issue.
- commands:
VLLM_HOST_IP=HEAD-IP:6379 vllm serve NousResearch/Meta-Llama-3.1-8B-Instruct --max-model-len 8192 --gpu-memory-utilization 0.8 \
--tensor-parallel-size 1 --pipeline-parallel-size 2 --distributed-executor-backend ray
VLLM_HOST_IP=HEAD-IP:6379 vllm serve NousResearch/Meta-Llama-3.1-8B-Instruct --max-model-len 8192 --gpu-memory-utilization 0.8 \
--tensor-parallel-size 2 --distributed-executor-backend ray
Before submitting a new issueβ¦
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 26 (5 by maintainers)
Also cc @richardliaw @kouroshHakha for Ray Serve LLM related issues above.
@youkaichao I used VLLM_HOST_IP by only the head IP, but it also did not work