vllm: [Bug]: Error: No available node types can fulfill resource request
Your current environment
The output of `python collect_env.py`
Your output of `python collect_env.py` here
π Describe the bug
Iβm testing a Llama3.1-70B on two nodes with 2x8 L4 GPUs. It works fine with v0.5.4.
python -m vllm.entrypoints.openai.api_server --model /secondary/thies/Hermes-3-Llama-3.1-70B/ --tensor-parallel-size 16 --max-num-batched-tokens 8192 --gpu-memory-utilization 0.85 --distributed-executor-backend=ray
Now when I upgrade to master I get this error:
(autoscaler +7s) Error: No available node types can fulfill resource request {'node:<IP>': 0.001, 'GPU': 1.0}. Add suitable node types to this cluster to resolve this issue.
INFO 08-23 10:16:22 ray_utils.py:174] Waiting for creating a placement group of specs for 10 seconds. specs=[{'node:<IP>': 0.001, 'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}]. Check `ray status` to see if you have enough resources.
ray status says:
Node status
---------------------------------------------------------------
Active:
1 node_fd78035d54d28a1357f3b6d412a3a6d11595cdb60f962e3b6ba51777
1 node_493f441c8cd5c582361ee8634f5cd1c2cb3413983a27ae9a083d1d55
Pending:
(no pending nodes)
Recent failures:
(no failures)
Resources
---------------------------------------------------------------
Usage:
0.0/128.0 CPU
0.0/16.0 GPU
0B/1.59TiB memory
0B/372.53GiB object_store_memory
Demands:
(no resource demands)```
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 17 (2 by maintainers)
Commits related to this issue
- [doc][misc] clarify VLLM_HOST_IP for multi-node inference (#12667) As more and more people are trying deepseek models with multi-node inference, https://github.com/vllm-project/vllm/issues/7815 beco... — committed to vllm-project/vllm by youkaichao 2 years ago
- [doc][misc] clarify VLLM_HOST_IP for multi-node inference (#12667) As more and more people are trying deepseek models with multi-node inference, https://github.com/vllm-project/vllm/issues/7815 becom... — committed to srikanthsrnvs/vllm by youkaichao 2 years ago
- [doc][misc] clarify VLLM_HOST_IP for multi-node inference (#12667) As more and more people are trying deepseek models with multi-node inference, https://github.com/vllm-project/vllm/issues/7815 beco... — committed to krai/vllm by youkaichao 2 years ago
- [doc][misc] clarify VLLM_HOST_IP for multi-node inference (#12667) As more and more people are trying deepseek models with multi-node inference, https://github.com/vllm-project/vllm/issues/7815 beco... — committed to GWS0428/VARserve by youkaichao 2 years ago
- [doc][misc] clarify VLLM_HOST_IP for multi-node inference (#12667) As more and more people are trying deepseek models with multi-node inference, https://github.com/vllm-project/vllm/issues/7815 beco... — committed to shreyankg/vllm by youkaichao 2 years ago
- [doc][misc] clarify VLLM_HOST_IP for multi-node inference (#12667) As more and more people are trying deepseek models with multi-node inference, https://github.com/vllm-project/vllm/issues/7815 beco... — committed to kon-pap/vllm by youkaichao 2 years ago
Okay, the outbound IP addresses of my two containers are not compatible. Thank you for your replyοΌ
https://github.com/vllm-project/vllm/blob/80c751e7f68ade3d4c6391a0f3fce9ce970ddad0/vllm/v1/executor/ray_utils.py#L250 This line is the issue. There could be a mismatch with the actual IP address used to setup ray. We can use
VLLM_HOST_IPto override the IP.