vllm: [Bug]: Error: No available node types can fulfill resource request

Your current environment

The output of `python collect_env.py`
Your output of `python collect_env.py` here

πŸ› Describe the bug

I’m testing a Llama3.1-70B on two nodes with 2x8 L4 GPUs. It works fine with v0.5.4. python -m vllm.entrypoints.openai.api_server --model /secondary/thies/Hermes-3-Llama-3.1-70B/ --tensor-parallel-size 16 --max-num-batched-tokens 8192 --gpu-memory-utilization 0.85 --distributed-executor-backend=ray

Now when I upgrade to master I get this error:

(autoscaler +7s) Error: No available node types can fulfill resource request {'node:<IP>': 0.001, 'GPU': 1.0}. Add suitable node types to this cluster to resolve this issue.
INFO 08-23 10:16:22 ray_utils.py:174] Waiting for creating a placement group of specs for 10 seconds. specs=[{'node:<IP>': 0.001, 'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}, {'GPU': 1.0}]. Check `ray status` to see if you have enough resources.

ray status says:

Node status
---------------------------------------------------------------
Active:
 1 node_fd78035d54d28a1357f3b6d412a3a6d11595cdb60f962e3b6ba51777
 1 node_493f441c8cd5c582361ee8634f5cd1c2cb3413983a27ae9a083d1d55
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/128.0 CPU
 0.0/16.0 GPU
 0B/1.59TiB memory
 0B/372.53GiB object_store_memory

Demands:
 (no resource demands)```

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 17 (2 by maintainers)

Commits related to this issue

Most upvoted comments

Okay, the outbound IP addresses of my two containers are not compatible. Thank you for your reply!

https://github.com/vllm-project/vllm/blob/80c751e7f68ade3d4c6391a0f3fce9ce970ddad0/vllm/v1/executor/ray_utils.py#L250 This line is the issue. There could be a mismatch with the actual IP address used to setup ray. We can use VLLM_HOST_IP to override the IP.

VLLM_HOST_IP=xxx.xxx.xxx.xxx vllm serve deepseek-ai/DeepSeek-V3 ...