vllm: [Doc]: Offline Inference Distributed Broken for TP

📚 The doc issue

https://docs.vllm.ai/en/latest/getting_started/examples/offline_inference_distributed.html This document suggests that it can be run vLLM in tensor parallel setting with Ray for use in map_batches but it is not possible currently to run a distributed inference engine with TP > 1.

Related Issue: https://github.com/vllm-project/vllm/issues/3190

Currently there is no way to get around the

ValueError: Ray does not allocate any GPUs on the driver node. Consider adjusting the Ray placement group or running the driver on a GPU node

Even if you put GPUs on the head node. The only way I can imagine this works right now is with a head group only Ray cluster which is unusable for production use cases.

This may not be present in earlier versions of vLLM (these docs were added in 0.3.1 trying to confirm that version works for TP on Ray)

Suggest a potential alternative/fix

Unsure. Really looking for advice on how to do this from @zhuohan123 who has worked on the GPU Ray Executor and @c21 who submitted the example.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 15 (1 by maintainers)

Most upvoted comments

@rkooo567 I wasn’t able to get this to work on earlier versions, so I don’t think it has ever worked (but I could be wrong).

@nkwangleiGIT thanks for the tip! I was able to use it for a solution. @sam-h-bean I think I figured out what’s going wrong and how to fix it.

In my case, the vLLM engine is inside a Ray Actor that is assigned no logical GPUs. This actor (the driver) gets scheduled onto some arbitrary node, while the vLLM workers that are children of the driver actor get scheduled onto a node that has GPUs, as they require one logical GPU per worker. In this case, no worker shares the same IP address as the driver and vLLM fails.

To fix it, you need to ensure the driver gets scheduled onto the same node as at least one of the workers.

The solution that worked for me was to do this:

  1. Manually calculate a placement group based on the tensor parallelism and pipeline parallelism.
  2. Use a PlacementGroupSchedulingStrategy with pack=STRICT_PACK to ensure all bundles in the placement group would be placed on the same physical node and placement_group_capture_child_tasks=True to ensure that all child tasks would use the placement group I created.
  3. Instantiate vLLM Engine inside an actor that is assigned the placement group, so that the vLLM engine and all its workers are forced to use this placement group and get scheduled onto the same physical node.

I had to use this solution because the create_engine_config does not use the placement_group variable — it is not defined on EngineArgs even though it appears in ParallelConfig.

I’ve posted some example code at the bottom (my setup is a little different than yours). But what you want to make sure is that the call to LLM() happens inside of a ray actor that is assigned a placement group as above. You can do this, like I did, by wrapping the LLM class inside a ray.remote decorator.

from ray.util.placement_group import (
    placement_group,
    placement_group_table,
    remove_placement_group,
)
from ray.util.scheduling_strategies import PlacementGroupSchedulingStrategy
import ray
from vllm import LLM

ray.init(address="auto")

# Create a placement group with 4 GPUs and at least 1 CPU.
# (assuming a tensor parallelism of 4 and pipeline parallelism of 1)
pg = placement_group(
    [
        {"GPU": 1, "CPU": 1},
        {"GPU": 1},
        {"GPU": 1},
        {"GPU": 1},
    ],
    # Ensure all bundles get placed onto the same physical node.
    strategy="STRICT_PACK",
)
# Block until the placement group is ready.
ray.get(pg.ready(), timeout=10)
# Placement group is ready.
llm_constructor = ray.remote(LLM).options(
    num_cpus=1, # Required so that the placement group is not ignored.
    scheduling_strategy=PlacementGroupSchedulingStrategy(
        # IMPORTANT: This will ensure that all child tasks spawned by
        # this actor (so all of VLLMs RayGpuExecutors) will scheduled
        # occupy resources of this placement group.
        placement_group_capture_child_tasks=True,
        placement_group=pg,
    ),
)
llm = llm_constructor.remote(
    model=...,
    tensor_parallel_size=4,
    worker_use_ray=True 
)