vllm: [Doc]: Offline Inference Distributed Broken for TP
📚 The doc issue
https://docs.vllm.ai/en/latest/getting_started/examples/offline_inference_distributed.html This document suggests that it can be run vLLM in tensor parallel setting with Ray for use in map_batches but it is not possible currently to run a distributed inference engine with TP > 1.
Related Issue: https://github.com/vllm-project/vllm/issues/3190
Currently there is no way to get around the
ValueError: Ray does not allocate any GPUs on the driver node. Consider adjusting the Ray placement group or running the driver on a GPU node
Even if you put GPUs on the head node. The only way I can imagine this works right now is with a head group only Ray cluster which is unusable for production use cases.
This may not be present in earlier versions of vLLM (these docs were added in 0.3.1 trying to confirm that version works for TP on Ray)
Suggest a potential alternative/fix
Unsure. Really looking for advice on how to do this from @zhuohan123 who has worked on the GPU Ray Executor and @c21 who submitted the example.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 15 (1 by maintainers)
@rkooo567 I wasn’t able to get this to work on earlier versions, so I don’t think it has ever worked (but I could be wrong).
@nkwangleiGIT thanks for the tip! I was able to use it for a solution. @sam-h-bean I think I figured out what’s going wrong and how to fix it.
In my case, the
vLLMengine is inside a Ray Actor that is assigned no logical GPUs. This actor (the driver) gets scheduled onto some arbitrary node, while thevLLMworkers that are children of the driver actor get scheduled onto a node that has GPUs, as they require one logical GPU per worker. In this case, no worker shares the same IP address as the driver and vLLM fails.To fix it, you need to ensure the driver gets scheduled onto the same node as at least one of the workers.
The solution that worked for me was to do this:
PlacementGroupSchedulingStrategywith pack=STRICT_PACKto ensure all bundles in the placement group would be placed on the same physical node andplacement_group_capture_child_tasks=Trueto ensure that all child tasks would use the placement group I created.I had to use this solution because the
create_engine_configdoes not use theplacement_groupvariable — it is not defined onEngineArgseven though it appears inParallelConfig.I’ve posted some example code at the bottom (my setup is a little different than yours). But what you want to make sure is that the call to
LLM()happens inside of a ray actor that is assigned a placement group as above. You can do this, like I did, by wrapping theLLMclass inside aray.remotedecorator.