vllm: Unable to specify GPU usage in VLLM code
I am facing difficulties in specifying GPU usage for different models for LLM inference pipeline using vLLM. Specifically, I have 4 RTX 4090 GPUs available, and I aim to run a LLM with a size of 42GB on 2 RTX 4090 GPUs (~48GB) and a separate model with a size of 22GB on 1 RTX 4090 GPU(`24GB). This is my code for running 42GB model on two GPUs.
from vllm import LLM
llm = LLM(model_name, max_model_len=50, tensor_parallel_size=2)
output = llm.generate(text)
However, I haven’t found a straightforward method within the VLLM library to specify which GPU should be used for each model.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 3
- Comments: 17
I’ve had your exact same scenario, my solution was to run on docker-compose, because in there you can specify which GPU ids to make available to each instance
@simon-mo Separatly they work but my goal is to run two different LLMs. One LLM on 2 GPUs and Second LLM on 3rd GPU
I found that specifying GPU ids for ray-executor could be achieved by modifying
worker_node_and_gpu_idsin vllm/executor/ray_gpu_executor.pyI’m sorry for my late response on the topic, as @sAviOr287 mentioned, there is a ray implementation out there, however I could not find much information about it.
So far my approach to the problem was just using docker and different instances for different models, like so:
It is a load balancing approach, however a different model gets hit depending on which port you are using. My dockerfile is pretty much just installing vllm + some other stuff, however it could be completely replaced with something like the OpenAI implementation vllm has.
@simon-mo
this still loads 2nd llm on 1 and 2 gpu and gives memory error