vllm: vllm hangs when reinitializing ray
I’d like to be able to unload a vllm model and re-load it later, in the same script. However, the following (on 0.1.7) causes the script to hang (disclaimer: this isn’t my particular workload, but a minimal reproducible example):
from vllm import LLM, SamplingParams
def process_prompts(prompts):
llm = LLM(
model="meta-llama/Llama-2-70b-chat-hf",
tensor_parallel_size=2,
trust_remote_code=True,
load_format="pt")
sampling_params = SamplingParams(temperature=0.0, top_p=1.0, max_tokens=500)
return llm.generate(prompts, sampling_params)
prompt_batch_1 = ["Hello, my name is", "The president of the United States is"]
prompt_batch_2 = ["The capital of France is", "The future of AI is"]
batch_1_output = process_prompts(prompt_batch_1)
batch_2_output = process_prompts(prompt_batch_2)
Results in:
2023-09-15 11:43:25,943 INFO worker.py:1621 -- Started a local Ray instance.
INFO 09-15 11:43:51 llm_engine.py:72] Initializing an LLM engine with config: model='meta-llama/Llama-2
-70b-chat-hf', tokenizer='meta-llama/Llama-2-70b-chat-hf', tokenizer_mode=auto, trust_remote_code=True,
dtype=torch.float16, download_dir='/scr/biggest/nfliu/cache/huggingface/', load_format=pt, tensor_para
llel_size=2, seed=0)
INFO 09-15 11:43:51 tokenizer.py:30] For some LLaMA-based models, initializing the fast tokenizer may t
ake a long time. To eliminate the initialization time, consider using 'hf-internal-testing/llama-tokeni
zer' instead of the original tokenizer.
INFO 09-15 11:45:58 llm_engine.py:199] # GPU blocks: 2561, # CPU blocks: 1638
Processed prompts: 100%|█████████████████████████████████████████████████| 2/2 [00:14<00:00, 7.17s/it]
2023-09-15 11:46:28,348 INFO worker.py:1453 -- Calling ray.init() again after it has already been called.
Then, it just hangs forever (been waiting 10 minutes, with no sign of life). Checking the GPUs shows that the model is indeed unloaded from the GPUs.
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.54.03 Driver Version: 535.54.03 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA A100-SXM4-80GB On | 00000000:C7:00.0 Off | 0 |
| N/A 30C P0 61W / 350W | 4MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 1 NVIDIA A100-SXM4-80GB On | 00000000:CA:00.0 Off | 0 |
| N/A 31C P0 57W / 350W | 4MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| No running processes found |
+---------------------------------------------------------------------------------------+
I’m fairly sure this is related to ray, since this doesn’t happen if tensor parallelism is set to 1 (e.g., if you’re running a smaller model). When I ctrl+c out of the script after it hangs, it shows that it’s stuck on ray.get(current_placement_group.ready(), timeout=1800) https://github.com/vllm-project/vllm/blob/main/vllm/engine/ray_utils.py#L112C9-L112C63 .
Is there any way to “reset” the ray state, such that it initializes from scratch the second time?
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 3
- Comments: 21 (3 by maintainers)
Commits related to this issue
- Update ray_utils.py FIX #1058 — committed to premsa/vllm by premsa 3 years ago
I encountered the same issue. It runs fine when I use
tensor_parallel_size=1, but it hangs when I usetensor_parallel_size>1. I have tried reinstalling many times but it didn’t help.The final solution for me was to modify the
vllm/engine/ray_utils.pyfile and limit the number of CPUs used. After making this change, it works properly. The modified code is:ray.init(num_cpus=32, num_gpus=4, address=ray_address, ignore_reinit_error=True).Note: I encountered hanging issues while using
tensor_parallel_size>1on a 128-core machine. However, runningtensor_parallel_size>1on a 96-core machine works normallyyou should load model outside the function to keep model only load once
from vllm import LLM, SamplingParams
llm = LLM( model=“meta-llama/Llama-2-70b-chat-hf”, tensor_parallel_size=2, trust_remote_code=True, load_format=“pt”)
def process_prompts(prompts): sampling_params = SamplingParams(temperature=0.0, top_p=1.0, max_tokens=500) return llm.generate(prompts, sampling_params)
prompt_batch_1 = [“Hello, my name is”, “The president of the United States is”] prompt_batch_2 = [“The capital of France is”, “The future of AI is”]
batch_1_output = process_prompts(prompt_batch_1) batch_2_output = process_prompts(prompt_batch_2)
I also fix the problem by setting the ray num cpus to 32.
ray start --head --num-cpus=32It also works when I set cpus to 49 (since I have two pysical cpus, each have 48 cores)TLDR: Don’t set the
num_gpusvalue for vLLM, only settensor_parallel_size.I encountered the same problem, and here’s what I found out:
According to Ray’s documentation, the framework itself will allocate the necessary GPUs (based on
num_gpus) and set theCUDA_VISIBLE_DEVICESvalue. When I started running vLLM through Ray, I found that Ray sets theCUDA_VISIBLE_DEVICESvalue to0,1,2,3(I hadnum_gpus = 4specified), however, when I called nvidia-smi, I found that vLLM uses4,5,6,7. Therefore, vLLM ignores theCUDA_VISIBLE_DEVICESvalue and chooses other devices. In my case, I have 8 GPUs, I allocated 4 for vLLM, and 2 for another model, leaving 2 free. But vLLM requested another 4 GPUs, and since Ray couldn’t satisfy this request, vLLM started waiting for the GPUs to free up. As soon as I removed the second model, which requested 2 GPUs, everything started working. Everything also worked when I allocated 2 GPUs for the vLLM model.If you try to run two identical applications using vLLM through Ray (in one
serve), everything will break. The applications will not use different GPUs, but will start loading data into the memory of the same GPUs, while the other GPUs will be idle. Ultimately, this will lead to OOM. I believe this is related to the incorrect handling of thenum_gpusvalue.In my case I have 4 GPUs and 3 RayServe deployments, 2 of which require 1 logical GPU with tensor_parallelism=1 and another one which requires 2 logical GPUs with tensor_parallelism=2. Looks like when vLLM tries to handle the tensor_parallelism=2 it got stuck because of not enough resources.
I was just triaging the issues. I’m not that involved with the use of Ray in vLLM so I won’t be of much assistance here.
Fenkail’s solution for setting the ‘num_cpus’ parameter up to a correct amount (i.e. 10 out of 10 available in my case) solved my problem. A fix for slurm jobs:
num_cpus = int(os.environ.get('SLURM_CPUS_PER_TASK'))I just tried using 32 cores and it solved my problem.
Yes, I did modify ray_utils.py, installed in my conda environment for vllm
maybe you can try insert
os.system("ray stop --force")somewhere between unload and reload