vllm: Is there a way to terminate vllm.LLM and release the GPU memory
After below code, is there an api(maybe like llm.terminate) to kill llm and release the GPU memory?
from vllm import LLM, SamplingParams
prompts = [
"Hello, my name is",
"The president of the United States is",
"The capital of France is",
"The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
outputs = llm.generate(prompts, sampling_params)
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 44 (10 by maintainers)
I understand your point. However, this feature is extremely useful for situations where you need to switch between models. For instance, reinforcement learning loops. I am writing an off-policy RL loop, requiring me to train one model (target policy) while its previous version performs inference (behavior policy). As a result, I frequently load and unload models. While I know vLLM is not intended for training, using
transformerswould be too slow, making my technique unviable.Let me know if this is a feature that’s wanted and the team would be interested in maintaining it. I can open a separate issue and start working on it.
Please check the codes below. It works.
Best regards,
Shuyue Dec. 3rd, 2023
In the latest version of vLLM
destroy_model_parallelhas moved tovllm.distributed.parallel_state. The objects you have to delete have also changed:I tried inferring multiple models consecutively with vLLM v0.5.2. As mentioned above, the behavior differs depending on the value of TP.
export VLLM_WORKER_MULTIPROC_METHOD=spawnin https://github.com/vllm-project/vllm/issues/6152#issuecomment-2211709345, and runvllm.distributed.parallel_state.destroy_distributed_environment().ray.shutdown()but it did not work.I use this function in a pipeline where I explain images with VLM and then summarize them with LLM. I hope that this kind of processing will be officially provided and become common.
We tried this in version 0.4.2, but GPU memory did not released.
This setup works perfectly on my end when using multiple GPUs! FYI, I am using
pip install vllm==0.6.3.post1:Even after executing the code above, the GPU memory is not freed with the latest vllm built from source. Any recommendations?
Glad to see you here @cassanof and to hear that you have been using vLLM in this kind of workflow!
Given how much wanted this feature seems to be, I will bring this back to the team to discuss! If multi-gpu instance is prone to deadlocks, then perhaps we can at least start with single-gpu instances. Everyone on the maintainer team does have limited bandwidth and we have a lot of things to work on, so contributions are very welcomed as always!
I find that we need to explicitly run “del llm.llm_engine.driver_worker” to release in when using a single worker. Can anybody explain why this is the case?
Got it working on vllm 0.5.x Adding these two flags to my LLM call:
and
using this method after llm.generate works for me
In general it is very difficult to clean up all resources correctly, especially when we use multiple GPUs, and might be prone to deadlocks .
I would say, the most stable way to terminate vLLM is to shut down the process.
When debugging, I found that when using ‘spawn’ the main GPU used (if using PCI_BUS) would still keep some small amount of memory allocated, indicating that the clean up is unsuccessful.
When I then check all available GPUs that have zero memory allocated and export
CUDA_VISIBLE_DEVICES, I am able to progress into changing into another model until another issue hits that seemingly indicates the workers distributed state is still somewhere in memory.There should be a built-in way! We cannot keep writing code that breaks on the next minor release 😦
No longer works for v1 as model_executor cannot be directly accessed. Instead try: llm.llm_engine.engine_core.shutdown()
Another update:
The problem is the global
_WORLDvariable that stays None. When overriding the backend to be ‘nccl’ hardcoded atvllm/distributed/parallel_state.py", line 920, in ensure_model_parallel_initialized, the whole state goes in deadlock:@bhavnicksm I can reproduce the same error you have.
UPDATE: Works with
rayat TP=2 but for some reason not withmultiprocessingwith either fork or spawn. Also, only unload works withraybut not reload right now.It works for me.
This code is worked for me
vllm==0.4.0.post1
Then I do not have a clue either. Meanwhile, I should add an information: the vllm version with which I succeeded with the above code was 0.4.0.post1
Tried this including
ray.shutdown()but the memory is not released on my end, any other suggestion?vLLM seems to hang to the first allocated LLM() instance. It does not hang to later instances. Maybe that helps with diagnosing the issue?
The
llm1consumes more thanllm0but you can see that the allocated memory stays at llm0 level.update: the following code would work better, without the possible dead lock warning.