vllm: [Bug]:The parameter gpu_memory_utilization does not take effect

Your current environment

The output of `python collect_env.py`
Your output of `python collect_env.py` here

Model Input Dumps

Google Colab with T4 GPU

šŸ› Describe the bug

I set gpu_memory_utilization=0.9, but I find the memory used rate is 0.99 sometimes . It is higher than the value I set before and it sounds like a bug .

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (5 by maintainers)

Most upvoted comments

Is there a formula I can use to get this value before I launch vllm?

I’d love it if you figured one out! At the moment the memory profiling is all empirical- we run the model and track the memory usage. There’s no formula under the hood to calculate memory usage

fragmentation - not sure what you mean exactly, but guess it can be a memory leak cause

I’m not talking about memory leaks though, what I mean here is that the pytorch allocator will reserve blocks of memory to allocate tensors within and as tensors are allocated and freed those blocks may become more fragmented. When you actually end up with an OOM error you can see some info about the memory that torch has reserved but not allocated, like this:

CUDA out of memory. Tried to allocate 524.00 MiB. GPU 0 has a total capacity of 15.77 GiB of which 137.12 MiB is free. Process 6513 has 15.63 GiB memory in use. Of the allocated memory 13.81 GiB is allocated by PyTorch, with 106.00 MiB allocated in private pools (e.g., CUDA Graphs), and 121.43 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)')

That shows that torch was holding onto 121MiB of memory that didn’t actually have anything allocated in it, which is likely just fragmented space in vram.

Also, I strongly believe that those extra MiB are somehow related to the LoRA adapter usage

We do try to account for LoRA usage in the memory profiling by creating dummy adapters but it sounds very reasonable that using real adapters could cause more memory fragmentation issues, or lead to more things sitting around and not getting garbage collected for a while. I haven’t looked into that for myself yet