vllm: [Bug]:The parameter gpu_memory_utilization does not take effect
Your current environment
The output of `python collect_env.py`
Your output of `python collect_env.py` here
Model Input Dumps
Google Colab with T4 GPU
š Describe the bug
I set gpu_memory_utilization=0.9, but I find the memory used rate is 0.99 sometimes . It is higher than the value I set before and it sounds like a bug .
Before submitting a new issueā¦
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 16 (5 by maintainers)
Iād love it if you figured one out! At the moment the memory profiling is all empirical- we run the model and track the memory usage. Thereās no formula under the hood to calculate memory usage
Iām not talking about memory leaks though, what I mean here is that the pytorch allocator will reserve blocks of memory to allocate tensors within and as tensors are allocated and freed those blocks may become more fragmented. When you actually end up with an OOM error you can see some info about the memory that torch has reserved but not allocated, like this:
That shows that torch was holding onto 121MiB of memory that didnāt actually have anything allocated in it, which is likely just fragmented space in vram.
We do try to account for LoRA usage in the memory profiling by creating dummy adapters but it sounds very reasonable that using real adapters could cause more memory fragmentation issues, or lead to more things sitting around and not getting garbage collected for a while. I havenāt looked into that for myself yet