vllm: [Bug]: TRACKING ISSUE: CUDA OOM with Logprobs
Your current environment
The output of `python collect_env.py`
🐛 Describe the bug
vLLM has an issue where we can go OOM if too many logprobs are requested.
The reason that this happens is that there are three sources of memory usage:
- Model weights
- KV caches
- Activations
When determining the KV cache size, we calculate peak memory running a long prefill * without logprobs *
If a prompt requests many logprobs, however, this is an additional source of memory usage which is not considered during warmup and can cause OOM because we have nothing in scheduler to prevent this
We have received several examples of this:
- https://github.com/vllm-project/vllm/issues/5890
- https://github.com/vllm-project/vllm/issues/5060 << some of the OOM issues in
AsyncEngineDeadError - Anyone running MMLU or ARC in lm-eval-harness
Attempt to fix this:
I am working on a design to address this issue
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 15
- Comments: 20 (7 by maintainers)
“chunked logits processing” looks like the ideal solution @tjohnson31415
Right now, adding
--enable-chunked-prefilland set--max-num-batched-tokensto be the same as--max-num-seqs(256 by default) should solve the problem, although it will make the prefill slower.@tjohnson31415 is looking into this.