vllm: [Bug]: TRACKING ISSUE: CUDA OOM with Logprobs

Your current environment

The output of `python collect_env.py`

🐛 Describe the bug

vLLM has an issue where we can go OOM if too many logprobs are requested.

The reason that this happens is that there are three sources of memory usage:

  • Model weights
  • KV caches
  • Activations

When determining the KV cache size, we calculate peak memory running a long prefill * without logprobs *

If a prompt requests many logprobs, however, this is an additional source of memory usage which is not considered during warmup and can cause OOM because we have nothing in scheduler to prevent this

We have received several examples of this:

Attempt to fix this:

I am working on a design to address this issue

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 15
  • Comments: 20 (7 by maintainers)

Commits related to this issue

Most upvoted comments

I measured the peak GPU memory usage during the processing in Sampler. I found that the memory usage is balooned to 9x the size of the input logits tensor. The increase comes from upscaling logits to float32 (+2x), copying the tensor for probs and logprobs (+4x), and processing in _sample creating another temporary copy (+2x). I’m sure there are ways to reduce this spike in memory, but we’d still need the input tesnor limited.

The main challenge is if any prompt_logprobs are requested since the logits tensor will have logits for every token in the prompt. We will hit limits for a model with a large context and vocab size in the size of the logits tensor even before processing in Sampler. With Llama 3.1 models with a vocab_size of 128256 and a max sequence length of 131072, a single request with prompt_logprobs requested could produce a logits tensor of 128256 * 131072 * 2B =~ 31 GiB.

Limiting the number of tokens processed at a time with chunked prefill seems like the right solution to me. However, for cases where chunked prefill is not supported, we may still need “chunked logits processing” to limit the number of tokens in the processing from hidden states -> logprobs output. This may be difficult to implement though.

@robertgshaw2-neuralmagic: What do you think about this approach of “chunked logits processing”?

“chunked logits processing” looks like the ideal solution @tjohnson31415

Right now, adding --enable-chunked-prefill and set --max-num-batched-tokens to be the same as --max-num-seqs (256 by default) should solve the problem, although it will make the prefill slower.

@tjohnson31415 is looking into this.