vllm: vLLM stops all processing when CPU KV cache is used, has to be shut down and restarted.

Hi

The issue: with --swap-space X specified, as soon as CPU KV cache is used, vLLM stops all processing. CPU and GPU usage go to 0%, and the request never returns. Any future requests are also never answered. There is no error.

I am testing the latest vLLM code (commit 6fc2a38) in a Docker container. I have experienced the issue since I first started using vLLM about 4 days ago, so it’s not specific to the latest commits.

I am launching vLLM with the following args:

--model lmsys/vicuna-7b-v1.3 --host 0.0.0.0 --tokenizer hf-internal-testing/llama-tokenizer --swap-space 100 

I am currently testing on a 1 x 4090 system, but I have experienced it on all GPU types I’ve tried, including A6000 and H100.

The following test code will quickly trigger the issue on a 1 x 4090 system:

import time
import requests

url = 'http://localhost:8000/v1/completions'
headers = {'Content-Type': 'application/json'}
data = {
    "model": "lmsys/vicuna-7b-v1.3",
    "prompt": " Write a story about a cat named George."*40,
    "max_tokens": 950,
    "temperature": 0.7,
    "n":125
}
s=time.time()
response = requests.post(url, headers=headers, json=data)
print(time.time()-s)

Here’s a screenshot demonstrating the issue: image

In the screenshot you can see that only 7.9% of CPU KV cache is used, but this is enough to cause all processing to stop. The server will now never answer this request, and never answer any new requests either. It is effectively dead.

If I leave out --swap-space X then the server aborts RuntimeError: Aborted due to the lack of CPU swap space. Please increase the swap space to avoid this error., which is what I’m trying to avoid. It would be nice to be able to use CPU RAM as an overflow buffer, in case I occasionally exceed VRAM.

Thanks in advance.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 2
  • Comments: 17 (3 by maintainers)

Commits related to this issue

Most upvoted comments

this issue makes vllm impossible for production use

--swap-space 0 - docs

@TheBloke are you still experiencing this issue?