vllm: vLLM stops all processing when CPU KV cache is used, has to be shut down and restarted.
Hi
The issue: with --swap-space X specified, as soon as CPU KV cache is used, vLLM stops all processing. CPU and GPU usage go to 0%, and the request never returns. Any future requests are also never answered. There is no error.
I am testing the latest vLLM code (commit 6fc2a38) in a Docker container. I have experienced the issue since I first started using vLLM about 4 days ago, so it’s not specific to the latest commits.
I am launching vLLM with the following args:
--model lmsys/vicuna-7b-v1.3 --host 0.0.0.0 --tokenizer hf-internal-testing/llama-tokenizer --swap-space 100
I am currently testing on a 1 x 4090 system, but I have experienced it on all GPU types I’ve tried, including A6000 and H100.
The following test code will quickly trigger the issue on a 1 x 4090 system:
import time
import requests
url = 'http://localhost:8000/v1/completions'
headers = {'Content-Type': 'application/json'}
data = {
"model": "lmsys/vicuna-7b-v1.3",
"prompt": " Write a story about a cat named George."*40,
"max_tokens": 950,
"temperature": 0.7,
"n":125
}
s=time.time()
response = requests.post(url, headers=headers, json=data)
print(time.time()-s)
Here’s a screenshot demonstrating the issue:
In the screenshot you can see that only 7.9% of CPU KV cache is used, but this is enough to cause all processing to stop. The server will now never answer this request, and never answer any new requests either. It is effectively dead.
If I leave out --swap-space X then the server aborts RuntimeError: Aborted due to the lack of CPU swap space. Please increase the swap space to avoid this error., which is what I’m trying to avoid. It would be nice to be able to use CPU RAM as an overflow buffer, in case I occasionally exceed VRAM.
Thanks in advance.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 2
- Comments: 17 (3 by maintainers)
Commits related to this issue
- [Scratch] Fix iteration bug (#546) Same issue as https://github.com/anyscale/vllm/pull/542 — committed to rickyyx/vllm by rkooo567 2 years ago
- Update sh scripts (#546) — committed to Xaenalt/vllm by michalkuligowski 2 years ago
- Updated README.md (#546) * Updated README.md Waiting on benchmark results, do not publish yet * Changed "OOM" to "Out of memory" * Added throughput results * Added latency results * Trying to fi... — committed to scarecr0w12/vllm by Mcirino1 2 years ago
this issue makes vllm impossible for production use
https://github.com/vllm-project/vllm/blob/66c54aa9c33555a6b41421d57d3ad6c1bf004ec9/vllm/engine/async_llm_engine.py#L67-L75
I comment this
await asyncio.sleep(0)and it seems to temporarily solve the stuck.--swap-space 0- docs@TheBloke are you still experiencing this issue?