vllm: [Bug]: VLLM 0.8.2 OOM error (No error in 0.7.3 version)
Your current environment
Databricks VLLM version: 0.8.2
π Describe the bug
I have been using VLLM for over 6 months with no problem, until recently which I started with vllm 0.8.2 version.
I am installing the vllm using:
pip install --upgrade vllm
then any model I try to load I immediately get OOM with the following error:
The Python process exited with exit code 137 (SIGKILL). This may have been cause by OOM error.
The model can correctly be loaded with vllm==0.7.3 version. I have tried increasing and decreasing gpu_memory_utilization from 0.9 to 0.5 and 0.96 and I have changed max_num_seqs to 256, still no luck. I have tried even to set environmental variable through: export VLLM_USE_V1 = 1 and 0 for v0 version, still no luck. May I know what cause this problem with 0.8.2 version which did not exist in 0.7.3 version, and is there any solution for this problem.
Before submitting a new issueβ¦
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 2
- Comments: 36 (12 by maintainers)
Hi @manitadayon , nice I was able to reproduce the error.
Same machine, 2x Nvidia L40s,
vllm 0.8.3Now, I can investigate.
@DarkLight1337 FYI