vllm: [Bug]: VLLM 0.8.2 OOM error (No error in 0.7.3 version)

Your current environment

Databricks VLLM version: 0.8.2

πŸ› Describe the bug

I have been using VLLM for over 6 months with no problem, until recently which I started with vllm 0.8.2 version. I am installing the vllm using: pip install --upgrade vllm then any model I try to load I immediately get OOM with the following error: The Python process exited with exit code 137 (SIGKILL). This may have been cause by OOM error.

The model can correctly be loaded with vllm==0.7.3 version. I have tried increasing and decreasing gpu_memory_utilization from 0.9 to 0.5 and 0.96 and I have changed max_num_seqs to 256, still no luck. I have tried even to set environmental variable through: export VLLM_USE_V1 = 1 and 0 for v0 version, still no luck. May I know what cause this problem with 0.8.2 version which did not exist in 0.7.3 version, and is there any solution for this problem.

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 2
  • Comments: 36 (12 by maintainers)

Most upvoted comments

Hi @manitadayon , nice I was able to reproduce the error.

Same machine, 2x Nvidia L40s, vllm 0.8.3

  1. V0 works as follows:
CUDA_VISIBLE_DEVICES=0 VLLM_USE_V1=0 vllm serve Llama-3_3-Nemotron-Super-49B-v1-4bit-GPTQ/ --trust-remote-code -q gptq --max-model-len 32768

INFO 04-11 09:34:18 [model_runner.py:1598] Graph capturing finished in 29 secs, took 3.34 GiB
INFO 04-11 09:34:18 [llm_engine.py:448] init engine (profile, create kv cache, warmup model) took 46.05 seconds
INFO 04-11 09:34:18 [api_server.py:1081] Starting vLLM API server on http://0.0.0.0:8000
INFO 04-11 09:34:18 [launcher.py:26] Available routes are:
INFO 04-11 09:34:18 [launcher.py:34] Route: /openapi.json, Methods: HEAD, GET
INFO 04-11 09:34:18 [launcher.py:34] Route: /docs, Methods: HEAD, GET
INFO 04-11 09:34:18 [launcher.py:34] Route: /docs/oauth2-redirect, Methods: HEAD, GET
INFO 04-11 09:34:18 [launcher.py:34] Route: /redoc, Methods: HEAD, GET
INFO 04-11 09:34:18 [launcher.py:34] Route: /health, Methods: GET
INFO 04-11 09:34:18 [launcher.py:34] Route: /load, Methods: GET
INFO 04-11 09:34:18 [launcher.py:34] Route: /ping, Methods: POST, GET
INFO 04-11 09:34:18 [launcher.py:34] Route: /tokenize, Methods: POST
INFO 04-11 09:34:18 [launcher.py:34] Route: /detokenize, Methods: POST
INFO 04-11 09:34:18 [launcher.py:34] Route: /v1/models, Methods: GET
INFO 04-11 09:34:18 [launcher.py:34] Route: /version, Methods: GET
INFO 04-11 09:34:18 [launcher.py:34] Route: /v1/chat/completions, Methods: POST
INFO 04-11 09:34:18 [launcher.py:34] Route: /v1/completions, Methods: POST
INFO 04-11 09:34:18 [launcher.py:34] Route: /v1/embeddings, Methods: POST
INFO 04-11 09:34:18 [launcher.py:34] Route: /pooling, Methods: POST
INFO 04-11 09:34:18 [launcher.py:34] Route: /score, Methods: POST
INFO 04-11 09:34:18 [launcher.py:34] Route: /v1/score, Methods: POST
INFO 04-11 09:34:18 [launcher.py:34] Route: /v1/audio/transcriptions, Methods: POST
INFO 04-11 09:34:18 [launcher.py:34] Route: /rerank, Methods: POST
INFO 04-11 09:34:18 [launcher.py:34] Route: /v1/rerank, Methods: POST
INFO 04-11 09:34:18 [launcher.py:34] Route: /v2/rerank, Methods: POST
INFO 04-11 09:34:18 [launcher.py:34] Route: /invocations, Methods: POST
INFO 04-11 09:34:18 [launcher.py:34] Route: /metrics, Methods: GET
INFO:     Started server process [2096773]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
  1. V1 fails:
CUDA_VISIBLE_DEVICES=1 VLLM_USE_V1=1 vllm serve Llama-3_3-Nemotron-Super-49B-v1-4bit-GPTQ/ --trust-remote-code -q gptq --max-model-len 32768

INFO 04-11 09:30:46 [loader.py:447] Loading weights took 3.71 seconds
INFO 04-11 09:30:47 [gpu_model_runner.py:1273] Model loading took 27.0754 GiB and 4.111081 seconds
INFO 04-11 09:30:59 [backends.py:416] Using cache directory: /.cache/vllm/torch_compile_cache/1a15678859/rank_0_0 for vLLM's torch.compile
INFO 04-11 09:30:59 [backends.py:426] Dynamo bytecode transform time: 12.46 s
INFO 04-11 09:31:02 [backends.py:132] Cache the graph of shape None for later use
INFO 04-11 09:31:47 [backends.py:144] Compiling a graph for general shape takes 47.02 s
INFO 04-11 09:32:07 [monitor.py:33] torch.compile takes 59.47 s in total
ERROR 04-11 09:32:08 [core.py:390] EngineCore hit an exception: Traceback (most recent call last):
ERROR 04-11 09:32:08 [core.py:390]   File "/environments/quantization_env/lib64/python3.11/site-packages/vllm/v1/engine/core.py", line 378, in run_engine_core
ERROR 04-11 09:32:08 [core.py:390]     engine_core = EngineCoreProc(*args, **kwargs)
ERROR 04-11 09:32:08 [core.py:390]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-11 09:32:08 [core.py:390]   File "/environments/quantization_env/lib64/python3.11/site-packages/vllm/v1/engine/core.py", line 319, in __init__
ERROR 04-11 09:32:08 [core.py:390]     super().__init__(vllm_config, executor_class, log_stats)
ERROR 04-11 09:32:08 [core.py:390]   File "/environments/quantization_env/lib64/python3.11/site-packages/vllm/v1/engine/core.py", line 71, in __init__
ERROR 04-11 09:32:08 [core.py:390]     self._initialize_kv_caches(vllm_config)
ERROR 04-11 09:32:08 [core.py:390]   File "/environments/quantization_env/lib64/python3.11/site-packages/vllm/v1/engine/core.py", line 136, in _initialize_kv_caches
ERROR 04-11 09:32:08 [core.py:390]     kv_cache_configs = [
ERROR 04-11 09:32:08 [core.py:390]                        ^
ERROR 04-11 09:32:08 [core.py:390]   File "/environments/quantization_env/lib64/python3.11/site-packages/vllm/v1/engine/core.py", line 137, in <listcomp>
ERROR 04-11 09:32:08 [core.py:390]     get_kv_cache_config(vllm_config, kv_cache_spec_one_worker,
ERROR 04-11 09:32:08 [core.py:390]   File "/environments/quantization_env/lib64/python3.11/site-packages/vllm/v1/core/kv_cache_utils.py", line 643, in get_kv_cache_config
ERROR 04-11 09:32:08 [core.py:390]     check_enough_kv_cache_memory(vllm_config, kv_cache_spec, available_memory)
ERROR 04-11 09:32:08 [core.py:390]   File "/environments/quantization_env/lib64/python3.11/site-packages/vllm/v1/core/kv_cache_utils.py", line 490, in check_enough_kv_cache_memor
y
ERROR 04-11 09:32:08 [core.py:390]     raise ValueError(
ERROR 04-11 09:32:08 [core.py:390] ValueError: To serve at least one request with the models's max seq len (32768), (6.12 GiB KV cache is needed, which is larger than the available KV cache
memory (5.63 GiB). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine.
ERROR 04-11 09:32:08 [core.py:390]
CRITICAL 04-11 09:32:08 [core_client.py:361] Got fatal signal from worker processes, shutting down. See stack trace above for root cause issue.
Killed

Now, I can investigate.

@DarkLight1337 FYI