vllm: [Bug]: qwen2.5-vl-72b oom in 4 A100 in 0.8.3

Your current environment

The output of `python collect_env.py`
- Platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.35
- Python version: 3.12.3
- PyTorch version: 2.6.0+cu124 (GPU)
- Transformers version: 4.50.3
- Datasets version: 3.2.0
- Accelerate version: 1.2.1
- PEFT version: 0.15.0
- TRL version: 0.9.6
- GPU type: NVIDIA A100-PCIE-40GB
- GPU number: 8
- GPU memory: 39.38GB
- vLLM version: 0.8.3

🐛 Describe the bug

VLLM_WORKER_MULTIPROC_METHOD=spawn python -m vllm.entrypoints.openai.api_server
–dtype auto
–port 8009
–trust-remote-code
–served-model-name qwen2vl
–model /home/ps/data/pretrained_model/Qwen/Qwen2.5-VL-72B-Instruct/
–tensor-parallel-size 4
–gpu_memory_utilization 0.95
–max_num_seqs 2
–max_model_len 8192
–mm_processor_kwargs ‘{“max_pixels”:1280, “min_pixels”:256}’

above is my running code. I find that it caused 36GB memory each card, but i don’t know the later process and vllm caused oom error. Can anyone know why? Thank you very much

(VllmWorker rank=1 pid=52940) INFO 04-14 14:23:05 [loader.py:447] Loading weights took 20.16 seconds
(VllmWorker rank=2 pid=52983) INFO 04-14 14:23:05 [loader.py:447] Loading weights took 20.27 seconds
(VllmWorker rank=1 pid=52940) INFO 04-14 14:23:05 [gpu_model_runner.py:1273] Model loading took 34.4340 GiB and 20.495027 seconds
(VllmWorker rank=3 pid=53006) INFO 04-14 14:23:05 [loader.py:447] Loading weights took 20.42 seconds
Loading safetensors checkpoint shards:  97% Completed | 37/38 [00:20<00:00,  1.98it/s]
(VllmWorker rank=2 pid=52983) INFO 04-14 14:23:05 [gpu_model_runner.py:1273] Model loading took 34.4340 GiB and 20.602629 seconds
(VllmWorker rank=3 pid=53006) INFO 04-14 14:23:05 [gpu_model_runner.py:1273] Model loading took 34.4340 GiB and 20.795247 seconds
Loading safetensors checkpoint shards: 100% Completed | 38/38 [00:20<00:00,  2.02it/s]
Loading safetensors checkpoint shards: 100% Completed | 38/38 [00:20<00:00,  1.81it/s]
(VllmWorker rank=0 pid=52924)
(VllmWorker rank=0 pid=52924) INFO 04-14 14:23:06 [loader.py:447] Loading weights took 21.01 seconds
(VllmWorker rank=0 pid=52924) INFO 04-14 14:23:06 [gpu_model_runner.py:1273] Model loading took 34.4340 GiB and 21.353455 seconds
(VllmWorker rank=3 pid=53006) INFO 04-14 14:23:06 [gpu_model_runner.py:1542] Encoder cache will be initialized with a budget of 2048 tokens, and profiled with 256 video items o
f the maximum feature size.
(VllmWorker rank=1 pid=52940) INFO 04-14 14:23:06 [gpu_model_runner.py:1542] Encoder cache will be initialized with a budget of 2048 tokens, and profiled with 256 video items o
f the maximum feature size.
(VllmWorker rank=2 pid=52983) INFO 04-14 14:23:06 [gpu_model_runner.py:1542] Encoder cache will be initialized with a budget of 2048 tokens, and profiled with 256 video items o
f the maximum feature size.
(VllmWorker rank=0 pid=52924) INFO 04-14 14:23:06 [gpu_model_runner.py:1542] Encoder cache will be initialized with a budget of 2048 tokens, and profiled with 256 video items o
f the maximum feature size.
(VllmWorker rank=2 pid=52983) INFO 04-14 14:23:29 [backends.py:416] Using cache directory: /root/.cache/vllm/torch_compile_cache/9cfb59d003/rank_2_0 for vLLM's torch.compile
(VllmWorker rank=2 pid=52983) INFO 04-14 14:23:29 [backends.py:426] Dynamo bytecode transform time: 18.25 s
(VllmWorker rank=1 pid=52940) INFO 04-14 14:23:29 [backends.py:416] Using cache directory: /root/.cache/vllm/torch_compile_cache/9cfb59d003/rank_1_0 for vLLM's torch.compile
(VllmWorker rank=1 pid=52940) INFO 04-14 14:23:29 [backends.py:426] Dynamo bytecode transform time: 18.55 s
(VllmWorker rank=0 pid=52924) INFO 04-14 14:23:29 [backends.py:416] Using cache directory: /root/.cache/vllm/torch_compile_cache/9cfb59d003/rank_0_0 for vLLM's torch.compile
(VllmWorker rank=0 pid=52924) INFO 04-14 14:23:29 [backends.py:426] Dynamo bytecode transform time: 18.66 s
(VllmWorker rank=3 pid=53006) INFO 04-14 14:23:29 [backends.py:416] Using cache directory: /root/.cache/vllm/torch_compile_cache/9cfb59d003/rank_3_0 for vLLM's torch.compile
(VllmWorker rank=3 pid=53006) INFO 04-14 14:23:29 [backends.py:426] Dynamo bytecode transform time: 18.80 s
(VllmWorker rank=2 pid=52983) INFO 04-14 14:23:30 [backends.py:115] Directly load the compiled graph for shape None from the cache
(VllmWorker rank=1 pid=52940) INFO 04-14 14:23:30 [backends.py:115] Directly load the compiled graph for shape None from the cache
(VllmWorker rank=3 pid=53006) INFO 04-14 14:23:30 [backends.py:115] Directly load the compiled graph for shape None from the cache
(VllmWorker rank=0 pid=52924) INFO 04-14 14:23:31 [backends.py:115] Directly load the compiled graph for shape None from the cache
(VllmWorker rank=1 pid=52940) INFO 04-14 14:23:48 [monitor.py:33] torch.compile takes 18.55 s in total
(VllmWorker rank=2 pid=52983) INFO 04-14 14:23:48 [monitor.py:33] torch.compile takes 18.25 s in total
(VllmWorker rank=0 pid=52924) INFO 04-14 14:23:48 [monitor.py:33] torch.compile takes 18.66 s in total
(VllmWorker rank=3 pid=53006) INFO 04-14 14:23:48 [monitor.py:33] torch.compile takes 18.80 s in total
INFO 04-14 14:23:49 [kv_cache_utils.py:578] GPU KV cache size: 9,168 tokens
INFO 04-14 14:23:49 [kv_cache_utils.py:581] Maximum concurrency for 8,192 tokens per request: 1.12x
INFO 04-14 14:23:49 [kv_cache_utils.py:578] GPU KV cache size: 9,072 tokens
INFO 04-14 14:23:49 [kv_cache_utils.py:581] Maximum concurrency for 8,192 tokens per request: 1.11x
INFO 04-14 14:23:49 [kv_cache_utils.py:578] GPU KV cache size: 9,072 tokens
INFO 04-14 14:23:49 [kv_cache_utils.py:581] Maximum concurrency for 8,192 tokens per request: 1.11x
INFO 04-14 14:23:49 [kv_cache_utils.py:578] GPU KV cache size: 9,584 tokens
INFO 04-14 14:23:49 [kv_cache_utils.py:581] Maximum concurrency for 8,192 tokens per request: 1.17x
(VllmWorker rank=2 pid=52983) INFO 04-14 14:24:52 [gpu_model_runner.py:1608] Graph capturing finished in 62 secs, took 2.65 GiB
(VllmWorker rank=0 pid=52924) INFO 04-14 14:24:52 [gpu_model_runner.py:1608] Graph capturing finished in 62 secs, took 2.65 GiB
(VllmWorker rank=1 pid=52940) INFO 04-14 14:24:52 [gpu_model_runner.py:1608] Graph capturing finished in 62 secs, took 2.65 GiB
(VllmWorker rank=3 pid=53006) INFO 04-14 14:24:52 [gpu_model_runner.py:1608] Graph capturing finished in 62 secs, took 2.65 GiB
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383] WorkerProc hit an exception: %s
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383] Traceback (most recent call last):
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383]   File "/data/ps/vllm/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1466, in _dummy_sampler_run
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383]     sampler_output = self.model.sample(
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383]                      ^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383]   File "/data/ps/vllm/venv/lib/python3.12/site-packages/vllm/model_executor/models/qwen2_5_vl.py", line 1107, in sample
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383]     return self.language_model.sample(logits, sampling_metadata)
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383]   File "/data/ps/vllm/venv/lib/python3.12/site-packages/vllm/model_executor/models/qwen2.py", line 480, in sample
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383]     next_tokens = self.sampler(logits, sampling_metadata)
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383]   File "/data/ps/vllm/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383]   File "/data/ps/vllm/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383]     return forward_call(*args, **kwargs)
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383]   File "/data/ps/vllm/venv/lib/python3.12/site-packages/vllm/v1/sample/sampler.py", line 49, in forward
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383]     sampled = self.sample(logits, sampling_metadata)
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383]   File "/data/ps/vllm/venv/lib/python3.12/site-packages/vllm/v1/sample/sampler.py", line 115, in sample
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383]     random_sampled = self.topk_topp_sampler(
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383]                      ^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383]   File "/data/ps/vllm/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383]   File "/data/ps/vllm/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383]     return forward_call(*args, **kwargs)
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383]   File "/data/ps/vllm/venv/lib/python3.12/site-packages/vllm/v1/sample/ops/topk_topp_sampler.py", line 98, in forward_native
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383]     logits = apply_top_k_top_p(logits, k, p)
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383]   File "/data/ps/vllm/venv/lib/python3.12/site-packages/vllm/v1/sample/ops/topk_topp_sampler.py", line 186, in apply_top_k_top_p
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383]     logits_sort, logits_idx = logits.sort(dim=-1, descending=False)
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383]                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=1 pid=52940) ERROR 04-14 14:24:52 [multiproc_executor.py:383] torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 298.00 MiB. GPU 1 has a total capacity of 39.38 GiB of which 65.38 MiB is free. Including non-PyTorch memory, this process has 39.31 GiB memory in use. Of the allocated memory 35.91 GiB is allocated by PyTorch, with 73.88 MiB allocated in private pools (e.g., CUDA Graphs), and 120.72 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 37 (15 by maintainers)

Most upvoted comments

In V0, the multi-modal embeddings are truncated to the context length, but this is not the case in V1. So the activation peak memory is greater which causes OOM. cc @ywang96