vllm: CUDA error: out of memory
I successfully installed vLLM in WSL2, when I was trying to run the sample code, I got error info like this:
from vllm import LLM, SamplingParams
prompts = [
"Hello, my name is",
"The president of the United States is",
"The capital of France is",
"The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
llm = LLM(model="/mnt/d/github/text-generation-webui/models/facebook_opt-125m")
outputs = llm.generate(prompts, sampling_params)
# Print the outputs.
for output in outputs:
prompt = output.prompt
generated_text = output.outputs[0].text
print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
INFO 06-21 21:40:02 llm_engine.py:59] Initializing an LLM engine with config: model=‘/mnt/d/github/text-generation-webui/models/facebook_opt-125m’, dtype=torch.float16, use_dummy_weights=False, download_dir=None, use_np_weights=False, tensor_parallel_size=1, seed=0)
INFO 06-21 21:40:12 llm_engine.py:128] # GPU blocks: 37375, # CPU blocks: 7281
Traceback (most recent call last):
File “/mnt/d/01Projects/vllm/prac_1.py”, line 11, in <module>
llm = LLM(model=“/mnt/d/github/text-generation-webui/models/facebook_opt-125m”)
File “/mnt/d/github/vllm/vllm/entrypoints/llm.py”, line 55, in init
self.llm_engine = LLMEngine.from_engine_args(engine_args)
File “/mnt/d/github/vllm/vllm/engine/llm_engine.py”, line 145, in from_engine_args
engine = cls(*engine_configs, distributed_init_method, devices,
File “/mnt/d/github/vllm/vllm/engine/llm_engine.py”, line 102, in init
self._init_cache()
File “/mnt/d/github/vllm/vllm/engine/llm_engine.py”, line 134, in _init_cache
self._run_workers(“init_cache_engine”, cache_config=self.cache_config)
File “/mnt/d/github/vllm/vllm/engine/llm_engine.py”, line 307, in _run_workers
output = executor(*args, **kwargs)
File “/mnt/d/github/vllm/vllm/worker/worker.py”, line 126, in init_cache_engine
self.cache_engine = CacheEngine(
File “/mnt/d/github/vllm/vllm/worker/cache_engine.py”, line 41, in init
self.cpu_cache = self.allocate_cpu_cache()
File “/mnt/d/github/vllm/vllm/worker/cache_engine.py”, line 89, in allocate_cpu_cache
key_blocks = torch.empty(
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
Python: 3.10.11 GPU: RTX 3090 24G Linux: WSL2, Ubuntu 20.04.6 LTS Can anyone help to answer this?
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 54 (6 by maintainers)
Links to this issue
Commits related to this issue
- remove redundant slice; match decode PA partition size with csrc (#188) — committed to mht-sharma/vllm by sanyalington 2 years ago
- Revert "remove redundant slice; match decode PA partition size with csrc (#188)" (#194) This reverts commit c68c242bdaf3df4fd85f508605365d58efc171f1. — committed to mht-sharma/vllm by gshtras 2 years ago
- [CI] upgrade to newest pta (#187) (#188) see: https://github.com/vllm-project/vllm-ascend/pull/187 Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com> Co-authored-by: angazenn <zengyanjia@huawei.co... — committed to wuhuikx/vllm by wangxiyuan 2 years ago
- fix quant path import error, due to recent upstream code change. (#188) Signed-off-by: Kunshang Ji <kunshang.ji@intel.com> — committed to yma11/vllm by jikunshang a year ago
- fix quant path import error, due to recent upstream code change. (#188) Signed-off-by: Kunshang Ji <kunshang.ji@intel.com> — committed to yma11/vllm by jikunshang a year ago
- fix quant path import error, due to recent upstream code change. (#188) Signed-off-by: Kunshang Ji <kunshang.ji@intel.com> — committed to zhenwei-intel/vllm by jikunshang a year ago
Just pair-debugged with @SunixLiu and we successfully locate the issue. As a temporary fix, please comment out
pin_memory=Truein vLLM code when allocating CPU cache:https://github.com/vllm-project/vllm/blob/4026a049d3ad510bea8e177bf71722bd510fbb46/vllm/worker/cache_engine.py#L89-L97
pin_memoryhas a limit in WSL (official doc) and the limit seems to be 2GB. After commenting out this, vLLM should work properly.After setting
--max-model-len 8192the OOM went away. I also use an AWQ quant with--quantization awqparameter. Works amazing!@nosolosoft, what if you try using
--max-model-len 4096?@allanwakes @prashantskit have you come across a solution?
I’m seeing same with Mistral 7B on RTX 3090 24GB
I’m from the future, December of the year 2024. And I’m still getting this CUDA Out of Memory error. I’ve got 5 RTX 3090s with 24 GB each. When I use a small model that loads easily in my other work flows I’m running into this error. I’ve tried decreasing gpu_memory_utilization and awq but still getting the error
With Mistral as well on Ubuntu, I’m getting the CUDA out of memory error, and playing with
--gpu-memory-utilizationdoesn’t seem to make a difference.results in:
I tried to follow the AWQ advice above and this works:
(Ubuntu 22.04, RTX 2080 Ti, 32GB RAM)
But I’m not sure how to get the original Mistral running. Anything I’m missing?
Got OOM too as well, on 32GB v100 and using 7B llama model, it shouldn’t OOM, why?
Same error on serving mistralai/Mistral-7B-Instruct-v0.3 on a server with two RTX 4090 24GB
Ok, I’ve done some debugging, this function is literally designed to allocate GPU memory as max as possible (up to what’s set in the
gpu_memory_utilization). What’s the reasoning behind this design decision?Edit: I know see that this is by design, you occupy all available GPU memory in case new instances are spawned? So then it must be some other bug. I stepped into the code and OPT-125M by itself only occupied a small amount of memory until this GPU allocation happened. I’ll limit the
gpu_memory_utilizationas a tmp workaround.@AlpinDale Good question. You can use the
tensor_parallel_sizeargument for multi-GPU inference. First, initialize your Ray cluster by executingThen, use the
tensor_parallel_sizeargument in the LLM class:I’m hitting this same issue on Ubuntu, 48GB of VRAM (2xRTX3090), using OPT-125M model.
I’m comfortably using large ML systems on my rig in general.
the pin memory hack works for ubuntu as well, I’ve just set it to False as a tmp workaround.
Mistral 7b on RTX4060 16GB fails. With ollama for example it doesn’t
@SunixLiu @dcruiz01 We believe this is actually a bug. We are not sure whether this is caused by WSL or vLLM. We don’t have a Windows environment with GPUs. How about we set up a 30-min meeting to pair-debug on your environments? Can you send me an email at
zhuohan[at]berkeley.eduto set up a call? Thanks!Got same issue when running
vllm serve "agentica-org/DeepCoder-14B-Preview".Got same thing for this sudo docker run --runtime nvidia --gpus all
-v ~/.cache/huggingface:/root/.cache/huggingface
–env “HUGGING_FACE_HUB_TOKEN=”
-p 8003:8000
–ipc=host
–name phi4
vllm/vllm-openai:latest
–model microsoft/phi-4
–gpu_memory_utilization=0.60
–max_model_len=4096
Just getting the same issue here! 😦
any update this issues? same issues i hit
Any update on this? constant OOM
[rank0]: Traceback (most recent call last): [rank0]: File “/usr/lib/python3.10/code.py”, line 90, in runcode [rank0]: exec(code, self.locals) [rank0]: File “”, line 1, in <module> [rank0]: File “/home/tom/Documents/PycharmProjects/TI_RAG/venv/lib/python3.10/site-packages/aphrodite/endpoints/llm.py”, line 180, in init [rank0]: self.llm_engine = AphroditeEngine.from_engine_args(engine_args) [rank0]: File “/home/tom/Documents/PycharmProjects/TI_RAG/venv/lib/python3.10/site-packages/aphrodite/engine/aphrodite_engine.py”, line 521, in from_engine_args [rank0]: engine = cls( [rank0]: File “/home/tom/Documents/PycharmProjects/TI_RAG/venv/lib/python3.10/site-packages/aphrodite/engine/aphrodite_engine.py”, line 347, in init [rank0]: self._initialize_kv_caches() [rank0]: File “/home/tom/Documents/PycharmProjects/TI_RAG/venv/lib/python3.10/site-packages/aphrodite/engine/aphrodite_engine.py”, line 448, in _initialize_kv_caches [rank0]: self.model_executor.initialize_cache(num_gpu_blocks, num_cpu_blocks) [rank0]: File “/home/tom/Documents/PycharmProjects/TI_RAG/venv/lib/python3.10/site-packages/aphrodite/executor/gpu_executor.py”, line 126, in initialize_cache [rank0]: self.driver_worker.initialize_cache(num_gpu_blocks, num_cpu_blocks) [rank0]: File “/home/tom/Documents/PycharmProjects/TI_RAG/venv/lib/python3.10/site-packages/aphrodite/worker/worker.py”, line 269, in initialize_cache [rank0]: self._init_cache_engine() [rank0]: File “/home/tom/Documents/PycharmProjects/TI_RAG/venv/lib/python3.10/site-packages/aphrodite/worker/worker.py”, line 274, in _init_cache_engine [rank0]: self.cache_engine = [ [rank0]: File “/home/tom/Documents/PycharmProjects/TI_RAG/venv/lib/python3.10/site-packages/aphrodite/worker/worker.py”, line 275, in <listcomp> [rank0]: CacheEngine(self.cache_config, self.model_config, [rank0]: File “/home/tom/Documents/PycharmProjects/TI_RAG/venv/lib/python3.10/site-packages/aphrodite/worker/cache_engine.py”, line 65, in init [rank0]: self.cpu_cache = self._allocate_kv_cache(self.num_cpu_blocks, “cpu”) [rank0]: File “/home/tom/Documents/PycharmProjects/TI_RAG/venv/lib/python3.10/site-packages/aphrodite/worker/cache_engine.py”, line 82, in _allocate_kv_cache [rank0]: torch.zeros(kv_cache_shape, [rank0]: RuntimeError: CUDA error: invalid argument [rank0]: Compile with
TORCH_USE_CUDA_DSAto enable device-side assertions.same error here 👎
lower the gpu-memory-utilization works for me(8*A800 80GB).
python -m vllm.entrypoints.openai.api_server --model /Qwen-7B-Chat --dtype bfloat16 --api-key token-abc123 --trust-remote-code --gpu-memory-utilization 0.3 --max-model-len 4096
Same situation here, tried every single option with gpu_memory_utilization:[0.2:0.9], enforce_eager, batch_size, tried smaller model like “pretrained=facebook/opt-125m”, max_model_len=[128:4096], etc, etc, etc, and the only thing that worked was to hack manually in the code
pin_memory to Falsein theanaconda3/envs/vllm_p39/lib/python3.9/site-packages/vllm/worker/cache_engine.pyLLM engine (v0.4.2)