vllm: CUDA error: out of memory

I successfully installed vLLM in WSL2, when I was trying to run the sample code, I got error info like this:

from vllm import LLM, SamplingParams

prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
llm = LLM(model="/mnt/d/github/text-generation-webui/models/facebook_opt-125m")

outputs = llm.generate(prompts, sampling_params)

# Print the outputs.
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

INFO 06-21 21:40:02 llm_engine.py:59] Initializing an LLM engine with config: model=‘/mnt/d/github/text-generation-webui/models/facebook_opt-125m’, dtype=torch.float16, use_dummy_weights=False, download_dir=None, use_np_weights=False, tensor_parallel_size=1, seed=0) INFO 06-21 21:40:12 llm_engine.py:128] # GPU blocks: 37375, # CPU blocks: 7281 Traceback (most recent call last): File “/mnt/d/01Projects/vllm/prac_1.py”, line 11, in <module> llm = LLM(model=“/mnt/d/github/text-generation-webui/models/facebook_opt-125m”) File “/mnt/d/github/vllm/vllm/entrypoints/llm.py”, line 55, in init self.llm_engine = LLMEngine.from_engine_args(engine_args) File “/mnt/d/github/vllm/vllm/engine/llm_engine.py”, line 145, in from_engine_args engine = cls(*engine_configs, distributed_init_method, devices, File “/mnt/d/github/vllm/vllm/engine/llm_engine.py”, line 102, in init self._init_cache() File “/mnt/d/github/vllm/vllm/engine/llm_engine.py”, line 134, in _init_cache self._run_workers(“init_cache_engine”, cache_config=self.cache_config) File “/mnt/d/github/vllm/vllm/engine/llm_engine.py”, line 307, in _run_workers output = executor(*args, **kwargs) File “/mnt/d/github/vllm/vllm/worker/worker.py”, line 126, in init_cache_engine self.cache_engine = CacheEngine( File “/mnt/d/github/vllm/vllm/worker/cache_engine.py”, line 41, in init self.cpu_cache = self.allocate_cpu_cache() File “/mnt/d/github/vllm/vllm/worker/cache_engine.py”, line 89, in allocate_cpu_cache key_blocks = torch.empty( RuntimeError: CUDA error: out of memory CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Python: 3.10.11 GPU: RTX 3090 24G Linux: WSL2, Ubuntu 20.04.6 LTS Can anyone help to answer this?

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 54 (6 by maintainers)

Commits related to this issue

Most upvoted comments

Just pair-debugged with @SunixLiu and we successfully locate the issue. As a temporary fix, please comment out pin_memory=True in vLLM code when allocating CPU cache:

https://github.com/vllm-project/vllm/blob/4026a049d3ad510bea8e177bf71722bd510fbb46/vllm/worker/cache_engine.py#L89-L97

pin_memory has a limit in WSL (official doc) and the limit seems to be 2GB. After commenting out this, vLLM should work properly.

After setting --max-model-len 8192 the OOM went away. I also use an AWQ quant with --quantization awq parameter. Works amazing!

@nosolosoft, what if you try using --max-model-len 4096?

@allanwakes @prashantskit have you come across a solution?

I’m seeing same with Mistral 7B on RTX 3090 24GB

I’m from the future, December of the year 2024. And I’m still getting this CUDA Out of Memory error. I’ve got 5 RTX 3090s with 24 GB each. When I use a small model that loads easily in my other work flows I’m running into this error. I’ve tried decreasing gpu_memory_utilization and awq but still getting the error

With Mistral as well on Ubuntu, I’m getting the CUDA out of memory error, and playing with --gpu-memory-utilization doesn’t seem to make a difference.

python -u -m vllm.entrypoints.openai.api_server --host 0.0.0.0 --model mistralai/Mistral-7B-v0.1 --dtype half --gpu-memory-utilization 0.8  --max-model-len 4096

results in:

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 224.00 MiB (GPU 0; 10.73 GiB total capacity; 9.85 GiB already allocated; 46.44 MiB free; 9.86 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.

I tried to follow the AWQ advice above and this works:

python -u -m vllm.entrypoints.openai.api_server --host 0.0.0.0 --model TheBloke/Mistral-7B-OpenOrca-AWQ --dtype half --gpu-memory-utilization 0.7  --max-model-len 4096 --quantization awq

(Ubuntu 22.04, RTX 2080 Ti, 32GB RAM)

But I’m not sure how to get the original Mistral running. Anything I’m missing?

Got OOM too as well, on 32GB v100 and using 7B llama model, it shouldn’t OOM, why?

Same error on serving mistralai/Mistral-7B-Instruct-v0.3 on a server with two RTX 4090 24GB

Ok, I’ve done some debugging, this function is literally designed to allocate GPU memory as max as possible (up to what’s set in the gpu_memory_utilization). What’s the reasoning behind this design decision?

Edit: I know see that this is by design, you occupy all available GPU memory in case new instances are spawned? So then it must be some other bug. I stepped into the code and OPT-125M by itself only occupied a small amount of memory until this GPU allocation happened. I’ll limit the gpu_memory_utilization as a tmp workaround.

image

@AlpinDale Good question. You can use the tensor_parallel_size argument for multi-GPU inference. First, initialize your Ray cluster by executing

$ ray start --head

Then, use the tensor_parallel_size argument in the LLM class:

llm = LLM(model=<your model>, tensor_parallel_size=2)  # Inference with 2 GPUs 

I’m hitting this same issue on Ubuntu, 48GB of VRAM (2xRTX3090), using OPT-125M model.

I’m comfortably using large ML systems on my rig in general.

the pin memory hack works for ubuntu as well, I’ve just set it to False as a tmp workaround.

Mistral 7b on RTX4060 16GB fails. With ollama for example it doesn’t

@SunixLiu @dcruiz01 We believe this is actually a bug. We are not sure whether this is caused by WSL or vLLM. We don’t have a Windows environment with GPUs. How about we set up a 30-min meeting to pair-debug on your environments? Can you send me an email at zhuohan[at]berkeley.edu to set up a call? Thanks!

Got same issue when running vllm serve "agentica-org/DeepCoder-14B-Preview".

Got same thing for this sudo docker run --runtime nvidia --gpus all
-v ~/.cache/huggingface:/root/.cache/huggingface
–env “HUGGING_FACE_HUB_TOKEN=”
-p 8003:8000
–ipc=host
–name phi4
vllm/vllm-openai:latest
–model microsoft/phi-4
–gpu_memory_utilization=0.60
–max_model_len=4096

Just getting the same issue here! 😦

any update this issues? same issues i hit

Any update on this? constant OOM

[rank0]: Traceback (most recent call last): [rank0]: File “/usr/lib/python3.10/code.py”, line 90, in runcode [rank0]: exec(code, self.locals) [rank0]: File “”, line 1, in <module> [rank0]: File “/home/tom/Documents/PycharmProjects/TI_RAG/venv/lib/python3.10/site-packages/aphrodite/endpoints/llm.py”, line 180, in init [rank0]: self.llm_engine = AphroditeEngine.from_engine_args(engine_args) [rank0]: File “/home/tom/Documents/PycharmProjects/TI_RAG/venv/lib/python3.10/site-packages/aphrodite/engine/aphrodite_engine.py”, line 521, in from_engine_args [rank0]: engine = cls( [rank0]: File “/home/tom/Documents/PycharmProjects/TI_RAG/venv/lib/python3.10/site-packages/aphrodite/engine/aphrodite_engine.py”, line 347, in init [rank0]: self._initialize_kv_caches() [rank0]: File “/home/tom/Documents/PycharmProjects/TI_RAG/venv/lib/python3.10/site-packages/aphrodite/engine/aphrodite_engine.py”, line 448, in _initialize_kv_caches [rank0]: self.model_executor.initialize_cache(num_gpu_blocks, num_cpu_blocks) [rank0]: File “/home/tom/Documents/PycharmProjects/TI_RAG/venv/lib/python3.10/site-packages/aphrodite/executor/gpu_executor.py”, line 126, in initialize_cache [rank0]: self.driver_worker.initialize_cache(num_gpu_blocks, num_cpu_blocks) [rank0]: File “/home/tom/Documents/PycharmProjects/TI_RAG/venv/lib/python3.10/site-packages/aphrodite/worker/worker.py”, line 269, in initialize_cache [rank0]: self._init_cache_engine() [rank0]: File “/home/tom/Documents/PycharmProjects/TI_RAG/venv/lib/python3.10/site-packages/aphrodite/worker/worker.py”, line 274, in _init_cache_engine [rank0]: self.cache_engine = [ [rank0]: File “/home/tom/Documents/PycharmProjects/TI_RAG/venv/lib/python3.10/site-packages/aphrodite/worker/worker.py”, line 275, in <listcomp> [rank0]: CacheEngine(self.cache_config, self.model_config, [rank0]: File “/home/tom/Documents/PycharmProjects/TI_RAG/venv/lib/python3.10/site-packages/aphrodite/worker/cache_engine.py”, line 65, in init [rank0]: self.cpu_cache = self._allocate_kv_cache(self.num_cpu_blocks, “cpu”) [rank0]: File “/home/tom/Documents/PycharmProjects/TI_RAG/venv/lib/python3.10/site-packages/aphrodite/worker/cache_engine.py”, line 82, in _allocate_kv_cache [rank0]: torch.zeros(kv_cache_shape, [rank0]: RuntimeError: CUDA error: invalid argument [rank0]: Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

same error here 👎

lower the gpu-memory-utilization works for me(8*A800 80GB).

python -m vllm.entrypoints.openai.api_server --model /Qwen-7B-Chat --dtype bfloat16 --api-key token-abc123 --trust-remote-code --gpu-memory-utilization 0.3 --max-model-len 4096

Same situation here, tried every single option with gpu_memory_utilization:[0.2:0.9], enforce_eager, batch_size, tried smaller model like “pretrained=facebook/opt-125m”, max_model_len=[128:4096], etc, etc, etc, and the only thing that worked was to hack manually in the code pin_memory to False in the anaconda3/envs/vllm_p39/lib/python3.9/site-packages/vllm/worker/cache_engine.py

LLM engine (v0.4.2)