vllm: [Bug]: cpu memory not released when wake up the vLLM instance

Your current environment

The output of `python collect_env.py`
root@a3c882d4b9de:/workspace# python3 collect_env.py 
INFO 04-15 11:22:23 [__init__.py:239] Automatically detected platform cuda.
Collecting environment information...
PyTorch version: 2.6.0+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.4 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.10.12 (main, Feb  4 2025, 14:57:36) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.10.134-17.2.al8.x86_64-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.4.131
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA A800 80GB PCIe
GPU 1: NVIDIA A800 80GB PCIe
GPU 2: NVIDIA A800 80GB PCIe
GPU 3: NVIDIA A800 80GB PCIe

Nvidia driver version: 565.77
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.1.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-cusparselt-cu12==0.6.2
[pip3] nvidia-ml-py==12.570.86
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] pynvml==12.0.0
[pip3] pyzmq==26.3.0
[pip3] torch==2.6.0
[pip3] torchaudio==2.6.0
[pip3] torchvision==0.21.0
[pip3] transformers==4.50.1
[pip3] triton==3.2.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.8.2
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
	GPU0	GPU1	GPU2	GPU3	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	PHB	PHB	PHB	0-63	0		N/A
GPU1	PHB	 X 	PHB	PHB	0-63	0		N/A
GPU2	PHB	PHB	 X 	PHB	0-63	0		N/A
GPU3	PHB	PHB	PHB	 X 	0-63	0		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NVIDIA_VISIBLE_DEVICES=all
NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536
NCCL_VERSION=2.21.5-1
NVIDIA_DRIVER_CAPABILITIES=compute,utility
NVIDIA_PRODUCT_NAME=CUDA
CUDA_VERSION=12.4.1
LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
NCCL_CUMEM_ENABLE=0
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

šŸ› Describe the bug

model weights offloads to cpu memory when called /sleep api. But after call /wake_up api, the cpu memory do not release. Is this on purpose to fast sleep vllm engine for the next time? In multi model switching scenario, this could bring big pressure on cpu memory for each slept model still occupy cpu memory.

[root@host ~]# free -h
              total        used        free      shared  buff/cache   available
Mem:          225Gi       4.1Gi       174Gi        21Mi        46Gi       219Gi
Swap:          63Gi       245Mi        63Gi
[root@host ~]# curl -XPOST http://localhost:8090/sleep?level=1
[root@host ~]# curl http://localhost:8090/is_sleeping
{"is_sleeping":true}
[root@host ~]# free -h
              total        used        free      shared  buff/cache   available
Mem:          225Gi       5.3Gi       124Gi        49Gi        96Gi       169Gi
Swap:          63Gi       245Mi        63Gi
[root@host ~]# curl -XPOST http://localhost:8090/wake_up
[root@host ~]# curl http://localhost:8090/is_sleeping
{"is_sleeping":false}
[root@host ~]# free -h
              total        used        free      shared  buff/cache   available
Mem:          225Gi       5.3Gi       124Gi        49Gi        96Gi       169Gi
Swap:          63Gi       245Mi        63Gi

when check the memory of progress, it seems many deleted /dev/zero file handle.

[root@host ~]# ps aux | grep spawn
root      198907  1.4 22.7 161240136 53878080 pts/0 Sl+ 16:27   4:14 /usr/bin/python3 -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=16, pipe_handle=18) --multiprocessing-fork
[root@host ~]# lsof -p 198907  | wc -l
485
[root@host ~]# lsof -p 198907  | grep DEL
python3 198907 root  DEL       REG                0,1             16611 /dev/zero
python3 198907 root  DEL       REG                0,1             27140 /dev/zero
python3 198907 root  DEL       REG                0,1             27138 /dev/zero
python3 198907 root  DEL       REG                0,1             27139 /dev/zero
python3 198907 root  DEL       REG                0,1             27137 /dev/zero
python3 198907 root  DEL       REG                0,1             27136 /dev/zero
python3 198907 root  DEL       REG                0,1             27134 /dev/zero
...
[root@host ~]# cat  /proc/198907/smaps
...
7f42c0000000-7f42e0000000 rw-s 00000000 00:01 28385                      /dev/zero (deleted)
Size:             524288 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:              524288 kB
Pss:              524288 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:    524288 kB
Referenced:       524288 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
FilePmdMapped:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
DupText:               0 kB
THPeligible:    0
VmFlags: rd wr sh mr mw me ms sd 
7f42e4000000-7f42f4000000 rw-s 00000000 00:01 28386                      /dev/zero (deleted)
Size:             262144 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:              262144 kB
Pss:              262144 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:    262144 kB
Referenced:       262144 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
FilePmdMapped:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
DupText:               0 kB
THPeligible:    0
VmFlags: rd wr sh mr mw me ms sd 
7f42f4000000-7f42f8000000 rw-s 00000000 00:01 28384                      /dev/zero (deleted)
Size:              65536 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:               65536 kB
Pss:               65536 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:     65536 kB
Referenced:        65536 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
FilePmdMapped:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
DupText:               0 kB
THPeligible:    0
...

We expect cpu memory will be released after wake up the vllm server, is there any solutions?Thanks~

By the way, when offload weights, the used memory seems to be much larger than the size of the model weight. For example, the above model is Qwen2.5-14B-Instruct(about 28GiB), but the offloaded memory takes about 50GiB. Is there any optimization for the memory usage?

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 15 (9 by maintainers)

Most upvoted comments

it seems I can confirm pytorch caches pinned host memory until we call torch._C._host_emptyCache():

import psutil
import os

process = psutil.Process(os.getpid())

print(f"RSS: {process.memory_info().rss / 1024 ** 2:.2f} MB")  # Actual physical memory usage

import torch

# Create a pinned memory tensor of size 100 MiB
pin_memory_tensor = torch.empty(100 * 1024 * 1024 // 4, dtype=torch.int32, device='cpu', pin_memory=True)
print(f"RSS after warmup pytorch: {process.memory_info().rss / 1024 ** 2:.2f} MB")  # Actual physical memory usage

pin_memory_tensor2 = torch.empty(100 * 1024 * 1024 // 4, dtype=torch.int32, device='cpu', pin_memory=True)
print(f"RSS after allocation: {process.memory_info().rss / 1024 ** 2:.2f} MB")  # Actual physical memory usage

del pin_memory_tensor2
print(f"RSS after free from python side: {process.memory_info().rss / 1024 ** 2:.2f} MB")  # Actual physical memory usage

torch._C._host_emptyCache()
print(f"RSS after free from pytorch side: {process.memory_info().rss / 1024 ** 2:.2f} MB")  # Actual physical memory usage

It prints:

RSS: 52.71 MB
RSS after warmup pytorch: 648.66 MB
RSS after allocation: 776.57 MB <- increases
RSS after free from python side: 776.57 MB <- does not release memory
RSS after free from pytorch side: 648.68 MB <- releases memory

with this function, wakeup time increase a lot for releasing cpu memory.(for ex, QwQ-32B wakeup time in my environment increase from 4s to 21s)

that’s quite a long time. then let’s keep the current status.

for API users who want to quickly switch models, quick wake-up is important.

for RLHF users, they can switch to sleep with level 2, after @fingertap fixes the buffer issue.