vllm: [Bug]: cpu memory not released when wake up the vLLM instance
Your current environment
The output of `python collect_env.py`
root@a3c882d4b9de:/workspace# python3 collect_env.py
INFO 04-15 11:22:23 [__init__.py:239] Automatically detected platform cuda.
Collecting environment information...
PyTorch version: 2.6.0+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A
OS: Ubuntu 22.04.4 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35
Python version: 3.10.12 (main, Feb 4 2025, 14:57:36) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.10.134-17.2.al8.x86_64-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.4.131
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA A800 80GB PCIe
GPU 1: NVIDIA A800 80GB PCIe
GPU 2: NVIDIA A800 80GB PCIe
GPU 3: NVIDIA A800 80GB PCIe
Nvidia driver version: 565.77
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.1.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-cusparselt-cu12==0.6.2
[pip3] nvidia-ml-py==12.570.86
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] pynvml==12.0.0
[pip3] pyzmq==26.3.0
[pip3] torch==2.6.0
[pip3] torchaudio==2.6.0
[pip3] torchvision==0.21.0
[pip3] transformers==4.50.1
[pip3] triton==3.2.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.8.2
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0 GPU1 GPU2 GPU3 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X PHB PHB PHB 0-63 0 N/A
GPU1 PHB X PHB PHB 0-63 0 N/A
GPU2 PHB PHB X PHB 0-63 0 N/A
GPU3 PHB PHB PHB X 0-63 0 N/A
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
NVIDIA_VISIBLE_DEVICES=all
NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536
NCCL_VERSION=2.21.5-1
NVIDIA_DRIVER_CAPABILITIES=compute,utility
NVIDIA_PRODUCT_NAME=CUDA
CUDA_VERSION=12.4.1
LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
NCCL_CUMEM_ENABLE=0
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY
š Describe the bug
model weights offloads to cpu memory when called /sleep api. But after call /wake_up api, the cpu memory do not release. Is this on purpose to fast sleep vllm engine for the next time? In multi model switching scenario, this could bring big pressure on cpu memory for each slept model still occupy cpu memory.
[root@host ~]# free -h
total used free shared buff/cache available
Mem: 225Gi 4.1Gi 174Gi 21Mi 46Gi 219Gi
Swap: 63Gi 245Mi 63Gi
[root@host ~]# curl -XPOST http://localhost:8090/sleep?level=1
[root@host ~]# curl http://localhost:8090/is_sleeping
{"is_sleeping":true}
[root@host ~]# free -h
total used free shared buff/cache available
Mem: 225Gi 5.3Gi 124Gi 49Gi 96Gi 169Gi
Swap: 63Gi 245Mi 63Gi
[root@host ~]# curl -XPOST http://localhost:8090/wake_up
[root@host ~]# curl http://localhost:8090/is_sleeping
{"is_sleeping":false}
[root@host ~]# free -h
total used free shared buff/cache available
Mem: 225Gi 5.3Gi 124Gi 49Gi 96Gi 169Gi
Swap: 63Gi 245Mi 63Gi
when check the memory of progress, it seems many deleted /dev/zero file handle.
[root@host ~]# ps aux | grep spawn
root 198907 1.4 22.7 161240136 53878080 pts/0 Sl+ 16:27 4:14 /usr/bin/python3 -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=16, pipe_handle=18) --multiprocessing-fork
[root@host ~]# lsof -p 198907 | wc -l
485
[root@host ~]# lsof -p 198907 | grep DEL
python3 198907 root DEL REG 0,1 16611 /dev/zero
python3 198907 root DEL REG 0,1 27140 /dev/zero
python3 198907 root DEL REG 0,1 27138 /dev/zero
python3 198907 root DEL REG 0,1 27139 /dev/zero
python3 198907 root DEL REG 0,1 27137 /dev/zero
python3 198907 root DEL REG 0,1 27136 /dev/zero
python3 198907 root DEL REG 0,1 27134 /dev/zero
...
[root@host ~]# cat /proc/198907/smaps
...
7f42c0000000-7f42e0000000 rw-s 00000000 00:01 28385 /dev/zero (deleted)
Size: 524288 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB
Rss: 524288 kB
Pss: 524288 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 524288 kB
Referenced: 524288 kB
Anonymous: 0 kB
LazyFree: 0 kB
AnonHugePages: 0 kB
ShmemPmdMapped: 0 kB
FilePmdMapped: 0 kB
Shared_Hugetlb: 0 kB
Private_Hugetlb: 0 kB
Swap: 0 kB
SwapPss: 0 kB
Locked: 0 kB
DupText: 0 kB
THPeligible: 0
VmFlags: rd wr sh mr mw me ms sd
7f42e4000000-7f42f4000000 rw-s 00000000 00:01 28386 /dev/zero (deleted)
Size: 262144 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB
Rss: 262144 kB
Pss: 262144 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 262144 kB
Referenced: 262144 kB
Anonymous: 0 kB
LazyFree: 0 kB
AnonHugePages: 0 kB
ShmemPmdMapped: 0 kB
FilePmdMapped: 0 kB
Shared_Hugetlb: 0 kB
Private_Hugetlb: 0 kB
Swap: 0 kB
SwapPss: 0 kB
Locked: 0 kB
DupText: 0 kB
THPeligible: 0
VmFlags: rd wr sh mr mw me ms sd
7f42f4000000-7f42f8000000 rw-s 00000000 00:01 28384 /dev/zero (deleted)
Size: 65536 kB
KernelPageSize: 4 kB
MMUPageSize: 4 kB
Rss: 65536 kB
Pss: 65536 kB
Shared_Clean: 0 kB
Shared_Dirty: 0 kB
Private_Clean: 0 kB
Private_Dirty: 65536 kB
Referenced: 65536 kB
Anonymous: 0 kB
LazyFree: 0 kB
AnonHugePages: 0 kB
ShmemPmdMapped: 0 kB
FilePmdMapped: 0 kB
Shared_Hugetlb: 0 kB
Private_Hugetlb: 0 kB
Swap: 0 kB
SwapPss: 0 kB
Locked: 0 kB
DupText: 0 kB
THPeligible: 0
...
We expect cpu memory will be released after wake up the vllm server, is there any solutions?Thanks~
By the way, when offload weights, the used memory seems to be much larger than the size of the model weight. For example, the above model is Qwen2.5-14B-Instruct(about 28GiB), but the offloaded memory takes about 50GiB. Is there any optimization for the memory usage?
Before submitting a new issueā¦
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 15 (9 by maintainers)
it seems I can confirm pytorch caches pinned host memory until we call
torch._C._host_emptyCache():It prints:
thatās quite a long time. then letās keep the current status.
for API users who want to quickly switch models, quick wake-up is important.
for RLHF users, they can switch to sleep with level 2, after @fingertap fixes the buffer issue.
see https://github.com/vllm-project/vllm/blob/fe742aef5aaf406c62cafa248068818bfe517d6e/vllm/distributed/parallel_state.py#L1062-L1066