CogVideo: CUDA OOM and possible solution -- diffusers cli_demo.py with Nvidia 3090 24GB
System Info / 系統信息
Thanks very much for releasing this great work!
In case this helps anyone else:
The diffusers cli_demo.py raised the CUDA OOM error below on an RTX 3090 with 24GB of VRAM using this command:
python cli_demo.py --prompt "A fish swimming underwater through a colorful coral reef. Sun is shining brightly through the water. It is a beautiful scene suitable for use in an eye-catching television advertisement." --model_path THUDM/CogVideoX-2b --num_inference_steps 50
… but works with barely enough free VRAM with this small adjustment – set the env var PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True prior to running:
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True python cli_demo.py --prompt "A fish swimming underwater through a colorful coral reef. Sun is shining brightly through the water. It is a beautiful scene suitable for use in an eye-catching television advertisement." --model_path THUDM/CogVideoX-2b --num_inference_steps 50
CUDA OOM error:
return torch._C._nn.pad(input, pad, mode, value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.66 GiB. GPU 0 has a total capacity of 23.48 GiB of which 1.53 GiB is free. Including non-PyTorch memory, this process has 21.81 GiB memory in use. Of the allocated memory 18.92 GiB is allocated by PyTorch, and 2.58 GiB is reserved by PyTorch but unallocated.
nvidia-smi before running – ony 135MiB VRAM used:
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 3090 Off | 00000000:09:00.0 Off | N/A |
| 0% 32C P8 17W / 350W | 135MiB / 24576MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
nvidia-smi while running with PYTORCH_CUDA_ALLOC_CONF=exandable_segments:True – 23GB used but no CUDA OOM.
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 3090 Off | 00000000:09:00.0 Off | N/A |
| 0% 40C P2 157W / 350W | 23335MiB / 24576MiB | 4% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
This was using a fresh conda environment with dependencies installed by pip from requirements.txt (side note: please include imageio in requirements.txt and also open the version range for opencv-python to >=4.10 as 4.10 was yanked from upstream)
Information / 问题信息
- The official example scripts / 官方的示例脚本
- My own modified scripts / 我自己修改的脚本和任务
Reproduction / 复现过程
Details above.
Expected behavior / 期待表现
Details above.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 27 (10 by maintainers)
Not now, we will try tied vae, we tested balance in 3 GPU in 20GB. you can try if you can run at 3 * 16G GPU
must one or more gpus >= 20GB beacuse of VAE.
Installed flash_attn 2.0.4 built from source, not sure how to force torch to let me use it
Output
EDIT: SDPA is disabled on AMD cards because they decided that only cool kids can play with fun toys https://github.com/pytorch/pytorch/issues/112997