CogVideo: Slow loading of model and very delay in image to video
Hii,
I am facing issue with delay in model loading and also the time taken to generate the video from Image. Currently it is taking 8minutes for 8 seconds video, I have 48GB VRAM , but still it is very slow.
Please let me know , if there is any way to solve this.
This is the code im using .
import torch
from diffusers import CogVideoXImageToVideoPipeline
from diffusers.utils import export_to_video, load_image
from diffusers import (
CogVideoXPipeline,
CogVideoXDPMScheduler,
CogVideoXVideoToVideoPipeline,
CogVideoXImageToVideoPipeline,
CogVideoXTransformer3DModel,
)
print('loading I2V model...')
pipe_image = CogVideoXImageToVideoPipeline.from_pretrained(
"THUDM/CogVideoX-5b-I2V",
transformer=CogVideoXTransformer3DModel.from_pretrained(
"THUDM/CogVideoX-5b-I2V", subfolder="transformer", torch_dtype=torch.bfloat16
),
torch_dtype=torch.bfloat16
).to("cuda")
import random
seed = random.randint(0, 2**8 - 1)
print('loading image..')
image = load_image(
"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg"
)
prompt = "An astronaut hatching from an egg, on the surface of the moon, the darkness and depth of space realised in the background. High quality, ultrarealistic detail and breath-taking movie-like camera shot."
negative_prompt ="The video is not of a high quality, it has a low resolution. Strange motion trajectory. Flickering, Blurriness, Face restore.Deformation, anime, cartoon, graphic, text, painting, crayon, graphite, abstract, glitch, deformed, mutated, ugly, disfigured "
video_pt = pipe_image(
image=image,
prompt=prompt,
negative_prompt=negative_prompt,
num_inference_steps=50,
num_videos_per_prompt=1,
use_dynamic_cfg=True,
output_type="pt",
guidance_scale=7.0,
num_frames=49,
generator=torch.Generator(device="cuda").manual_seed(seed),
).frames
batch_video_frames = []
batch_size = video_pt.shape[0]
from diffusers.image_processor import VaeImageProcessor
for batch_idx in range(batch_size):
pt_image = video_pt[batch_idx]
pt_image = torch.stack([pt_image[i] for i in range(pt_image.shape[0])])
image_np = VaeImageProcessor.pt_to_numpy(pt_image)
image_pil = VaeImageProcessor.numpy_to_pil(image_np)
batch_video_frames.append(image_pil)
export_to_video(batch_video_frames[0], "videos/output.mp4", fps=8)
Thanks in Advance
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 23 (8 by maintainers)
https://github.com/THUDM/CogVideo/blob/main/inference/cli_demo.py follow this and remove the
pipe_image.enable_sequential_cpu_offload()and usepipe.to("cuda")For 4090, you can completely remove
and just move pipe.to(“cuda”), should work Currently, there is indeed no way to visualize the intermediate results