CogVideo: Slow loading of model and very delay in image to video

Hii,

I am facing issue with delay in model loading and also the time taken to generate the video from Image. Currently it is taking 8minutes for 8 seconds video, I have 48GB VRAM , but still it is very slow.

Please let me know , if there is any way to solve this.

This is the code im using .

import torch
from diffusers import CogVideoXImageToVideoPipeline
from diffusers.utils import export_to_video, load_image

from diffusers import (
    CogVideoXPipeline,
    CogVideoXDPMScheduler,
    CogVideoXVideoToVideoPipeline,
    CogVideoXImageToVideoPipeline,
    CogVideoXTransformer3DModel,
)
print('loading I2V model...')
pipe_image = CogVideoXImageToVideoPipeline.from_pretrained(
    "THUDM/CogVideoX-5b-I2V",
    transformer=CogVideoXTransformer3DModel.from_pretrained(
        "THUDM/CogVideoX-5b-I2V", subfolder="transformer", torch_dtype=torch.bfloat16
    ),
    torch_dtype=torch.bfloat16
).to("cuda")

import random
seed = random.randint(0, 2**8 - 1)
print('loading image..')
image = load_image(
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg"
    )
prompt = "An astronaut hatching from an egg, on the surface of the moon, the darkness and depth of space realised in the background. High quality, ultrarealistic detail and breath-taking movie-like camera shot."

negative_prompt ="The video is not of a high quality, it has a low resolution. Strange motion trajectory. Flickering, Blurriness, Face restore.Deformation, anime, cartoon, graphic, text, painting, crayon, graphite, abstract, glitch, deformed, mutated, ugly, disfigured "
video_pt = pipe_image(
            image=image,
            prompt=prompt,
            negative_prompt=negative_prompt,
            num_inference_steps=50,
            num_videos_per_prompt=1,
            use_dynamic_cfg=True,
            output_type="pt",
            guidance_scale=7.0,
            num_frames=49,
            generator=torch.Generator(device="cuda").manual_seed(seed),
        ).frames

batch_video_frames = []
batch_size = video_pt.shape[0]
from diffusers.image_processor import VaeImageProcessor
for batch_idx in range(batch_size):
    pt_image = video_pt[batch_idx]
    pt_image = torch.stack([pt_image[i] for i in range(pt_image.shape[0])])

    image_np = VaeImageProcessor.pt_to_numpy(pt_image)
    image_pil = VaeImageProcessor.numpy_to_pil(image_np)
    batch_video_frames.append(image_pil)
export_to_video(batch_video_frames[0], "videos/output.mp4", fps=8)

Thanks in Advance

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 23 (8 by maintainers)

Most upvoted comments

https://github.com/THUDM/CogVideo/blob/main/inference/cli_demo.py follow this and remove the pipe_image.enable_sequential_cpu_offload() and use pipe.to("cuda")

For 4090, you can completely remove

pipe_image.enable_sequential_cpu_offload()

and just move pipe.to(“cuda”), should work Currently, there is indeed no way to visualize the intermediate results