diffusers: Black image on SDXL 1.0 + RuntimeWarning: invalid value encountered in cast images = (images * 255).round().astype("uint8")

Describe the bug

I’m trying to run SDXL in a container environment (Debian). I tried both diffusers[torch]==0.18.0 and diffusers[torch]==0.19.0 with Python 3.10 on A10g and A100 but I get a black image back and RuntimeWarning: invalid value encountered in cast images = (images * 255).round().astype(“uint8”) when running 0.18.0.

Basically, when building the container’s image I’m doing:

 pipe = diffusers.DiffusionPipeline.from_pretrained(
        model_id,
        use_auth_token=hugging_face_token,
        torch_dtype=torch.float16,
        use_safetensors=True, 
        variant="fp16",
        cache_dir=cache_path
    )
pipe.save_pretrained(cache_path, safe_serialization=True)

After that, when running the function I’m calling:

self.pipe = diffusers.DiffusionPipeline.from_pretrained(
            cache_path).to("cuda")

With 0.19.0 I’m getting Out of memory errors. .enable_model_cpu_offload() didn’t work for me. Anyway, with both versions in the end I get a black image.

Reproduction

pipe = diffusers.DiffusionPipeline.from_pretrained(
        model_id,
        use_auth_token=hugging_face_token,
        torch_dtype=torch.float16,
        use_safetensors=True, 
        variant="fp16",
        cache_dir=cache_path
    )
pipe.save_pretrained(cache_path, safe_serialization=True)
    
pipe = diffusers.DiffusionPipeline.from_pretrained(
            cache_path).to("cuda")

Logs

The config attributes {'force_upcast': True} were passed to AutoencoderKL, but are not expected and will be ignored. Please verify your config.json configuration file.
 98%|█████████▊| 49/50 [00:10<00:00,  5.28it/s]100%|██████████| 50/50 [00:10<00:00,  5.27it/s]100%|██████████| 50/50 [00:10<00:00,  4.90it/s]
/usr/local/lib/python3.10/site-packages/diffusers/image_processor.py:65: RuntimeWarning: invalid value encountered in cast
  images = (images * 255).round().astype("uint8")

System Info

0.18.0 and 0.19.0. Linux Debian (don’t have access to the container now).

Who can help?

@patrickvonplaten

About this issue

Original URL
State: closed
Created a year ago
Reactions: 1
Comments: 17 (10 by maintainers)

Most upvoted comments

OK, I am only seeing this when generating validation images for wandb using the dreambooth_sdxl_lora training script. Using master version of diffusers and the training script. Here are my params:
“pretrained_model_name_or_path”: “stabilityai/stable-diffusion-xl-base-1.0”,

“instance_data_dir”: “/local/images”,

“output_dir”: “/lora/path”,

“mixed_precision”: “fp16”,

“instance_prompt”: “a man”,

“resolution”: 1024,

“train_batch_size”: 1,

“gradient_accumulation_steps”: 4,

“learning_rate”: 0.001,

“report_to”: “wandb”,

“lr_scheduler”: “constant”,

“lr_warmup_steps”: 0,

“max_train_steps”: 500,

“validation_prompt”: f"An illustration of a man",

“validation_epochs”: 25,

“seed”: 0,
All the images in wandb turn out black. I am running this in a container, so there is no pre-cached model I’m using. I looked into the pipeline that the training script uses for validation images and there’s nothing out of the ordinary. Here’s the output:
Loaded scheduler as EulerDiscreteScheduler from scheduler subfolder of stabilityai/stable-diffusion-xl-base-1.0.

Loaded tokenizer as CLIPTokenizer from tokenizer subfolder of stabilityai/stable-diffusion-xl-base-1.0.

Loaded tokenizer_2 as CLIPTokenizer from tokenizer_2 subfolder of stabilityai/stable-diffusion-xl-base-1.0.]

Loading pipeline components…: 100%|██████████| 7/7 [00:00<00:00, 36.25it/s]

torch_dtype: torch.float16, device: cuda

{‘algorithm_type’, ‘lambda_min_clipped’, ‘dynamic_thresholding_ratio’, ‘solver_order’, ‘thresholding’, ‘variance_type’, ‘solver_type’, ‘lower_order_final’} was not found in config. Values will be initialized to default values.

/usr/local/lib/python3.9/site-packages/diffusers/image_processor.py:65: RuntimeWarning: invalid value encountered in cast

images = (images * 255).round().astype(“uint8”)
I am facing this same issue. I can see the model getting fine-tuned but validation and test images are all black. Switching the VAE can work. But I am using an already fine-tuned model as the base model and keeping the original VAE has benefits(Please let me know if I am wrong). Since the validation images are corrupt not sure if fine-tuning is done properly. I am using tensorboard and not WANDB. I tried disabling mixed precision but it didn’t work.

Your setup is correct. The mixed precision you’re using is probably fp16 and the vae is broken with fp16. You can either set the vae to run in fp32, or use a fixed vae like https://huggingface.co/madebyollin/sdxl-vae-fp16-fix

levi on Mar 23, 2024

Ah ok the last code snippet seems correct. Note that you can speed it up by using less inference steps and potentially a better scheduler such as: https://huggingface.co/docs/diffusers/api/schedulers/unipc

patrickvonplaten on Aug 3, 2023

The solution was to switch out the VAE to (https://huggingface.co/madebyollin/sdxl-vae-fp16-fix) based on this note: https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/README_sdxl.md#specifying-a-better-vae

levi on Jul 28, 2023