diffusers: LoRA quality degradation -> xformers?

Describe the bug

I have a forked diffusers repo that I try to keep up to date. My latest sync was from around ~3 weeks ago. This week I synced again and tried to do some quality tests with my Stable Diffusion XL LoRA inference engine. However, I noticed some really strange quality degradation.

This table corresponds to the images below:

previous inference	current inference
LoRA trained w/ old diffusers	LoRA trained w/ old diffusers
LoRA trained w/ new diffusers	LoRA trained w/ new diffusers

There are slight differences in the old and new training scripts (vertical), but there are large and breaking differences in the inference scripts.

The old inference script is based on a custom SDXLControlnetImg2Img class. As a sanity check, I ditched my newly synced branch and installed the latest huggingface/diffusers commit, still the same behavior.

I uninstalled the latest commit and installed the latest release package (v0.21.2), still the same.

To be complete, LoRAs are loading like this: self.pipeline.load_lora_weights(lora_dir_path)

Any insights would be appreciated. I suspect it’s the LoRA loading refactor.

Update: Running the same prompt more than once gives some weird results: When I run "a render of <mp>" at a fixed seed I get this:

For each exact subsequent call with the exact same parameters I get:

Something recursive?

Reproduction

Test a LoRA on:

(good that I know of) https://github.com/huggingface/diffusers/commit/4909b1e3ac8446dc4ebc340cde8e5b236689ff0b

Vs new e.g.: https://github.com/huggingface/diffusers/commit/157c9011d87e52632113024c1dc5125426971556

Logs

Getting this after loading loras:

diffusers/models/attention_processor.py:313: FutureWarning: set_processor to offload LoRA is deprecated and will be removed in version 0.26.0. In detail, removing LoRA layers via calling set_processor or set_default_attn_processor is deprecated. Please make sure to call pipe.unload_lora_weights() instead. deprecate(

System Info

• diffusers version: 0.22.0.dev0
• Platform: Linux-6.2.0-32-generic-x86_64-with-glibc2.35
• Python version: 3.10.12
• PyTorch version (GPU?): 2.0.1+cu118 (True)
• Huggingface_hub version: 0.16.4
• Transformers version: 4.31.0
• Accelerate version: 0.21.0
• xFormers version: 0.0.20
• Using GPU in script?: True
• Using distributed or parallel set-up in script?: False

Who can help?

@patrickvonplaten @sayakpaul