diffusers: The loaders load_lora_into_unet() function throw out "AttributeError: 'ModuleList' object has no attribute '4'"

Describe the bug

In Diffusers Version: 0.20.0.dev0

When loading SDXL LoRA to SDXL pipeline.

Works by

lora_path = r"path/to/SDXLLoRA.safetensors"
base_pipe.load_lora_weights(lora_path)

However, when extracting state_dict, and network_alphas manually feed into unet like this:

state_dict, network_alphas = base_pipe.lora_state_dict(
    lora_path
)
base_pipe.load_lora_into_unet(
    state_dict
    , network_alphas = network_alphas
    , unet = base_pipe.unet
)

Throws out errors

[1612](file:///d%3A/az_git_folder/azcode/azcode_venv_win_p310/lib/site-packages/torch/nn/modules/module.py?line=1611)     if name in modules:
   [1613](file:///d%3A/az_git_folder/azcode/azcode_venv_win_p310/lib/site-packages/torch/nn/modules/module.py?line=1612)         return modules[name]
-> [1614](file:///d%3A/az_git_folder/azcode/azcode_venv_win_p310/lib/site-packages/torch/nn/modules/module.py?line=1613) raise AttributeError("'{}' object has no attribute '{}'".format(
   [1615](file:///d%3A/az_git_folder/azcode/azcode_venv_win_p310/lib/site-packages/torch/nn/modules/module.py?line=1614)     type(self).__name__, name))

AttributeError: 'ModuleList' object has no attribute '4'

Is this a bug, or anything I missed somewhere? my understanding is that load_lora_weights() is basically take the state_dict, network_alphas then call load_lora_into_unet and load_lora_into_text_encoder, why would this exception been thrown out? Thanks

Reproduction

Sample code to reproduce the issue:

import torch
from diffusers import StableDiffusionXLPipeline

# load SDXL base pipeline
base_model_checkpoint_path = r"path/to/sd_xl_base_1.0.safetensors"
base_pipe = StableDiffusionXLPipeline.from_single_file(
    base_model_checkpoint_path
    , torch_dtype = torch.float16
    , use_safetensors = True
    , load_safety_checker = False
)

# add LoRA
lora_path = r"path/to/SDXLLoRA.safetensors"
state_dict, network_alphas = base_pipe.lora_state_dict(
    lora_path
)
base_pipe.load_lora_into_unet(
    state_dict
    , network_alphas = network_alphas
    , unet = base_pipe.unet
)

Logs

No response

System Info

Diffusers Version: 0.20.0.dev0 Python version: Python 3.10.11 Torch version: 2.0.0+cu117

Who can help?

@patrickvonplaten @sayakpaul

About this issue

Original URL
State: closed
Created a year ago
Comments: 16 (14 by maintainers)

Most upvoted comments

https://colab.research.google.com/gist/sayakpaul/7c1d04ffd94483b5b99c2b24f910c5ca/scratchpad.ipynb

see if this works 😃

OMG, your solution works! Thank You so much @sayakpaul Your sample code tells everything!

I go back to the StableDiffusionXLPipeline code and found the overrided load_lora_weights function used the unet_config=self.unet.config

...
def load_lora_weights(self, pretrained_model_name_or_path_or_dict: Union[str, Dict[str, torch.Tensor]], **kwargs):
      # We could have accessed the unet config from `lora_state_dict()` too. We pass
      # it here explicitly to be able to tell that it's coming from an SDXL
      # pipeline.
      state_dict, network_alphas = self.lora_state_dict(
          pretrained_model_name_or_path_or_dict,
          unet_config=self.unet.config,
          **kwargs,
      )
      self.load_lora_into_unet(state_dict, network_alphas=network_alphas, unet=self.unet)
     ....

Again, Thank You @sayakpaul !

xhinker on Aug 6, 2023

https://colab.research.google.com/gist/sayakpaul/7c1d04ffd94483b5b99c2b24f910c5ca/scratchpad.ipynb

see if this works 😃

sayakpaul on Aug 6, 2023

Can you also check if you can successfully call load_lora_into_unet into a pipeline when the pipeline is initialised with from_pretrained() and NOT using the single checkpoint loading method? Will help us to track this issue down.

sayakpaul on Aug 4, 2023

SDXL has some implications on the loading side, that is why. For now, we find it easier to always promote load_lora_weights().

The thing I really don’t understand is the load_lora_weights function is just a wrapper calls lora_state_dict, load_lora_into_unet, and then load_lora_into_text_encoder. here is a copy of the function body from the source code:

def load_lora_weights(self, pretrained_model_name_or_path_or_dict: Union[str, Dict[str, torch.Tensor]], **kwargs):
      state_dict, network_alphas = self.lora_state_dict(pretrained_model_name_or_path_or_dict, **kwargs)
      self.load_lora_into_unet(state_dict, network_alphas=network_alphas, unet=self.unet)
      self.load_lora_into_text_encoder(
          state_dict,
          network_alphas=network_alphas,
          text_encoder=self.text_encoder,
          lora_scale=self.lora_scale,
      )

Why it works inside of load_lora_weights, but not working by calling it outside? even the outside call load_lora_into_text_encoder works without a problem. but only load_lora_into_unet always throw out exception, which really puzzles me.

xhinker on Aug 4, 2023