diffusers: Error while loading Lora

Describe the bug

According to the guidance , I try to finetune the SD with lora, and use the lora weight, but the issue appears.

Traceback (most recent call last):
  File "AutoPipelineForTex.py", line 17, in <module>
    pipe.unet.load_attn_procs("/code/pytorch_lora_weights.safetensors")
  File "/usr/local/lib/python3.8/dist-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/diffusers/loaders/unet.py", line 264, in load_attn_procs
    rank = value_dict["lora.down.weight"].shape[0]
KeyError: 'lora.down.weight

When I try the following codes, the above isssue appears。

from diffusers import AutoPipelineForText2Image
import torch

pipeline = AutoPipelineForText2Image.from_pretrained("runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16).to("cuda")
pipeline.load_lora_weights("/sddata/finetune/lora/pokemon/", weight_name="pytorch_lora_weights.safetensors")
image = pipeline("Green pokemon with menacing face").images[0]
image.save("green_pokemon.png")

The safetensors file was created when run the following cmd: python train_text_to_image_lora.py --pretrained_model_name_or_path=$MODEL_NAME --dataset_name=$DATASET_NAME --dataloader_num_workers=8 --resolution=512 --center_crop --random_flip --train_batch_size=1 --gradient_accumulation_steps=4 --max_train_steps=15000 --learning_rate=1e-04 --max_grad_norm=1 --lr_scheduler=“cosine” --lr_warmup_steps=0 --output_dir=${OUTPUT_DIR} --checkpointing_steps=500 --validation_prompt=“A pokemon with blue eyes.” --seed=1337

Reproduction

from diffusers import AutoPipelineForText2Image
import torch

pipeline = AutoPipelineForText2Image.from_pretrained("runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16).to("cuda")
pipeline.load_lora_weights("/sddata/finetune/lora/pokemon/", weight_name="pytorch_lora_weights.safetensors")
image = pipeline("Green pokemon with menacing face").images[0]
image.save("green_pokemon.png")

Logs

No response

System Info

Python: 3.8.10 diffusers: 0.25.0.dev0

Who can help?

No response