stable-diffusion.cpp: SDXL : LoRa problem

When I use the following command:

sd.exe -m ..\models\sd_xl_base_1.0.safetensors --vae ..\models\sdxl_vae.safetensors --lora-model-dir ..\models -H 1024 -W 1024 -p "a lovely cat<lora:sd_xl_offset_example-lora_1.0:1>" -v

The lora model apparently can not be used:

[INFO ] model.cpp:705  - load ..\models/sd_xl_offset_example-lora_1.0.safetensors using safetensors format
[DEBUG] model.cpp:771  - init from '..\models/sd_xl_offset_example-lora_1.0.safetensors'
[INFO ] lora.hpp:38   - loading LoRA from '..\models/sd_xl_offset_example-lora_1.0.safetensors'
[DEBUG] model.cpp:1343 - loading tensors from ..\models/sd_xl_offset_example-lora_1.0.safetensors
[DEBUG] ggml_extend.hpp:884  - lora params backend buffer size =  47.01 MB(VRAM) (10240 tensors)
[DEBUG] model.cpp:1343 - loading tensors from ..\models/sd_xl_offset_example-lora_1.0.safetensors
[DEBUG] lora.hpp:74   - finished loaded lora
[WARN ] lora.hpp:160  - unused lora tensor lora.unet_input_blocks_1_0_emb_layers_1.alpha
[WARN ] lora.hpp:160  - unused lora tensor lora.unet_input_blocks_1_0_emb_layers_1.lora_down.weight
[WARN ] lora.hpp:160  - unused lora tensor lora.unet_input_blocks_1_0_emb_layers_1.lora_up.weight
[WARN ] lora.hpp:160  - unused lora tensor lora.unet_input_blocks_1_0_in_layers_2.alpha
[WARN ] lora.hpp:160  - unused lora tensor lora.unet_input_blocks_1_0_in_layers_2.lora_down.weight
[WARN ] lora.hpp:160  - unused lora tensor lora.unet_input_blocks_1_0_in_layers_2.lora_up.weight
...
(hundreds of same warnings)

It’s the same problem as https://github.com/leejet/stable-diffusion.cpp/pull/117#issuecomment-1856518605. The lora model is eventually not used at all.

I’m using the latest master-a469688 release.

About this issue

Original URL
State: closed
Created 3 months ago
Reactions: 1
Comments: 17 (15 by maintainers)

Most upvoted comments

Alright, in that case it sounds like a separate quantization issue distinct from this one. I propose that this issue be marked as resolved.

grauho on Mar 20, 2024

Fixed in the new release master-90e9178 Thanks!

piallai on Apr 3, 2024

I tried with the new release master-48bcce4. The program now crashes when loading LoRa. Here is the log, with the same command as in the OP.

[INFO ] stable-diffusion.cpp:553  - Attempting to apply 1 LoRAs
[INFO ] model.cpp:726  - load ..\models\sd_xl_offset_example-lora_1.0.safetensors using safetensors format
[DEBUG] model.cpp:792  - init from '..\models\sd_xl_offset_example-lora_1.0.safetensors'
[INFO ] lora.hpp:38   - loading LoRA from '..\models\sd_xl_offset_example-lora_1.0.safetensors'
[DEBUG] model.cpp:1364 - loading tensors from ..\models\sd_xl_offset_example-lora_1.0.safetensors
Assertion failed: n_dims >= 1 && n_dims <= GGML_MAX_DIMS, file C:\***\ggml\src\ggml.c, line 2745

With n_dims valued 0. Not exactly the same problem, but still related to LoRa loading. I suppose it’s worth leaving the issue open.

piallai on Mar 22, 2024

Now it is crashing with

ggml_cuda_op_bin_bcast: unsupported types: dst: q8_0, src0: q8_0, src1: f32

probably because loras contains some f32 AND q8_0+f32->q8_0 is not supported. conversion seems to work but its not loading it (only looking for safetensors and ckpt)

Green-Sky on Mar 20, 2024

I had the same issue and addressed it in my pending pull request #200

From what I can tell it is because the SDXL LoRAs use a slightly different naming convention that the current code isn’t set up to properly convert to the internally used convention. Also, it seems like the existing memory allocated for the GGML graph is insufficient to accommodate adding a SDXL LoRA so I had to bump that up as well.

grauho on Mar 16, 2024