fast-stable-diffusion: Error when trying to use BLIP, Dreambooth, Hypernetwork and Embeddings. SD 2.1 - Automatic1111 Updated.

@cerarslan I’m working on a fix

yeh same, btw any ideas on this once a training finishes, why i am getting it:

Returning result: Exception training model: ‘Repo id must be in the form ‘repo_name’ or ‘namespace/repo_name’: ‘/content/gdrive/MyDrive/sd/stable-diffusion-webui/models/dreambooth/mayadragon/working/tokenizer’. Use repo_type argument if needed.’.

what is this repo id or name i need to pass, i cant make sense of it

Hey mate, I am facing exactly this issue rn. If you find a solution please share here

@jvkap

EDIT: The following solved the ‘Dreambooth tab not showing in the web UI’ problem.

I solved the HeunDiscreteScheduler error by !pip install git+https://github.com/huggingface/diffusers. This should install a compatible version of Diffusers and fix the error.

But if then throws a transformers error do !pip install git+https://github.com/huggingface/transformers. This should install a compatible version of Transformers.

Having done that you should be left with one last error message :

def convert_text_enc_state_dict_v20(text_enc_dict: dict[str, torch.Tensor]):
TypeError: 'type' object is not subscriptable

This seem to be a Python3.8 problem (i think doesn’t happen with Python 3.11). So open the file "/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/sd_dreambooth_extension/dreambooth/diff_to_sd.py" and then delete the type hint from the parameter of the function so change def convert_text_enc_state_dict_v20(text_enc_dict: dict[str, torch.Tensor]): to def convert_text_enc_state_dict_v20(text_enc_dict). Also change def convert_text_enc_state_dict(text_enc_dict: dict[str, torch.Tensor]) to def convert_text_enc_state_dict(text_enc_dict):. These two deletions are 100% safe.

Save those two edited files, re-run everything and voila! You should finally get the Dreambooth tab.

PS. I’m using model 1.5 but this should work for 2.X

@TheLastBen I was able to fix Hypernetworks and Embedding using this fix.

https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/5523#issuecomment-1343041303 I've added self.logvar = self.logvar.to(self.device) in ddpm.py above line 903 before starting anything.

However, when it goes to generate the example images, they turn black. The normal generation in txt2img is normal. In the automatic’s repo it says that:

If 2.0 or 2.1 is generating black images, enable full precision with --no-half or try using the --xformers optimization.

However, I do not know how to add this to the part only to the generation of previews of Hypernetworks and Embedding. You know and could help me?

Sure! Let me state this again though: this is a bodge, and not only it is a bodge but it is also probably not going to work with SD 2.x! A more order-agnostic process should be made instead of this mess.

First thing is I opened the .ckpt of a model i knew worked and from that I extracted the keys of the state_dict and wrote them in order on a .txt file. Then I changed the code for the converter such that, when a .safetensors model is loaded, the .txt too is opened. And instead of building a dictionary by just reading the .safetensors file and writing the tensors as they are read, I instead read the keys from the .txt in their own order and retrieve the relative tensor from the file.

Essentially, instead of doing this:

model_dict = {}
with safetensors.safe_open("model.safetensors", framework="pt", device="cpu") as f:
    for k in f.keys():
        model_dict[k] = f.get_tensor(k)

I do:

with open("correct_order.txt") as f:
    mylist = f.read().splitlines()

model_dict = {}

with safetensors.safe_open("model.safetensors", framework="pt", device="cpu") as f:
    for l in mylist:
        model_dict[l] = f.get_tensor(l)

This is what I did, sorry if I don’t know this codebase therefore I don’t know how you’re doing the loading 😛. At any rate I suggest in doing this stuff in a way better manner- as said the conversion in my opinion should be agnostic to the order of the given dict (after all it’s a dictionary… its order shouldn’t matter!), otherwise it may create a mess for portability!

Ohh you uptaded your tweaks. Thank you for guidance. I will try today!

Sure, I’ll give it a shot tonight. (Makes me think its worth having a tiny Creative Commons dateset for training with specified captions to keep it standardized). I’ll update once I’ve tested it. I’ll update the Web UI / sd_dreambooth_extention and let you know.

Also thinking about it. Obviously, it depends on the goals of TheLastBen etc. I think having some additional options to update the extensions outside of the SD Webu ui would be useful. I did previously create a subprocess that did this in one of the tweaked colabs I fiddled with.

After much frustration with the syntax with the find program. This could be inserted and do a git pull on each of the extensions that can help updated them. As updating within Colab isn’t the fastest process and forces a restart of the whole GUI in some instances.

#@markdown Update extensions with inbuilt find and execute only on git repositories
%cd /content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions
!find $pwd -mindepth 1 -maxdepth 1 -type d -exec git --git-dir={}/.git --work-tree=$PWD/{} pull \;

Just tested it and works a dream, a little easier to read (maybe) than the sub process version.

I already use your tweaks 1 and 2 so I will try soon.

Nope, surprisingly, this one I don’t think is my fault. 😛

This is the eternal joy that is the “safe unpickle” module of SD-Webui. The goal is to make it so that arbitrary code can’t be executed when unpickling a model. The reality is that it just breaks stuff for no reason.

I’ve got around it in my code by creating a pair of methods that I call to set and unset the --disable-safe-unpickle command when I’m loading a module I specifically know is safe to run, like the YoloV5 library for image detection.

In this instance, I’d be hesitant to use this workaround, as we can’t be sure what ckpt the user is trying to load. As such, this would be one of those discretionary things where it’s up to the user to say “Yes, this is safe” and add the flag manually.

On a separate note - I’ve noticed an uptick in issues from people trying to run this stuff in your collab. I’m totally cool with you using the code - the more the merrier - but it might be a bit easier for both of us if we could talk directly. If you have any questions or issues updating from my nightmare commits, I can answer them, and if I accidentally add another typehint that’s not compatible with Python 3.8, you can let me know easier.

The link is in the “create an issue” section in my repo if you want to hop on and chat. I check in there pretty frequently.

Cool, thanks. I’ll add this upstream.

@markojak

Edit file /content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/sd_dreambooth_extension/dreambooth/finetune_utils.py

In line 330

Change def generate_images(self, prompt_data: list[PromptData]) -> [Image]:

To def generate_images(self, prompt_data):

That will remove the TypeError.

Delete they typehint at 359 as well (->tuple…)

On Mon, Dec 19, 2022 at 9:41 PM mediocreatmybest @.***> wrote:

Just did an update to diffusers and transformers with: !pip install diffusers==0.10.2 transformers==4.25.1

Changed those two lines above as mentioned.

Looks like those two lines fixed the original colab with the extension.

Line 214 to:

def convert_text_enc_state_dict_v20(text_enc_dict):

Line 256 to:

def convert_text_enc_state_dict(text_enc_dict):

[image: dream] https://user-images.githubusercontent.com/80406625/208576918-8f9b7e3a-4914-41bf-a9ae-65438f213f20.png

Figured I should do a quick test run to see if any other errors.

Initializing dreambooth training… Patching transformers to fix kwargs errors. Traceback (most recent call last): File “/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/sd_dreambooth_extension/dreambooth/dreambooth.py”, line 361, in start_training from extensions.sd_dreambooth_extension.dreambooth.train_dreambooth import main File “/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/sd_dreambooth_extension/dreambooth/train_dreambooth.py”, line 359, in <module> def main(args: DreamboothConfig, memory_record, use_subdir, lora_model=None, lora_alpha=1.0, lora_txt_alpha=1.0, custom_model_name=“”) -> tuple[ TypeError: ‘type’ object is not subscriptable Training completed, reloading SD Model.

This is with LORA Off, SD2.1 x768, Xformers, FP16, 8bitAdam enabled

— Reply to this email directly, view it on GitHub https://github.com/TheLastBen/fast-stable-diffusion/issues/914#issuecomment-1358797093, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMO4NHPFFPSUDF5346WOD3WOETFBANCNFSM6AAAAAASXLDMKI . You are receiving this because you were mentioned.Message ID: @.***>

Oooohhh…

I wonder, is it the type hint that’s making it mad?

You could try editing line 214 to read

def convert_text_enc_state_dict_v20(text_enc_dict):

Removing the hint. I bet that fixes it.

On Mon, Dec 19, 2022 at 9:02 PM mediocreatmybest @.***> wrote:

Hey all!

So, I think the issue here (at least from the first post) is that the latest transformers version has some check that throws an issue if it finds extra kwargs.

“/usr/local/lib/python3.8/dist-packages/transformers/generation_utils.py”, line 964, in _validate_model_kwargs raise ValueError( ValueError: The followingmodel_kwargs are not used by the model: [‘encoder_hidden_states’, ‘encoder_attention_mask’] (note: typos in the generate arguments will also show up in this list)

I had that patched out using the “xattention.py” file in my repo, which is just sort of a generic catchall for monkey-patching various bugs I found along the way. But, I bumped my diffusers and transformers versions to match the official diffusers repo (as of ver 0.10.2), and didn’t have to monkey-patch this any more. So, I think the right answer is “check the diffusers and transfomers” versions.

If I missed some other issue in the discussion, throw me an error message and I’ll let you know if it’s anything else I hacked in to fix stuff.

@ekeric13 https://github.com/ekeric13 - “Launch” refers to launch.py from the auto1111 script. It’s just a wrapper for the run_command call, which I should probably just remove and add my own, as I’m already working to make less of the extension depend directly on Auto1111, just for compatibility and sanity’s sake. 😉

Looks like it is still an issue even when updating transformers in the standard colab as mentioned this seems to be mainly a python 3.8 issue?

Dreambooth API layer loaded Error loading script: main.py Traceback (most recent call last): File “/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/scripts.py”, line 195, in load_scripts module = script_loading.load_module(scriptfile.path) File “/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/script_loading.py”, line 13, in load_module exec(compiled, module.dict) File “/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/sd_dreambooth_extension/scripts/main.py”, line 5, in <module> from extensions.sd_dreambooth_extension.dreambooth.diff_to_sd import compile_checkpoint File “/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/sd_dreambooth_extension/dreambooth/diff_to_sd.py”, line 214, in <module> def convert_text_enc_state_dict_v20(text_enc_dict: dict[str, torch.Tensor]): TypeError: ‘type’ object is not subscriptable

— Reply to this email directly, view it on GitHub https://github.com/TheLastBen/fast-stable-diffusion/issues/914#issuecomment-1358774551, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMO4NC5TTYN22WVZUD5FADWOEOTVANCNFSM6AAAAAASXLDMKI . You are receiving this because you were mentioned.Message ID: @.***>

Hey all!

So, I think the issue here (at least from the first post) is that the latest transformers version has some check that throws an issue if it finds extra kwargs.

"/usr/local/lib/python3.8/dist-packages/transformers/generation_utils.py", line 964, in _validate_model_kwargs raise ValueError( ValueError: The followingmodel_kwargs are not used by the model: ['encoder_hidden_states', 'encoder_attention_mask'] (note: typos in the generate arguments will also show up in this list)

I had that patched out using the “xattention.py” file in my repo, which is just sort of a generic catchall for monkey-patching various bugs I found along the way. But, I bumped my diffusers and transformers versions to match the official diffusers repo (as of ver 0.10.2), and didn’t have to monkey-patch this any more. So, I think the right answer is “check the diffusers and transfomers” versions.

If I missed some other issue in the discussion, throw me an error message and I’ll let you know if it’s anything else I hacked in to fix stuff.

@ekeric13 - “Launch” refers to launch.py from the auto1111 script. It’s just a wrapper for the run_command call, which I should probably just remove and add my own, as I’m already working to make less of the extension depend directly on Auto1111, just for compatibility and sanity’s sake. 😉

Ok. So we fixed the Textual Inversion part, but we’re still buggy in Lora and Dreambooth. Is that correct?

It would be interesting to have help from @d8ahazard for that Dreambooth issue.

And @cloneofsimo for Lora problem.

Would anyone be interested in opening an issue in their repo?

Dreambooth works for me but only after making the changes I suggested above.

Wait what? I didn’t know this even existed. You guys have adapted LoRA here?

@mediocreatmybest

Amazing! I’ll test it and get back if anything goes wrong. Thanks!