ml-stable-diffusion: Image generation not working with controlNet on iPadOS. Out of memory crash
Hi! 😃
I’m running into an out-of-memory crash on a physical iPad Pro M2 (2022) when trying to generate an image with a controlNet model.
Specs: iPad Pro 11-inch (4th generation) iPadOS 17.0 Xcode 15.0.0 ml-stable-diffusion 1.0.0
Models used: Stable Diffusion 1.5: https://huggingface.co/runwayml/stable-diffusion-v1-5 ControlNet Canny: https://huggingface.co/lllyasviel/sd-controlnet-canny
Parameters: imageCount = 1 stepCount = 15 scheduler = .dpmSolverMultistepScheduler disableSafety = true reduceMemory = true computeUnits = .cpuAndNeuralEngine
Converted the models using the provided python package with this command:
python -m python_coreml_stable_diffusion.torch2coreml --convert-unet --convert-text-encoder --convert-vae-decoder --convert-safety-checker --model-version "runwayml/stable-diffusion-v1-5" --chunk-unet --unet-support-controlnet --convert-controlnet "lllyasviel/sd-controlnet-canny" --bundle-resources-for-swift-cli -o "/path/to/save/the/converted/model"
Also tried with a model generated with an added --quantize-nbits 6 argument, same result.
The StableDiffusionPipeline is successfully created, and the out-of-memory crash happens when the generator is already sending progress callbacks.
Without controlNet the generation process is smooth and really fast. The crash happens only on a physical device, on an iPad simulator it successfully generates an image (though takes ages).
The heaviest stuff is happening inside Unet.predictions(from:) method
P.S. Thanks for such an exciting package! Hoping to get to play around with controlNet soon 😃
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 1
- Comments: 19 (3 by maintainers)
Hey @TimYao18 I’m running this on iPadOS, so my model files are inside the project directory, and I access them like so:
Where SD_Models is a folder in the project root, and SD_1.5_canny_openpose is a folder inside it (with the models)
You can check out the MochiDiffusion repo to see how they do it on macOS
Hi @atiorh! I realized that I was missing the
VAEEncoder.mlmodelc, adding the--convert-vae-encoderto my command converted it successfully, and I’m able to run controlNet withstartingImagenow.If I’m not missing anything, the
--convert-vae-encoderparameter is not listed anywhere in the “Converting Models to Core ML” section of the readme, so it was a lucky guess 😁Closing the issue as it’s resolved. Also here’s the final conversion command used, maybe it’ll help somebody like me out 😃 :
Have you tried with a conversion that uses both
--quantize-nbits 6and--attention-implementation SPLIT_EINSUM_V2? Along with--chunk-unet, that should get you every optimization available for mobile devices.