ml-stable-diffusion: Image generation not working with controlNet on iPadOS. Out of memory crash

Hi! 😃

I’m running into an out-of-memory crash on a physical iPad Pro M2 (2022) when trying to generate an image with a controlNet model.

Specs: iPad Pro 11-inch (4th generation) iPadOS 17.0 Xcode 15.0.0 ml-stable-diffusion 1.0.0

Models used: Stable Diffusion 1.5: https://huggingface.co/runwayml/stable-diffusion-v1-5 ControlNet Canny: https://huggingface.co/lllyasviel/sd-controlnet-canny

Parameters: imageCount = 1 stepCount = 15 scheduler = .dpmSolverMultistepScheduler disableSafety = true reduceMemory = true computeUnits = .cpuAndNeuralEngine

Converted the models using the provided python package with this command:

python -m python_coreml_stable_diffusion.torch2coreml --convert-unet --convert-text-encoder --convert-vae-decoder --convert-safety-checker --model-version "runwayml/stable-diffusion-v1-5" --chunk-unet --unet-support-controlnet --convert-controlnet "lllyasviel/sd-controlnet-canny" --bundle-resources-for-swift-cli -o "/path/to/save/the/converted/model"

Also tried with a model generated with an added --quantize-nbits 6 argument, same result.

The StableDiffusionPipeline is successfully created, and the out-of-memory crash happens when the generator is already sending progress callbacks.

Without controlNet the generation process is smooth and really fast. The crash happens only on a physical device, on an iPad simulator it successfully generates an image (though takes ages).

The heaviest stuff is happening inside Unet.predictions(from:) method CleanShot 2023-06-21 at 11 24 15@2x

P.S. Thanks for such an exciting package! Hoping to get to play around with controlNet soon 😃

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 1
  • Comments: 19 (3 by maintainers)

Most upvoted comments

Hey @TimYao18 I’m running this on iPadOS, so my model files are inside the project directory, and I access them like so:

guard let path = Bundle.main.path(forResource: "SD_1.5_canny_openpose", ofType: nil, inDirectory: "SD_Models") else {
    fatalError("Fatal error: failed to find the CoreML models.")
}
let resourceURL = URL(fileURLWithPath: path)

Where SD_Models is a folder in the project root, and SD_1.5_canny_openpose is a folder inside it (with the models)

You can check out the MochiDiffusion repo to see how they do it on macOS

Hi @atiorh! I realized that I was missing the VAEEncoder.mlmodelc, adding the --convert-vae-encoder to my command converted it successfully, and I’m able to run controlNet with startingImage now.

If I’m not missing anything, the --convert-vae-encoder parameter is not listed anywhere in the “Converting Models to Core ML” section of the readme, so it was a lucky guess 😁

Closing the issue as it’s resolved. Also here’s the final conversion command used, maybe it’ll help somebody like me out 😃 :

python -m python_coreml_stable_diffusion.torch2coreml --convert-unet --convert-text-encoder --convert-vae-decoder --convert-vae-encoder --convert-safety-checker --model-version "runwayml/stable-diffusion-v1-5" --unet-support-controlnet --quantize-nbits 6 --attention-implementation SPLIT_EINSUM_V2 --convert-controlnet "lllyasviel/sd-controlnet-canny" --bundle-resources-for-swift-cli -o "/path/to/save/the/model"

Have you tried with a conversion that uses both --quantize-nbits 6 and --attention-implementation SPLIT_EINSUM_V2 ? Along with --chunk-unet, that should get you every optimization available for mobile devices.