onnxruntime: [Performance] Why is dynamic shape not supported with the CoreML provider, while CoreML 2+ supports it ?

From what I understand, CoreML allows 3 approaches for dynamic shapes: -a set of predetermined shapes -bounded ranges -unbounded ranges

Support for any of these mode would be a significant improvement over the single fixed shape situation we have currently. I don’t have any personal preference as long as I can at least play with a couple different shapes.

I suppose the easiest would be to implement the unbounded range mode, which is a straightforward application of the ONNX’s unbounded dimensions.

I added some checks to limit the shapes to at most rank 5 in the CoreML EP. That limit seems to be inherent to CoreML at the moment. ORT should no longer fail to load the model because of CoreML compilation errors due to that, but it limits the nodes that are assigned to the CoreML EP.

https://github.com/microsoft/onnxruntime/pull/17086

This SAM model has some shapes with rank greater than 5, so those nodes won’t be supported by the CoreML EP. Maybe the model can be updated to work around this limitation.

Performance is not the priority, at least for me, thus I think that basic support for dynamic shape with CoreMLExecutionProvider is enough, for instance using unbounded ranges only.

Basic dynamic shape support is enabled in the CoreML EP now. Note that the CPU EP is also an option if performance is not a priority.

Except for Einsum and Erf I’ve never encountered before, all the other ops you listed here seems pretty common to me, I see them in a lot of models. So it might be worth adding CoreML support for theses ?

We’ll look into adding more op support. PRs are welcome too.

Seconding the above. We’re trying to run a transformer sentence embedding model using the CoreML Execution Provider. Without support for dynamic shapes we have to pad each sequence in a batch to the maximum model sequence length.

Do you have a production use case for this model?

SAM models and its variants (FastSAM) are now widely used for semi-automated segmentation tasks. For instance the AnyLabeling tool uses such models to assist the semantic segmentation annotation task and this tool is quite popular (not notion of “production use case” is quite subjective). SAM models are also used in concrete applications such as medical image segmentation with SAMed.

Apart from transformers models, dynamic shapes are widely used in standard CNNs in order to allow different speed-accuracy trade-offs. For instance, one would use a CNN (let say for object detection) with input shapes of 256x256 or 512x512 depending on the need of real-time inference.

I also often use dynamic shapes to allow multiple batch sizes. This allows ones to adapt to different loads dynamically, for instances on servers.

ONNX, TensortRT, CoreML and others support dynamic shape because there is a demand for such capability I guess. Performance is not the priority, at least for me, thus I think that basic support for dynamic shape with CoreMLExecutionProvider is enough, for instance using unbounded ranges only. This avoids crashes at runtime and improves the interoperability.

Here is a link to the ONNX model that caused this error.

Looking into it. Is the model private or can I discuss details about it here?

The model is public, it’s one from the AnyLabeling software. They simply repackaged the public SAM model as far as I know.

No it’s supposed to work with any network, at least networks using convolutions: https://coremltools.readme.io/docs/flexible-inputs

In my use case (audio processing) I adapt the input shape based both on the rendering type (real-time preview : smaller input shape for a more dynamic feedback, offline render: longer input shape to reduce borders artefacts) and sample rate (to not tile excessively where the spectrum contains nothing).