onnxruntime: Bug: Converting from ONNX to ORT fails when setting Device=Direct ML [C++] [ONNX2ORT converter] [Direct ML]

When we persist with setOptFilePath=true, it will not do the fusing of partitions of DML nodes into a single DML_GRAPH.

@diablodale Correct, nodes will remain distinct operators (or fused operators).

It will instead persist a slightly less optimized model lacking that fuse.

Yes, it will have operator fusions (e.g. Conv + Relu -> ConvRelu), but not whole-graph-fusion.

–>

When this persisted model is loaded, will Ort do that final fuse optimization?

Yes, that final whole-graph-fusion will be done upon reload.

Or, is this the tradeoff to have a faster load?

That final fusion happens in either case, loading the original model or loading the pre-operator-fused model. Exporting to .onnx file and reloading, I noticed a time saving during session load of like 5-15% depending on the model, and run time is the same. Exporting to .ort file format and reloading, I noticed a substantial time saving in session load, from 2-7x depending on the model, but as enticing as that is, beware .ort is just recently enabled by https://github.com/microsoft/onnxruntime/pull/13913, and I can’t yet vouch for it’s robustness without further more exhuastive testing (I just tried it with a few models), because interaction with the DML EP might call new code paths. Also, we should verify whether this issue applies still: https://github.com/microsoft/onnxruntime/issues/13535.

Got it, I’ve already code in place to invalidate a persisted optimized model if any config changes.

Great. I’d also include the driver version too in your hash, just in case updating the driver changes registered data type support.

@diablodale @gedoensmax can you elaborate on your use case where you want/need DML to be enabled when creating an optimized model vs. doing that at runtime?

I create a DLL plugin for the Cycling74 Max runtime patching system. My customers are educators, researchers, artists, musicians, etc. I provide one onnx model for a specific use case plus the ability to run any onnx model. My DLL transforms in/outs between native Max data. My plugin allows running the model on the cpu, directml, cuda, or tensorRT providers with a single setting change. I hide all the technical complexities so my customers can focus on their art/research/education.

The Max environment is always running, it is a graphical hack/patch environment where nodes are connected by patchcords. Patchcords and nodes are reshaped/connected hundreds of times a day as customers experiment and try ideas. This realtime iteration necessitates caching and reuse. The time burden of running the onnx optimization process every time they connect a patchcord or click “go” hampers their creativity and kills their “flow”.

I know when hardware, models, or settings change…therefore I can cache models after they go through the optimization process. I already do this successfully with the TensorRT provider. A similar ability with DirectML is desired and I attempted it with SetOptimizedModelFilePath() but ran into this same OP…the saved DirectML model is unusable.

@gedoensmax If you have a model with dynamic dimensions and want to make them fixed, you could use this tool: https://onnxruntime.ai/docs/reference/mobile/make-dynamic-shape-fixed.html

I don’t quite understand how model load time would be affected by having fixed shapes. If anything, I would expect more optimizations to be possible when shapes are fixed.

I would suggest running the ‘basic’ level optimizations on the model with just the CPU EP enabled to do those optimizations ahead of time. They are not specific to any EP, only use official ONNX operators, and cover things like constant folding and common subexpression elimination.

Beyond the ‘basic’ level you get into EP specific optimizations which may involve compiling nodes or fusing nodes that will use a custom operator. Currently there’s no general purpose way to save a compiled node like TensorRT engine caching does. An inference session is intended to be re-used though, so this cost during loading is not per-inference.

POC for adding support for DML when using an ORT format model: https://github.com/microsoft/onnxruntime/compare/skottmckay/ORT_model_support_with_DML_EP

Technically we could create the ORT format model with just basic optimizations and DML disabled to not require the changes in the DML graph partitioning. At runtime, if DML was enabled it could still execute the same nodes.

Thanks to those last answers we were able to feed the ONNX buffer into ORT directly, which is a working workaround for us!

We will keep an eye to this post to know when the DML-ORT file issue is solved, as we’d need to switch to it once it’s working, but we are no longer blocked.

Thanks for the quick answers and the great work!

Example of reading bytes from file: https://github.com/microsoft/onnxruntime/blob/894fc828587c919d815918c4da6cde314e5d54ed/onnxruntime/test/shared_lib/test_model_loading.cc#L21-L31

The bytes are just passed directly when creating the inference session.

https://github.com/microsoft/onnxruntime/blob/894fc828587c919d815918c4da6cde314e5d54ed/onnxruntime/test/shared_lib/test_model_loading.cc#L41

We’ll look into the DML issue as it should be possible to use that with an ORT format model.

ONNX format files are supported on all platforms. It’s just that the binary size of the ORT library will be bigger vs. a minimal build that only supports ORT format models (by a few MB). For that you get a lot more flexibility though, such as the ability to use CPU or GPU depending on what’s available at runtime.

Can you provide more details on how you were trying to feed the ONNX format file at runtime? InferenceSession has an API where raw bytes can be provided, which can be used for both ONNX and ORT format models. Given that, I’m not quite following how ‘the ORT API seems to open the onnx file in many places’ given it’s only seeing bytes and not a filename if that API is used.

https://github.com/microsoft/onnxruntime/blob/894fc828587c919d815918c4da6cde314e5d54ed/onnxruntime/core/session/inference_session.cc#L686

I did a quick test using the python API and it seemed to work fine with the ONNX format model being provided as bytes.

import onnxruntime as ort
import numpy as np

model_path = r'my_test_model.onnx'

so = ort.SessionOptions()
s = ort.InferenceSession(model_path, so)

# random input matching what the model requires
input_data = np.zeros((1, 5, 512, 867), dtype=np.float32)
inputs = { 'input': input_data }

# run with filename
o1 = s.run(None, inputs)

with open(model_path, 'rb') as infile:
    bytes = infile.read()
    # run with bytes
    s2 = ort.InferenceSession(bytes, so)
    o2 = s2.run(None, inputs)

    # this model produces a single output so compare the run via filename with the run with bytes
    print(np.array_equal(o1[0],o2[0]))