onnxruntime: 1 - onnxruntime_test_all fails when building with OpenVINO support

Describe the bug When I try to build onnxruntime with OpenVINO support, as instructed in the official instructions, 1 - onnxruntime_test_all fails.

Urgency Standard.

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 20.04
ONNX Runtime installed from (source or binary): source
ONNX Runtime version: 1.9.1
Python version: 3.8.10
GCC/Compiler version (if compiling from source): 9.4.0
OpenVINO version: 2021.4.1 LTS (latest)

To Reproduce As I <hardware_option> I specify MYRIAD_FP16 since I attend to build it for Intel NCS2 which is listed as MYRIAD_FP16 according to the Build instructions.

The exactc ommand I use is:

./build.sh --config RelWithDebInfo --use_openvino MYRIAD_FP16 --build_shared_lib

Expected behavior It is expected that all tests pass successfully, but only 83% tests passed, 1 tests failed out of 6.

About this issue

Original URL
State: open
Created 3 years ago
Comments: 15 (11 by maintainers)

Most upvoted comments

@drux007 I will work with our team internally and see if we can enable the change temporarily from your comment above.

@skottmckay Yeah, that would be a nice change to have. Meanwhile, we have to figure out what would be the ideal tolerance rate setting to avoid accuracy mismatches w.r.t MYRIAD_FP16. once we get there, TensorCheck helper can be modified to accept EP Type and set the right threshold w.r.t the EP’s.

MaajidKhan on Oct 22, 2021

@MaajidKhan As a long term solution can the TensorCheck helper in the unit tests be updated to take the current EP as an input and adjust tolerances instead? Disabling the tests seems like a good way to miss issues.

That would be better than the current ugly setup where we hardcode tolerances in multiple places and apply them to all tests (and not just ones where the particular EP is used):

https://github.com/microsoft/onnxruntime/blob/66ceb6926df6c7a81dc1f241c6bf59cfe58d3a72/onnxruntime/test/providers/provider_test_utils.cc#L167-L170

https://github.com/microsoft/onnxruntime/blob/66ceb6926df6c7a81dc1f241c6bf59cfe58d3a72/onnxruntime/test/providers/provider_test_utils.cc#L225-L229

https://github.com/microsoft/onnxruntime/blob/66ceb6926df6c7a81dc1f241c6bf59cfe58d3a72/onnxruntime/test/providers/provider_test_utils.cc#L291-L294

skottmckay on Oct 21, 2021

https://github.com/microsoft/onnxruntime/commit/b7129305be21111ba9d503019e8b0ede9c6834e3 seems to have disabled tests when building for MYRIAD and building NuGet packages:

    # Disabling unit tests for GPU and MYRIAD on nuget creation
    if args.use_openvino != "CPU_FP32" and args.build_nuget:
        args.test = False

Is the reduced precision of MYRIAD the reason why the tests are disabled because they would otherwise fail? If yes, why are they only disabled if building NuGet packages and not always? Does it mean the tests are expected to fail when building for MYRIAD?

/cc @MaajidKhan

chausner-audeering on Oct 21, 2021

But shouldn’t then the tests be always deactivated if building for something other than CPU_FP32, irrespective of whether NuGet builds are enabled or not?

As a temporary solution, we could adjust the code to always skip tests when using “CPU_FP32” and not only on nuget creation, i.e. changing lines to:

    # Disabling unit tests for GPU and MYRIAD
    if args.use_openvino != "CPU_FP32":
        args.test = False

I adjusted it accordingly and the build finished successfully. It’s not ideal, but still better than simply skipping tests with an additional flag --skip_tests which is not even documented somewhere (correct me if I am wrong).

drux007 on Oct 22, 2021