openvino: [Bug] CPU and MYRIAD output difference
System information (version)
- OpenVINO => 2021.4
- Operating System / Platform => Linux / CPU and Myriad
Detailed description
Hey, I’m having another issue with CPU and MYRIAD output being way off. I am using Selfie Segmentation Landscape model from MediaPipe and I use tflite2tensorflow to generate the ONNX model (I am attaching ONNX in the zip).
When comparing MediaPipe, ONNX, and OpenVINO IE output (CPU), it’s pretty much the same with really small absolute error (< 2e-4). However, when comparing the output of OpenVINO IE Myriad output to the original one, it’s way off (absolute error can be 1, so maximum possible). I am attaching the image of absolute differences between MediaPipe results and inference from bin and xml on Myriad (which you can also see in the Colab that reproduces these results):
Similar issue has also been described here: https://github.com/PINTO0309/tflite2tensorflow/issues/9
There was also a similar issue with YoloV5 Pytorch model, which was solved by calling model.half(), which set the precision to FP16, BEFORE exporting the torch model to ONNX. So I am assuming there must be something wrong with quantization to FP16, as simply setting data_type FP16 when calling mo.py didn’t work and results were bad, until model.half() was called.
However, in tensorflow, it’s not possible to save the model with FP16 precision, so I have to directly rely on data_type FP16, which is not working as it should for me. Do you have some insight into why this would happen?
Steps to reproduce
I am providing the Colab which goes over all steps: https://colab.research.google.com/drive/19Ka407n6tZIs0kJn2mY5IHXPRbniZ2uk?usp=sharing
I am also attaching ZIP with onnx model and npy which contains results of inference on Myriad.
Thanks in advance!
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 37 (12 by maintainers)
no action has been planned for the purpose of this ticket for OV 22.3.1 release. however, if I understand correctly Maksim’s remark on the ticket, it is possible to be solved by FW upgrade so I would like to ask you to try and confirm/deny if the ticket is still valid in the context of the new 22.3.1 release where the fw is updated.
@tersekmatija As per https://www.intel.com/content/www/us/en/developer/articles/release-notes/openvino-2022-3-lts-relnotes.html Myriad will be in the maintenance mode and updates are expected for 2022.3 LTS iterations.
I’ll let @Maxim-Doronin to comment on that.
yes
Hi @tersekmatija
Thank you for providing all the details to reproduce the issue. I am able to see the difference when comparing MYRIAD inference results with CPU and ONNX runtime. I have opened a bug with the development team to get their input.
I will let you know what I find out.
Regards, Jesus
Ref. 66793
@tersekmatija you are right, if the issues lies in the plugin it doesn’t matter which path you used to get IR/nGraph representation.
@AndreeaAtudorei any thoughts on why the issue is still there? Maybe some additional steps are required?
I don’t think API matters since the same MX plugin is used for inference? The results are the same when using ONNX like that yes. @avitial
Hi @tersekmatija
Apologies for the delay, I don’t have an update just yet. Let me follow up with the development team.
Regards, Jesus
The conversion within OpenVINO would involve the Model Optimizer.
This Model Optimizer would generally:
In this case, getting a much better result is expected since that is the purpose of optimization.
However, if compared by the result of inferencing FP16 IR with the native format, I believe they should have the same pattern as CPU and GPU. In terms of fps I could see that there’s a slight improvement for IR compared to the native format. We’ll look further into this.