openvino: [Bug] Exception when switching between Myriad and CPU execution: Failed to allocate graph: NC_ERROR

System information (version)

• OpenVINO => 2021.4.582
• Operating System / Platform => Windows 64 Bit
• Compiler => Visual Studio 2019
• Problem classification: Model Inference
• Model name: person-detection-retail-0013

Detailed description

We discovered an exception from OpenVino in some particular cases. Initially this happened while testing with our production-like test-hardware and with the production software. We then tried to reproduce the situation within our development environment, and then within a small code example.

I would like copy the information provided by our development team:

OpenVino throws an exception, which we do not understand. Unfortunatly, we have not found a good way to reproduce it, but it happens sometimes in the productive environment. We would like to fix this. So we are reaching out for more information and possible solution approaches.

We were able to reproduce the exception under some constraints in an debug-Program. We are not 100% sure if it is the same reason as in the productive environment, but it is the same exception.

The following screenshot shows the exception. The colored text has to be something from OpenVINO, or other libraries. The white text is from the debug-Program. I have also attached the cpp file.

Steps to reproduce

#include <iostream>
#include <thread>
#include <ostream>

#pragma warning(push, 0)
#include <ie_core.hpp>
#pragma warning(pop)

int main()
{
	const std::string modelPath = ".\\OpenVINO-models\\FP16\\person-detection-retail-0013.xml";
	const std::string myriadConfig = "MULTI:MYRIAD";
	const std::string cpuConfig = "MULTI:CPU";

	std::shared_ptr<InferenceEngine::Core> core = std::make_shared<InferenceEngine::Core>();

	while (true)
	{
		try
		{
			std::cout << "A" << std::endl;

			InferenceEngine::CNNNetwork network = core->ReadNetwork(modelPath);
			InferenceEngine::ExecutableNetwork executableNetwork = core->LoadNetwork(network, myriadConfig);
			std::this_thread::sleep_for(std::chrono::milliseconds(1000));
		}
		catch (const std::exception& ex)
		{
			std::cout << ex.what() << std::endl;
		}

		std::this_thread::sleep_for(std::chrono::milliseconds(50));

		try
		{
			std::cout << "B" << std::endl;
			InferenceEngine::CNNNetwork network = core->ReadNetwork(modelPath);
			InferenceEngine::ExecutableNetwork executableNetwork = core->LoadNetwork(network, cpuConfig);
			std::this_thread::sleep_for(std::chrono::milliseconds(1000));
		}
		catch (const std::exception& ex)
		{
			std::cout << ex.what() << std::endl;
		}

		std::this_thread::sleep_for(std::chrono::milliseconds(50));
	}

}

It looks like, the exception (and also the additional command line output), only happens with the debug-Program if these constraints are fulfilled:

1. The VisualStudio Project settings has „Common Language Runtime Support (/clr)“ enabled.
2. The Debug-Application is started from VisualStudio.

Some more details:

• Version: openvino_2021.4.582
• Network: https://download.01.org/opencv/2020/openvinotoolkit/2020.1/open_model_zoo/models_bin/1/person-detection-retail-0013/FP16/
• There are two Myriad USB Sticks in the PC
• Microsoft Visual Studio Professional 2019, Version 16.11.5
• Visual Studio 2019 v142 runtime
• _MSC_VER=1929

Have you ever observed such exceptions? If yes, can you tell us more about it? What are possible reasons for this exception? Are there typical situations where this exception can appear? Unfortunately our google researched not let to a meaningful result. So we need help on this topic from your side.

Issue submission checklist

• I report the issue, it’s not a question
• I checked the problem with documentation, FAQ, open issues, Stack Overflow, etc and have not found solution
• There is reproducer code and related data files: images, videos, models, etc.