openvino: Crash at clGetDeviceIDs in OpenVINO 2022.3

System information (version)

OpenVINO Source=> Custom built 2022.3 for Electron node addon
OpenVINO Version=> 2022.3
Operating System / Platform => Windows 10
Compiler => Visual Studio
Problem classification => Crash
Device use: => GPU and OpenCL
Framework => Electron

Detailed description

Crash callstack:

 	OpenCL.dll!00007ffefe564658()	Unknown
>	[Inline Frame] openvino_intel_gpu_plugin.dll!cl::Platform::getDevices(unsigned __int64) Line 2580	C++
 	openvino_intel_gpu_plugin.dll!cldnn::ocl::ocl_device_detector::create_device_list() Line 205	C++
 	openvino_intel_gpu_plugin.dll!cldnn::ocl::ocl_device_detector::get_available_devices(void * user_context, void * user_device, int ctx_device_id, int target_tile_id) Line 160	C++
 	openvino_intel_gpu_plugin.dll!cldnn::device_query::device_query(cldnn::engine_types engine_type, cldnn::runtime_types runtime_type, void * user_context, void * user_device, int ctx_device_id, int target_tile_id) Line 25	C++
 	openvino_intel_gpu_plugin.dll!ov::intel_gpu::Plugin::Plugin() Line 132	C++
 	[Inline Frame] openvino_intel_gpu_plugin.dll!std::_Construct_in_place(ov::intel_gpu::Plugin &) Line 158	C++
 	[Inline Frame] openvino_intel_gpu_plugin.dll!std::_Ref_count_obj2<ov::intel_gpu::Plugin>::{ctor}() Line 2029	C++
 	[Inline Frame] openvino_intel_gpu_plugin.dll!std::make_shared() Line 2748	C++
 	openvino_intel_gpu_plugin.dll!CreatePluginEngine(std::shared_ptr<InferenceEngine::IInferencePlugin> & plugin) Line 1011	C++
 	openvino.dll!ov::CoreImpl::GetCPPPluginByName(const std::string & pluginName) Line 1168	C++
 	openvino.dll!ov::CoreImpl::GetMetric(const std::string & deviceName, const std::string & name, const std::map<std::string,ov::Any,std::less<std::string>,std::allocator<std::pair<std::string const ,ov::Any>>> & options) Line 980	C++
 	openvino.dll!ov::CoreImpl::GetAvailableDevices() Line 1073	C++
 	openvino.dll!ov::Core::get_available_devices() Line 2120	C++
 	RCVNativeVBG.dll!VBG::OpenVinoProcessing::loadModel(const std::wstring & modelPath) Line 39	C++
 	[Inline Frame] RCVNativeVBG.dll!VBG::IVBGMLProcessing::loadBindModel(const std::wstring &) Line 164	C++
 	RCVNativeVBG.dll!initInternalOpenVino(VBG::ConfigParser * ptrConfigParser) Line 291	C++
 	[Inline Frame] RCVNativeVBG.dll!std::invoke(void(*)(VBG::ConfigParser *) &&) Line 1534	C++
 	RCVNativeVBG.dll!std::thread::_Invoke<std::tuple<void (__cdecl*)(VBG::ConfigParser *),VBG::ConfigParser *>,0,1>(void * _RawVals) Line 56	C++
 	ucrtbase.dll!00007fff76aa1bb2()	Unknown
 	kernel32.dll!00007fff78ac7034()	Unknown
 	ntdll.dll!00007fff78c02651()	Unknown

Crashing function inside OpenVINO, filename: opencl.hpp, line no: 2580

        **cl_int err = ::clGetDeviceIDs(object_, type, 0, NULL, &n);**
        if (err != CL_SUCCESS  && err != CL_DEVICE_NOT_FOUND) {
            return detail::errHandler(err, __GET_DEVICE_IDS_ERR);
        }

Attaching dxdiag of one the system where issue was observed, looks like issue is observed on all systems with OpenCL 2.2.8 DxDiag.txt

OpenVINO was custom compiled on 2022.3, this was done to use/consume from Electron application, there were some path issues which required some fixing.

Most likely issue seems to due to incompatible OpenCL version, where ion my system OpenCL version 3.0.3

About this issue

Original URL
State: closed
Created a year ago
Comments: 30 (30 by maintainers)

Most upvoted comments

@peterchen-intel, @songbell, @avitial Looks like technically we can add a custom handler for signals, but it seems that it may be not a good idea, at least for linux:

SIGSEGV means that a program might be in a corrupted state
POSIX standard says that The behavior of a process is undefined after it returns normally from a signal-catching function for a SIGBUS, SIGFPE, SIGILL, or SIGSEGV signal that was not generated by kill(), sigqueue(), raise()
Signal handlers seem to be set for the whole process which might impact other components (even if we reset handler to default after attempt to detect devices - a parallel thread can be impacted)

For windows it seems that Structured exception handling api may allow to safely continue process after catching certain signal types, but there are also noncontinuable exceptions, not sure which one is signaled in this particular case. However I haven’t learned this SEH win api deeply, so not sure if it has any side effects similar to posix signals.

From my point of view, it’s better to update drivers or manually disable GPU plugin (remove dll for example) on such systems rather than have a risk of unwanted side effects due to recovery attempt. But if someone knows a good and safe (and ideally portable) way to recover after segmentation fault, feel free to create PR, contributions are welcomed

vladimir-paramuzov on Aug 9, 2023

DxDiag (1).txt attached DxDiag details from system. We need to reach out to customer for hello_query_device.exe, which will take some time

venki-thiyag on Jun 2, 2023