openvino: Valgrind error using int8 model

Hi!

I’ve encountered Valgrind error running the following sample on OpenVINO 2020.4.

#include <stdio.h>

#include <inference_engine.hpp>

using namespace InferenceEngine;

void testOV()
{
    // Create an IR reader
    CNNNetReader network_reader;

    // Hardocode name of the models to be tested
    const std::string Model_xml = "2020_4/face-detection-0102.xml";
    const std::string Model_bin = "2020_4/face-detection-0102.bin";
    
    // Read the model description and weights
    network_reader.ReadNetwork(Model_xml);
    network_reader.ReadWeights(Model_bin);

    // Fetch network and setup CPU plugin
    auto network = network_reader.getNetwork();
    InferenceEngine::InferencePlugin plugin(InferenceEngine::PluginDispatcher({ "" }).getPluginByDevice("CPU"));
    InferenceEngine::ExecutableNetwork executable_network(
			plugin.LoadNetwork(network,
				{
						{
							PluginConfigParams::KEY_CPU_THREADS_NUM, std::to_string(1)
						}
				}
			)
    );
    
    // Setup inference request
    InferenceEngine::InferRequest infer_request = executable_network.CreateInferRequest();
    
    // Print out first and last layer just for fun
    std::string inputLayerName = network.getInputsInfo().begin()->first;
    std::string outputLayerName = network.getOutputsInfo().begin()->first;
    printf("inputLayerName : %s outputLayerName: %s\n", inputLayerName.c_str(), outputLayerName.c_str());
    
    // Create input_data buffer to be filled with zeros
    Blob::Ptr input = infer_request.GetBlob(inputLayerName);
    auto input_data = input->buffer().as<PrecisionTrait<Precision::FP32>::value_type *>();
    
    // Fill with zeros
    // It's important to put correct number here width*height*nChannels for input layer
    memset(input_data, 0, 96 * 96 * 1 *  sizeof(float));
    
    // Request inference
    // Note: For OpenVINO 2019 R3.1 cpu_affinity valgrind error will be raised - it's safe to ignore
    infer_request.Infer();
    
    // Fetch output buffer to read the inference results
    auto output_blob = infer_request.GetBlob(outputLayerName);
    auto output_data = output_blob->cbuffer().as<PrecisionTrait<Precision::FP32>::value_type*>();
    
    // Reading the ouput data triggers valgrind error because output data is 
    // uninitialized for quantized models (for some reason)
    // Should quantized model be used this will NOT cause a valgrind error!
    if (output_data[0] > 1.0f)
    {
	printf("Just a test\n");
    }	
}

//Entry point
int main()
{
    testOV();
}

Compiled with:

g++ -g -pthread -std=c++11 -I/home/osboxes/intel/openvino_2020.4.287/inference_engine/include -L/home/osboxes/intel/openvino_2020.4.287/inference_engine/lib/intel64 -L//home/osboxes/intel/openvino_2020.4.287/inference_engine/external/tbb/lib test.cpp -linference_engine -ltbb -ldl -linference_engine_legacy -o vino_test_2020_4

The error is:

==12147==  Uninitialised value was created by a heap allocation
==12147==    at 0x4C3089F: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==12147==    by 0x52B0A1B: ??? (in /home/osboxes/intel/openvino_2020.4.287/deployment_tools/inference_engine/lib/intel64/libinference_engine_legacy.so)
==12147==    by 0x51B8B28: ??? (in /home/osboxes/intel/openvino_2020.4.287/deployment_tools/inference_engine/lib/intel64/libinference_engine_legacy.so)
==12147==    by 0xBC475A6: ??? (in /home/osboxes/intel/openvino_2020.4.287/deployment_tools/inference_engine/lib/intel64/libMKLDNNPlugin.so)
==12147==    by 0xBC48227: ??? (in /home/osboxes/intel/openvino_2020.4.287/deployment_tools/inference_engine/lib/intel64/libMKLDNNPlugin.so)
==12147==    by 0xBBFC6EB: ??? (in /home/osboxes/intel/openvino_2020.4.287/deployment_tools/inference_engine/lib/intel64/libMKLDNNPlugin.so)
==12147==    by 0xBBFBB2F: ??? (in /home/osboxes/intel/openvino_2020.4.287/deployment_tools/inference_engine/lib/intel64/libMKLDNNPlugin.so)
==12147==    by 0xBC692B4: ??? (in /home/osboxes/intel/openvino_2020.4.287/deployment_tools/inference_engine/lib/intel64/libMKLDNNPlugin.so)
==12147==    by 0x10F856: InferenceEngine::ExecutableNetwork::CreateInferRequest() (ie_executable_network.hpp:109)
==12147==    by 0x10BD0B: testOV() (test.cpp:34)
==12147==    by 0x10C264: main (test.cpp:69)

Model used for this is available at: https://download.01.org/opencv/2020/openvinotoolkit/2020.4/open_model_zoo/models_bin/3/face-detection-0102/FP16-INT8/

How can this error be solved?

Thanks in advance for the help.

Warm regards, Nikola

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 22 (10 by maintainers)

Most upvoted comments

Memory leak reproduced with AddresSanitizer:

  1. Build with -DENABLE_SANITIZER=ON
  2. LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libasan.so.4:/usr/lib/x86_64-linux-gnu/libstdc++.so.6:lib/libMKLDNNPlugin.so ./benchmark_app -m face-detection-0102.xml -nireq 1 -niter 1

Memory allocated by post_ops_t::append_quantization is never freed

Suspected code: https://github.com/openvinotoolkit/oneDNN/blob/b73474c80c21ae170b112803a1fc315e1549bdab/src/common/primitive_attr.hpp#L231

==32359==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 20480 byte(s) in 256 object(s) allocated from:
    #0 0x7fcfbbe4e7a0 in posix_memalign (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdf7a0)
    #1 0x7fcfb528156c in mkldnn::impl::malloc(unsigned long, int) ../inference-engine/thirdparty/mkl-dnn/src/common/utils.cpp:132
    #2 0x7fcfb52549ed in mkldnn::impl::c_compatible::operator new(unsigned long) ../inference-engine/thirdparty/mkl-dnn/src/common/utils.hpp:318
    #3 0x7fcfb52549ed in mkldnn_post_ops::append_quantization(mkldnn_alg_kind_t, int, float const*, int, float const*, int, float const*, int, float const*, int, float const*, int, float const*) ../inference-engine/thirdparty/mkl-dnn/src/common/primitive_attr.cpp:199
    #4 0x7fcfb52549ed in mkldnn_post_ops_append_quantization ../inference-engine/thirdparty/mkl-dnn/src/common/primitive_attr.cpp:570
    #5 0x7fcfb46a117d in mkldnn::post_ops::append_quantization(mkldnn::algorithm, std::vector<float, std::allocator<float> > const&, std::vector<float, std::allocator<float> > const&, std::vector<float, std::allocator<float> > const&, std::vector<float, std::allocator<float> > const&, std::vector<float, std::allocator<float> > const&, std::vector<float, std::allocator<float> > const&) ../inference-engine/thirdparty/mkl-dnn/include/mkldnn.hpp:482
    #6 0x7fcfb46a117d in MKLDNNPlugin::MKLDNNQuantizeNode::appendPostOps(mkldnn::post_ops&) ../inference-engine/src/mkldnn_plugin/nodes/mkldnn_quantize_node.cpp:520
    #7 0x7fcfb38043c1 in MKLDNNPlugin::MKLDNNConvolutionNode::setPostOps(mkldnn::primitive_attr&, bool) ../inference-engine/src/mkldnn_plugin/nodes/mkldnn_conv_node.cpp:457
    #8 0x7fcfb3821d55 in MKLDNNPlugin::MKLDNNConvolutionNode::createPrimitive() ../inference-engine/src/mkldnn_plugin/nodes/mkldnn_conv_node.cpp:643
    #9 0x7fcfb2ffed8b in MKLDNNPlugin::MKLDNNGraph::CreatePrimitives() ../inference-engine/src/mkldnn_plugin/mkldnn_graph.cpp:700
    #10 0x7fcfb30813b1 in MKLDNNPlugin::MKLDNNGraph::InitGraph() ../inference-engine/src/mkldnn_plugin/mkldnn_graph.cpp:361
    #11 0x7fcfb308475d in void MKLDNNPlugin::MKLDNNGraph::CreateGraph<InferenceEngine::ICNNNetwork>(InferenceEngine::ICNNNetwork const&, std::shared_ptr<MKLDNNPlugin::MKLDNNExtensionManager> const&, std::shared_ptr<MKLDNNPlugin::MKLDNNWeightsSharing>&) ../inference-engine/src/mkldnn_plugin/mkldnn_graph.cpp:98
    #12 0x7fcfb2facae6 in construct ../inference-engine/src/mkldnn_plugin/mkldnn_exec_network.cpp:136
    #13 0x7fcfb2facae6 in construct ../inference-engine/temp/tbb/include/tbb/enumerable_thread_specific.h:709
    #14 0x7fcfb2f8367a in tbb::interface6::enumerable_thread_specific<std::shared_ptr<MKLDNNPlugin::MKLDNNGraph>, tbb::cache_aligned_allocator<std::shared_ptr<MKLDNNPlugin::MKLDNNGraph> >, (tbb::ets_key_usage_type)1>::create_local() ../inference-engine/temp/tbb/include/tbb/enumerable_thread_specific.h:831
    #15 0x7fcfb2f8a602 in tbb::interface6::internal::ets_base<(tbb::ets_key_usage_type)1>::table_lookup(bool&) ../inference-engine/temp/tbb/include/tbb/enumerable_thread_specific.h:234
    #16 0x7fcfb2f8baab in tbb::interface6::enumerable_thread_specific<std::shared_ptr<MKLDNNPlugin::MKLDNNGraph>, tbb::cache_aligned_allocator<std::shared_ptr<MKLDNNPlugin::MKLDNNGraph> >, (tbb::ets_key_usage_type)1>::local(bool&) ../inference-engine/temp/tbb/include/tbb/enumerable_thread_specific.h:935
    #17 0x7fcfb2f8baab in tbb::interface6::enumerable_thread_specific<std::shared_ptr<MKLDNNPlugin::MKLDNNGraph>, tbb::cache_aligned_allocator<std::shared_ptr<MKLDNNPlugin::MKLDNNGraph> >, (tbb::ets_key_usage_type)1>::local() ../inference-engine/temp/tbb/include/tbb/enumerable_thread_specific.h:930
    #18 0x7fcfb2f8baab in operator() ../inference-engine/src/mkldnn_plugin/mkldnn_exec_network.cpp:140
    #19 0x7fcfb2f8baab in _M_invoke /usr/include/c++/7/bits/std_function.h:316
    #20 0x7fcfbcebcec9 in std::function<void ()>::operator()() const /usr/include/c++/7/bits/std_function.h:706
    #21 0x7fcfbcebcec9 in operator() ../inference-engine/src/inference_engine/threading/ie_itask_executor.cpp:18
    #22 0x7fcfbcebcec9 in __invoke_impl<void, InferenceEngine::ITaskExecutor::runAndWait(const std::vector<std::function<void()> >&)::<lambda()>&> /usr/include/c++/7/bits/invoke.h:60
    #23 0x7fcfbcebcec9 in __invoke<InferenceEngine::ITaskExecutor::runAndWait(const std::vector<std::function<void()> >&)::<lambda()>&> /usr/include/c++/7/bits/invoke.h:95
    #24 0x7fcfbcebcec9 in operator() /usr/include/c++/7/future:1421
    #25 0x7fcfbcebcec9 in operator() /usr/include/c++/7/future:1362
    #26 0x7fcfbcebcec9 in _M_invoke /usr/include/c++/7/bits/std_function.h:302
    #27 0x7fcfbcebe4c1 in std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>::operator()() const /usr/include/c++/7/bits/std_function.h:706
    #28 0x7fcfbcebe4c1 in std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) /usr/include/c++/7/future:561
    #29 0x7fcfb2ef3c24 in void std::__invoke_impl<void, void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::__invoke_memfun_deref, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/7/bits/invoke.h:73
    #30 0x7fcfb2ef3c24 in std::__invoke_result<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>::type std::__invoke<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/7/bits/invoke.h:95
    #31 0x7fcfb2ef3c24 in void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#1}::operator()() const /usr/include/c++/7/mutex:672
    #32 0x7fcfb2ef3c24 in void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::operator()() const /usr/include/c++/7/mutex:677
    #33 0x7fcfb2ef3c24 in void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::_FUN() /usr/include/c++/7/mutex:677
    #34 0x7fcfb0a23826 in __pthread_once_slow (/lib/x86_64-linux-gnu/libpthread.so.0+0xf826)