onnxruntime: Problems caused by GetInputNameAllocated after upgrading from 1.12 to 1.13

Describe the issue

When upgrading from 1.12.x to 1.13.x,GetInputName and GetOutputName need to be replaced with GetInputNameAllocated and GetOutputNameAllocated， I encountered a very strange bug here.
onnx mode export from yolov5-seg.pt: https://drive.google.com/file/d/1tV2moQxNfLzNf6yXm5Zev5CVj2o9zuaz/view?usp=share_link

Run the following code, and everything is OK for TestONNX(), but when running TestONNX2(), the input and output nodes names become strange after session->Run()

They just have different ways of obtaining node names. One is for loop, and the other is useless.

An error will be reported even if nothing is modified beyond the bracket of the node name：

So, if I have multiple inputs and outputs and cannot use the for loop, how can I solve this problem？

To reproduce

int TestONNX(std::string modelPath, bool useCuda=true) {

	Ort::Env ortEnv = Ort::Env(OrtLoggingLevel::ORT_LOGGING_LEVEL_ERROR, "Yolov5-Seg");
	Ort::SessionOptions sessionOptions = Ort::SessionOptions();
	std::vector<std::string> available_providers = GetAvailableProviders();
	auto cuda_available = std::find(available_providers.begin(), available_providers.end(), "CUDAExecutionProvider");
	OrtCUDAProviderOptions cudaOption;
	if (useCuda && (cuda_available == available_providers.end()))
	{
		std::cout << "Your ORT build without GPU. Change to CPU." << std::endl;
	  std::cout << " Infer model on CPU "<< std::endl;
	}
	else if (useCuda && (cuda_available != available_providers.end()))
	{
		std::cout << "* Infer model on GPU! " << std::endl;
		OrtStatus* status = OrtSessionOptionsAppendExecutionProvider_CUDA(sessionOptions, 0);
	}
	else
	{
		std::cout << " Infer model on CPU! " << std::endl;
	}
	//
	sessionOptions.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_EXTENDED);

#ifdef _WIN32
	std::wstring model_path(modelPath.begin(), modelPath.end());
	Ort::Session* session = new Ort::Session(ortEnv, model_path.c_str(), sessionOptions);
#else
	Ort::Session* session = new Ort::Session(_OrtEnv, modelPath.c_str(), sessionOptions);
#endif

	Ort::AllocatorWithDefaultOptions allocator;
	//init input

	std::vector<const char*> inputNodeNames; //
	std::vector<const char*> outputNodeNames;//
	std::vector<int64_t> inputTensorShape; //
	std::vector<int64_t> outputTensorShape;
	std::vector<int64_t> outputMaskTensorShape;

	auto inputNodesNum = session->GetInputCount();

	auto temp_input_name0 = session->GetInputNameAllocated(0, allocator);
	inputNodeNames.push_back(temp_input_name0.get());


	Ort::TypeInfo inputTypeInfo = session->GetInputTypeInfo(0);
	auto input_tensor_info = inputTypeInfo.GetTensorTypeAndShapeInfo();
	//inputNodeDataType = input_tensor_info.GetElementType();
	inputTensorShape = input_tensor_info.GetShape();
	//init output
	auto outputNodesNum = session->GetOutputCount();

	auto temp_output_name0 = session->GetOutputNameAllocated(0, allocator);
	auto temp_output_name1 = session->GetOutputNameAllocated(1, allocator);
	Ort::TypeInfo type_info_output0(nullptr);
	Ort::TypeInfo type_info_output1(nullptr);
	type_info_output0 = session->GetOutputTypeInfo(0);  //output0
	type_info_output1 = session->GetOutputTypeInfo(1);  //output1
	outputNodeNames.push_back(temp_output_name0.get());
	outputNodeNames.push_back(temp_output_name1.get());


	auto tensor_info_output0 = type_info_output0.GetTensorTypeAndShapeInfo();
	//outputNodeDataType = tensor_info_output0.GetElementType();
	outputTensorShape = tensor_info_output0.GetShape();
	auto tensor_info_output1 = type_info_output1.GetTensorTypeAndShapeInfo();
	//_outputMaskNodeDataType = tensor_info_output1.GetElementType(); //the same as output0
	outputMaskTensorShape = tensor_info_output1.GetShape();

	auto memoryInfo = Ort::MemoryInfo::CreateCpu(OrtAllocatorType::OrtDeviceAllocator, OrtMemType::OrtMemTypeCPUOutput);
	for (int i = 0; i < 3; i++) {
		std::cout << "Start warming up" << endl;
		size_t input_tensor_length = 640 * 640 * 3;
		float* temp = new float[input_tensor_length];
		std::vector<Ort::Value> input_tensors;
		std::vector<Ort::Value> output_tensors;
		std::cout << "################### befor run:##############" << endl;
		std::cout << "input node name:" << inputNodeNames[0] << endl;
		std::cout << "output0 node name:" << outputNodeNames[0] << endl;
		std::cout << "output1 node name:" << outputNodeNames[1] << endl;

		input_tensors.push_back(Ort::Value::CreateTensor<float>(
			memoryInfo, temp, input_tensor_length, inputTensorShape.data(),
			inputTensorShape.size()));
		output_tensors = session->Run(Ort::RunOptions{ nullptr },
			inputNodeNames.data(),
			input_tensors.data(),
			inputNodeNames.size(),
			outputNodeNames.data(),
			outputNodeNames.size());
		std::cout << "################### after run:##############" << endl;
		std::cout << "input node name:" << inputNodeNames[0] << endl;
		std::cout << "output0 node name:" << outputNodeNames[0] << endl;
		std::cout << "output1 node name:" << outputNodeNames[1] << endl;
	}

	std::cout << "*********************************** test onnx ok  ***************************************" << endl;

	return 0;
}

int TestONNX2(std::string modelPath, bool useCuda = true) {

	Ort::Env ortEnv = Ort::Env(OrtLoggingLevel::ORT_LOGGING_LEVEL_ERROR, "Yolov5-Seg2");
	Ort::SessionOptions sessionOptions = Ort::SessionOptions();
	std::vector<std::string> available_providers = GetAvailableProviders();
	auto cuda_available = std::find(available_providers.begin(), available_providers.end(), "CUDAExecutionProvider");
	OrtCUDAProviderOptions cudaOption;
	if (useCuda && (cuda_available == available_providers.end()))
	{
		std::cout << "Your ORT build without GPU. Change to CPU." << std::endl;
		std::cout << " Infer model on CPU " << std::endl;
	}
	else if (useCuda && (cuda_available != available_providers.end()))
	{
		std::cout << "* Infer model on GPU! " << std::endl;
		OrtStatus* status = OrtSessionOptionsAppendExecutionProvider_CUDA(sessionOptions, 0);
	}
	else
	{
		std::cout << " Infer model on CPU! " << std::endl;
	}
	//
	sessionOptions.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_EXTENDED);

#ifdef _WIN32
	std::wstring model_path(modelPath.begin(), modelPath.end());
	Ort::Session* session = new Ort::Session(ortEnv, model_path.c_str(), sessionOptions);
#else
	Ort::Session* session = new Ort::Session(_OrtEnv, modelPath.c_str(), sessionOptions);
#endif

	Ort::AllocatorWithDefaultOptions allocator;
	//init input

	std::vector<const char*> inputNodeNames; //
	std::vector<const char*> outputNodeNames;//
	std::vector<int64_t> inputTensorShape; //
	std::vector<int64_t> outputTensorShape;
	std::vector<int64_t> outputMaskTensorShape;

	auto inputNodesNum = session->GetInputCount();
	for (int i = 0; i < inputNodesNum; i++) {
		auto temp_input_name = session->GetInputNameAllocated(i, allocator);
		inputNodeNames.push_back(temp_input_name.get());
	}

	Ort::TypeInfo inputTypeInfo = session->GetInputTypeInfo(0);
	auto input_tensor_info = inputTypeInfo.GetTensorTypeAndShapeInfo();
	//inputNodeDataType = input_tensor_info.GetElementType();
	inputTensorShape = input_tensor_info.GetShape();
	//init output
	auto outputNodesNum = session->GetOutputCount();

	for (int i = 0; i < outputNodesNum; i++) {
		auto temp_output_name = session->GetOutputNameAllocated(i, allocator);
		Ort::TypeInfo type_info_output(nullptr);
		type_info_output = session->GetOutputTypeInfo(i);  //output0
		outputNodeNames.push_back(temp_output_name.get());

		auto tensor_info_output = type_info_output.GetTensorTypeAndShapeInfo();
		if (i == 0) {
			outputTensorShape = tensor_info_output.GetShape();
		}
		else {
			outputMaskTensorShape = tensor_info_output.GetShape();
		}

	}
	auto memoryInfo = Ort::MemoryInfo::CreateCpu(OrtAllocatorType::OrtDeviceAllocator, OrtMemType::OrtMemTypeCPUOutput);
	for (int i = 0; i < 3; i++) {
		std::cout << "Start warming up" << endl;
		size_t input_tensor_length = 640 * 640 * 3;
		float* temp = new float[input_tensor_length];
		std::vector<Ort::Value> input_tensors;
		std::vector<Ort::Value> output_tensors;
		std::cout << "################### befor run:##############" << endl;
		std::cout << "input node name:" << inputNodeNames[0] << endl;
		std::cout << "output0 node name:" << outputNodeNames[0] << endl;
		std::cout << "output1 node name:" << outputNodeNames[1] << endl;

		input_tensors.push_back(Ort::Value::CreateTensor<float>(
			memoryInfo, temp, input_tensor_length, inputTensorShape.data(),
			inputTensorShape.size()));
		output_tensors = session->Run(Ort::RunOptions{ nullptr },
			inputNodeNames.data(),
			input_tensors.data(),
			inputNodeNames.size(),
			outputNodeNames.data(),
			outputNodeNames.size());
		std::cout << "################### after run:##############" << endl;
		std::cout << "input node name:" << inputNodeNames[0] << endl;
		std::cout << "output0 node name:" << outputNodeNames[0] << endl;
		std::cout << "output1 node name:" << outputNodeNames[1] << endl;
	}

	return 0;
}
int main()
{
	string model_path = "./yolov5s-seg.onnx";
	TestONNX(model_path, true);
	TestONNX2(model_path, true);
	return 0;

}

Urgency

No response

Platform

Windows

OS Version

WIN10 22H2

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.13.1 from https://github.com/microsoft/onnxruntime/releases/download/v1.13.1/onnxruntime-win-x64-gpu-1.13.1.zip

ONNX Runtime API

C++

Architecture

X64

Execution Provider

CUDA

Execution Provider Library Version

CUDA 11.4

About this issue

Original URL
State: closed
Created a year ago
Comments: 16 (3 by maintainers)

Most upvoted comments

I’ve tried std::vector<AllocatedStringPtr>, but it’s the same problem. It doesn’t work after only session.run() once.

To clarify, in this approach there is an additional vector.

	std::vector<const char*> inputNodeNames; //
	std::vector<AllocatedStringPtr> inputNodeNameAllocatedStrings; // <-- newly added
	...

	auto inputNodesNum = session->GetInputCount();
	for (int i = 0; i < inputNodesNum; i++) {
		auto input_name = session->GetInputNameAllocated(i, allocator);
		inputNodeNameAllocatedStrings.push_back(std::move(input_name));
		inputNodeNames.push_back(inputNodeNameAllocatedStrings.back().get());
	}

So the memory pointed to by inputNodeNames[i] is owned by inputNodeNameAllocatedStrings[i].

Because there is only one model, and the input and output names are fixed, I don’t want to get the input and output names every time forward model, but I want to get them once when I read the model, which will be used later.

You can do this after loading the model before the call(s) to Session::Run().

edgchen1 on Jan 6, 2023

@edgchen1 Thanks for your help, it works.

UNeedCryDear on Jan 7, 2023