tensorrtx: Loading plugins from libmyplugins.so with trtexec fails

I’m trying to run the serialized yolov4 engine with trtexec, which comes included in the TensorRT NGC docker containers. But anyway: it basically loads the serialized engine and runs inference - not much different from what yolov4.cpp does.

docker run --gpus all -it --rm -v $(pwd)/tensorrtx:/tensorrtx nvcr.io/nvidia/tensorrt:19.10-py3 cd /workspace/tensorrt/bin ./trtexec --loadEngine=/tensorrtx/yolov4/build/yolov4.engine --verbose

...
[06/26/2020-16:59:06] [E] [TRT] INVALID_ARGUMENT: getPluginCreator could not find plugin Mish_TRT version 1
[06/26/2020-16:59:06] [E] [TRT] safeDeserializationUtils.cpp (259) - Serialization Error in load: 0 (Cannot deserialize plugin since corresponding IPluginCreator not found in Plugin Registry)
[06/26/2020-16:59:06] [E] [TRT] INVALID_STATE: std::exception
[06/26/2020-16:59:06] [E] [TRT] INVALID_CONFIG: Deserialize the cuda engine failed.
[06/26/2020-16:59:06] [E] Engine could not be created
&&&& FAILED TensorRT.trtexec # ./trtexec --loadEngine=/tensorrtx/yolov4/build/yolov4.engine --verbose

This obviously fails because TensorRT does not load the plugin containing Mish Activation and Yolo Layer. So next I tried:

./trtexec --loadEngine=/tensorrtx/yolov4/build/yolov4.engine --plugins=/tensorrtx/yolov4/build/libmyplugins.so --verbose

...
[06/26/2020-17:02:26] [I] Loading supplied plugin library: /tensorrtx/yolov4/build/libmyplugins.so
[06/26/2020-17:02:27] [E] [TRT] INVALID_ARGUMENT: getPluginCreator could not find plugin Mish_TRT version 1
[06/26/2020-17:02:27] [E] [TRT] safeDeserializationUtils.cpp (259) - Serialization Error in load: 0 (Cannot deserialize plugin since corresponding IPluginCreator not found in Plugin Registry)
[06/26/2020-17:02:27] [E] [TRT] INVALID_STATE: std::exception
[06/26/2020-17:02:27] [E] [TRT] INVALID_CONFIG: Deserialize the cuda engine failed.
[06/26/2020-17:02:27] [E] Engine could not be created
&&&& FAILED TensorRT.trtexec # ./trtexec --loadEngine=/tensorrtx/yolov4/build/yolov4.engine --plugins=/tensorrtx/yolov4/build/libmyplugins.so --verbose

So it loads the library containing the plugin but it does not seem to register the plugin correctly. I checked the same approach with a different tensorrt engine using custom layers and it worked. Is there any trick to loading your custom plugins @wang-xinyu ? Is selecting the correct arch=compute_30;code=sm_30 in the CMakeList.txt important? Is it possible to run the engine in a “standardized” way outside of your custom loading code in yolov4.cpp? Any hint greatly appreciated! 😃

Edit: I tried with different TensorRT versions, including the most recent 20.06 container which comes with 7.1.2

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 17 (7 by maintainers)

Most upvoted comments

I can write a little PR on how to build and deploy this to Triton if you want. Triton really steps up the game I think. It takes care of Model Deployment and runs the model on multiple GPUs with overlapping data transfer and scheduling etc.

philipp-schmidt on Jul 27, 2020

Hi,

I implemented everything to deploy to Triton and put it here: https://github.com/isarsoft/yolov4-triton-tensorrt

If Triton is an alternative for you @cesarandreslopez instead of using TensorRT directly (Triton has some amazing benefits and it has a python SDK), maybe you give it a try.

@wang-xinyu I included your license and mentioned you in the README. Feel free to copy the tutorial in my README for your repo if you want. Or you can link my repo.

philipp-schmidt on Aug 16, 2020

Client code will be the next thing I’ll add. It’s pretty straightforward so it should not take long.

@sudapure Can you add an issue in my repo and tell me whether you want C++ or Python first?

philipp-schmidt on Aug 16, 2020

@philipp-schmidt , appreciate your work, but without having client infer code , it looks incomplete, looking forward for client infer script if you wish to opensource it.

sudapure on Aug 16, 2020

Can you try move REGISTER_TENSORRT_PLUGIN(pluginCreator) to yololayer.h and rebuild the libmyplugins.so.

wang-xinyu on Jul 27, 2020

@philipp-schmidt Thanks for sharing. I will add a link in readme.

wang-xinyu on Aug 17, 2020

@philipp-schmidt looks very promising! Will check out soon and perhaps even contribute to some versions with other neural networks. Thanks,

cesarandreslopez on Aug 16, 2020

Hi @cesarandreslopez @wang-xinyu I’m working on it now. I planning to make a few more changes to optimize the engine (including INT8 opt and calibration, which in my tests currently gave me no speedup because the custom layers dont support INT8 and will introduce a lot of FP32 output layers in between) and I’m thinking about using the ReLU version instead of the MISH activations, so tensorrt can optimize conv/relu better.

Currently my yolov3 implementation is twice as fast as our yolov4 here (its with int8 and without combining the three yolo layers at the end though so some of it is expected)

Anyway, there will be so many changes that I might be putting it in a new repo. Ofc I will share any goodies with you here.

philipp-schmidt on Aug 14, 2020