TensorRT: Myelin error when load ONNX model, Assertion `false && "Invalid size written"' failed

Description

We’re preparing to use Orin with TensorRT8.4 for better performance, but it failed when creating a Builder. Error occurred in myelin which is close-sourced, we’ve gotten stuck here for days and had no clue where to start debugging, so ask for your help:)

Environment

TensorRT Version: 8400 NVIDIA GPU: Orin NVIDIA Driver Version: 11.4 CUDA Version: 11.4 Operating System: Ubunut20.4

Key error

It can be reproduced by just running trtexec with ONNX model, it aborted at error: trtexec: /root/gpgpu/MachineLearning/myelin/src/compiler/optimizer/cublas_impl.cpp:479: void add_heuristic_results_to_tactics(std::vector<cublasLtMatmulHeuristicResult_t>&, std::vectormyelin::ir::tactic_attribute_t&, myelin::ir::tactic_attribute_t&, bool): Assertion `false && “Invalid size written”’ failed.

Detailed log were attached, please check:

test_model.log

Simpilifed onnx model file for reproduction: test_model.tar.gz