mmdeploy: [Bug] Manually building RTMDet-Inst tensorRT engine fails with internal error

Checklist

• I have searched related issues but cannot get the expected help.
• 2. I have read the FAQ documentation but cannot get the expected help.
• 3. The bug has not been fixed in the latest version.

Describe the bug

I want to parse the RTMDet-Inst model and build the serialized model from .onnx file representation in order to control which tensorRT version is being used (and not necessarily 8.2 which is default for mmdeploy)

When I try to load the RTMDet-Inst end2end.onnx model created using mmdeploy into a tensorRT python script to build the engine I get the following error:

[TRT] [E] 4: [graphShapeAnalyzer.cpp::nvinfer1::builder::`anonymous-namespace’::ShapeAnalyzerImpl::processCheck::862] Error Code 4: Internal Error (/TopK: K exceeds the maximum value allowed (3840).)

Reproduction

import tensorrt as trt

logger = trt.Logger(trt.Logger.WARNING)
trt.init_libnvinfer_plugins(logger, '')
runtime = trt.Runtime(logger)
builder = trt.Builder(logger)
network = builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH))
parser = trt.OnnxParser(network, logger)
success = parser.parse_from_file("end2end.onnx")
for idx in range(parser.num_errors):
    print(parser.get_error(idx))

if not success:
    return None # Error handling code here

config = builder.create_builder_config()
config.set_memory_pool_limit(trt.MemoryPoolType.WORKSPACE, 1 << 20)
serialized_engine = builder.build_serialized_network(network, config) # <--- this fails with internal error

Environment

07/03 10:52:46 - mmengine - INFO - **********Environmental information**********
07/03 10:52:49 - mmengine - INFO - sys.platform: win32
07/03 10:52:49 - mmengine - INFO - Python: 3.8.16 (default, Mar  2 2023, 03:18:16) [MSC v.1916 64 bit (AMD64)]
07/03 10:52:49 - mmengine - INFO - CUDA available: True
07/03 10:52:49 - mmengine - INFO - numpy_random_seed: 2147483648
07/03 10:52:49 - mmengine - INFO - GPU 0: NVIDIA GeForce RTX 3060 Laptop GPU
07/03 10:52:49 - mmengine - INFO - CUDA_HOME: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6        
07/03 10:52:49 - mmengine - INFO - NVCC: Cuda compilation tools, release 11.6, V11.6.124
07/03 10:52:49 - mmengine - INFO - MSVC: Microsoft (R) C/C++ Optimizing Compiler Version 19.34.31937 for x64  
07/03 10:52:49 - mmengine - INFO - GCC: n/a
07/03 10:52:49 - mmengine - INFO - PyTorch: 1.13.1+cu116
07/03 10:52:49 - mmengine - INFO - PyTorch compiling details: PyTorch built with:
  - C++ Version: 199711
  - MSVC 192829337
  - Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
  - OpenMP 2019
  - LAPACK is enabled (usually provided by MKL)        
  - CPU capability usage: AVX2
  - CUDA Runtime 11.6
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
07/03 10:52:49 - mmengine - INFO - OpenCV: 4.7.0
07/03 10:52:49 - mmengine - INFO - MMEngine: 0.7.4
07/03 10:52:49 - mmengine - INFO - MMCV: 2.0.0
07/03 10:52:49 - mmengine - INFO - MMCV Compiler: MSVC 193431937
07/03 10:52:49 - mmengine - INFO - MMCV CUDA Compiler: 11.6
07/03 10:52:49 - mmengine - INFO - MMDeploy: 1.0.0rc3+
07/03 10:52:49 - mmengine - INFO -

07/03 10:52:49 - mmengine - INFO - **********Backend information**********
07/03 10:52:49 - mmengine - INFO - tensorrt:    8.6.1
07/03 10:52:49 - mmengine - INFO - tensorrt custom opsAvailable
07/03 10:52:49 - mmengine - INFO - ONNXRuntime: 1.14.1
07/03 10:52:49 - mmengine - INFO - ONNXRuntime-gpu:   1.14.1
07/03 10:52:49 - mmengine - INFO - ONNXRuntime custom ops:     NotAvailable
07/03 10:52:49 - mmengine - INFO - pplnn:       None
07/03 10:52:49 - mmengine - INFO - ncnn:        None
07/03 10:52:49 - mmengine - INFO - snpe:        None
07/03 10:52:49 - mmengine - INFO - openvino:    None
07/03 10:52:49 - mmengine - INFO - torchscript: 1.13.1+cu116
07/03 10:52:49 - mmengine - INFO - torchscript custom ops:     NotAvailable
07/03 10:52:49 - mmengine - INFO - rknn-toolkit:      None
07/03 10:52:49 - mmengine - INFO - rknn-toolkit2:     None
07/03 10:52:49 - mmengine - INFO - ascend:      None
07/03 10:52:49 - mmengine - INFO - coreml:      None
07/03 10:52:49 - mmengine - INFO - tvm: None
07/03 10:52:49 - mmengine - INFO -

07/03 10:52:49 - mmengine - INFO - **********Codebase information**********
07/03 10:52:49 - mmengine - INFO - mmdet:       3.0.0
07/03 10:52:49 - mmengine - INFO - mmseg:       None
07/03 10:52:49 - mmengine - INFO - mmcls:       None
07/03 10:52:49 - mmengine - INFO - mmocr:       None
07/03 10:52:49 - mmengine - INFO - mmedit:      None
07/03 10:52:49 - mmengine - INFO - mmdet3d:     None
07/03 10:52:49 - mmengine - INFO - mmpose:      None
07/03 10:52:49 - mmengine - INFO - mmrotate:    None
07/03 10:52:49 - mmengine - INFO - mmaction:    None
(trt_export_38)

Error traceback

[TRT] [E] 4: [graphShapeAnalyzer.cpp::nvinfer1::builder::`anonymous-namespace'::ShapeAnalyzerImpl::processCheck::862] Error Code 4: Internal Error (/TopK: K exceeds the maximum value allowed (3840).)