mmdeploy: [Bug] visualize onnxruntime model failed. all output are zeros onnx/tensorrt

Checklist

• I have searched related issues but cannot get the expected help.
• 2. I have read the FAQ documentation but cannot get the expected help.
• 3. The bug has not been fixed in the latest version.

Describe the bug

Hi, when I try to convert models trained from mmsegmentation (unet, deeplabv3plus, swin) using tensorrt, the outputs are all zeros. However, if I use the same backbone in a classification model trained in mmpretrain, it is okay. To resolve this problem, I tried the official example stated in here. but encountered “visualize onnxruntime model failed.” Is the anything wrong with my setting?

Reproduction

for tensorrt: DEPLOY_CFG=“configs/mmseg/segmentation_tensorrt_static-512x512.py” for onnx: DEPLOY_CFG=“configs/mmseg/segmentation_onnxruntime_static-512x512.py”

python tools/deploy.py
${DEPLOY_CFG}
${MODEL_CFG}
${CHECKPOINT}
${INPUT_IMG}
–device ${DEVICE}
–dump-info
–work-dir ${WORK_DIR}

Environment

06/07 15:43:29 - mmengine - INFO -

06/07 15:43:29 - mmengine - INFO - **********Environmental information**********
06/07 15:43:30 - mmengine - INFO - sys.platform: linux
06/07 15:43:30 - mmengine - INFO - Python: 3.8.16 (default, Mar  2 2023, 03:21:46) [GCC 11.2.0]
06/07 15:43:30 - mmengine - INFO - CUDA available: True
06/07 15:43:30 - mmengine - INFO - numpy_random_seed: 2147483648
06/07 15:43:30 - mmengine - INFO - GPU 0,1: NVIDIA GeForce RTX 3060
06/07 15:43:30 - mmengine - INFO - CUDA_HOME: /usr/local/cuda
06/07 15:43:30 - mmengine - INFO - NVCC: Cuda compilation tools, release 11.1, V11.1.74
06/07 15:43:30 - mmengine - INFO - GCC: x86_64-conda_cos6-linux-gnu-gcc (conda-forge gcc 12.1.0-17) 12.1.0
06/07 15:43:30 - mmengine - INFO - PyTorch: 1.10.1+cu111
06/07 15:43:30 - mmengine - INFO - PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.2.3 (Git Hash 7336ca9f055cf1bfa13efb658fe15dc9b41f0740)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 11.1
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
  - CuDNN 8.0.5
  - Magma 2.5.2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.10.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

06/07 15:43:30 - mmengine - INFO - TorchVision: 0.11.2+cu111
06/07 15:43:30 - mmengine - INFO - OpenCV: 4.7.0
06/07 15:43:30 - mmengine - INFO - MMEngine: 0.7.2
06/07 15:43:30 - mmengine - INFO - MMCV: 2.0.0
06/07 15:43:30 - mmengine - INFO - MMCV Compiler: GCC 7.3
06/07 15:43:30 - mmengine - INFO - MMCV CUDA Compiler: not available
06/07 15:43:30 - mmengine - INFO - MMDeploy: 1.1.0+faf05fe
06/07 15:43:30 - mmengine - INFO -

06/07 15:43:30 - mmengine - INFO - **********Backend information**********
06/07 15:43:30 - mmengine - INFO - tensorrt:    8.6.1
06/07 15:43:30 - mmengine - INFO - tensorrt custom ops: Available
06/07 15:43:30 - mmengine - INFO - ONNXRuntime: None
06/07 15:43:30 - mmengine - INFO - ONNXRuntime-gpu:     1.8.1
06/07 15:43:30 - mmengine - INFO - ONNXRuntime custom ops:      Available
06/07 15:43:30 - mmengine - INFO - pplnn:       None
06/07 15:43:30 - mmengine - INFO - ncnn:        None
06/07 15:43:30 - mmengine - INFO - snpe:        None
06/07 15:43:30 - mmengine - INFO - openvino:    None
06/07 15:43:30 - mmengine - INFO - torchscript: 1.10.1+cu111
06/07 15:43:30 - mmengine - INFO - torchscript custom ops:      NotAvailable
06/07 15:43:30 - mmengine - INFO - rknn-toolkit:        None
06/07 15:43:30 - mmengine - INFO - rknn-toolkit2:       None
06/07 15:43:30 - mmengine - INFO - ascend:      None
06/07 15:43:30 - mmengine - INFO - coreml:      None
06/07 15:43:30 - mmengine - INFO - tvm: None
06/07 15:43:30 - mmengine - INFO - vacc:        None
06/07 15:43:30 - mmengine - INFO -

06/07 15:43:30 - mmengine - INFO - **********Codebase information**********
06/07 15:43:30 - mmengine - INFO - mmdet:       None
06/07 15:43:30 - mmengine - INFO - mmseg:       1.0.0
06/07 15:43:30 - mmengine - INFO - mmpretrain:  1.0.0rc7
06/07 15:43:30 - mmengine - INFO - mmocr:       None
06/07 15:43:30 - mmengine - INFO - mmagic:      1.0.1.dev0
06/07 15:43:30 - mmengine - INFO - mmdet3d:     None
06/07 15:43:30 - mmengine - INFO - mmpose:      None
06/07 15:43:30 - mmengine - INFO - mmrotate:    None
06/07 15:43:30 - mmengine - INFO - mmaction:    None
06/07 15:43:30 - mmengine - INFO - mmrazor:     None

Error traceback

06/07 15:44:33 - mmengine - WARNING - Failed to search registry with scope "mmseg" in the "Codebases" registry tree. As a workaround, the current "Codebases" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmseg" is a correct scope, or whether the registry is initialized.
06/07 15:44:33 - mmengine - WARNING - Failed to search registry with scope "mmseg" in the "mmseg_tasks" registry tree. As a workaround, the current "mmseg_tasks" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmseg" is a correct scope, or whether the registry is initialized.
06/07 15:44:34 - mmengine - INFO - Start pipeline mmdeploy.apis.pytorch2onnx.torch2onnx in subprocess
06/07 15:44:35 - mmengine - WARNING - Failed to search registry with scope "mmseg" in the "Codebases" registry tree. As a workaround, the current "Codebases" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmseg" is a correct scope, or whether the registry is initialized.
06/07 15:44:35 - mmengine - WARNING - Failed to search registry with scope "mmseg" in the "mmseg_tasks" registry tree. As a workaround, the current "mmseg_tasks" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmseg" is a correct scope, or whether the registry is initialized.
/home/huangyh/code/mmsegmentation/mmseg/models/builder.py:36: UserWarning: ``build_loss`` would be deprecated soon, please use ``mmseg.registry.MODELS.build()``
  warnings.warn('``build_loss`` would be deprecated soon, please use '
/home/huangyh/code/mmsegmentation/mmseg/models/losses/cross_entropy_loss.py:235: UserWarning: Default ``avg_non_ignore`` is False, if you would like to ignore the certain label and average loss over non-ignore labels, which is the same with PyTorch official cross_entropy, set ``avg_non_ignore=True``.
  warnings.warn(
Loads checkpoint by local backend from path: fcn_unet_s5-d16_4x4_512x1024_160k_cityscapes_20211210_145204-6860854e.pth
06/07 15:44:35 - mmengine - WARNING - DeprecationWarning: get_onnx_config will be deprecated in the future.
06/07 15:44:35 - mmengine - INFO - Export PyTorch model to ONNX: mmdeploy_model/mmseg/back_seg/fcn/onnx/end2end.onnx.
06/07 15:44:35 - mmengine - WARNING - Can not find torch._C._jit_pass_onnx_autograd_function_process, function rewrite will not be applied
/home/huangyh/code/mmdeploy/mmdeploy/core/optimizers/function_marker.py:160: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  ys_shape = tuple(int(s) for s in ys.shape)
/home/huangyh/code/mmsegmentation/mmseg/models/backbones/unet.py:431: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert (h % whole_downsample_rate == 0) \
06/07 15:44:42 - mmengine - INFO - Execute onnx optimize passes.
06/07 15:44:43 - mmengine - INFO - Finish pipeline mmdeploy.apis.pytorch2onnx.torch2onnx
06/07 15:44:43 - mmengine - INFO - Start pipeline mmdeploy.apis.utils.utils.to_backend in main process
06/07 15:44:43 - mmengine - INFO - Finish pipeline mmdeploy.apis.utils.utils.to_backend
06/07 15:44:43 - mmengine - INFO - visualize onnxruntime model start.
06/07 15:44:44 - mmengine - WARNING - Failed to search registry with scope "mmseg" in the "Codebases" registry tree. As a workaround, the current "Codebases" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmseg" is a correct scope, or whether the registry is initialized.
06/07 15:44:44 - mmengine - WARNING - Failed to search registry with scope "mmseg" in the "mmseg_tasks" registry tree. As a workaround, the current "mmseg_tasks" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmseg" is a correct scope, or whether the registry is initialized.
06/07 15:44:44 - mmengine - WARNING - Failed to search registry with scope "mmseg" in the "backend_segmentors" registry tree. As a workaround, the current "backend_segmentors" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmseg" is a correct scope, or whether the registry is initialized.
06/07 15:44:44 - mmengine - INFO - Successfully loaded onnxruntime custom ops from /home/huangyh/code/mmdeploy/mmdeploy/lib/libmmdeploy_onnxruntime_ops.so
/opt/rh/devtoolset-8/root/usr/include/c++/8/bits/stl_vector.h:932: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::operator[](std::vector<_Tp, _Alloc>::size_type) [with _Tp = int; _Alloc = std::allocator<int>; std::vector<_Tp, _Alloc>::reference = int&; std::vector<_Tp, _Alloc>::size_type = long unsigned int]: Assertion '__builtin_expect(__n < this->size(), true)' failed.
06/07 15:44:44 - mmengine - ERROR - tools/deploy.py - create_process - 82 - visualize onnxruntime model failed.