mmdeploy: [Bug] Wrong output_tensorrt.jpg generated.

Checklist

• I have searched related issues but cannot get the expected help.
• 2. I have read the FAQ documentation but cannot get the expected help.
• 3. The bug has not been fixed in the latest version.

Describe the bug

I pick fcn_r0-d8 to train a segmentation model for CHASE_DB1 dataset in mmsegmentation (v0.30.0). It is a success. After that, I use mmdeploy (0.14.0) and try to deploy it to tensorrt. The conversion is a success (I got both end2end.engine and end2end.onnx successfully), however, the inference output for tensorrt is the same as the input image. Specifically, output_pytorch.jpg is correct after deploy.py command, but output_tensorrt.jpg has no prediction. There is no crashes/errors happening during running deploy.py.

Reproduction

1. Command:

python tools/deploy.py configs/mmseg/segmentation_tensorrt_dynamic-512x512-2048x2048.py ../mmsegmentation/configs/fcn/fcn_r50-d8_128x128_20k_chasedb1.py ../mmsegmentation/work_dirs/fcn_r50-d8_128x128_20k_chasedb1/latest.pth ../mmsegmentation/data/CHASE_DB1/images/validation/Image_11L.png --test-img ../mmsegmentation/data/CHASE_DB1/images/validation/Image_11L.png --device cuda:0 --work-dir ./work-dir/ --dump-info

1. See my fcn_r50-d8_128x128_20k_chasedb1.py below, simple modification to CHASE_DB1 dataset to support 2 classes.

_base_ = [
    '../_base_/models/fcn_r50-d8.py', '../_base_/datasets/chase_db1.py',
    '../_base_/default_runtime.py', '../_base_/schedules/schedule_20k.py'
]
model = dict(
        decode_head=dict(num_classes=2, out_channels=1, loss_decode=dict(
        type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0)),
        auxiliary_head=dict(num_classes=2, out_channels=1, loss_decode=dict(
        type='CrossEntropyLoss', use_sigmoid=True, loss_weight=0.4))
        )

Environment

1. 

/opt/conda/envs/openmmlab/lib/python3.8/site-packages/mmcv/__init__.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details.
  warnings.warn(
2023-04-11 20:39:14,363 - mmdeploy - INFO - 

2023-04-11 20:39:14,363 - mmdeploy - INFO - **********Environmental information**********
2023-04-11 20:39:14,879 - mmdeploy - INFO - sys.platform: linux
2023-04-11 20:39:14,879 - mmdeploy - INFO - Python: 3.8.16 | packaged by conda-forge | (default, Feb  1 2023, 16:01:55) [GCC 11.3.0]
2023-04-11 20:39:14,880 - mmdeploy - INFO - CUDA available: True
2023-04-11 20:39:14,880 - mmdeploy - INFO - GPU 0,1,2,3,4,5,6,7: Tesla V100-SXM2-16GB
2023-04-11 20:39:14,880 - mmdeploy - INFO - CUDA_HOME: /usr/local/cuda
2023-04-11 20:39:14,880 - mmdeploy - INFO - NVCC: Cuda compilation tools, release 11.7, V11.7.99
2023-04-11 20:39:14,880 - mmdeploy - INFO - GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
2023-04-11 20:39:14,880 - mmdeploy - INFO - PyTorch: 1.13.1+cu117
2023-04-11 20:39:14,880 - mmdeploy - INFO - PyTorch compiling details: PyTorch built with:
  - GCC 9.3
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 11.7
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
  - CuDNN 8.6  (built against CUDA 11.8)
    - Built with CuDNN 8.5
  - Magma 2.6.1
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.7, CUDNN_VERSION=8.5.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.13.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 

2023-04-11 20:39:14,880 - mmdeploy - INFO - TorchVision: 0.14.1+cu117
2023-04-11 20:39:14,880 - mmdeploy - INFO - OpenCV: 4.7.0
2023-04-11 20:39:14,880 - mmdeploy - INFO - MMCV: 1.7.1
2023-04-11 20:39:14,880 - mmdeploy - INFO - MMCV Compiler: GCC 9.3
2023-04-11 20:39:14,880 - mmdeploy - INFO - MMCV CUDA Compiler: 11.7
2023-04-11 20:39:14,880 - mmdeploy - INFO - MMDeploy: 0.14.0+c737563
2023-04-11 20:39:14,880 - mmdeploy - INFO - 

2023-04-11 20:39:14,880 - mmdeploy - INFO - **********Backend information**********
2023-04-11 20:39:14,930 - mmdeploy - INFO - tensorrt:   8.6.0
2023-04-11 20:39:14,930 - mmdeploy - INFO - tensorrt custom ops:        Available
2023-04-11 20:39:15,023 - mmdeploy - INFO - ONNXRuntime:        None
2023-04-11 20:39:15,023 - mmdeploy - INFO - ONNXRuntime-gpu:    1.14.1
2023-04-11 20:39:15,023 - mmdeploy - INFO - ONNXRuntime custom ops:     Available
2023-04-11 20:39:15,024 - mmdeploy - INFO - pplnn:      None
2023-04-11 20:39:15,026 - mmdeploy - INFO - ncnn:       None
2023-04-11 20:39:15,027 - mmdeploy - INFO - snpe:       None
2023-04-11 20:39:15,028 - mmdeploy - INFO - openvino:   None
2023-04-11 20:39:15,029 - mmdeploy - INFO - torchscript:        1.13.1+cu117
2023-04-11 20:39:15,030 - mmdeploy - INFO - torchscript custom ops:     NotAvailable
2023-04-11 20:39:15,054 - mmdeploy - INFO - rknn-toolkit:       None
2023-04-11 20:39:15,055 - mmdeploy - INFO - rknn2-toolkit:      None
2023-04-11 20:39:15,055 - mmdeploy - INFO - ascend:     None
2023-04-11 20:39:15,056 - mmdeploy - INFO - coreml:     None
2023-04-11 20:39:15,057 - mmdeploy - INFO - tvm:        None
2023-04-11 20:39:15,057 - mmdeploy - INFO - 

2023-04-11 20:39:15,057 - mmdeploy - INFO - **********Codebase information**********
2023-04-11 20:39:15,059 - mmdeploy - INFO - mmdet:      None
2023-04-11 20:39:15,059 - mmdeploy - INFO - mmseg:      0.30.0
2023-04-11 20:39:15,059 - mmdeploy - INFO - mmcls:      0.25.0
2023-04-11 20:39:15,059 - mmdeploy - INFO - mmocr:      None
2023-04-11 20:39:15,059 - mmdeploy - INFO - mmedit:     None
2023-04-11 20:39:15,059 - mmdeploy - INFO - mmdet3d:    None
2023-04-11 20:39:15,059 - mmdeploy - INFO - mmpose:     None
2023-04-11 20:39:15,059 - mmdeploy - INFO - mmrotate:   None
2023-04-11 20:39:15,059 - mmdeploy - INFO - mmaction:   None

### Error traceback

```Shell
N/A