server: error running simple example

Description A clear and concise description of what the bug is.

triton_python_backend_stub /tensorrt/triton-repos/trtibf-Trendyol-LLM-7b-chat-v1.0/preprocessing/1/model.py triton_python_backend_shm_region_6 1048576 1048576 2796 /opt/tritonserver/backends/python 336 preprocessing_0_0 DEFAULT
I0415 13:01:09.290891 2796 python_be.cc:2384] TRITONBACKEND_ModelInstanceInitialize: instance initialization successful postprocessing_0_0 (device 0)
I0415 13:01:09.291048 2796 backend_model_instance.cc:772] Starting backend thread for postprocessing_0_0 at nice 0 on device 0...
I0415 13:01:09.291151 2796 backend_model.cc:618] Created model instance named 'postprocessing_0_0' with device id '0'
I0415 13:01:09.291257 2796 model_lifecycle.cc:675] OnLoadComplete() 'postprocessing' version 1
I0415 13:01:09.291277 2796 model_lifecycle.cc:713] OnLoadFinal() 'postprocessing' for all version(s)
I0415 13:01:09.291284 2796 model_lifecycle.cc:818] successfully loaded 'postprocessing'
I0415 13:01:09.291368 2796 model_lifecycle.cc:286] VersionStates() 'postprocessing'
I0415 13:01:09.414078 2796 python_be.cc:2384] TRITONBACKEND_ModelInstanceInitialize: instance initialization successful preprocessing_0_0 (device 0)
I0415 13:01:09.414210 2796 backend_model_instance.cc:772] Starting backend thread for preprocessing_0_0 at nice 0 on device 0...
I0415 13:01:09.414317 2796 backend_model.cc:618] Created model instance named 'preprocessing_0_0' with device id '0'
I0415 13:01:09.414441 2796 model_lifecycle.cc:675] OnLoadComplete() 'preprocessing' version 1
I0415 13:01:09.414469 2796 model_lifecycle.cc:713] OnLoadFinal() 'preprocessing' for all version(s)
I0415 13:01:09.414492 2796 model_lifecycle.cc:818] successfully loaded 'preprocessing'
I0415 13:01:09.414545 2796 model_lifecycle.cc:286] VersionStates() 'preprocessing'
[TensorRT-LLM][INFO] TRTGptModel maxNumSequences: 2
[TensorRT-LLM][INFO] TRTGptModel maxBatchSize: 1
[TensorRT-LLM][INFO] TRTGptModel mMaxAttentionWindowSize: 2560
[TensorRT-LLM][INFO] TRTGptModel enableTrtOverlap: 1
[TensorRT-LLM][WARNING] Device 0 peer access Device 1 is not available.
[TensorRT-LLM][WARNING] Device 0 peer access Device 2 is not available.
[TensorRT-LLM][WARNING] Device 0 peer access Device 3 is not available.
[TensorRT-LLM][INFO] Loaded engine size: 14008 MiB
[TensorRT-LLM][ERROR] tensorrt_llm::common::TllmException: [TensorRT-LLM][ERROR] Assertion failed: d == a + length (/app/tensorrt_llm/cpp/tensorrt_llm/plugins/gemmPlugin/gemmPlugin.cpp:156)
1       0x7fd1a11c612f /opt/tritonserver/backends/tensorrtllm/libnvinfer_plugin_tensorrt_llm.so.9(+0x4512f) [0x7fd1a11c612f]
2       0x7fd1a1256ac8 tensorrt_llm::plugins::GemmPlugin::GemmPlugin(void const*, unsigned long, std::shared_ptr<tensorrt_llm::plugins::CublasLtGemmPluginProfiler> const&) + 440
3       0x7fd1a1256bdf tensorrt_llm::plugins::GemmPluginCreator::deserializePlugin(char const*, void const*, unsigned long) + 191
4       0x7fd1e8b118a6 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10d68a6) [0x7fd1e8b118a6]
5       0x7fd1e8b0966e /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10ce66e) [0x7fd1e8b0966e]
6       0x7fd1e8aa4217 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x1069217) [0x7fd1e8aa4217]
7       0x7fd1e8aa219e /usr/local/tensorrt/lib/libnvinfer.so.9(+0x106719e) [0x7fd1e8aa219e]
8       0x7fd1e8ab9c2b /usr/local/tensorrt/lib/libnvinfer.so.9(+0x107ec2b) [0x7fd1e8ab9c2b]
9       0x7fd1e8abce32 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x1081e32) [0x7fd1e8abce32]
10      0x7fd1e8abd20c /usr/local/tensorrt/lib/libnvinfer.so.9(+0x108220c) [0x7fd1e8abd20c]
11      0x7fd1e8af09b1 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10b59b1) [0x7fd1e8af09b1]
12      0x7fd1e8af1777 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10b6777) [0x7fd1e8af1777]
13      0x7fd1f7579022 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x127022) [0x7fd1f7579022]
14      0x7fd1f74d29cf /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x809cf) [0x7fd1f74d29cf]
15      0x7fd1f74c299f /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x7099f) [0x7fd1f74c299f]
16      0x7fd1f74b9222 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x67222) [0x7fd1f74b9222]
17      0x7fd1f749acbc /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x48cbc) [0x7fd1f749acbc]
18      0x7fd1f749bd62 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x49d62) [0x7fd1f749bd62]
19      0x7fd1f748bbb5 TRITONBACKEND_ModelInstanceInitialize + 101
20      0x7fd2b1589226 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x1a7226) [0x7fd2b1589226]
21      0x7fd2b158a466 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x1a8466) [0x7fd2b158a466]
22      0x7fd2b156d165 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x18b165) [0x7fd2b156d165]
23      0x7fd2b156d7a6 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x18b7a6) [0x7fd2b156d7a6]
24      0x7fd2b1579a1d /opt/tritonserver/bin/../lib/libtritonserver.so(+0x197a1d) [0x7fd2b1579a1d]
25      0x7fd2b0be4ee8 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fd2b0be4ee8]
26      0x7fd2b1563feb /opt/tritonserver/bin/../lib/libtritonserver.so(+0x181feb) [0x7fd2b1563feb]
27      0x7fd2b1573dc5 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x191dc5) [0x7fd2b1573dc5]
28      0x7fd2b1578d36 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x196d36) [0x7fd2b1578d36]
29      0x7fd2b1669330 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x287330) [0x7fd2b1669330]
30      0x7fd2b166ca23 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x28aa23) [0x7fd2b166ca23]
31      0x7fd2b17c0d82 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x3ded82) [0x7fd2b17c0d82]
32      0x7fd2b0e4f253 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7fd2b0e4f253]
33      0x7fd2b0bdfac3 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7fd2b0bdfac3]
34      0x7fd2b0c70814 clone + 68
[TensorRT-LLM][ERROR] tensorrt_llm::common::TllmException: [TensorRT-LLM][ERROR] Assertion failed: d == a + length (/app/tensorrt_llm/cpp/tensorrt_llm/plugins/gptAttentionCommon/gptAttentionCommon.cpp:418)
1       0x7fd1a11c612f /opt/tritonserver/backends/tensorrtllm/libnvinfer_plugin_tensorrt_llm.so.9(+0x4512f) [0x7fd1a11c612f]
2       0x7fd1a1232846 tensorrt_llm::plugins::GPTAttentionPluginCommon::GPTAttentionPluginCommon(void const*, unsigned long) + 870
3       0x7fd1a12498b3 tensorrt_llm::plugins::GPTAttentionPlugin::GPTAttentionPlugin(void const*, unsigned long) + 19
4       0x7fd1a1249932 tensorrt_llm::plugins::GPTAttentionPluginCreator::deserializePlugin(char const*, void const*, unsigned long) + 50
5       0x7fd1e8b118a6 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10d68a6) [0x7fd1e8b118a6]
6       0x7fd1e8b0966e /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10ce66e) [0x7fd1e8b0966e]
7       0x7fd1e8aa4217 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x1069217) [0x7fd1e8aa4217]
8       0x7fd1e8aa219e /usr/local/tensorrt/lib/libnvinfer.so.9(+0x106719e) [0x7fd1e8aa219e]
9       0x7fd1e8ab9c2b /usr/local/tensorrt/lib/libnvinfer.so.9(+0x107ec2b) [0x7fd1e8ab9c2b]
10      0x7fd1e8abce32 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x1081e32) [0x7fd1e8abce32]
11      0x7fd1e8abd20c /usr/local/tensorrt/lib/libnvinfer.so.9(+0x108220c) [0x7fd1e8abd20c]
12      0x7fd1e8af09b1 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10b59b1) [0x7fd1e8af09b1]
13      0x7fd1e8af1777 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10b6777) [0x7fd1e8af1777]
14      0x7fd1f7579022 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x127022) [0x7fd1f7579022]
15      0x7fd1f74d29cf /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x809cf) [0x7fd1f74d29cf]
16      0x7fd1f74c299f /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x7099f) [0x7fd1f74c299f]
17      0x7fd1f74b9222 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x67222) [0x7fd1f74b9222]
18      0x7fd1f749acbc /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x48cbc) [0x7fd1f749acbc]
19      0x7fd1f749bd62 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x49d62) [0x7fd1f749bd62]
20      0x7fd1f748bbb5 TRITONBACKEND_ModelInstanceInitialize + 101
21      0x7fd2b1589226 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x1a7226) [0x7fd2b1589226]
22      0x7fd2b158a466 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x1a8466) [0x7fd2b158a466]
23      0x7fd2b156d165 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x18b165) [0x7fd2b156d165]
24      0x7fd2b156d7a6 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x18b7a6) [0x7fd2b156d7a6]
25      0x7fd2b1579a1d /opt/tritonserver/bin/../lib/libtritonserver.so(+0x197a1d) [0x7fd2b1579a1d]
26      0x7fd2b0be4ee8 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fd2b0be4ee8]
27      0x7fd2b1563feb /opt/tritonserver/bin/../lib/libtritonserver.so(+0x181feb) [0x7fd2b1563feb]
28      0x7fd2b1573dc5 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x191dc5) [0x7fd2b1573dc5]
29      0x7fd2b1578d36 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x196d36) [0x7fd2b1578d36]
30      0x7fd2b1669330 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x287330) [0x7fd2b1669330]
31      0x7fd2b166ca23 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x28aa23) [0x7fd2b166ca23]
32      0x7fd2b17c0d82 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x3ded82) [0x7fd2b17c0d82]
33      0x7fd2b0e4f253 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7fd2b0e4f253]
34      0x7fd2b0bdfac3 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7fd2b0bdfac3]
35      0x7fd2b0c70814 clone + 68
[TensorRT-LLM][ERROR] tensorrt_llm::common::TllmException: [TensorRT-LLM][ERROR] Assertion failed: d == a + length (/app/tensorrt_llm/cpp/tensorrt_llm/plugins/gemmPlugin/gemmPlugin.cpp:156)
1       0x7fd1a11c612f /opt/tritonserver/backends/tensorrtllm/libnvinfer_plugin_tensorrt_llm.so.9(+0x4512f) [0x7fd1a11c612f]
2       0x7fd1a1256ac8 tensorrt_llm::plugins::GemmPlugin::GemmPlugin(void const*, unsigned long, std::shared_ptr<tensorrt_llm::plugins::CublasLtGemmPluginProfiler> const&) + 440
3       0x7fd1a1256bdf tensorrt_llm::plugins::GemmPluginCreator::deserializePlugin(char const*, void const*, unsigned long) + 191
4       0x7fd1e8b118a6 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10d68a6) [0x7fd1e8b118a6]
5       0x7fd1e8b0966e /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10ce66e) [0x7fd1e8b0966e]
6       0x7fd1e8aa4217 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x1069217) [0x7fd1e8aa4217]
7       0x7fd1e8aa219e /usr/local/tensorrt/lib/libnvinfer.so.9(+0x106719e) [0x7fd1e8aa219e]
8       0x7fd1e8ab9c2b /usr/local/tensorrt/lib/libnvinfer.so.9(+0x107ec2b) [0x7fd1e8ab9c2b]
9       0x7fd1e8abce32 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x1081e32) [0x7fd1e8abce32]
10      0x7fd1e8abd20c /usr/local/tensorrt/lib/libnvinfer.so.9(+0x108220c) [0x7fd1e8abd20c]
11      0x7fd1e8af09b1 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10b59b1) [0x7fd1e8af09b1]
12      0x7fd1e8af1777 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10b6777) [0x7fd1e8af1777]
13      0x7fd1f7579022 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x127022) [0x7fd1f7579022]
14      0x7fd1f74d29cf /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x809cf) [0x7fd1f74d29cf]
15      0x7fd1f74c299f /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x7099f) [0x7fd1f74c299f]
16      0x7fd1f74b9222 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x67222) [0x7fd1f74b9222]
17      0x7fd1f749acbc /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x48cbc) [0x7fd1f749acbc]
18      0x7fd1f749bd62 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x49d62) [0x7fd1f749bd62]
19      0x7fd1f748bbb5 TRITONBACKEND_ModelInstanceInitialize + 101
20      0x7fd2b1589226 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x1a7226) [0x7fd2b1589226]
21      0x7fd2b158a466 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x1a8466) [0x7fd2b158a466]
22      0x7fd2b156d165 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x18b165) [0x7fd2b156d165]
23      0x7fd2b156d7a6 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x18b7a6) [0x7fd2b156d7a6]
24      0x7fd2b1579a1d /opt/tritonserver/bin/../lib/libtritonserver.so(+0x197a1d) [0x7fd2b1579a1d]
25      0x7fd2b0be4ee8 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fd2b0be4ee8]
26      0x7fd2b1563feb /opt/tritonserver/bin/../lib/libtritonserver.so(+0x181feb) [0x7fd2b1563feb]
27      0x7fd2b1573dc5 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x191dc5) [0x7fd2b1573dc5]
28      0x7fd2b1578d36 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x196d36) [0x7fd2b1578d36]
29      0x7fd2b1669330 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x287330) [0x7fd2b1669330]
30      0x7fd2b166ca23 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x28aa23) [0x7fd2b166ca23]
31      0x7fd2b17c0d82 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x3ded82) [0x7fd2b17c0d82]
32      0x7fd2b0e4f253 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7fd2b0e4f253]
33      0x7fd2b0bdfac3 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7fd2b0bdfac3]
34      0x7fd2b0c70814 clone + 68
[TensorRT-LLM][ERROR] tensorrt_llm::common::TllmException: [TensorRT-LLM][ERROR] Assertion failed: d == a + length (/app/tensorrt_llm/cpp/tensorrt_llm/plugins/gemmPlugin/gemmPlugin.cpp:156)
1       0x7fd1a11c612f /opt/tritonserver/backends/tensorrtllm/libnvinfer_plugin_tensorrt_llm.so.9(+0x4512f) [0x7fd1a11c612f]
2       0x7fd1a1256ac8 tensorrt_llm::plugins::GemmPlugin::GemmPlugin(void const*, unsigned long, std::shared_ptr<tensorrt_llm::plugins::CublasLtGemmPluginProfiler> const&) + 440
3       0x7fd1a1256bdf tensorrt_llm::plugins::GemmPluginCreator::deserializePlugin(char const*, void const*, unsigned long) + 191
4       0x7fd1e8b118a6 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10d68a6) [0x7fd1e8b118a6]
5       0x7fd1e8b0966e /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10ce66e) [0x7fd1e8b0966e]
6       0x7fd1e8aa4217 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x1069217) [0x7fd1e8aa4217]
7       0x7fd1e8aa219e /usr/local/tensorrt/lib/libnvinfer.so.9(+0x106719e) [0x7fd1e8aa219e]
8       0x7fd1e8ab9c2b /usr/local/tensorrt/lib/libnvinfer.so.9(+0x107ec2b) [0x7fd1e8ab9c2b]
9       0x7fd1e8abce32 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x1081e32) [0x7fd1e8abce32]
10      0x7fd1e8abd20c /usr/local/tensorrt/lib/libnvinfer.so.9(+0x108220c) [0x7fd1e8abd20c]
11      0x7fd1e8af09b1 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10b59b1) [0x7fd1e8af09b1]
12      0x7fd1e8af1777 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10b6777) [0x7fd1e8af1777]
13      0x7fd1f7579022 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x127022) [0x7fd1f7579022]
14      0x7fd1f74d29cf /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x809cf) [0x7fd1f74d29cf]
15      0x7fd1f74c299f /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x7099f) [0x7fd1f74c299f]
16      0x7fd1f74b9222 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x67222) [0x7fd1f74b9222]
17      0x7fd1f749acbc /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x48cbc) [0x7fd1f749acbc]
18      0x7fd1f749bd62 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x49d62) [0x7fd1f749bd62]
19      0x7fd1f748bbb5 TRITONBACKEND_ModelInstanceInitialize + 101
20      0x7fd2b1589226 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x1a7226) [0x7fd2b1589226]
21      0x7fd2b158a466 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x1a8466) [0x7fd2b158a466]
22      0x7fd2b156d165 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x18b165) [0x7fd2b156d165]
23      0x7fd2b156d7a6 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x18b7a6) [0x7fd2b156d7a6]
24      0x7fd2b1579a1d /opt/tritonserver/bin/../lib/libtritonserver.so(+0x197a1d) [0x7fd2b1579a1d]
25      0x7fd2b0be4ee8 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fd2b0be4ee8]
26      0x7fd2b1563feb /opt/tritonserver/bin/../lib/libtritonserver.so(+0x181feb) [0x7fd2b1563feb]
27      0x7fd2b1573dc5 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x191dc5) [0x7fd2b1573dc5]
28      0x7fd2b1578d36 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x196d36) [0x7fd2b1578d36]
29      0x7fd2b1669330 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x287330) [0x7fd2b1669330]
30      0x7fd2b166ca23 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x28aa23) [0x7fd2b166ca23]
31      0x7fd2b17c0d82 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x3ded82) [0x7fd2b17c0d82]
32      0x7fd2b0e4f253 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7fd2b0e4f253]
33      0x7fd2b0bdfac3 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7fd2b0bdfac3]
34      0x7fd2b0c70814 clone + 68
[TensorRT-LLM][ERROR] tensorrt_llm::common::TllmException: [TensorRT-LLM][ERROR] Assertion failed: d == a + length (/app/tensorrt_llm/cpp/tensorrt_llm/plugins/gemmPlugin/gemmPlugin.cpp:156)
1       0x7fd1a11c612f /opt/tritonserver/backends/tensorrtllm/libnvinfer_plugin_tensorrt_llm.so.9(+0x4512f) [0x7fd1a11c612f]
2       0x7fd1a1256ac8 tensorrt_llm::plugins::GemmPlugin::GemmPlugin(void const*, unsigned long, std::shared_ptr<tensorrt_llm::plugins::CublasLtGemmPluginProfiler> const&) + 440
3       0x7fd1a1256bdf tensorrt_llm::plugins::GemmPluginCreator::deserializePlugin(char const*, void const*, unsigned long) + 191
4       0x7fd1e8b118a6 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10d68a6) [0x7fd1e8b118a6]
5       0x7fd1e8b0966e /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10ce66e) [0x7fd1e8b0966e]
6       0x7fd1e8aa4217 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x1069217) [0x7fd1e8aa4217]
7       0x7fd1e8aa219e /usr/local/tensorrt/lib/libnvinfer.so.9(+0x106719e) [0x7fd1e8aa219e]
8       0x7fd1e8ab9c2b /usr/local/tensorrt/lib/libnvinfer.so.9(+0x107ec2b) [0x7fd1e8ab9c2b]
9       0x7fd1e8abce32 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x1081e32) [0x7fd1e8abce32]
10      0x7fd1e8abd20c /usr/local/tensorrt/lib/libnvinfer.so.9(+0x108220c) [0x7fd1e8abd20c]
11      0x7fd1e8af09b1 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10b59b1) [0x7fd1e8af09b1]
12      0x7fd1e8af1777 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10b6777) [0x7fd1e8af1777]
13      0x7fd1f7579022 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x127022) [0x7fd1f7579022]
14      0x7fd1f74d29cf /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x809cf) [0x7fd1f74d29cf]
15      0x7fd1f74c299f /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x7099f) [0x7fd1f74c299f]
16      0x7fd1f74b9222 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x67222) [0x7fd1f74b9222]
17      0x7fd1f749acbc /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x48cbc) [0x7fd1f749acbc]
18      0x7fd1f749bd62 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x49d62) [0x7fd1f749bd62]
19      0x7fd1f748bbb5 TRITONBACKEND_ModelInstanceInitialize + 101
20      0x7fd2b1589226 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x1a7226) [0x7fd2b1589226]
21      0x7fd2b158a466 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x1a8466) [0x7fd2b158a466]
22      0x7fd2b156d165 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x18b165) [0x7fd2b156d165]
23      0x7fd2b156d7a6 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x18b7a6) [0x7fd2b156d7a6]
24      0x7fd2b1579a1d /opt/tritonserver/bin/../lib/libtritonserver.so(+0x197a1d) [0x7fd2b1579a1d]
25      0x7fd2b0be4ee8 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fd2b0be4ee8]
26      0x7fd2b1563feb /opt/tritonserver/bin/../lib/libtritonserver.so(+0x181feb) [0x7fd2b1563feb]
27      0x7fd2b1573dc5 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x191dc5) [0x7fd2b1573dc5]
28      0x7fd2b1578d36 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x196d36) [0x7fd2b1578d36]
29      0x7fd2b1669330 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x287330) [0x7fd2b1669330]
30      0x7fd2b166ca23 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x28aa23) [0x7fd2b166ca23]
31      0x7fd2b17c0d82 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x3ded82) [0x7fd2b17c0d82]
32      0x7fd2b0e4f253 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7fd2b0e4f253]
33      0x7fd2b0bdfac3 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7fd2b0bdfac3]
34      0x7fd2b0c70814 clone + 68
[TensorRT-LLM][ERROR] tensorrt_llm::common::TllmException: [TensorRT-LLM][ERROR] Assertion failed: d == a + length (/app/tensorrt_llm/cpp/tensorrt_llm/plugins/gemmPlugin/gemmPlugin.cpp:156)
1       0x7fd1a11c612f /opt/tritonserver/backends/tensorrtllm/libnvinfer_plugin_tensorrt_llm.so.9(+0x4512f) [0x7fd1a11c612f]
2       0x7fd1a1256ac8 tensorrt_llm::plugins::GemmPlugin::GemmPlugin(void const*, unsigned long, std::shared_ptr<tensorrt_llm::plugins::CublasLtGemmPluginProfiler> const&) + 440
3       0x7fd1a1256bdf tensorrt_llm::plugins::GemmPluginCreator::deserializePlugin(char const*, void const*, unsigned long) + 191
4       0x7fd1e8b118a6 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10d68a6) [0x7fd1e8b118a6]
5       0x7fd1e8b0966e /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10ce66e) [0x7fd1e8b0966e]
6       0x7fd1e8aa4217 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x1069217) [0x7fd1e8aa4217]
7       0x7fd1e8aa219e /usr/local/tensorrt/lib/libnvinfer.so.9(+0x106719e) [0x7fd1e8aa219e]
8       0x7fd1e8ab9c2b /usr/local/tensorrt/lib/libnvinfer.so.9(+0x107ec2b) [0x7fd1e8ab9c2b]
9       0x7fd1e8abce32 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x1081e32) [0x7fd1e8abce32]
10      0x7fd1e8abd20c /usr/local/tensorrt/lib/libnvinfer.so.9(+0x108220c) [0x7fd1e8abd20c]
11      0x7fd1e8af09b1 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10b59b1) [0x7fd1e8af09b1]
12      0x7fd1e8af1777 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10b6777) [0x7fd1e8af1777]
13      0x7fd1f7579022 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x127022) [0x7fd1f7579022]
14      0x7fd1f74d29cf /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x809cf) [0x7fd1f74d29cf]
15      0x7fd1f74c299f /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x7099f) [0x7fd1f74c299f]
16      0x7fd1f74b9222 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x67222) [0x7fd1f74b9222]
17      0x7fd1f749acbc /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x48cbc) [0x7fd1f749acbc]
18      0x7fd1f749bd62 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x49d62) [0x7fd1f749bd62]
19      0x7fd1f748bbb5 TRITONBACKEND_ModelInstanceInitialize + 101
20      0x7fd2b1589226 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x1a7226) [0x7fd2b1589226]
21      0x7fd2b158a466 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x1a8466) [0x7fd2b158a466]
22      0x7fd2b156d165 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x18b165) [0x7fd2b156d165]
23      0x7fd2b156d7a6 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x18b7a6) [0x7fd2b156d7a6]
24      0x7fd2b1579a1d /opt/tritonserver/bin/../lib/libtritonserver.so(+0x197a1d) [0x7fd2b1579a1d]
25      0x7fd2b0be4ee8 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fd2b0be4ee8]
26      0x7fd2b1563feb /opt/tritonserver/bin/../lib/libtritonserver.so(+0x181feb) [0x7fd2b1563feb]
27      0x7fd2b1573dc5 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x191dc5) [0x7fd2b1573dc5]
28      0x7fd2b1578d36 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x196d36) [0x7fd2b1578d36]
29      0x7fd2b1669330 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x287330) [0x7fd2b1669330]
30      0x7fd2b166ca23 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x28aa23) [0x7fd2b166ca23]
31      0x7fd2b17c0d82 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x3ded82) [0x7fd2b17c0d82]
32      0x7fd2b0e4f253 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7fd2b0e4f253]
33      0x7fd2b0bdfac3 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7fd2b0bdfac3]
34      0x7fd2b0c70814 clone + 68
[TensorRT-LLM][ERROR] tensorrt_llm::common::TllmException: [TensorRT-LLM][ERROR] Assertion failed: d == a + length (/app/tensorrt_llm/cpp/tensorrt_llm/plugins/gemmPlugin/gemmPlugin.cpp:156)
1       0x7fd1a11c612f /opt/tritonserver/backends/tensorrtllm/libnvinfer_plugin_tensorrt_llm.so.9(+0x4512f) [0x7fd1a11c612f]
2       0x7fd1a1256ac8 tensorrt_llm::plugins::GemmPlugin::GemmPlugin(void const*, unsigned long, std::shared_ptr<tensorrt_llm::plugins::CublasLtGemmPluginProfiler> const&) + 440
3       0x7fd1a1256bdf tensorrt_llm::plugins::GemmPluginCreator::deserializePlugin(char const*, void const*, unsigned long) + 191
4       0x7fd1e8b118a6 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10d68a6) [0x7fd1e8b118a6]
5       0x7fd1e8b0966e /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10ce66e) [0x7fd1e8b0966e]
6       0x7fd1e8aa4217 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x1069217) [0x7fd1e8aa4217]
7       0x7fd1e8aa219e /usr/local/tensorrt/lib/libnvinfer.so.9(+0x106719e) [0x7fd1e8aa219e]
8       0x7fd1e8ab9c2b /usr/local/tensorrt/lib/libnvinfer.so.9(+0x107ec2b) [0x7fd1e8ab9c2b]
9       0x7fd1e8abce32 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x1081e32) [0x7fd1e8abce32]
10      0x7fd1e8abd20c /usr/local/tensorrt/lib/libnvinfer.so.9(+0x108220c) [0x7fd1e8abd20c]
11      0x7fd1e8af09b1 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10b59b1) [0x7fd1e8af09b1]
12      0x7fd1e8af1777 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10b6777) [0x7fd1e8af1777]
13      0x7fd1f7579022 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x127022) [0x7fd1f7579022]
14      0x7fd1f74d29cf /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x809cf) [0x7fd1f74d29cf]
15      0x7fd1f74c299f /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x7099f) [0x7fd1f74c299f]
16      0x7fd1f74b9222 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x67222) [0x7fd1f74b9222]
17      0x7fd1f749acbc /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x48cbc) [0x7fd1f749acbc]
18      0x7fd1f749bd62 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x49d62) [0x7fd1f749bd62]
19      0x7fd1f748bbb5 TRITONBACKEND_ModelInstanceInitialize + 101
20      0x7fd2b1589226 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x1a7226) [0x7fd2b1589226]
21      0x7fd2b158a466 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x1a8466) [0x7fd2b158a466]
22      0x7fd2b156d165 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x18b165) [0x7fd2b156d165]
23      0x7fd2b156d7a6 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x18b7a6) [0x7fd2b156d7a6]
24      0x7fd2b1579a1d /opt/tritonserver/bin/../lib/libtritonserver.so(+0x197a1d) [0x7fd2b1579a1d]
25      0x7fd2b0be4ee8 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fd2b0be4ee8]
26      0x7fd2b1563feb /opt/tritonserver/bin/../lib/libtritonserver.so(+0x181feb) [0x7fd2b1563feb]
27      0x7fd2b1573dc5 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x191dc5) [0x7fd2b1573dc5]
28      0x7fd2b1578d36 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x196d36) [0x7fd2b1578d36]
29      0x7fd2b1669330 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x287330) [0x7fd2b1669330]
30      0x7fd2b166ca23 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x28aa23) [0x7fd2b166ca23]
31      0x7fd2b17c0d82 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x3ded82) [0x7fd2b17c0d82]
32      0x7fd2b0e4f253 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7fd2b0e4f253]
33      0x7fd2b0bdfac3 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7fd2b0bdfac3]
34      0x7fd2b0c70814 clone + 68
[TensorRT-LLM][ERROR] tensorrt_llm::common::TllmException: [TensorRT-LLM][ERROR] Assertion failed: d == a + length (/app/tensorrt_llm/cpp/tensorrt_llm/plugins/gptAttentionCommon/gptAttentionCommon.cpp:418)
1       0x7fd1a11c612f /opt/tritonserver/backends/tensorrtllm/libnvinfer_plugin_tensorrt_llm.so.9(+0x4512f) [0x7fd1a11c612f]
2       0x7fd1a1232846 tensorrt_llm::plugins::GPTAttentionPluginCommon::GPTAttentionPluginCommon(void const*, unsigned long) + 870
3       0x7fd1a12498b3 tensorrt_llm::plugins::GPTAttentionPlugin::GPTAttentionPlugin(void const*, unsigned long) + 19
4       0x7fd1a1249932 tensorrt_llm::plugins::GPTAttentionPluginCreator::deserializePlugin(char const*, void const*, unsigned long) + 50
5       0x7fd1e8b118a6 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10d68a6) [0x7fd1e8b118a6]
6       0x7fd1e8b0966e /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10ce66e) [0x7fd1e8b0966e]
7       0x7fd1e8aa4217 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x1069217) [0x7fd1e8aa4217]
8       0x7fd1e8aa219e /usr/local/tensorrt/lib/libnvinfer.so.9(+0x106719e) [0x7fd1e8aa219e]
9       0x7fd1e8ab9c2b /usr/local/tensorrt/lib/libnvinfer.so.9(+0x107ec2b) [0x7fd1e8ab9c2b]
10      0x7fd1e8abce32 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x1081e32) [0x7fd1e8abce32]
11      0x7fd1e8abd20c /usr/local/tensorrt/lib/libnvinfer.so.9(+0x108220c) [0x7fd1e8abd20c]
12      0x7fd1e8af09b1 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10b59b1) [0x7fd1e8af09b1]
13      0x7fd1e8af1777 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10b6777) [0x7fd1e8af1777]
14      0x7fd1f7579022 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x127022) [0x7fd1f7579022]
15      0x7fd1f74d29cf /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x809cf) [0x7fd1f74d29cf]
16      0x7fd1f74c299f /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x7099f) [0x7fd1f74c299f]
17      0x7fd1f74b9222 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x67222) [0x7fd1f74b9222]
18      0x7fd1f749acbc /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x48cbc) [0x7fd1f749acbc]
19      0x7fd1f749bd62 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x49d62) [0x7fd1f749bd62]
20      0x7fd1f748bbb5 TRITONBACKEND_ModelInstanceInitialize + 101
21      0x7fd2b1589226 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x1a7226) [0x7fd2b1589226]
22      0x7fd2b158a466 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x1a8466) [0x7fd2b158a466]
23      0x7fd2b156d165 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x18b165) [0x7fd2b156d165]
24      0x7fd2b156d7a6 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x18b7a6) [0x7fd2b156d7a6]
25      0x7fd2b1579a1d /opt/tritonserver/bin/../lib/libtritonserver.so(+0x197a1d) [0x7fd2b1579a1d]
26      0x7fd2b0be4ee8 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fd2b0be4ee8]
27      0x7fd2b1563feb /opt/tritonserver/bin/../lib/libtritonserver.so(+0x181feb) [0x7fd2b1563feb]
28      0x7fd2b1573dc5 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x191dc5) [0x7fd2b1573dc5]
29      0x7fd2b1578d36 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x196d36) [0x7fd2b1578d36]
30      0x7fd2b1669330 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x287330) [0x7fd2b1669330]
31      0x7fd2b166ca23 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x28aa23) [0x7fd2b166ca23]
32      0x7fd2b17c0d82 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x3ded82) [0x7fd2b17c0d82]
33      0x7fd2b0e4f253 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7fd2b0e4f253]
34      0x7fd2b0bdfac3 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7fd2b0bdfac3]
35      0x7fd2b0c70814 clone + 68
[TensorRT-LLM][ERROR] tensorrt_llm::common::TllmException: [TensorRT-LLM][ERROR] Assertion failed: d == a + length (/app/tensorrt_llm/cpp/tensorrt_llm/plugins/gemmPlugin/gemmPlugin.cpp:156)
1       0x7fd1a11c612f /opt/tritonserver/backends/tensorrtllm/libnvinfer_plugin_tensorrt_llm.so.9(+0x4512f) [0x7fd1a11c612f]
2       0x7fd1a1256ac8 tensorrt_llm::plugins::GemmPlugin::GemmPlugin(void const*, unsigned long, std::shared_ptr<tensorrt_llm::plugins::CublasLtGemmPluginProfiler> const&) + 440
3       0x7fd1a1256bdf tensorrt_llm::plugins::GemmPluginCreator::deserializePlugin(char const*, void const*, unsigned long) + 191
4       0x7fd1e8b118a6 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10d68a6) [0x7fd1e8b118a6]
5       0x7fd1e8b0966e /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10ce66e) [0x7fd1e8b0966e]
6       0x7fd1e8aa4217 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x1069217) [0x7fd1e8aa4217]
7       0x7fd1e8aa219e /usr/local/tensorrt/lib/libnvinfer.so.9(+0x106719e) [0x7fd1e8aa219e]
8       0x7fd1e8ab9c2b /usr/local/tensorrt/lib/libnvinfer.so.9(+0x107ec2b) [0x7fd1e8ab9c2b]
9       0x7fd1e8abce32 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x1081e32) [0x7fd1e8abce32]
10      0x7fd1e8abd20c /usr/local/tensorrt/lib/libnvinfer.so.9(+0x108220c) [0x7fd1e8abd20c]
11      0x7fd1e8af09b1 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10b59b1) [0x7fd1e8af09b1]
12      0x7fd1e8af1777 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10b6777) [0x7fd1e8af1777]
13      0x7fd1f7579022 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x127022) [0x7fd1f7579022]
14      0x7fd1f74d29cf /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x809cf) [0x7fd1f74d29cf]
15      0x7fd1f74c299f /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x7099f) [0x7fd1f74c299f]
16      0x7fd1f74b9222 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x67222) [0x7fd1f74b9222]
17      0x7fd1f749acbc /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x48cbc) [0x7fd1f749acbc]
18      0x7fd1f749bd62 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x49d62) [0x7fd1f749bd62]
19      0x7fd1f748bbb5 TRITONBACKEND_ModelInstanceInitialize + 101
20      0x7fd2b1589226 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x1a7226) [0x7fd2b1589226]
21      0x7fd2b158a466 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x1a8466) [0x7fd2b158a466]
22      0x7fd2b156d165 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x18b165) [0x7fd2b156d165]
23      0x7fd2b156d7a6 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x18b7a6) [0x7fd2b156d7a6]
24      0x7fd2b1579a1d /opt/tritonserver/bin/../lib/libtritonserver.so(+0x197a1d) [0x7fd2b1579a1d]
25      0x7fd2b0be4ee8 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fd2b0be4ee8]
26      0x7fd2b1563feb /opt/tritonserver/bin/../lib/libtritonserver.so(+0x181feb) [0x7fd2b1563feb]
27      0x7fd2b1573dc5 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x191dc5) [0x7fd2b1573dc5]
28      0x7fd2b1578d36 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x196d36) [0x7fd2b1578d36]
29      0x7fd2b1669330 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x287330) [0x7fd2b1669330]
30      0x7fd2b166ca23 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x28aa23) [0x7fd2b166ca23]
31      0x7fd2b17c0d82 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x3ded82) [0x7fd2b17c0d82]
32      0x7fd2b0e4f253 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7fd2b0e4f253]
33      0x7fd2b0bdfac3 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7fd2b0bdfac3]
34      0x7fd2b0c70814 clone + 68
[TensorRT-LLM][ERROR] tensorrt_llm::common::TllmException: [TensorRT-LLM][ERROR] Assertion failed: d == a + length (/app/tensorrt_llm/cpp/tensorrt_llm/plugins/gemmPlugin/gemmPlugin.cpp:156)
1       0x7fd1a11c612f /opt/tritonserver/backends/tensorrtllm/libnvinfer_plugin_tensorrt_llm.so.9(+0x4512f) [0x7fd1a11c612f]
2       0x7fd1a1256ac8 tensorrt_llm::plugins::GemmPlugin::GemmPlugin(void const*, unsigned long, std::shared_ptr<tensorrt_llm::plugins::CublasLtGemmPluginProfiler> const&) + 440
3       0x7fd1a1256bdf tensorrt_llm::plugins::GemmPluginCreator::deserializePlugin(char const*, void const*, unsigned long) + 191
4       0x7fd1e8b118a6 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10d68a6) [0x7fd1e8b118a6]
5       0x7fd1e8b0966e /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10ce66e) [0x7fd1e8b0966e]
6       0x7fd1e8aa4217 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x1069217) [0x7fd1e8aa4217]
7       0x7fd1e8aa219e /usr/local/tensorrt/lib/libnvinfer.so.9(+0x106719e) [0x7fd1e8aa219e]
8       0x7fd1e8ab9c2b /usr/local/tensorrt/lib/libnvinfer.so.9(+0x107ec2b) [0x7fd1e8ab9c2b]
9       0x7fd1e8abce32 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x1081e32) [0x7fd1e8abce32]
10      0x7fd1e8abd20c /usr/local/tensorrt/lib/libnvinfer.so.9(+0x108220c) [0x7fd1e8abd20c]
11      0x7fd1e8af09b1 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10b59b1) [0x7fd1e8af09b1]
12      0x7fd1e8af1777 /usr/local/tensorrt/lib/libnvinfer.so.9(+0x10b6777) [0x7fd1e8af1777]
13      0x7fd1f7579022 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x127022) [0x7fd1f7579022]
14      0x7fd1f74d29cf /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x809cf) [0x7fd1f74d29cf]
15      0x7fd1f74c299f /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x7099f) [0x7fd1f74c299f]
16      0x7fd1f74b9222 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x67222) [0x7fd1f74b9222]
17      0x7fd1f749acbc /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x48cbc) [0x7fd1f749acbc]
18      0x7fd1f749bd62 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x49d62) [0x7fd1f749bd62]
19      0x7fd1f748bbb5 TRITONBACKEND_ModelInstanceInitialize + 101
20      0x7fd2b1589226 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x1a7226) [0x7fd2b1589226]
21      0x7fd2b158a466 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x1a8466) [0x7fd2b158a466]
22      0x7fd2b156d165 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x18b165) [0x7fd2b156d165]
23      0x7fd2b156d7a6 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x18b7a6) [0x7fd2b156d7a6]
24      0x7fd2b1579a1d /opt/tritonserver/bin/../lib/libtritonserver.so(+0x197a1d) [0x7fd2b1579a1d]
25      0x7fd2b0be4ee8 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x99ee8) [0x7fd2b0be4ee8]
26      0x7fd2b1563feb /opt/tritonserver/bin/../lib/libtritonserver.so(+0x181feb) [0x7fd2b1563feb]
27      0x7fd2b1573dc5 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x191dc5) [0x7fd2b1573dc5]
28      0x7fd2b1578d36 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x196d36) [0x7fd2b1578d36]
29      0x7fd2b1669330 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x287330) [0x7fd2b1669330]
30      0x7fd2b166ca23 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x28aa23) [0x7fd2b166ca23]
31      0x7fd2b17c0d82 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x3ded82) [0x7fd2b17c0d82]
32      0x7fd2b0e4f253 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7fd2b0e4f253]
33      0x7fd2b0bdfac3 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7fd2b0bdfac3]
34      0x7fd2b0c70814 clone + 68
...

Triton Information What version of Triton are you using? Triton: 2.41 tensorrtllm_backend: 0.8.0

Are you using the Triton container or did you build it yourself? i used docker: nvcr.io/nvidia/tritonserver:23.12-trtllm-python-py3 - running on ubuntu 22.04

To Reproduce all the steps to reproduce are described here: https://github.com/mtezgider/triton-tensorrt-llm-model-preparation-and-deployment than i started the server: tritonserver --model-repository=/tensorrt/triton-repos/trtibf-Trendyol-LLM-7b-chat-v1.0 --model-control-mode=explicit --load-model=preprocessing --load-model=postprocessing --load-model=tensorrt_llm --load-model=tensorrt_llm_bls --load-model=ensemble --log-verbose=2 --log-info=1 --log-warning=1 --log-error=1

Describe the models (framework, inputs, outputs), ideally include the model configuration file (if using an ensemble include the model configuration file for that as well). see here: https://github.com/mtezgider/triton-tensorrt-llm-model-preparation-and-deployment

Expected behavior A clear and concise description of what you expected to happen. no error running the model

the entire logs: logs.txt

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Comments: 15 (8 by maintainers)

Most upvoted comments

@krishung5 thanks for quick reply. i created an issue for the TRT-LLM team: https://github.com/NVIDIA/TensorRT-LLM/issues/1487 - they said its only a warning and it should work still for 1 or 4 gpus?