tensorrtllm_backend: Input tensor 'host_sink_token_length' not found when launch llama2-7b.

I installed tensorrtllm_backend in the follow way:

  1. docker pull nvcr.io/nvidia/tritonserver:23.12-trtllm-python-py3
  2. docker run -v /data2/share/:/data/ -v /mnt/sdb/benchmark/xiangrui:/root -it -d --cap-add=SYS_PTRACE --cap-add=SYS_ADMIN --security-opt seccomp=unconfined --gpus=all --shm-size=16g --privileged --ulimit memlock=-1 --name=develop nvcr.io/nvidia/tritonserver:23.12-trtllm-python-py3 bash
  3. git clone git@github.com:triton-inference-server/tensorrtllm_backend.git --recursive
  4. apt-get update && apt-get -y install git git-lfs pip install cmake cd tensorrtllm_backend git submodule update --init --recursive git lfs install git lfs pull
  5. cd tensorrt_llm python3 ./scripts/build_wheel.py --trt_root /usr/local/tensorrt pip installtensorrt_llm-0.7.1-cp310-cp310-linux_x86_64.whl
  6. now, I installed tensorrt_llm
>>> import tensorrt
>>> import tensorrt_llm
>>> tensorrt.__version__
'9.2.0.post12.dev5'
>>> tensorrt_llm.__version__
'0.7.1'
  1. follow this doc https://github.com/triton-inference-server/tensorrtllm_backend/blob/main/docs/llama.md
  2. until Launch server, I meet this issue https://github.com/NVIDIA/TensorRT-LLM/issues/656 ,Assertion failed: d == a + length (/app/tensorrt_llm/cpp/tensorrt_llm/plugins/gptAttentionCommon/gptAttentionCommon.cpp:326) then, I do this cp tensorrt_llm/build/lib/tensorrt_llm/libs/* /opt/tritonserver/backends/tensorrtllm/, solved problem.
  3. After that, I launched server successfully. use this command python3 scripts/launch_triton_server.py --world_size 1 --model_repo=triton_model_repo/ messages shown blow:
TensorRT-LLM][INFO] Allocate 1342177280 bytes for k/v cache. 
[TensorRT-LLM][INFO] Using 2560 total tokens in paged KV cache, and 20 blocks per sequence
I0107 12:18:03.563733 4223 model_lifecycle.cc:818] successfully loaded 'tensorrt_llm'
I0107 12:18:03.565248 4223 model_lifecycle.cc:461] loading: ensemble:1
I0107 12:18:03.565653 4223 model_lifecycle.cc:818] successfully loaded 'ensemble'
I0107 12:18:03.565748 4223 server.cc:606] 
+------------------+------+
| Repository Agent | Path |
+------------------+------+
+------------------+------+

I0107 12:18:03.565813 4223 server.cc:633] 
+-------------+-------------------------------------------------------------+-------------------------------------------------------------+
| Backend     | Path                                                        | Config                                                      |
+-------------+-------------------------------------------------------------+-------------------------------------------------------------+
| python      | /opt/tritonserver/backends/python/libtriton_python.so       | {"cmdline":{"auto-complete-config":"false","backend-directo |
|             |                                                             | ry":"/opt/tritonserver/backends","min-compute-capability":" |
|             |                                                             | 6.000000","shm-region-prefix-name":"prefix0_","default-max- |
|             |                                                             | batch-size":"4"}}                                           |
| tensorrtllm | /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtll | {"cmdline":{"auto-complete-config":"false","backend-directo |
|             | m.so                                                        | ry":"/opt/tritonserver/backends","min-compute-capability":" |
|             |                                                             | 6.000000","default-max-batch-size":"4"}}                    |
|             |                                                             |                                                             |
+-------------+-------------------------------------------------------------+-------------------------------------------------------------+

I0107 12:18:03.565860 4223 server.cc:676] 
+------------------+---------+--------+
| Model            | Version | Status |
+------------------+---------+--------+
| ensemble         | 1       | READY  |
| postprocessing   | 1       | READY  |
| preprocessing    | 1       | READY  |
| tensorrt_llm     | 1       | READY  |
| tensorrt_llm_bls | 1       | READY  |
+------------------+---------+--------+

I0107 12:18:03.712204 4223 metrics.cc:817] Collecting metrics for GPU 0: NVIDIA RTX A6000
I0107 12:18:03.726368 4223 metrics.cc:710] Collecting CPU metrics
I0107 12:18:03.726577 4223 tritonserver.cc:2483] 
+----------------------------------+------------------------------------------------------------------------------------------------------+
| Option                           | Value                                                                                                |
+----------------------------------+------------------------------------------------------------------------------------------------------+
| server_id                        | triton                                                                                               |
| server_version                   | 2.41.0                                                                                               |
| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_c |
|                                  | onfiguration system_shared_memory cuda_shared_memory binary_tensor_data parameters statistics trace  |
|                                  | logging                                                                                              |
| model_repository_path[0]         | triton_model_repo/                                                                                   |
| model_control_mode               | MODE_NONE                                                                                            |
| strict_model_config              | 1                                                                                                    |
| rate_limit                       | OFF                                                                                                  |
| pinned_memory_pool_byte_size     | 268435456                                                                                            |
| cuda_memory_pool_byte_size{0}    | 67108864                                                                                             |
| min_supported_compute_capability | 6.0                                                                                                  |
| strict_readiness                 | 1                                                                                                    |
| exit_timeout                     | 30                                                                                                   |
| cache_enabled                    | 0                                                                                                    |
+----------------------------------+------------------------------------------------------------------------------------------------------+

I0107 12:18:03.744520 4223 grpc_server.cc:2495] Started GRPCInferenceService at 0.0.0.0:8001
I0107 12:18:03.744823 4223 http_server.cc:4619] Started HTTPService at 0.0.0.0:8000
I0107 12:18:03.804746 4223 http_server.cc:282] Started Metrics Service at 0.0.0.0:8002
  1. Test, when I use command curl -X POST localhost:8000/v2/models/ensemble/generate -d '{"text_input": "What is machine learning?", "max_tokens": 20, "bad_words": "", "stop_words": "", "pad_id": 2, "end_id": 2}' Error happen, they are:
root@d0b11d0dea8b:/tensorrtllm_backend# curl -X POST localhost:8000/v2/models/ensemble/generate -d '{"text_input": "What is machine learning?", "max_tokens": 20, "bad_words": "", "stop_words": "", "pad_id": 2, "end_id": 2}'
[TensorRT-LLM][ERROR] Encountered an error in forward function: Input tensor 'host_sink_token_length' not found; expected shape: (1) (/app/tensorrt_llm/cpp/tensorrt_llm/runtime/tllmRuntime.cpp:124)
1       0x7f830f5793e3 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x1273e3) [0x7f830f5793e3]
2       0x7f830f4cbeb1 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x79eb1) [0x7f830f4cbeb1]
3       0x7f830f4ccfa6 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x7afa6) [0x7f830f4ccfa6]
4       0x7f830f4d0f0d /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x7ef0d) [0x7f830f4d0f0d]
5       0x7f830f4bba28 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x69a28) [0x7f830f4bba28]
6       0x7f830f4bffb5 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x6dfb5) [0x7f830f4bffb5]
7       0x7f838604f253 /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f838604f253]
8       0x7f8385ddfac3 /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f8385ddfac3]
9       0x7f8385e71660 /lib/x86_64-linux-gnu/libc.so.6(+0x126660) [0x7f8385e71660]
[TensorRT-LLM][ERROR] Encountered error for requestId 1804289384: Encountered an error in forward function: Input tensor 'host_sink_token_length' not found; expected shape: (1) (/app/tensorrt_llm/cpp/tensorrt_llm/runtime/tllmRuntime.cpp:124)
1       0x7f830f5793e3 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x1273e3) [0x7f830f5793e3]
2       0x7f830f4cbeb1 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x79eb1) [0x7f830f4cbeb1]
3       0x7f830f4ccfa6 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x7afa6) [0x7f830f4ccfa6]
4       0x7f830f4d0f0d /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x7ef0d) [0x7f830f4d0f0d]
5       0x7f830f4bba28 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x69a28) [0x7f830f4bba28]
6       0x7f830f4bffb5 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x6dfb5) [0x7f830f4bffb5]
7       0x7f838604f253 /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f838604f253]
8       0x7f8385ddfac3 /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f8385ddfac3]
9       0x7f8385e71660 /lib/x86_64-linux-gnu/libc.so.6(+0x126660) [0x7f8385e71660]
[TensorRT-LLM][WARNING] Step function failed, continuing.
{"error":"in ensemble 'ensemble', Encountered error for requestId 1804289384: Encountered an error in forward function: Input tensor 'host_sink_token_length' not found; expected shape: (1) (/app/tensorrt_llm/cpp/tensorrt_llm/runtime/tllmRuntime.cpp:124)\n1       0x7f830f5793e3 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x1273e3) [0x7f830f5793e3]\n2       0x7f830f4cbeb1 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x79eb1) [0x7f830f4cbeb1]\n3       0x7f830f4ccfa6 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x7afa6) [0x7f830f4ccfa6]\n4       0x7f830f4d0f0d /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x7ef0d) [0x7f830f4d0f0d]\n5       0x7f830f4bba28 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x69a28) [0x7f830f4bba28]\n6       0x7f830f4bffb5 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x6dfb5) [0x7f830f4bffb5]\n7       0x7f838604f253 /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f838604f253]\n8       0x7f8385ddfac3 /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f8385ddfac3]\n9       0x7f8385e71660 /lib/x86_64-linux-gnu/libc.so.6(+0x126660) [0x7f8385e71660]"}root@d0b11d0dea8b:/tensorrtllm_backend# 
root@d0b11d0dea8b:/tensorrtllm_backend# tmux a

Can anyone help me? plsssss.

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Comments: 18 (1 by maintainers)

Most upvoted comments

This may be a bug of grpcio, both exit method and del method invoke close, which cause double free. Delete del method of class InferenceServerClient can solve.