tensorrtllm_backend: Input tensor 'host_sink_token_length' not found when launch llama2-7b.
I installed tensorrtllm_backend in the follow way:
docker pull nvcr.io/nvidia/tritonserver:23.12-trtllm-python-py3docker run -v /data2/share/:/data/ -v /mnt/sdb/benchmark/xiangrui:/root -it -d --cap-add=SYS_PTRACE --cap-add=SYS_ADMIN --security-opt seccomp=unconfined --gpus=all --shm-size=16g --privileged --ulimit memlock=-1 --name=develop nvcr.io/nvidia/tritonserver:23.12-trtllm-python-py3 bash- git clone git@github.com:triton-inference-server/tensorrtllm_backend.git --recursive
- apt-get update && apt-get -y install git git-lfs pip install cmake cd tensorrtllm_backend git submodule update --init --recursive git lfs install git lfs pull
- cd tensorrt_llm python3 ./scripts/build_wheel.py --trt_root /usr/local/tensorrt pip installtensorrt_llm-0.7.1-cp310-cp310-linux_x86_64.whl
- now, I installed tensorrt_llm
>>> import tensorrt
>>> import tensorrt_llm
>>> tensorrt.__version__
'9.2.0.post12.dev5'
>>> tensorrt_llm.__version__
'0.7.1'
- follow this doc https://github.com/triton-inference-server/tensorrtllm_backend/blob/main/docs/llama.md
- until
Launch server, I meet this issue https://github.com/NVIDIA/TensorRT-LLM/issues/656 ,Assertion failed: d == a + length (/app/tensorrt_llm/cpp/tensorrt_llm/plugins/gptAttentionCommon/gptAttentionCommon.cpp:326)then, I do thiscp tensorrt_llm/build/lib/tensorrt_llm/libs/* /opt/tritonserver/backends/tensorrtllm/, solved problem. - After that, I launched server successfully. use this command
python3 scripts/launch_triton_server.py --world_size 1 --model_repo=triton_model_repo/messages shown blow:
TensorRT-LLM][INFO] Allocate 1342177280 bytes for k/v cache.
[TensorRT-LLM][INFO] Using 2560 total tokens in paged KV cache, and 20 blocks per sequence
I0107 12:18:03.563733 4223 model_lifecycle.cc:818] successfully loaded 'tensorrt_llm'
I0107 12:18:03.565248 4223 model_lifecycle.cc:461] loading: ensemble:1
I0107 12:18:03.565653 4223 model_lifecycle.cc:818] successfully loaded 'ensemble'
I0107 12:18:03.565748 4223 server.cc:606]
+------------------+------+
| Repository Agent | Path |
+------------------+------+
+------------------+------+
I0107 12:18:03.565813 4223 server.cc:633]
+-------------+-------------------------------------------------------------+-------------------------------------------------------------+
| Backend | Path | Config |
+-------------+-------------------------------------------------------------+-------------------------------------------------------------+
| python | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"false","backend-directo |
| | | ry":"/opt/tritonserver/backends","min-compute-capability":" |
| | | 6.000000","shm-region-prefix-name":"prefix0_","default-max- |
| | | batch-size":"4"}} |
| tensorrtllm | /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtll | {"cmdline":{"auto-complete-config":"false","backend-directo |
| | m.so | ry":"/opt/tritonserver/backends","min-compute-capability":" |
| | | 6.000000","default-max-batch-size":"4"}} |
| | | |
+-------------+-------------------------------------------------------------+-------------------------------------------------------------+
I0107 12:18:03.565860 4223 server.cc:676]
+------------------+---------+--------+
| Model | Version | Status |
+------------------+---------+--------+
| ensemble | 1 | READY |
| postprocessing | 1 | READY |
| preprocessing | 1 | READY |
| tensorrt_llm | 1 | READY |
| tensorrt_llm_bls | 1 | READY |
+------------------+---------+--------+
I0107 12:18:03.712204 4223 metrics.cc:817] Collecting metrics for GPU 0: NVIDIA RTX A6000
I0107 12:18:03.726368 4223 metrics.cc:710] Collecting CPU metrics
I0107 12:18:03.726577 4223 tritonserver.cc:2483]
+----------------------------------+------------------------------------------------------------------------------------------------------+
| Option | Value |
+----------------------------------+------------------------------------------------------------------------------------------------------+
| server_id | triton |
| server_version | 2.41.0 |
| server_extensions | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_c |
| | onfiguration system_shared_memory cuda_shared_memory binary_tensor_data parameters statistics trace |
| | logging |
| model_repository_path[0] | triton_model_repo/ |
| model_control_mode | MODE_NONE |
| strict_model_config | 1 |
| rate_limit | OFF |
| pinned_memory_pool_byte_size | 268435456 |
| cuda_memory_pool_byte_size{0} | 67108864 |
| min_supported_compute_capability | 6.0 |
| strict_readiness | 1 |
| exit_timeout | 30 |
| cache_enabled | 0 |
+----------------------------------+------------------------------------------------------------------------------------------------------+
I0107 12:18:03.744520 4223 grpc_server.cc:2495] Started GRPCInferenceService at 0.0.0.0:8001
I0107 12:18:03.744823 4223 http_server.cc:4619] Started HTTPService at 0.0.0.0:8000
I0107 12:18:03.804746 4223 http_server.cc:282] Started Metrics Service at 0.0.0.0:8002
- Test, when I use command
curl -X POST localhost:8000/v2/models/ensemble/generate -d '{"text_input": "What is machine learning?", "max_tokens": 20, "bad_words": "", "stop_words": "", "pad_id": 2, "end_id": 2}'Error happen, they are:
root@d0b11d0dea8b:/tensorrtllm_backend# curl -X POST localhost:8000/v2/models/ensemble/generate -d '{"text_input": "What is machine learning?", "max_tokens": 20, "bad_words": "", "stop_words": "", "pad_id": 2, "end_id": 2}'
[TensorRT-LLM][ERROR] Encountered an error in forward function: Input tensor 'host_sink_token_length' not found; expected shape: (1) (/app/tensorrt_llm/cpp/tensorrt_llm/runtime/tllmRuntime.cpp:124)
1 0x7f830f5793e3 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x1273e3) [0x7f830f5793e3]
2 0x7f830f4cbeb1 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x79eb1) [0x7f830f4cbeb1]
3 0x7f830f4ccfa6 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x7afa6) [0x7f830f4ccfa6]
4 0x7f830f4d0f0d /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x7ef0d) [0x7f830f4d0f0d]
5 0x7f830f4bba28 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x69a28) [0x7f830f4bba28]
6 0x7f830f4bffb5 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x6dfb5) [0x7f830f4bffb5]
7 0x7f838604f253 /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f838604f253]
8 0x7f8385ddfac3 /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f8385ddfac3]
9 0x7f8385e71660 /lib/x86_64-linux-gnu/libc.so.6(+0x126660) [0x7f8385e71660]
[TensorRT-LLM][ERROR] Encountered error for requestId 1804289384: Encountered an error in forward function: Input tensor 'host_sink_token_length' not found; expected shape: (1) (/app/tensorrt_llm/cpp/tensorrt_llm/runtime/tllmRuntime.cpp:124)
1 0x7f830f5793e3 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x1273e3) [0x7f830f5793e3]
2 0x7f830f4cbeb1 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x79eb1) [0x7f830f4cbeb1]
3 0x7f830f4ccfa6 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x7afa6) [0x7f830f4ccfa6]
4 0x7f830f4d0f0d /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x7ef0d) [0x7f830f4d0f0d]
5 0x7f830f4bba28 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x69a28) [0x7f830f4bba28]
6 0x7f830f4bffb5 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x6dfb5) [0x7f830f4bffb5]
7 0x7f838604f253 /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f838604f253]
8 0x7f8385ddfac3 /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f8385ddfac3]
9 0x7f8385e71660 /lib/x86_64-linux-gnu/libc.so.6(+0x126660) [0x7f8385e71660]
[TensorRT-LLM][WARNING] Step function failed, continuing.
{"error":"in ensemble 'ensemble', Encountered error for requestId 1804289384: Encountered an error in forward function: Input tensor 'host_sink_token_length' not found; expected shape: (1) (/app/tensorrt_llm/cpp/tensorrt_llm/runtime/tllmRuntime.cpp:124)\n1 0x7f830f5793e3 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x1273e3) [0x7f830f5793e3]\n2 0x7f830f4cbeb1 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x79eb1) [0x7f830f4cbeb1]\n3 0x7f830f4ccfa6 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x7afa6) [0x7f830f4ccfa6]\n4 0x7f830f4d0f0d /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x7ef0d) [0x7f830f4d0f0d]\n5 0x7f830f4bba28 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x69a28) [0x7f830f4bba28]\n6 0x7f830f4bffb5 /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so(+0x6dfb5) [0x7f830f4bffb5]\n7 0x7f838604f253 /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f838604f253]\n8 0x7f8385ddfac3 /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f8385ddfac3]\n9 0x7f8385e71660 /lib/x86_64-linux-gnu/libc.so.6(+0x126660) [0x7f8385e71660]"}root@d0b11d0dea8b:/tensorrtllm_backend#
root@d0b11d0dea8b:/tensorrtllm_backend# tmux a
Can anyone help me? plsssss.
About this issue
- Original URL
- State: open
- Created 3 years ago
- Comments: 18 (1 by maintainers)
This may be a bug of grpcio, both exit method and del method invoke
close, which cause double free. Delete del method of class InferenceServerClient can solve.