tensorrtllm_backend: ERROR: Failed to create instance: unexpected error when creating modelInstanceState: maxTokensInPagedKvCache must be large enough to process at least 1 sequence to completion (i.e. must be larger than beam_width * tokensPerBlock * maxBlocksPerSeq)
I referred this official blog to run the llama2 7b chat model
python3 tools/fill_template.py --in_place \
all_models/inflight_batcher_llm/tensorrt_llm/config.pbtxt \
decoupled_mode:true,engine_dir:/all_models/inflight_batcher_llm/tensorrt_llm/1,\
max_tokens_in_paged_kv_cache:,batch_scheduler_policy:guaranteed_completion,kv_cache_free_gpu_mem_fraction:0.2,\
max_num_sequences:4
python tools/fill_template.py --in_place \
all_models/inflight_batcher_llm/preprocessing/config.pbtxt \
tokenizer_type:llama,tokenizer_dir:meta-llama/Llama-2-7b-chat-hf
python tools/fill_template.py --in_place \
all_models/inflight_batcher_llm/postprocessing/config.pbtxt \
tokenizer_type:llama,tokenizer_dir:meta-llama/Llama-2-7b-chat-hf
docker run -it --rm --gpus all --network host --shm-size=1g \
-v $(pwd)/all_models:/all_models \
-v $(pwd)/scripts:/opt/scripts \
nvcr.io/nvidia/tritonserver:23.10-trtllm-python-py3
# Log in to huggingface-cli to get tokenizer
huggingface-cli login --token *****
# Install python dependencies
pip install sentencepiece protobuf
when i ran the final command
python /opt/scripts/launch_triton_server.py --model_repo /all_models/inflight_batcher_llm --world_size 1
I’m getting the below error
[TensorRT-LLM][INFO] Using 693 tokens in paged KV cache.
E1128 16:31:49.479096 227 backend_model.cc:634] ERROR: Failed to create instance: unexpected error when creating modelInstanceState: maxTokensInPagedKvCache must be large enough to process at least 1 sequence to completion (i.e. must be larger than beam_width * tokensPerBlock * maxBlocksPerSeq)
E1128 16:31:49.479182 227 model_lifecycle.cc:621] failed to load 'tensorrt_llm' version 1: Internal: unexpected error when creating modelInstanceState: maxTokensInPagedKvCache must be large enough to process at least 1 sequence to completion (i.e. must be larger than beam_width * tokensPerBlock * maxBlocksPerSeq)
I1128 16:31:49.479201 227 model_lifecycle.cc:756] failed to load 'tensorrt_llm'
E1128 16:31:49.479285 227 model_repository_manager.cc:563] Invalid argument: ensemble 'ensemble' depends on 'tensorrt_llm' which has no loaded version. Model 'tensorrt_llm' loading failed with error: version 1 is at UNAVAILABLE state: Internal: unexpected error when creating modelInstanceState: maxTokensInPagedKvCache must be large enough to process at least 1 sequence to completion (i.e. must be larger than beam_width * tokensPerBlock * maxBlocksPerSeq);
I1128 16:31:49.479354 227 server.cc:592]
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 15
you’re correct. the script isn’t very robust, but it gets the job done (assuming you configure things right the first time and/or backed up your original config files).
in the devs’ defense, i believe there are a lot of moving parts in this space. we are working with the early stages of TensorRT-LLM and should adjust our expectations as such.
FWIW, i’ve spent many hours configuring environments with Triton/TensorRT-LLM. i’ve run into a countless little issues, even if you faithfully follow the TensorRT-LLM documentation. there are many contradictions in the documentation, things that are out of date, pointers to things that are no longer relevant, etc. etc.
i assume NVIDIA will refine the documentation for TensorRT-LLM as their product matures. there are real performance gains to using TensorRT-LLM as a model backend, and businesses everywhere are scrambling to add LLM inference to their software offerings. i have to imagine there will be a lot more adoption soon.
working.
fill_template.py is not update the values if running for second time. I changed the kv_cache_free_gpu_mem_fraction :0.8 in the config file directly.
Changing directly in config.pbtxt fixed the issue