tensorrtllm_backend: ERROR: Failed to create instance: unexpected error when creating modelInstanceState: maxTokensInPagedKvCache must be large enough to process at least 1 sequence to completion (i.e. must be larger than beam_width * tokensPerBlock * maxBlocksPerSeq)

I referred this official blog to run the llama2 7b chat model

python3 tools/fill_template.py --in_place \
      all_models/inflight_batcher_llm/tensorrt_llm/config.pbtxt \
      decoupled_mode:true,engine_dir:/all_models/inflight_batcher_llm/tensorrt_llm/1,\
max_tokens_in_paged_kv_cache:,batch_scheduler_policy:guaranteed_completion,kv_cache_free_gpu_mem_fraction:0.2,\
max_num_sequences:4
 
python tools/fill_template.py --in_place \
    all_models/inflight_batcher_llm/preprocessing/config.pbtxt \
    tokenizer_type:llama,tokenizer_dir:meta-llama/Llama-2-7b-chat-hf
 
python tools/fill_template.py --in_place \
    all_models/inflight_batcher_llm/postprocessing/config.pbtxt \
    tokenizer_type:llama,tokenizer_dir:meta-llama/Llama-2-7b-chat-hf

docker run -it --rm --gpus all --network host --shm-size=1g \
-v $(pwd)/all_models:/all_models \
-v $(pwd)/scripts:/opt/scripts \
nvcr.io/nvidia/tritonserver:23.10-trtllm-python-py3
 
# Log in to huggingface-cli to get tokenizer
huggingface-cli login --token *****
 
# Install python dependencies
pip install sentencepiece protobuf

when i ran the final command python /opt/scripts/launch_triton_server.py --model_repo /all_models/inflight_batcher_llm --world_size 1

I’m getting the below error

[TensorRT-LLM][INFO] Using 693 tokens in paged KV cache.
E1128 16:31:49.479096 227 backend_model.cc:634] ERROR: Failed to create instance: unexpected error when creating modelInstanceState: maxTokensInPagedKvCache must be large enough to process at least 1 sequence to completion (i.e. must be larger than beam_width * tokensPerBlock * maxBlocksPerSeq)
E1128 16:31:49.479182 227 model_lifecycle.cc:621] failed to load 'tensorrt_llm' version 1: Internal: unexpected error when creating modelInstanceState: maxTokensInPagedKvCache must be large enough to process at least 1 sequence to completion (i.e. must be larger than beam_width * tokensPerBlock * maxBlocksPerSeq)
I1128 16:31:49.479201 227 model_lifecycle.cc:756] failed to load 'tensorrt_llm'
E1128 16:31:49.479285 227 model_repository_manager.cc:563] Invalid argument: ensemble 'ensemble' depends on 'tensorrt_llm' which has no loaded version. Model 'tensorrt_llm' loading failed with error: version 1 is at UNAVAILABLE state: Internal: unexpected error when creating modelInstanceState: maxTokensInPagedKvCache must be large enough to process at least 1 sequence to completion (i.e. must be larger than beam_width * tokensPerBlock * maxBlocksPerSeq);
I1128 16:31:49.479354 227 server.cc:592]

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 15

Most upvoted comments

python3 tools/fill_template.py is using string.Template.safe_substitute to replace values, meaning that only a value represented as “$foo” in the configuration file will be replaced.

Seems to me that once you use fill_template.py once, everything you specified becomes hardcoded. If you try to override anything later using python3 tools/fill_template.py --in_place, the override is just silently ignored.

That seems bad - is that really what is wanted? Or am I misunderstanding something big?

you’re correct. the script isn’t very robust, but it gets the job done (assuming you configure things right the first time and/or backed up your original config files).

in the devs’ defense, i believe there are a lot of moving parts in this space. we are working with the early stages of TensorRT-LLM and should adjust our expectations as such.

FWIW, i’ve spent many hours configuring environments with Triton/TensorRT-LLM. i’ve run into a countless little issues, even if you faithfully follow the TensorRT-LLM documentation. there are many contradictions in the documentation, things that are out of date, pointers to things that are no longer relevant, etc. etc.

i assume NVIDIA will refine the documentation for TensorRT-LLM as their product matures. there are real performance gains to using TensorRT-LLM as a model backend, and businesses everywhere are scrambling to add LLM inference to their software offerings. i have to imagine there will be a lot more adoption soon.

working.

fill_template.py is not update the values if running for second time. I changed the kv_cache_free_gpu_mem_fraction :0.8 in the config file directly.

Changing directly in config.pbtxt fixed the issue