vllm: ValueError: The prompt (total length 25938) is too long to fit into the model (context length 4096). Make sure that `max_model_len` is no smaller than the number of text tokens plus multimodal tokens.

Your current environment

Name Version Build Channel

_libgcc_mutex 0.1 main defaults _openmp_mutex 5.1 1_gnu defaults accelerate 1.0.1 pypi_0 pypi aiofiles 23.2.1 pypi_0 pypi aiohappyeyeballs 2.4.4 pypi_0 pypi aiohttp 3.11.11 pypi_0 pypi aiohttp-cors 0.7.0 pypi_0 pypi aiosignal 1.3.2 pypi_0 pypi airportsdata 20241001 pypi_0 pypi annotated-types 0.7.0 pypi_0 pypi anyio 4.8.0 pypi_0 pypi astor 0.8.1 pypi_0 pypi attrs 24.3.0 pypi_0 pypi av 14.0.1 pypi_0 pypi blake3 1.0.2 pypi_0 pypi bzip2 1.0.8 h5eee18b_6 defaults ca-certificates 2024.12.31 h06a4308_0 defaults cachetools 5.5.0 pypi_0 pypi certifi 2024.12.14 pypi_0 pypi charset-normalizer 3.4.1 pypi_0 pypi click 8.1.8 pypi_0 pypi cloudpickle 3.1.1 pypi_0 pypi colorful 0.5.6 pypi_0 pypi compressed-tensors 0.8.1 pypi_0 pypi contourpy 1.3.1 pypi_0 pypi cycler 0.12.1 pypi_0 pypi datasets 3.1.0 pypi_0 pypi depyf 0.18.0 pypi_0 pypi dill 0.3.8 pypi_0 pypi diskcache 5.6.3 pypi_0 pypi distlib 0.3.9 pypi_0 pypi distro 1.9.0 pypi_0 pypi docstring-parser 0.16 pypi_0 pypi einops 0.8.0 pypi_0 pypi fastapi 0.115.6 pypi_0 pypi ffmpy 0.5.0 pypi_0 pypi filelock 3.16.1 pypi_0 pypi fire 0.7.0 pypi_0 pypi fonttools 4.55.3 pypi_0 pypi frozenlist 1.5.0 pypi_0 pypi fsspec 2024.9.0 pypi_0 pypi gguf 0.10.0 pypi_0 pypi google-api-core 2.24.0 pypi_0 pypi google-auth 2.37.0 pypi_0 pypi googleapis-common-protos 1.66.0 pypi_0 pypi gradio 4.44.1 pypi_0 pypi gradio-client 1.3.0 pypi_0 pypi grpcio 1.69.0 pypi_0 pypi h11 0.14.0 pypi_0 pypi httpcore 1.0.7 pypi_0 pypi httptools 0.6.4 pypi_0 pypi httpx 0.28.1 pypi_0 pypi huggingface-hub 0.27.1 pypi_0 pypi idna 3.10 pypi_0 pypi importlib-metadata 8.5.0 pypi_0 pypi importlib-resources 6.5.2 pypi_0 pypi iniconfig 2.0.0 pypi_0 pypi interegular 0.3.3 pypi_0 pypi jieba 0.42.1 pypi_0 pypi jinja2 3.1.5 pypi_0 pypi jiter 0.8.2 pypi_0 pypi joblib 1.4.2 pypi_0 pypi jsonschema 4.23.0 pypi_0 pypi jsonschema-specifications 2024.10.1 pypi_0 pypi kiwisolver 1.4.8 pypi_0 pypi lark 1.2.2 pypi_0 pypi ld_impl_linux-64 2.40 h12ee557_0 defaults libffi 3.4.4 h6a678d5_1 defaults libgcc-ng 11.2.0 h1234567_1 defaults libgomp 11.2.0 h1234567_1 defaults libstdcxx-ng 11.2.0 h1234567_1 defaults libuuid 1.41.5 h5eee18b_0 defaults linkify-it-py 2.0.3 pypi_0 pypi llamafactory 0.9.2.dev0 pypi_0 pypi lm-format-enforcer 0.10.9 pypi_0 pypi markdown-it-py 3.0.0 pypi_0 pypi markupsafe 2.1.5 pypi_0 pypi matplotlib 3.10.0 pypi_0 pypi mdit-py-plugins 0.4.2 pypi_0 pypi mdurl 0.1.2 pypi_0 pypi memray 1.15.0 pypi_0 pypi mistral-common 1.5.1 pypi_0 pypi mpmath 1.3.0 pypi_0 pypi msgpack 1.1.0 pypi_0 pypi msgspec 0.19.0 pypi_0 pypi multidict 6.1.0 pypi_0 pypi multiprocess 0.70.16 pypi_0 pypi ncurses 6.4 h6a678d5_0 defaults nest-asyncio 1.6.0 pypi_0 pypi networkx 3.4.2 pypi_0 pypi nltk 3.9.1 pypi_0 pypi numpy 1.26.4 pypi_0 pypi nvidia-cublas-cu12 12.4.5.8 pypi_0 pypi nvidia-cuda-cupti-cu12 12.4.127 pypi_0 pypi nvidia-cuda-nvrtc-cu12 12.4.127 pypi_0 pypi nvidia-cuda-runtime-cu12 12.4.127 pypi_0 pypi nvidia-cudnn-cu12 9.1.0.70 pypi_0 pypi nvidia-cufft-cu12 11.2.1.3 pypi_0 pypi nvidia-curand-cu12 10.3.5.147 pypi_0 pypi nvidia-cusolver-cu12 11.6.1.9 pypi_0 pypi nvidia-cusparse-cu12 12.3.1.170 pypi_0 pypi nvidia-ml-py 12.560.30 pypi_0 pypi nvidia-nccl-cu12 2.21.5 pypi_0 pypi nvidia-nvjitlink-cu12 12.4.127 pypi_0 pypi nvidia-nvtx-cu12 12.4.127 pypi_0 pypi openai 1.59.7 pypi_0 pypi opencensus 0.11.4 pypi_0 pypi opencensus-context 0.1.3 pypi_0 pypi opencv-python-headless 4.11.0.86 pypi_0 pypi openssl 1.1.1w h7f8727e_0 defaults orjson 3.10.14 pypi_0 pypi outlines 0.1.11 pypi_0 pypi outlines-core 0.1.26 pypi_0 pypi packaging 24.2 pypi_0 pypi pandas 2.2.3 pypi_0 pypi partial-json-parser 0.2.1.1.post5 pypi_0 pypi peft 0.12.0 pypi_0 pypi pillow 10.4.0 pypi_0 pypi pip 24.2 pypi_0 pypi platformdirs 4.3.6 pypi_0 pypi pluggy 1.5.0 pypi_0 pypi prometheus-client 0.21.1 pypi_0 pypi prometheus-fastapi-instrumentator 7.0.2 pypi_0 pypi propcache 0.2.1 pypi_0 pypi proto-plus 1.25.0 pypi_0 pypi protobuf 5.29.3 pypi_0 pypi psutil 6.1.1 pypi_0 pypi py-cpuinfo 9.0.0 pypi_0 pypi py-spy 0.4.0 pypi_0 pypi pyarrow 18.1.0 pypi_0 pypi pyasn1 0.6.1 pypi_0 pypi pyasn1-modules 0.4.1 pypi_0 pypi pybind11 2.13.6 pypi_0 pypi pycountry 24.6.1 pypi_0 pypi pydantic 2.10.5 pypi_0 pypi pydantic-core 2.27.2 pypi_0 pypi pydevd 3.2.3 pypi_0 pypi pydub 0.25.1 pypi_0 pypi pygments 2.19.1 pypi_0 pypi pyparsing 3.2.1 pypi_0 pypi pytest 8.3.4 pypi_0 pypi python 3.11.0 h7a1cb2a_3 defaults python-dateutil 2.9.0.post0 pypi_0 pypi python-dotenv 1.0.1 pypi_0 pypi python-multipart 0.0.20 pypi_0 pypi pytz 2024.2 pypi_0 pypi pyyaml 6.0.2 pypi_0 pypi pyzmq 26.2.0 pypi_0 pypi ray 2.40.0 pypi_0 pypi readline 8.2 h5eee18b_0 defaults referencing 0.36.1 pypi_0 pypi regex 2024.11.6 pypi_0 pypi requests 2.32.3 pypi_0 pypi rich 13.9.4 pypi_0 pypi rouge-chinese 1.0.3 pypi_0 pypi rpds-py 0.22.3 pypi_0 pypi rsa 4.9 pypi_0 pypi ruff 0.9.1 pypi_0 pypi safetensors 0.5.2 pypi_0 pypi scipy 1.15.1 pypi_0 pypi semantic-version 2.10.0 pypi_0 pypi sentencepiece 0.2.0 pypi_0 pypi setuptools 75.1.0 pypi_0 pypi shellingham 1.5.4 pypi_0 pypi shtab 1.7.1 pypi_0 pypi six 1.17.0 pypi_0 pypi smart-open 7.1.0 pypi_0 pypi sniffio 1.3.1 pypi_0 pypi sqlite 3.45.3 h5eee18b_0 defaults sse-starlette 2.2.1 pypi_0 pypi starlette 0.41.3 pypi_0 pypi sympy 1.13.1 pypi_0 pypi termcolor 2.5.0 pypi_0 pypi textual 1.0.0 pypi_0 pypi tiktoken 0.7.0 pypi_0 pypi tk 8.6.14 h39e8969_0 defaults tokenizers 0.20.3 pypi_0 pypi tomlkit 0.12.0 pypi_0 pypi torch 2.5.1 pypi_0 pypi torchvision 0.20.1 pypi_0 pypi tqdm 4.67.1 pypi_0 pypi transformers 4.46.1 pypi_0 pypi triton 3.1.0 pypi_0 pypi trl 0.9.6 pypi_0 pypi typer 0.15.1 pypi_0 pypi typing-extensions 4.12.2 pypi_0 pypi tyro 0.8.14 pypi_0 pypi tzdata 2024.2 pypi_0 pypi uc-micro-py 1.0.3 pypi_0 pypi urllib3 2.3.0 pypi_0 pypi uvicorn 0.34.0 pypi_0 pypi uvloop 0.21.0 pypi_0 pypi virtualenv 20.29.0 pypi_0 pypi vllm 0.6.6.post1 pypi_0 pypi watchfiles 1.0.4 pypi_0 pypi websockets 12.0 pypi_0 pypi wheel 0.44.0 pypi_0 pypi wrapt 1.17.2 pypi_0 pypi xformers 0.0.28.post3 pypi_0 pypi xgrammar 0.1.10 pypi_0 pypi xxhash 3.5.0 pypi_0 pypi xz 5.4.6 h5eee18b_1 defaults yarl 1.18.3 pypi_0 pypi zipp 3.21.0 pypi_0 pypi zlib 1.2.13 h5eee18b_1 defaults

Model Input Dumps

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": text},
    {"role": "user", "content": base64_image}
]

text = “Describe this picture.” the image is a 224x224 png

🐛 Describe the bug

when I was trying to use vllm.LLM.generate to inference on llava1.5, I met this issue:

ValueError: The prompt (total length 25938) is too long to fit into the model (context length 4096). Make sure that max_model_len is no smaller than the number of text tokens plus multimodal tokens. For image inputs, the number of image tokens depends on the number of images, and possibly their aspect ratios as well.

I tried to set max_length in LLM.get_tokenizer.apply_chat_template to be 4096 to match the max tokens limit of llava, but it didn’t work. Moreover, I also tried to set max_model_len in vllm.LLM to be 11488 which is the max of my device, but the prompt length is too huge ( 25938 according to the report info ). I am really confused because I only put one image in the message and the size is not big, only 224x224.

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (6 by maintainers)

Most upvoted comments

I am not sure whether it is suitable for you. I just followed the doc link to modify the method about loading images and creating prompts.

I see. Could you share your modified code?

I know the Huggingface version of my code works but when I use VLLM it says I’ve exceeded the model context length, even with just image

def image_to_base64(image_path): base64_encoded = PIL.Image.open(image_path) return base64_encoded

def generate_messages(image_path): base64_image = image_to_base64(image_path) prompt = “USER: <image>\n Describe this image\nASSISTANT:” messages = {“prompt”:prompt, “multi_modal_data”:{“image”: base64_image},} return messages

def detect(self, messages): outputs = self.llm.generate(messages) return [output.outputs[0].text for output in outputs]

Well, only these three functions have been modified, compared with my code above.