vllm: [Bug]: “limit-mm-per-prompt” can't be set correctly as in 0.6.5

Your current environment

vllm==0.6.6.post1
cuda==12.4
ubuntu==22.04

Model Input Dumps

No response

🐛 Describe the bug

when I run the following server command in v0.6.6.post1:

python -m vllm.entrypoints.openai.api_server \
--model "./Qwen2-VL-7B-Instruct" \
--served-model-name "Qwen2-VL-7B-Instruct" \
--device cuda \
--limit-mm-per-prompt image=8 

it warns that:

WARNING 01-08 18:59:14 model_runner.py:1279] Computed max_num_seqs (min(256, 32768 // 180224)) to be less than 1. Setting it to the minimum value of 1.
Token indices sequence length is longer than the specified maximum sequence length for this model (163840 > 32768). Running this sequence through the model will result in indexing errors
WARNING 01-08 18:59:32 processing.py:878] The context length (32768) of the model is too short to hold the multi-modal embeddings in the worst case (163840 tokens in total, out of which {'image': 163840} are reserved for multi-modal embeddings). This may cause certain multi-modal inputs to fail during inference, even when the input text is short. To avoid this, you should increase `max_model_len`, reduce `max_num_seqs`, and/or reduce `mm_counts`.

and the the cuda error occurs:

......
../aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [8274,0,0], thread: [123,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [8274,0,0], thread: [124,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [8274,0,0], thread: [125,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [8274,0,0], thread: [126,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [8274,0,0], thread: [127,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
INFO 01-08 19:15:31 model_runner_base.py:120] Writing input of failed execution to /tmp/err_execute_model_input_20250108-191531.pkl...
WARNING 01-08 19:15:31 model_runner_base.py:143] Failed to pickle inputs of failed execution: CUDA error: device-side assert triggered
WARNING 01-08 19:15:31 model_runner_base.py:143] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
.....
WARNING 01-08 19:15:31 model_runner_base.py:143] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
WARNING 01-08 19:15:31 model_runner_base.py:143] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
WARNING 01-08 19:15:31 model_runner_base.py:143] 

But when i degrade the vllm to 0.6.5, everything runs ok. And i tried post 4 images to the server, it works well, and didn’t occur any OOM error.

Is their any reasons why 0.6.6 pre-compute multi-modal token counts ? It seems that the compute method is too restrictive, or not support Naive Dynamic Resolution ?

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 29 (15 by maintainers)

Most upvoted comments

We do have plans to expand the limit_mm_per_prompt option in the future to allow users to specify the maximum image size, among other things.