vllm: [Bug]: Confuse about the `computed max_num_seqs < 1` for Qwen2.5-VL-7B

Your current environment

The output of `python collect_env.py`
L20

🐛 Describe the bug

WARNING 03-31 16:53:06 [model_runner.py:1296] Computed max_num_seqs (min(64, 16384 // 32768)) to be less than 1. Setting it to the minimum value of 1.
WARNING 03-31 16:53:11 [profiling.py:222] The sequence length used for profiling (max_num_batched_tokens / max_num_seqs = 16384) is too short to hold the multi-modal embeddings in the worst case (32768 tokens in total, out of which {'image': 16384, 'video': 16384} are reserved for multi-modal embeddings). This may cause certain multi-modal inputs to fail during inference, even when the input text is short. To avoid this, you should increase `max_model_len`, reduce `max_num_seqs`, and/or reduce `mm_counts`.

How can i reduce mm_counts?

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (15 by maintainers)

Most upvoted comments

It means that the vision encoder can output at most 16384 tokens worth of embeddings per prompt.