vllm: [Bug]: ValueError: Attempted to assign 119 = 119 multimodal tokens to 120 placeholders

Your current environment

The output of python collect_env.py
Your output of `python collect_env.py` here

🐛 Describe the bug

qwen2-audio; A100 first request can get correct answer,then raise this error and shutting down service

INFO 06-03 17:19:10 [engine.py:310] Added request chatcmpl-dbc1b9d1833145889ac9303cf53b6ca7. ERROR 06-03 17:19:10 [serving_chat.py:885] Error in chat completion stream generator. ERROR 06-03 17:19:10 [serving_chat.py:885] Traceback (most recent call last): ERROR 06-03 17:19:10 [serving_chat.py:885] File "/data/miniconda3/envs/ljj_opencompass/lib/python3.10/site-packages/vllm/entrypoints/openai/serving_chat.py", line 487, in chat_completion_stream_generator ERROR 06-03 17:19:10 [serving_chat.py:885] async for res in result_generator: ERROR 06-03 17:19:10 [serving_chat.py:885] File "/data/miniconda3/envs/ljj_opencompass/lib/python3.10/site-packages/vllm/engine/multiprocessing/client.py", line 667, in _process_request ERROR 06-03 17:19:10 [serving_chat.py:885] raise request_output ERROR 06-03 17:19:10 [serving_chat.py:885] vllm.engine.multiprocessing.MQEngineDeadError: Engine loop is not running. Inspect the stacktrace to find the original error: ValueError('Attempted to assign 119 = 119 multimodal tokens to 120 placeholders'). ERROR 06-03 17:19:10 [engine.py:160] ValueError('Attempted to assign 119 = 119 multimodal tokens to 120 placeholders') ERROR 06-03 17:19:10 [engine.py:160] Traceback (most recent call last): ERROR 06-03 17:19:10 [engine.py:160] File "/data/miniconda3/envs/ljj_opencompass/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 158, in start ERROR 06-03 17:19:10 [engine.py:160] self.run_engine_loop() ERROR 06-03 17:19:10 [engine.py:160] File "/data/miniconda3/envs/ljj_opencompass/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 221, in run_engine_loop ERROR 06-03 17:19:10 [engine.py:160] request_outputs = self.engine_step() ERROR 06-03 17:19:10 [engine.py:160] File "/data/miniconda3/envs/ljj_opencompass/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 247, in engine_step ERROR 06-03 17:19:10 [engine.py:160] raise e ERROR 06-03 17:19:10 [engine.py:160] File "/data/miniconda3/envs/ljj_opencompass/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 230, in engine_step ERROR 06-03 17:19:10 [engine.py:160] return self.engine.step() ERROR 06-03 17:19:10 [engine.py:160] File "/data/miniconda3/envs/ljj_opencompass/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 1412, in step ERROR 06-03 17:19:10 [engine.py:160] outputs = self.model_executor.execute_model( ERROR 06-03 17:19:10 [engine.py:160] File "/data/miniconda3/envs/ljj_opencompass/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 140, in execute_model ERROR 06-03 17:19:10 [engine.py:160] output = self.collective_rpc("execute_model", ERROR 06-03 17:19:10 [engine.py:160] File "/data/miniconda3/envs/ljj_opencompass/lib/python3.10/site-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc ERROR 06-03 17:19:10 [engine.py:160] answer = run_method(self.driver_worker, method, args, kwargs) ERROR 06-03 17:19:10 [engine.py:160] File "/data/miniconda3/envs/ljj_opencompass/lib/python3.10/site-packages/vllm/utils.py", line 2456, in run_method ERROR 06-03 17:19:10 [engine.py:160] return func(*args, **kwargs) ERROR 06-03 17:19:10 [engine.py:160] File "/data/miniconda3/envs/ljj_opencompass/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 420, in execute_model ERROR 06-03 17:19:10 [engine.py:160] output = self.model_runner.execute_model( ERROR 06-03 17:19:10 [engine.py:160] File "/data/miniconda3/envs/ljj_opencompass/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context ERROR 06-03 17:19:10 [engine.py:160] return func(*args, **kwargs) ERROR 06-03 17:19:10 [engine.py:160] File "/data/miniconda3/envs/ljj_opencompass/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1764, in execute_model ERROR 06-03 17:19:10 [engine.py:160] hidden_or_intermediate_states = model_executable( ERROR 06-03 17:19:10 [engine.py:160] File "/data/miniconda3/envs/ljj_opencompass/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl ERROR 06-03 17:19:10 [engine.py:160] return self._call_impl(*args, **kwargs) ERROR 06-03 17:19:10 [engine.py:160] File "/data/miniconda3/envs/ljj_opencompass/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl ERROR 06-03 17:19:10 [engine.py:160] return forward_call(*args, **kwargs) ERROR 06-03 17:19:10 [engine.py:160] File "/data/miniconda3/envs/ljj_opencompass/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_audio.py", line 381, in forward ERROR 06-03 17:19:10 [engine.py:160] inputs_embeds = self.get_input_embeddings(input_ids, ERROR 06-03 17:19:10 [engine.py:160] File "/data/miniconda3/envs/ljj_opencompass/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_audio.py", line 360, in get_input_embeddings ERROR 06-03 17:19:10 [engine.py:160] inputs_embeds = merge_multimodal_embeddings( ERROR 06-03 17:19:10 [engine.py:160] File "/data/miniconda3/envs/ljj_opencompass/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 481, in merge_multimodal_embeddings ERROR 06-03 17:19:10 [engine.py:160] return _merge_multimodal_embeddings( ERROR 06-03 17:19:10 [engine.py:160] File "/data/miniconda3/envs/ljj_opencompass/lib/python3.10/site-packages/vllm/model_executor/models/utils.py", line 397, in _merge_multimodal_embeddings ERROR 06-03 17:19:10 [engine.py:160] raise ValueError( ERROR 06-03 17:19:10 [engine.py:160] ValueError: Attempted to assign 119 = 119 multimodal tokens to 120 placeholders INFO: 51.62.5.35:60880 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 51.62.5.35:60880 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 51.62.5.35:60880 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: 51.62.5.35:60880 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error INFO: Shutting down INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [862259]

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: open
  • Created a year ago
  • Comments: 39 (18 by maintainers)

Most upvoted comments

Ok, see when I have time to take a look (I’m busy until next week). cc @Isotr0py if you have time

INFO 06-04 08:59:43 [logger.py:39] Received request chatcmpl-7c57f27f7cf34d288a05147aeb8d7ace: prompt: ‘<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n<|AUDIO|><|AUDIO|>In this audio, what kind of sound can you hear? A: Laughter, B: Sigh, C: Cough, D: Throat clearing, E: Sneeze, F: Sniff, Please select the one closest to the correct answer. ASSISTANT:<|im_end|>\n<|im_start|>assistant\n’, params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.03, temperature=0.5, top_p=0.95, top_k=10, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=80, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None), prompt_token_ids: None, lora_request: None, prompt_adapter_request: None.

Seems the prompt is formatted with incorrect number of audio placeholder (2 placeholder vs 1 audio_input), perhaps the chat_template has some issues… 🤔