vllm: [Bug]: Mistrall Small 3.2 doesn't work with images

Your current environment

vLLM 0.9.1 (Docker)

mistralai/Mistral-Small-3.2-24B-Instruct-2506

Using mistralai/Mistral-Small-3.1-24B-Instruct-2503 with everything else unchanged works.

šŸ› Describe the bug

ERROR 06-24 14:40:09 [serving_chat.py:200] Error in preprocessing prompt inputs ERROR 06-24 14:40:09 [serving_chat.py:200] Traceback (most recent call last): ERROR 06-24 14:40:09 [serving_chat.py:200] File "/workspace/.venv/lib/python3.12/site-packages/vllm/entrypoints/openai/serving_chat.py", line 183, in create_chat_completion ERROR 06-24 14:40:09 [serving_chat.py:200] ) = await self._preprocess_chat( ERROR 06-24 14:40:09 [serving_chat.py:200] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 06-24 14:40:09 [serving_chat.py:200] File "/workspace/.venv/lib/python3.12/site-packages/vllm/entrypoints/openai/serving_engine.py", line 787, in _preprocess_chat ERROR 06-24 14:40:09 [serving_chat.py:200] conversation, mm_data_future = parse_chat_messages_futures( ERROR 06-24 14:40:09 [serving_chat.py:200] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 06-24 14:40:09 [serving_chat.py:200] File "/workspace/.venv/lib/python3.12/site-packages/vllm/entrypoints/chat_utils.py", line 1196, in parse_chat_messages_futures ERROR 06-24 14:40:09 [serving_chat.py:200] sub_messages = _parse_chat_message_content( ERROR 06-24 14:40:09 [serving_chat.py:200] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 06-24 14:40:09 [serving_chat.py:200] File "/workspace/.venv/lib/python3.12/site-packages/vllm/entrypoints/chat_utils.py", line 1120, in _parse_chat_message_content ERROR 06-24 14:40:09 [serving_chat.py:200] result = _parse_chat_message_content_parts( ERROR 06-24 14:40:09 [serving_chat.py:200] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 06-24 14:40:09 [serving_chat.py:200] File "/workspace/.venv/lib/python3.12/site-packages/vllm/entrypoints/chat_utils.py", line 1020, in _parse_chat_message_content_parts ERROR 06-24 14:40:09 [serving_chat.py:200] parse_res = _parse_chat_message_content_part( ERROR 06-24 14:40:09 [serving_chat.py:200] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 06-24 14:40:09 [serving_chat.py:200] File "/workspace/.venv/lib/python3.12/site-packages/vllm/entrypoints/chat_utils.py", line 1077, in _parse_chat_message_content_part ERROR 06-24 14:40:09 [serving_chat.py:200] mm_parser.parse_image(str_content) ERROR 06-24 14:40:09 [serving_chat.py:200] File "/workspace/.venv/lib/python3.12/site-packages/vllm/entrypoints/chat_utils.py", line 777, in parse_image ERROR 06-24 14:40:09 [serving_chat.py:200] placeholder = self._tracker.add("image", image_coro) ERROR 06-24 14:40:09 [serving_chat.py:200] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 06-24 14:40:09 [serving_chat.py:200] File "/workspace/.venv/lib/python3.12/site-packages/vllm/entrypoints/chat_utils.py", line 586, in add ERROR 06-24 14:40:09 [serving_chat.py:200] mm_processor = mm_registry.create_processor(model_config) ERROR 06-24 14:40:09 [serving_chat.py:200] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 06-24 14:40:09 [serving_chat.py:200] File "/workspace/.venv/lib/python3.12/site-packages/vllm/multimodal/registry.py", line 275, in create_processor ERROR 06-24 14:40:09 [serving_chat.py:200] model_cls = self._get_model_cls(model_config) ERROR 06-24 14:40:09 [serving_chat.py:200] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 06-24 14:40:09 [serving_chat.py:200] File "/workspace/.venv/lib/python3.12/site-packages/vllm/multimodal/registry.py", line 246, in _get_model_cls ERROR 06-24 14:40:09 [serving_chat.py:200] model_cls, _ = get_model_architecture(model_config) ERROR 06-24 14:40:09 [serving_chat.py:200] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 06-24 14:40:09 [serving_chat.py:200] File "/workspace/.venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/utils.py", line 240, in get_model_architecture ERROR 06-24 14:40:09 [serving_chat.py:200] model_cls, arch = ModelRegistry.resolve_model_cls(architectures) ERROR 06-24 14:40:09 [serving_chat.py:200] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 06-24 14:40:09 [serving_chat.py:200] File "/workspace/.venv/lib/python3.12/site-packages/vllm/model_executor/models/registry.py", line 489, in resolve_model_cls ERROR 06-24 14:40:09 [serving_chat.py:200] return self._raise_for_unsupported(architectures) ERROR 06-24 14:40:09 [serving_chat.py:200] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 06-24 14:40:09 [serving_chat.py:200] File "/workspace/.venv/lib/python3.12/site-packages/vllm/model_executor/models/registry.py", line 426, in _raise_for_unsupported ERROR 06-24 14:40:09 [serving_chat.py:200] raise ValueError( ERROR 06-24 14:40:09 [serving_chat.py:200] ValueError: Model architectures ['PixtralForConditionalGeneration'] failed to be inspected. Please check the logs for more details. INFO: 10.231.0.4:36046 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: open
  • Created a year ago
  • Comments: 16 (5 by maintainers)

Most upvoted comments

I’ve also had an issue with this specifically on blackwell

vllm-1 | CUDA error (/__w/xformers/xformers/third_party/flash-attention/hopper/flash_fwd_launch_template.h:175): no kernel image is available for execution on the device

Observing the same issue, using a 5090 and the 0.9.2 docker. It doesn’t seem specific to Mistral-Small though, for me it happens for several other VLMs like InternVL3, too. However, gemma-3 works fine on my end, so some things are working better for Blackwell on this version…

I’ve also had an issue with this specifically on blackwell

vllm-1 | CUDA error (/__w/xformers/xformers/third_party/flash-attention/hopper/flash_fwd_launch_template.h:175): no kernel image is available for execution on the device

vllm-1 | INFO 07-10 20:04:41 [__init__.py:244] Automatically detected platform cuda. vllm-1 | INFO 07-10 20:04:46 [api_server.py:1395] vLLM API server version 0.9.2rc2.dev35+g43813e636 vllm-1 | INFO 07-10 20:04:46 [cli_args.py:325] non-default args: {'model': '/models/Mistral-Small-3.2-24B-Instruct-2506', 'tokenizer_mode': 'mistral', 'served_model_name': ['the_model'], 'config_format': 'mistral', 'load_format': 'mistral', 'gpu_memory_utilization': 0.95, 'limit_mm_per_prompt': {'image': 10}} vllm-1 | INFO 07-10 20:04:53 [config.py:841] This model supports multiple tasks: {'generate', 'classify', 'embed', 'reward'}. Defaulting to 'generate'. vllm-1 | ERROR 07-10 20:04:53 [config.py:130] Error retrieving safetensors: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/models/Mistral-Small-3.2-24B-Instruct-2506'. Use `repo_type` argument if needed., retrying 1 of 2 vllm-1 | ERROR 07-10 20:04:55 [config.py:128] Error retrieving safetensors: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/models/Mistral-Small-3.2-24B-Instruct-2506'. Use `repo_type` argument if needed. vllm-1 | INFO 07-10 20:04:55 [config.py:3368] Downcasting torch.float32 to torch.bfloat16. vllm-1 | INFO 07-10 20:04:55 [config.py:1472] Using max model len 131072 vllm-1 | INFO 07-10 20:04:56 [config.py:2285] Chunked prefill is enabled with max_num_batched_tokens=8192. vllm-1 | /usr/local/lib/python3.12/dist-packages/mistral_common/tokens/tokenizers/tekken.py:304: FutureWarning: The attributed `special_token_policy` is deprecated and will be removed in 1.7.0. Please pass a special token policy explicitly to the relevant methods. vllm-1 | warnings.warn( vllm-1 | INFO 07-10 20:05:01 [__init__.py:244] Automatically detected platform cuda. vllm-1 | INFO 07-10 20:05:03 [core.py:526] Waiting for init message from front-end. vllm-1 | INFO 07-10 20:05:03 [core.py:69] Initializing a V1 LLM engine (v0.9.2rc2.dev35+g43813e636) with config: model='/models/Mistral-Small-3.2-24B-Instruct-2506', speculative_config=None, tokenizer='/models/Mistral-Small-3.2-24B-Instruct-2506', skip_tokenizer_init=False, tokenizer_mode=mistral, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=LoadFormat.MISTRAL, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=the_model, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":512,"local_cache_dir":null} vllm-1 | INFO 07-10 20:05:05 [parallel_state.py:1076] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0 vllm-1 | /usr/local/lib/python3.12/dist-packages/mistral_common/tokens/tokenizers/tekken.py:304: FutureWarning: The attributed `special_token_policy` is deprecated and will be removed in 1.7.0. Please pass a special token policy explicitly to the relevant methods. vllm-1 | warnings.warn( vllm-1 | WARNING 07-10 20:05:06 [registry.py:183] PixtralProcessorAdapter did not return `BatchFeature`. Make sure to match the behaviour of `ProcessorMixin` when implementing custom processors. vllm-1 | /usr/local/lib/python3.12/dist-packages/mistral_common/tokens/tokenizers/tekken.py:428: FutureWarning: Using the tokenizer's special token policy (SpecialTokenPolicy.IGNORE) is deprecated. It will be removed in 1.7.0. Please pass a special token policy explicitly. Future default will be SpecialTokenPolicy.IGNORE. vllm-1 | warnings.warn( vllm-1 | INFO 07-10 20:05:06 [topk_topp_sampler.py:49] Using FlashInfer for top-p & top-k sampling. vllm-1 | INFO 07-10 20:05:06 [gpu_model_runner.py:1769] Starting to load model /models/Mistral-Small-3.2-24B-Instruct-2506... vllm-1 | INFO 07-10 20:05:06 [gpu_model_runner.py:1774] Loading model from scratch... vllm-1 | INFO 07-10 20:05:06 [cuda.py:274] Using Flash Attention backend on V1 engine. Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s] Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:07<00:00, 7.04s/it] Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:07<00:00, 7.04s/it] vllm-1 | vllm-1 | INFO 07-10 20:05:16 [default_loader.py:272] Loading weights took 9.24 seconds vllm-1 | INFO 07-10 20:05:16 [gpu_model_runner.py:1800] Model loading took 44.7569 GiB and 9.505902 seconds vllm-1 | INFO 07-10 20:05:17 [gpu_model_runner.py:2237] Encoder cache will be initialized with a budget of 8192 tokens, and profiled with 3 image items of the maximum feature size. vllm-1 | CUDA error (/__w/xformers/xformers/third_party/flash-attention/hopper/flash_fwd_launch_template.h:175): no kernel image is available for execution on the device vllm-1 | Traceback (most recent call last): vllm-1 | File "<frozen runpy>", line 198, in _run_module_as_main vllm-1 | File "<frozen runpy>", line 88, in _run_code vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1495, in <module> vllm-1 | uvloop.run(run_server(args)) vllm-1 | File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 109, in run vllm-1 | return __asyncio.run( vllm-1 | ^^^^^^^^^^^^^^ vllm-1 | File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run vllm-1 | return runner.run(main) vllm-1 | ^^^^^^^^^^^^^^^^ vllm-1 | File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run vllm-1 | return self._loop.run_until_complete(task) vllm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm-1 | File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete vllm-1 | File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 61, in wrapper vllm-1 | return await main vllm-1 | ^^^^^^^^^^ vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1431, in run_server vllm-1 | await run_server_worker(listen_address, sock, args, **uvicorn_kwargs) vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1451, in run_server_worker vllm-1 | async with build_async_engine_client(args, client_config) as engine_client: vllm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm-1 | File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__ vllm-1 | return await anext(self.gen) vllm-1 | ^^^^^^^^^^^^^^^^^^^^^ vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 158, in build_async_engine_client vllm-1 | async with build_async_engine_client_from_engine_args( vllm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm-1 | File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__ vllm-1 | return await anext(self.gen) vllm-1 | ^^^^^^^^^^^^^^^^^^^^^ vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 194, in build_async_engine_client_from_engine_args vllm-1 | async_llm = AsyncLLM.from_vllm_config( vllm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 162, in from_vllm_config vllm-1 | return cls( vllm-1 | ^^^^ vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 124, in __init__ vllm-1 | self.engine_core = EngineCoreClient.make_async_mp_client( vllm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 96, in make_async_mp_client vllm-1 | return AsyncMPClient(*client_args) vllm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 666, in __init__ vllm-1 | super().__init__( vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 403, in __init__ vllm-1 | with launch_core_engines(vllm_config, executor_class, vllm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ vllm-1 | File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ vllm-1 | next(self.gen) vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 434, in launch_core_engines vllm-1 | wait_for_engine_startup( vllm-1 | File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 484, in wait_for_engine_startup vllm-1 | raise RuntimeError("Engine core initialization failed. " vllm-1 | RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {} vllm-1 | /usr/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown vllm-1 | warnings.warn('resource_tracker: There appear to be %d ' vllm-1 exited with code 1

@erikmargaronis Yes, there’s a PR already although it’s been pending for 3 weeks. Here https://github.com/vllm-project/vllm/pull/19425

@PedroMiolaSilva I get the same error when running tool calling in streaming mode. Thanks for the notice! There is an open issue on this https://github.com/vllm-project/vllm/issues/20028

@DarkLight1337 Can you assign this task to me, I noticed that the Mistral 3.2 seems different from Mistral 3.1, I reproduced the problem and noticed some code works on Mistral 3.1 but not works on Mistral 3.2, I will try to solve it.