vllm: [Bug]: Uncaught exception | ; Qwen2_5_VLModel has no vLLM implementation and the Transformers implementation is not compatible with vLLM

Your current environment

I just know it’s hosted on runpod serverless vLLM latest (today).

πŸ› Describe the bug

When trying to host my finetuned Qwen2.5 VL 7b 4bit dynamic quantization using unsloth, and after I have saved the trained model it as bf16, when I try to host the model, it gives me this error:


worker exited with exit code 1
j6zswihe185nfq[warning][rank0]:[W324 18:13:29.115599288 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present,  but this warning has only been added since PyTorch 2.4 (function operator())\n
j6zswihe185nfq[info]engine.py           :116  2025-03-24 18:13:28,839 Error initializing vLLM engine: Qwen2_5_VLModel has no vLLM implementation and the Transformers implementation is not compatible with vLLM.\n
j6zswihe185nfq[error]Uncaught exception | <class 'ValueError'>; Qwen2_5_VLModel has no vLLM implementation and the Transformers implementation is not compatible with vLLM.; <traceback object at 0x7f5beafb7900>;
j6zswihe185nfq[info]INFO 03-24 18:13:28 model_runner.py:1110] Starting to load model itztheking/FMAX-testrun-1.0...\n
j6zswihe185nfq[info]INFO 03-24 18:13:28 cuda.py:229] Using Flash Attention backend.\n
j6zswihe185nfq[info]
j6zswihe185nfq[info]INFO 03-24 18:13:27 config.py:549] This model supports multiple tasks: {'score', 'embed', 'classify', 'reward', 'generate'}. Defaulting to 'generate'.\n
j6zswihe185nfq[info]tokenizer_name_or_path: itztheking/FMAX-testrun-1.0, tokenizer_revision: None, trust_remote_code: False\n
j6zswihe185nfq[info]engine.py           :27   2025-03-24 18:13:18,801 Engine args: AsyncEngineArgs(model='itztheking/FMAX-testrun-1.0', served_model_name=None, tokenizer='itztheking/FMAX-testrun-1.0', task='auto', skip_tokenizer_init=False, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path='', download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='bfloat16', kv_cache_dtype='auto', seed=0, max_model_len=10000, distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, block_size=16, enable_prefix_caching=False, disable_sliding_window=False, use_v2_block_manager='true', swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.95, max_num_batched_tokens=None, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, revision=None, code_revision=None, rope_scaling=None, rope_theta=None, hf_overrides=None, tokenizer_revision=None, quantization=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, fully_sharded_loras=False, lora_extra_vocab_size=256, long_lora_scaling_factors=None, lora_dtype='auto', max_cpu_loras=None, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, ray_workers_use_nsight=False, num_gpu_blocks_override=None, num_lookahead_slots=0, model_loader_extra_config=None, ignore_patterns=None, preemption_mode=None, scheduler_delay_factor=0.0, enable_chunked_prefill=None, guided_decoding_backend='outlines', logits_processor_pattern=None, speculative_model=None, speculative_model_quantization=None, speculative_draft_tensor_parallel_size=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, qlora_adapter_name_or_path=None, disable_logprobs_during_spec_decoding=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, worker_cls='auto', kv_transfer_config=None, generation_config=None, override_generation_config=None, enable_sleep_mode=False, model_impl='auto', calculate_kv_scales=None, additional_config=None, disable_log_requests=False)\n
j6zswihe185nfq[info]INFO 03-24 18:13:17 __init__.py:207] Automatically detected platform cuda.\n

Before submitting a new issue…

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 15 (6 by maintainers)

Most upvoted comments

The architecture name for this model should be Qwen2_5_VLForConditionalGeneration, you may have to update your config.json.