llm-compressor: Quant Qwen2.5-VL with llmcompressor=0.5.1&Transformers 4.51.3 than infer with vLLM 0.8.1 got error

tokenizer.jsonof Qwen2.5-VL-7B-Instruct-W4A16-G128 is different with original model, from "truncation": null to

"truncation": {
    "direction": "Right",
    "max_length": 2048,
    "strategy": "LongestFirst",
    "stride": 0
  },

while infer with vLLM 0.8.1 got this error:

[rank0]: Traceback (most recent call last):
[rank0]:   File "/mnt/dolphinfs/ssd_pool/docker/user/hadoop-mlm/yangyang260/proj/quant_platform/meituan-pytorch-quantization/examples/test_qwen2_5_w4a16.py", line 3, in <module>
[rank0]:     model = mllminfer.AutoModel.create('./qwen2_5_w4a16_resources')
[rank0]:   File "/mnt/dolphinfs/ssd_pool/docker/user/hadoop-mlm/yangyang260/proj/quant_platform/mllm-infer/mllminfer/model/modeling_auto.py", line 121, in create
[rank0]:     raise e
[rank0]:   File "/mnt/dolphinfs/ssd_pool/docker/user/hadoop-mlm/yangyang260/proj/quant_platform/mllm-infer/mllminfer/model/modeling_auto.py", line 116, in create
[rank0]:     return dynamic_import(model_class)(resources_path)
[rank0]:   File "/mnt/dolphinfs/ssd_pool/docker/user/hadoop-mlm/yangyang260/proj/quant_platform/mllm-infer/mllminfer/model/special/qwen_2_vl_vllm.py", line 32, in __init__
[rank0]:     self.async_llm = AsyncLLMEngine.from_engine_args(engine_args)
[rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 678, in from_engine_args
[rank0]:     return async_engine_cls.from_vllm_config(
[rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 651, in from_vllm_config
[rank0]:     return cls(
[rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 606, in __init__
[rank0]:     self.engine = self._engine_class(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 267, in __init__
[rank0]:     super().__init__(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 283, in __init__
[rank0]:     self._initialize_kv_caches()
[rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 432, in _initialize_kv_caches
[rank0]:     self.model_executor.determine_num_available_blocks())
[rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 102, in determine_num_available_blocks
[rank0]:     results = self.collective_rpc("determine_num_available_blocks")
[rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
[rank0]:     answer = run_method(self.driver_worker, method, args, kwargs)
[rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/utils.py", line 2216, in run_method
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/worker/worker.py", line 229, in determine_num_available_blocks
[rank0]:     self.model_runner.profile_run()
[rank0]:   File "/usr/local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1243, in profile_run
[rank0]:     self._dummy_run(max_num_batched_tokens, max_num_seqs)
[rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1308, in _dummy_run
[rank0]:     .dummy_data_for_profiling(self.model_config,
[rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/inputs/registry.py", line 342, in dummy_data_for_profiling
[rank0]:     dummy_data = dummy_data_factory(seq_len)
[rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/multimodal/profiling.py", line 214, in get_decoder_dummy_data
[rank0]:     ) = self.get_and_validate_mm_inputs(seq_len)
[rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/multimodal/profiling.py", line 163, in get_and_validate_mm_inputs
[rank0]:     mm_inputs = self._get_dummy_mm_inputs(seq_len, mm_counts)
[rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/multimodal/profiling.py", line 140, in _get_dummy_mm_inputs
[rank0]:     return self.processor.apply(
[rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/multimodal/processing.py", line 1613, in apply
[rank0]:     self._validate_mm_placeholders(mm_placeholders, mm_item_counts)
[rank0]:   File "/usr/local/lib/python3.10/site-packages/vllm/multimodal/processing.py", line 1536, in _validate_mm_placeholders
[rank0]:     raise RuntimeError(
[rank0]: RuntimeError: Expected there to be 1 prompt updates corresponding to 1 image items, but instead found 0 prompt updates! Either the prompt text has missing/incorrect tokens for multi-modal inputs, or there is a problem with your implementation of merged multi-modal processor for this model (usually arising from an inconsistency between `_call_hf_processor` and `_get_prompt_updates`).

If downgrade transformers to 4.49.0, "truncation"in tokenizer.json after quantization remains the same and error gone.

I directly us the example of qwen_2_5_vl_example.py to act the quantization. I want to know should I upgrade vLLM or downgrade transformers to do it right.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 19

Most upvoted comments

@YangYang-DLUT The TP=2 issue is a well known issue with the VL-72B model and not related to the issue in this project. You need to first create a padded version of your model, then quantize. See the very bottom of this page for a python script that will pad it for you: https://qwen.readthedocs.io/en/latest/quantization/gptq.html

@HPUedCSLearner I’ll also note that I’m looking into the effect that may be caused by having saved a new tokenizer with the latest transformers. That is to say, the latest transformers checkpoints differ from the checkpoint present in the original base model

I tried vllm 0.7.3 can work


MAX_LENGTH=16384 
GPU_MEM_UTIL=0.5 

OMP_NUM_THREADS=32 \
CUDA_VISIBLE_DEVICES=2 \
python -m vllm.entrypoints.openai.api_server \
    --port 23333 \
    --served-model-name Qwen2.5-VL-7B-Instruct \
    --model ${MODEL_ID} \
    --trust-remote-code \
    --max-model-len $MAX_LENGTH \
    --gpu-memory-utilization $GPU_MEM_UTIL \
    --limit_mm_per_prompt 'image=20'
INFO 05-29 20:22:14 __init__.py:207] Automatically detected platform cuda.
INFO 05-29 20:22:15 api_server.py:912] vLLM API server version 0.7.3
INFO 05-29 20:22:15 api_server.py:913] args: Namespace(host=None, port=23333, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, enable_reasoning=False, reasoning_parser=None, tool_call_parser=None, tool_parser_plugin='', model='/juicefs-algorithm/workspace/vision/yongshuai_wang/codes/deploy/quantization/llmcompressor_2/juicefs-algorithm-W4A16-G128', task='auto', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=True, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', max_model_len=16384, guided_decoding_backend='xgrammar', logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.5, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt={'image': 20}, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=['Qwen2.5-VL-7B-Instruct'], qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', generation_config=None, override_generation_config=None, enable_sleep_mode=False, calculate_kv_scales=False, additional_config=None, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False)
INFO 05-29 20:22:15 api_server.py:209] Started engine process with PID 2746113
INFO 05-29 20:22:19 __init__.py:207] Automatically detected platform cuda.
INFO 05-29 20:22:21 config.py:549] This model supports multiple tasks: {'generate', 'reward', 'embed', 'score', 'classify'}. Defaulting to 'generate'.
INFO 05-29 20:22:26 config.py:549] This model supports multiple tasks: {'reward', 'classify', 'generate', 'score', 'embed'}. Defaulting to 'generate'.
INFO 05-29 20:22:27 llm_engine.py:234] Initializing a V0 LLM engine (v0.7.3) with config: model='/juicefs-algorithm/workspace/vision/yongshuai_wang/codes/deploy/quantization/llmcompressor_2/juicefs-algorithm-W4A16-G128', speculative_config=None, tokenizer='/juicefs-algorithm/workspace/vision/yongshuai_wang/codes/deploy/quantization/llmcompressor_2/juicefs-algorithm-W4A16-G128', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=16384, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=compressed-tensors, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=Qwen2.5-VL-7B-Instruct, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=True, 
INFO 05-29 20:22:27 cuda.py:229] Using Flash Attention backend.
INFO 05-29 20:22:28 model_runner.py:1110] Starting to load model /juicefs-algorithm/workspace/vision/yongshuai_wang/codes/deploy/quantization/llmcompressor_2/juicefs-algorithm-W4A16-G128...
WARNING 05-29 20:22:28 vision.py:94] Current `vllm-flash-attn` has a bug inside vision module, so we use xformers backend instead. You can run `pip install flash-attn` to use flash-attention backend.
INFO 05-29 20:22:28 config.py:3054] cudagraph sizes specified by model runner [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256] is overridden by config [256, 128, 2, 1, 4, 136, 8, 144, 16, 152, 24, 160, 32, 168, 40, 176, 48, 184, 56, 192, 64, 200, 72, 208, 80, 216, 88, 120, 224, 96, 232, 104, 240, 112, 248]
INFO 05-29 20:22:28 compressed_tensors_wNa16.py:85] Using MarlinLinearKernel for CompressedTensorsWNA16
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:03<00:00,  3.76s/it]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:03<00:00,  3.76s/it]

INFO 05-29 20:22:32 model_runner.py:1115] Loading model weights took 3.3127 GB
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
WARNING 05-29 20:22:33 model_runner.py:1288] Computed max_num_seqs (min(256, 16384 // 344064)) to be less than 1. Setting it to the minimum value of 1.
Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.


It looks like you are trying to rescale already rescaled images. If the input images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again.
WARNING 05-29 20:25:41 profiling.py:192] The context length (16384) of the model is too short to hold the multi-modal embeddings in the worst case (344064 tokens in total, out of which {'image': 327680, 'video': 16384} are reserved for multi-modal embeddings). This may cause certain multi-modal inputs to fail during inference, even when the input text is short. To avoid this, you should increase `max_model_len`, reduce `max_num_seqs`, and/or reduce `mm_counts`.
INFO 05-29 20:25:43 worker.py:267] Memory profiling takes 190.08 seconds
INFO 05-29 20:25:43 worker.py:267] the current vLLM instance can use total_gpu_memory (44.52GiB) x gpu_memory_utilization (0.50) = 22.26GiB
INFO 05-29 20:25:43 worker.py:267] model weights take 3.31GiB; non_torch_memory takes 0.08GiB; PyTorch activation peak memory takes 1.60GiB; the rest of the memory reserved for KV Cache is 17.26GiB.
INFO 05-29 20:25:43 executor_base.py:111] # cuda blocks: 31427, # CPU blocks: 7281
INFO 05-29 20:25:43 executor_base.py:116] Maximum concurrency for 16384 tokens per request: 30.69x
INFO 05-29 20:25:44 model_runner.py:1434] Capturing cudagraphs for decoding. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. If out-of-memory error occurs during cudagraph capture, consider decreasing `gpu_memory_utilization` or switching to eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
Capturing CUDA graph shapes: 100%|████████████████████████████████████████████████████████████████████████████| 35/35 [00:18<00:00,  1.89it/s]
INFO 05-29 20:26:03 model_runner.py:1562] Graph capturing finished in 19 secs, took 0.44 GiB
INFO 05-29 20:26:03 llm_engine.py:436] init engine (profile, create kv cache, warmup model) took 210.52 seconds
INFO 05-29 20:26:04 api_server.py:958] Starting vLLM API server on http://0.0.0.0:23333
INFO 05-29 20:26:04 launcher.py:23] Available routes are:
INFO 05-29 20:26:04 launcher.py:31] Route: /openapi.json, Methods: HEAD, GET
INFO 05-29 20:26:04 launcher.py:31] Route: /docs, Methods: HEAD, GET
INFO 05-29 20:26:04 launcher.py:31] Route: /docs/oauth2-redirect, Methods: HEAD, GET
INFO 05-29 20:26:04 launcher.py:31] Route: /redoc, Methods: HEAD, GET
INFO 05-29 20:26:04 launcher.py:31] Route: /health, Methods: GET
INFO 05-29 20:26:04 launcher.py:31] Route: /ping, Methods: POST, GET
INFO 05-29 20:26:04 launcher.py:31] Route: /tokenize, Methods: POST
INFO 05-29 20:26:04 launcher.py:31] Route: /detokenize, Methods: POST
INFO 05-29 20:26:04 launcher.py:31] Route: /v1/models, Methods: GET
INFO 05-29 20:26:04 launcher.py:31] Route: /version, Methods: GET
INFO 05-29 20:26:04 launcher.py:31] Route: /v1/chat/completions, Methods: POST
INFO 05-29 20:26:04 launcher.py:31] Route: /v1/completions, Methods: POST
INFO 05-29 20:26:04 launcher.py:31] Route: /v1/embeddings, Methods: POST
INFO 05-29 20:26:04 launcher.py:31] Route: /pooling, Methods: POST
INFO 05-29 20:26:04 launcher.py:31] Route: /score, Methods: POST
INFO 05-29 20:26:04 launcher.py:31] Route: /v1/score, Methods: POST
INFO 05-29 20:26:04 launcher.py:31] Route: /v1/audio/transcriptions, Methods: POST
INFO 05-29 20:26:04 launcher.py:31] Route: /rerank, Methods: POST
INFO 05-29 20:26:04 launcher.py:31] Route: /v1/rerank, Methods: POST
INFO 05-29 20:26:04 launcher.py:31] Route: /v2/rerank, Methods: POST
INFO 05-29 20:26:04 launcher.py:31] Route: /invocations, Methods: POST
INFO:     Started server process [2745835]
INFO:     Waiting for application startup.
INFO:     Application startup complete.