llm-compressor: Quant Qwen2.5-VL with llmcompressor=0.5.1&Transformers 4.51.3 than infer with vLLM 0.8.1 got error
tokenizer.jsonof Qwen2.5-VL-7B-Instruct-W4A16-G128 is different with original model,
from
"truncation": null
to
"truncation": {
"direction": "Right",
"max_length": 2048,
"strategy": "LongestFirst",
"stride": 0
},
while infer with vLLM 0.8.1 got this error:
[rank0]: Traceback (most recent call last):
[rank0]: File "/mnt/dolphinfs/ssd_pool/docker/user/hadoop-mlm/yangyang260/proj/quant_platform/meituan-pytorch-quantization/examples/test_qwen2_5_w4a16.py", line 3, in <module>
[rank0]: model = mllminfer.AutoModel.create('./qwen2_5_w4a16_resources')
[rank0]: File "/mnt/dolphinfs/ssd_pool/docker/user/hadoop-mlm/yangyang260/proj/quant_platform/mllm-infer/mllminfer/model/modeling_auto.py", line 121, in create
[rank0]: raise e
[rank0]: File "/mnt/dolphinfs/ssd_pool/docker/user/hadoop-mlm/yangyang260/proj/quant_platform/mllm-infer/mllminfer/model/modeling_auto.py", line 116, in create
[rank0]: return dynamic_import(model_class)(resources_path)
[rank0]: File "/mnt/dolphinfs/ssd_pool/docker/user/hadoop-mlm/yangyang260/proj/quant_platform/mllm-infer/mllminfer/model/special/qwen_2_vl_vllm.py", line 32, in __init__
[rank0]: self.async_llm = AsyncLLMEngine.from_engine_args(engine_args)
[rank0]: File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 678, in from_engine_args
[rank0]: return async_engine_cls.from_vllm_config(
[rank0]: File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 651, in from_vllm_config
[rank0]: return cls(
[rank0]: File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 606, in __init__
[rank0]: self.engine = self._engine_class(*args, **kwargs)
[rank0]: File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 267, in __init__
[rank0]: super().__init__(*args, **kwargs)
[rank0]: File "/usr/local/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 283, in __init__
[rank0]: self._initialize_kv_caches()
[rank0]: File "/usr/local/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 432, in _initialize_kv_caches
[rank0]: self.model_executor.determine_num_available_blocks())
[rank0]: File "/usr/local/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 102, in determine_num_available_blocks
[rank0]: results = self.collective_rpc("determine_num_available_blocks")
[rank0]: File "/usr/local/lib/python3.10/site-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
[rank0]: answer = run_method(self.driver_worker, method, args, kwargs)
[rank0]: File "/usr/local/lib/python3.10/site-packages/vllm/utils.py", line 2216, in run_method
[rank0]: return func(*args, **kwargs)
[rank0]: File "/usr/local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank0]: return func(*args, **kwargs)
[rank0]: File "/usr/local/lib/python3.10/site-packages/vllm/worker/worker.py", line 229, in determine_num_available_blocks
[rank0]: self.model_runner.profile_run()
[rank0]: File "/usr/local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank0]: return func(*args, **kwargs)
[rank0]: File "/usr/local/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1243, in profile_run
[rank0]: self._dummy_run(max_num_batched_tokens, max_num_seqs)
[rank0]: File "/usr/local/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1308, in _dummy_run
[rank0]: .dummy_data_for_profiling(self.model_config,
[rank0]: File "/usr/local/lib/python3.10/site-packages/vllm/inputs/registry.py", line 342, in dummy_data_for_profiling
[rank0]: dummy_data = dummy_data_factory(seq_len)
[rank0]: File "/usr/local/lib/python3.10/site-packages/vllm/multimodal/profiling.py", line 214, in get_decoder_dummy_data
[rank0]: ) = self.get_and_validate_mm_inputs(seq_len)
[rank0]: File "/usr/local/lib/python3.10/site-packages/vllm/multimodal/profiling.py", line 163, in get_and_validate_mm_inputs
[rank0]: mm_inputs = self._get_dummy_mm_inputs(seq_len, mm_counts)
[rank0]: File "/usr/local/lib/python3.10/site-packages/vllm/multimodal/profiling.py", line 140, in _get_dummy_mm_inputs
[rank0]: return self.processor.apply(
[rank0]: File "/usr/local/lib/python3.10/site-packages/vllm/multimodal/processing.py", line 1613, in apply
[rank0]: self._validate_mm_placeholders(mm_placeholders, mm_item_counts)
[rank0]: File "/usr/local/lib/python3.10/site-packages/vllm/multimodal/processing.py", line 1536, in _validate_mm_placeholders
[rank0]: raise RuntimeError(
[rank0]: RuntimeError: Expected there to be 1 prompt updates corresponding to 1 image items, but instead found 0 prompt updates! Either the prompt text has missing/incorrect tokens for multi-modal inputs, or there is a problem with your implementation of merged multi-modal processor for this model (usually arising from an inconsistency between `_call_hf_processor` and `_get_prompt_updates`).
If downgrade transformers to 4.49.0, "truncation"in tokenizer.json after quantization remains the same and error gone.
I directly us the example of qwen_2_5_vl_example.py to act the quantization.
I want to know should I upgrade vLLM or downgrade transformers to do it right.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 19
@liuy-2019 see example here – https://github.com/vllm-project/llm-compressor/blob/main/examples/multimodal_vision/qwen_2_5_vl_example.py#L80
@YangYang-DLUT The TP=2 issue is a well known issue with the VL-72B model and not related to the issue in this project. You need to first create a padded version of your model, then quantize. See the very bottom of this page for a python script that will pad it for you: https://qwen.readthedocs.io/en/latest/quantization/gptq.html
@HPUedCSLearner I’ll also note that I’m looking into the effect that may be caused by having saved a new tokenizer with the latest transformers. That is to say, the latest transformers checkpoints differ from the checkpoint present in the original base model
I tried vllm 0.7.3 can work