vllm: [Bug]: Failed to load the fine-tuned Qwen2.5-VL-7B-Instruct model.
Your current environment
The output of python collect_env.py
==============================
System Info
==============================
OS : Ubuntu 22.04.3 LTS (x86_64)
GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version : Could not collect
CMake version : Could not collect
Libc version : glibc-2.35
==============================
PyTorch Info
==============================
PyTorch version : 2.6.0+cu124
Is debug build : False
CUDA used to build PyTorch : 12.4
ROCM used to build PyTorch : N/A
==============================
Python Environment
==============================
Python version : 3.10.16 (main, Dec 11 2024, 16:24:50) [GCC 11.2.0] (64-bit runtime)
Python platform : Linux-5.15.0-134-generic-x86_64-with-glibc2.35
==============================
CUDA / GPU Info
==============================
Is CUDA available : True
CUDA runtime version : 12.1.105
CUDA_MODULE_LOADING set to : LAZY
GPU models and configuration : GPU 0: NVIDIA GeForce RTX 3090 Ti
Nvidia driver version : 535.183.06
cuDNN version : Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.0
HIP runtime version : N/A
MIOpen runtime version : N/A
==============================
Versions of relevant libraries
==============================
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-cufile-cu12==1.11.1.6
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-cusparselt-cu12==0.6.2
[pip3] nvidia-ml-py==12.575.51
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] pynvml==12.0.0
[pip3] pyzmq==26.4.0
[pip3] torch==2.6.0
[pip3] torchaudio==2.6.0
[pip3] torchvision==0.21.0
[pip3] transformers==4.51.3
[pip3] transformers-stream-generator==0.0.5
[pip3] triton==3.2.0
[conda] numpy 1.26.4 pypi_0 pypi
[conda] nvidia-cublas-cu12 12.4.5.8 pypi_0 pypi
[conda] nvidia-cuda-cupti-cu12 12.4.127 pypi_0 pypi
[conda] nvidia-cuda-nvrtc-cu12 12.4.127 pypi_0 pypi
[conda] nvidia-cuda-runtime-cu12 12.4.127 pypi_0 pypi
[conda] nvidia-cudnn-cu12 9.1.0.70 pypi_0 pypi
[conda] nvidia-cufft-cu12 11.2.1.3 pypi_0 pypi
[conda] nvidia-cufile-cu12 1.11.1.6 pypi_0 pypi
[conda] nvidia-curand-cu12 10.3.5.147 pypi_0 pypi
[conda] nvidia-cusolver-cu12 11.6.1.9 pypi_0 pypi
[conda] nvidia-cusparse-cu12 12.3.1.170 pypi_0 pypi
[conda] nvidia-cusparselt-cu12 0.6.2 pypi_0 pypi
[conda] nvidia-ml-py 12.575.51 pypi_0 pypi
[conda] nvidia-nccl-cu12 2.21.5 pypi_0 pypi
[conda] nvidia-nvjitlink-cu12 12.4.127 pypi_0 pypi
[conda] nvidia-nvtx-cu12 12.4.127 pypi_0 pypi
[conda] pynvml 12.0.0 pypi_0 pypi
[conda] pyzmq 26.4.0 pypi_0 pypi
[conda] torch 2.6.0 pypi_0 pypi
[conda] torchaudio 2.6.0 pypi_0 pypi
[conda] torchvision 0.21.0 pypi_0 pypi
[conda] transformers 4.51.3 pypi_0 pypi
[conda] transformers-stream-generator 0.0.5 pypi_0 pypi
[conda] triton 3.2.0 pypi_0 pypi
==============================
vLLM Info
==============================
ROCM Version : Could not collect
Neuron SDK Version : N/A
vLLM Version : 0.8.3
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X 0-5,12-17 0 N/A
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
==============================
Environment Variables
==============================
NVIDIA_VISIBLE_DEVICES=all
NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526
NCCL_VERSION=2.17.1-1
NVIDIA_DRIVER_CAPABILITIES=compute,utility
NVIDIA_PRODUCT_NAME=CUDA
NVIDIA_CUDA_END_OF_LIFE=1
CUDA_VERSION=12.1.0
LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
NCCL_CUMEM_ENABLE=0
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY
🐛 Describe the bug
I want to load this model which is exported from llama-factory:
Using this script:
llm = LLM(
model=self.model_name,
gpu_memory_utilization=gpu_memory_utilization,
max_model_len=max_model_len,
download_dir=download_dir,
dtype=dtype,
limit_mm_per_prompt={"image": 10},
allowed_local_media_path="/",
**kwargs,
)
Then I got this error:
Traceback (most recent call last):
File "/mlcv3/WorkingSpace/Personal/baotg/AICity25/Track2/src/inference/new_inference.py", line 176, in <module>
main(args)
File "/mlcv3/WorkingSpace/Personal/baotg/AICity25/Track2/src/inference/new_inference.py", line 106, in main
model = load_model(args)
File "/mlcv3/WorkingSpace/Personal/baotg/AICity25/Track2/src/inference/new_inference.py", line 78, in load_model
return VLLM(
File "/mlcv3/WorkingSpace/Personal/baotg/AICity25/Track2/src/models/vllm_generator.py", line 21, in __init__
self.llm = LLM(
File "/root/miniconda3/envs/main/lib/python3.10/site-packages/vllm/utils.py", line 1022, in inner
return fn(*args, **kwargs)
File "/root/miniconda3/envs/main/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 242, in __init__
self.llm_engine = self.engine_class.from_engine_args(
File "/root/miniconda3/envs/main/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 486, in from_engine_args
engine_config = engine_args.create_engine_config(usage_context)
File "/root/miniconda3/envs/main/lib/python3.10/site-packages/vllm/engine/arg_utils.py", line 1127, in create_engine_config
model_config = self.create_model_config()
File "/root/miniconda3/envs/main/lib/python3.10/site-packages/vllm/engine/arg_utils.py", line 1047, in create_model_config
return ModelConfig(
File "/root/miniconda3/envs/main/lib/python3.10/site-packages/vllm/config.py", line 316, in __init__
self.hf_text_config = get_hf_text_config(self.hf_config)
File "/root/miniconda3/envs/main/lib/python3.10/site-packages/vllm/transformers_utils/config.py", line 708, in get_hf_text_config
assert hasattr(config.text_config, "num_attention_heads")
AssertionError
llama-factory training and export config
Training config:
### model
model_name_or_path: Qwen/Qwen2.5-VL-7B-Instruct
image_max_pixels: 262144
video_max_pixels: 16384
trust_remote_code: true
### method
stage: sft
do_train: true
finetuning_type: lora
lora_rank: 8
lora_target: all
### dataset
dataset: combine_multi_bbox_pedes_appearance
template: qwen2_vl
dataset_dir: ./data_llama_factory
media_dir: ./
cutoff_len: 14000
overwrite_cache: true
preprocessing_num_workers: 4
dataloader_num_workers: 4
### output
output_dir: llama_factory_results/multi_bbox/pedes/qwen2.5_vl_7b_instruct_appearance
logging_steps: 10
save_steps: 500
plot_loss: true
save_total_limit: 2
overwrite_output_dir: true
save_only_model: false
report_to: wandb # choices: [none, wandb, tensorboard, swanlab, mlflow]
### train
per_device_train_batch_size: 1
gradient_accumulation_steps: 2
learning_rate: 1.0e-4
num_train_epochs: 3.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000
resume_from_checkpoint: null
flash_attn: fa2
# use_unsloth: True
### eval
# val_size: 0.1
# per_device_eval_batch_size: 1
# eval_strategy: steps
# eval_steps: 500
Export config:
### Note: DO NOT use quantized model or quantization_bit when merging lora adapters
### model
model_name_or_path: Qwen/Qwen2.5-VL-7B-Instruct
adapter_name_or_path: llama_factory_results/multi_bbox/pedes/qwen2.5_vl_7b_instruct_appearance
finetuning_type: lora
template: qwen2_vl
trust_remote_code: true
flash_attn: fa2
### export
export_dir: trained_models/multi_bbox/pedes/qwen2.5_vl_7b_instruct_appearance
export_size: 5
export_device: cpu # choices: [cpu, auto]
export_legacy_format: false
Before submitting a new issue…
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 15 (7 by maintainers)
Hi @DarkLight1337, it turns out the issue is from LLaMA-Factory (link), so I will close this issue here.