unsloth: [FIXED] `Qwen2VL` finetuning broken

Installing latest version of unsloth:

!pip uninstall unsloth -y && pip install --upgrade --no-cache-dir --no-deps git+https://github.com/unslothai/unsloth.git

breaks the Qwen2 7B Vision Colab it seems(?)

from unsloth import FastVisionModel # FastLanguageModel for LLMs
import torch

# 4bit pre quantized models we support for 4x faster downloading + no OOMs.
fourbit_models = [
    "unsloth/Llama-3.2-11B-Vision-Instruct-bnb-4bit", # Llama 3.2 vision support
    "unsloth/Llama-3.2-11B-Vision-bnb-4bit",
    "unsloth/Llama-3.2-90B-Vision-Instruct-bnb-4bit", # Can fit in a 80GB card!
    "unsloth/Llama-3.2-90B-Vision-bnb-4bit",

    "unsloth/Pixtral-12B-2409-bnb-4bit",              # Pixtral fits in 16GB!
    "unsloth/Pixtral-12B-Base-2409-bnb-4bit",         # Pixtral base model

    "unsloth/Qwen2-VL-2B-Instruct-bnb-4bit",          # Qwen2 VL support
    "unsloth/Qwen2-VL-7B-Instruct-bnb-4bit",
    "unsloth/Qwen2-VL-72B-Instruct-bnb-4bit",

    "unsloth/llava-v1.6-mistral-7b-hf-bnb-4bit",      # Any Llava variant works!
    "unsloth/llava-1.5-7b-hf-bnb-4bit",
] # More models at https://huggingface.co/unsloth

model, tokenizer = FastVisionModel.from_pretrained(
    "unsloth/Qwen2-VL-7B-Instruct",
    load_in_4bit = True, # Use 4bit to reduce memory use. False for 16bit LoRA.
    use_gradient_checkpointing = "unsloth", # True or "unsloth" for long context
)

Leading to:

🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning.
🦥 Unsloth Zoo will now patch everything to make training faster!
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[<ipython-input-2-dd19b029c094>](https://localhost:8080/#) in <cell line: 22>()
     20 ] # More models at https://huggingface.co/unsloth
     21 
---> 22 model, tokenizer = FastVisionModel.from_pretrained(
     23     "unsloth/Qwen2-VL-7B-Instruct",
     24     load_in_4bit = True, # Use 4bit to reduce memory use. False for 16bit LoRA.

3 frames
[/usr/local/lib/python3.10/dist-packages/unsloth_zoo/compiler.py](https://localhost:8080/#) in patch_gradient_accumulation(modeling_file, module)
    789     functions = dir(modeling_file)
    790     module = eval(f"modeling_file.{module}")
--> 791     forward = module.forward
    792     source = inspect.getsource(forward)
    793     has_kwargs = tuple(inspect.signature(forward).parameters.values())[-1].kind == inspect._VAR_KEYWORD

AttributeError: type object 'Qwen2VLCausalLMOutputWithPast' has no attribute 'forward'

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 24 (5 by maintainers)

Most upvoted comments

use pip install "unsloth==2024.12.11" until the problem solved

Hi I am seeing the same error in qwen2.5 vl…

Hopefully I finally fixed it! Would appreciate it if anyone could check - thanks a lot!

For local machines, please update Unsloth via:

pip install --upgrade --no-cache-dir --force-reinstall --no-deps unsloth unsloth_zoo

Colab / Kaggle should just restart

The latest version(=2024.12.7) of ‘usloth-zoo’ has an issue.

pip install "unsloth==2024.12.11" pip install "unsloth-zoo==2024.12.6"