unsloth: [Bug] google/gemma3. UnslothVisionDataCollator is only for image models!

I’m using related colab notebook for fine-tuning Gemma3 model for vision task. In docs I saw that I can use Qwen2.5-VL or Pixtral notebooks for tuning, just need to replace model to Gemma. But I’m facing error UnslothVisionDataCollator is only for image models!

how I can fix that?

Link to notebook: https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2.5_VL_(7B)-Vision.ipynb#scrollTo=95_Nn-89DhsL

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 15

Most upvoted comments

@hexdrx @sensai99 @vody-am

The qwen2-vl/qwen2.5-vl have now been fixed . gemma3 vision still doesn’t work. We’re working on releasing a fix in the upcoming week. Meanwhile if you’re working in a local environment please make sure to:

pip install --upgrade unsloth_zoo
pip install --upgrade unsloth

in colab install cell should be updated to

%%capture
import os
if "COLAB_" not in "".join(os.environ.keys()):
    !pip install unsloth
else:
    # Do this only in Colab notebooks! Otherwise use pip install unsloth
    !pip install --no-deps bitsandbytes accelerate xformers==0.0.30 trl peft transformers triton cut_cross_entropy unsloth_zoo
    !pip install sentencepiece protobuf "datasets>=3.4.1" huggingface_hub hf_transfer
    !pip install --no-deps unsloth

Meanwhile we will be updating the notebooks shortly

@hexdrx UnslothVisionDataCollator does not currently accept a mixed dataset where some samples do not have an input image.

@sensai99 I was running https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2.5_VL_(7B)-Vision.ipynb#scrollTo=upcOlWe7A1vc but as a jupyter notebook on my own system.

I was seeing issues on unsloth-zoo 2025.5.11 hence downgraded to 2025.5.1.

The current (working) pyproject.toml is:

[project]
name = "notebooks"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.10"
dependencies = [
    "jupyter>=1.1.1",
    "matplotlib>=3.10.3",
    "numpy>=2.2.6",
    "scipy>=1.15.3",
    "unsloth==2025.5.1",
    "unsloth-zoo==2025.5.1",
    "wandb>=0.19.11",
]

The configuration used was:

MODEL_ID = "unsloth/Qwen2.5-VL-32B-Instruct"
MODEL_OUT = "qwen-finetuned"


model, tokenizer = FastVisionModel.from_pretrained(
    MODEL_ID,
    load_in_4bit = False, # Use 4bit to reduce memory use. False for 16bit LoRA.
    load_in_8bit = False,
    use_gradient_checkpointing = "unsloth", # True or "unsloth" for long context
)
model = FastVisionModel.get_peft_model(
    model,
    finetune_vision_layers     = False, # False if not finetuning vision layers
    finetune_language_layers   = True, # False if not finetuning language layers
    finetune_attention_modules = True, # False if not finetuning attention layers
    finetune_mlp_modules       = True, # False if not finetuning MLP layers

    r = 16,           # The larger, the higher the accuracy, but might overfit
    lora_alpha = 16,  # Recommended alpha == r at least
    lora_dropout = 0,
    bias = "none",
    random_state = 3407,
    use_rslora = False,  # We support rank stabilized LoRA
    loftq_config = None, # And LoftQ
    # target_modules = "all-linear", # Optional now! Can specify a list if needed
)