llm-compressor: [AWQ] Gemma 3: ValueError: too many values to unpack (expected 4)

Greetings,

I have finetuned gemma 3 12b using unsloth. After merging into 16bit, I am now trying to compress it to awq using this code:

from llmcompressor.modifiers.awq import AWQModifier
from llmcompressor.modifiers.quantization import QuantizationModifier
from llmcompressor.modifiers.smoothquant import SmoothQuantModifier
from llmcompressor.modifiers.quantization import GPTQModifier
from llmcompressor import oneshot
from compressed_tensors.quantization import (
    QuantizationArgs,
    QuantizationScheme,
    QuantizationStrategy,
    QuantizationType,
)
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = '/data/models/gemma-3-finetuned_merged/'
DATASET_ID = "mit-han-lab/pile-val-backup"
DATASET_SPLIT = "validation"
NUM_CALIBRATION_SAMPLES = 2
MAX_SEQUENCE_LENGTH = 5000
OUTPUT_DIR = '/data/models/awq/model-awq'

recipe = [
    AWQModifier(num_bits=4, symmetric=False),
    QuantizationModifier(
        ignore=["lm_head",
                "re:.*embed_tokens",
                "re:multi_modal_projector.*",
                "re:vision_tower.*"],
        config_groups={
            "group_0": QuantizationScheme(
                targets=["Linear"],
                weights=QuantizationArgs(
                    num_bits=4,
                    type=QuantizationType.INT,
                    dynamic=False,
                    symmetric=False,
                    strategy=QuantizationStrategy.GROUP,
                    group_size=128,
                ),
            )
        },
    )
]


model = AutoModelForCausalLM.from_pretrained(MODEL_ID, device_map="auto", torch_dtype="auto", local_files_only=True)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True, local_files_only=True)

oneshot(
    model=model,
    dataset="open_platypus",
    recipe=recipe,
    output_dir=OUTPUT_DIR,
    max_seq_length=MAX_SEQUENCE_LENGTH, #3500
    num_calibration_samples=NUM_CALIBRATION_SAMPLES,
)

but I get the following error:

Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:04<00:00,  1.20it/s]
Using `use_fast=True` but `torchvision` is not available. Falling back to the slow image processor.
Tokenizing: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 24926/24926 [01:23<00:00, 299.17 examples/s]
2025-05-07T09:30:40.811904+0200 | reset | INFO - Compression lifecycle reset
2025-05-07T09:30:40.812334+0200 | from_modifiers | INFO - Creating recipe from modifiers
2025-05-07T09:30:43.675443+0200 | _set_resolved_mappings | INFO - Excluded 48 from resolved mappings due to shape mismatch
2025-05-07T09:30:43.849759+0200 | _calibrate | INFO - Running AWQModifier calibration with 2 samples...
  0%|                                                                                                                                                                                                      | 0/2 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/home/ignace/projects/compress_model/compress.py", line 53, in <module>
    recipe=recipe,
^^^^^^^^^^^^^^^^^^
  File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/llmcompressor/entrypoints/oneshot.py", line 179, in oneshot
    one_shot()
  File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/llmcompressor/entrypoints/oneshot.py", line 131, in __call__
    self.apply_recipe_modifiers(
  File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/llmcompressor/entrypoints/oneshot.py", line 173, in apply_recipe_modifiers
    session.initialize(**session_kwargs)
  File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/llmcompressor/core/session.py", line 118, in initialize
    mod_data = self._lifecycle.initialize(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/llmcompressor/core/lifecycle.py", line 105, in initialize
    data = mod.initialize(state=self.state, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/llmcompressor/modifiers/stage.py", line 83, in initialize
    modifier.initialize(state, **kwargs)
  File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/llmcompressor/modifiers/modifier.py", line 90, in initialize
    self.initialized_ = self.on_initialize(state=state, **kwargs)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/llmcompressor/modifiers/awq/base.py", line 123, in on_initialize
    self._calibrate(state.model, state.data.calib)
  File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/llmcompressor/modifiers/awq/base.py", line 262, in _calibrate
    run_calibration_forward(
  File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/llmcompressor/modifiers/utils/pytorch_helpers.py", line 82, in run_calibration_forward
    forward_fn(batch, module=model)
  File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/llmcompressor/pytorch/utils/helpers.py", line 154, in tensors_module_forward
    return module(**tensors)
           ^^^^^^^^^^^^^^^^^
  File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/accelerate/hooks.py", line 176, in new_forward
    output = module._old_forward(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/transformers/utils/generic.py", line 965, in wrapper
    output = func(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/transformers/models/gemma3/modeling_gemma3.py", line 942, in forward
    outputs: BaseModelOutputWithPast = self.model(
                                       ^^^^^^^^^^^
  File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/transformers/utils/generic.py", line 965, in wrapper
    output = func(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/transformers/models/gemma3/modeling_gemma3.py", line 722, in forward
    layer_outputs = decoder_layer(
                    ^^^^^^^^^^^^^^
  File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/accelerate/hooks.py", line 176, in new_forward
    output = module._old_forward(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/transformers/models/gemma3/modeling_gemma3.py", line 420, in forward
    hidden_states, self_attn_weights = self.self_attn(
                                       ^^^^^^^^^^^^^^^
  File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/accelerate/hooks.py", line 176, in new_forward
    output = module._old_forward(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/transformers/models/gemma3/modeling_gemma3.py", line 342, in forward
    attn_output, attn_weights = attention_interface(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/transformers/integrations/sdpa_attention.py", line 30, in sdpa_attention_forward
    key = repeat_kv(key, module.num_key_value_groups)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/transformers/integrations/sdpa_attention.py", line 11, in repeat_kv
    batch, num_key_value_heads, slen, head_dim = hidden_states.shape
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: too many values to unpack (expected 4)

Since Gemma 3 is multimodel, I added the "re:.*embed_tokens", "re:multi_modal_projector.*", "re:vision_tower.*"]

part that I have come across in previous issues but this did not help.

What am I doing wrong here?

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 20

Most upvoted comments

Just reinstalled and followed the updated awq guide and it’s working. Thanks guys!

@Gman3214 Probably the former. As described by the guide.

Hi @ignaceHelsen! Can you let me know what your model’s particular base model is? Running your script locally gives different results

Your error suggests that the gemma tokenizer is auto-batching the samples, you may want to add a data_collator to remove the extra batching dimension

# Define a oneshot data collator for multimodal inputs.
def data_collator(batch):
    assert len(batch) == 1
    return {key: torch.tensor(value) for key, value in batch[0].items()}

oneshot(
    model=model,
    dataset="open_platypus",
    recipe=recipe,
    output_dir=OUTPUT_DIR,
    max_seq_length=MAX_SEQUENCE_LENGTH, #3500
    num_calibration_samples=NUM_CALIBRATION_SAMPLES,
    splits={"calibration": f"{DATASET_SPLIT}[:{NUM_CALIBRATION_SAMPLES}]"},
    data_collator=data_collator,
)