llm-compressor: [AWQ] Gemma 3: ValueError: too many values to unpack (expected 4)
Greetings,
I have finetuned gemma 3 12b using unsloth. After merging into 16bit, I am now trying to compress it to awq using this code:
from llmcompressor.modifiers.awq import AWQModifier
from llmcompressor.modifiers.quantization import QuantizationModifier
from llmcompressor.modifiers.smoothquant import SmoothQuantModifier
from llmcompressor.modifiers.quantization import GPTQModifier
from llmcompressor import oneshot
from compressed_tensors.quantization import (
QuantizationArgs,
QuantizationScheme,
QuantizationStrategy,
QuantizationType,
)
from transformers import AutoModelForCausalLM, AutoTokenizer
MODEL_ID = '/data/models/gemma-3-finetuned_merged/'
DATASET_ID = "mit-han-lab/pile-val-backup"
DATASET_SPLIT = "validation"
NUM_CALIBRATION_SAMPLES = 2
MAX_SEQUENCE_LENGTH = 5000
OUTPUT_DIR = '/data/models/awq/model-awq'
recipe = [
AWQModifier(num_bits=4, symmetric=False),
QuantizationModifier(
ignore=["lm_head",
"re:.*embed_tokens",
"re:multi_modal_projector.*",
"re:vision_tower.*"],
config_groups={
"group_0": QuantizationScheme(
targets=["Linear"],
weights=QuantizationArgs(
num_bits=4,
type=QuantizationType.INT,
dynamic=False,
symmetric=False,
strategy=QuantizationStrategy.GROUP,
group_size=128,
),
)
},
)
]
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, device_map="auto", torch_dtype="auto", local_files_only=True)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True, local_files_only=True)
oneshot(
model=model,
dataset="open_platypus",
recipe=recipe,
output_dir=OUTPUT_DIR,
max_seq_length=MAX_SEQUENCE_LENGTH, #3500
num_calibration_samples=NUM_CALIBRATION_SAMPLES,
)
but I get the following error:
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:04<00:00, 1.20it/s]
Using `use_fast=True` but `torchvision` is not available. Falling back to the slow image processor.
Tokenizing: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 24926/24926 [01:23<00:00, 299.17 examples/s]
2025-05-07T09:30:40.811904+0200 | reset | INFO - Compression lifecycle reset
2025-05-07T09:30:40.812334+0200 | from_modifiers | INFO - Creating recipe from modifiers
2025-05-07T09:30:43.675443+0200 | _set_resolved_mappings | INFO - Excluded 48 from resolved mappings due to shape mismatch
2025-05-07T09:30:43.849759+0200 | _calibrate | INFO - Running AWQModifier calibration with 2 samples...
0%| | 0/2 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/home/ignace/projects/compress_model/compress.py", line 53, in <module>
recipe=recipe,
^^^^^^^^^^^^^^^^^^
File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/llmcompressor/entrypoints/oneshot.py", line 179, in oneshot
one_shot()
File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/llmcompressor/entrypoints/oneshot.py", line 131, in __call__
self.apply_recipe_modifiers(
File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/llmcompressor/entrypoints/oneshot.py", line 173, in apply_recipe_modifiers
session.initialize(**session_kwargs)
File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/llmcompressor/core/session.py", line 118, in initialize
mod_data = self._lifecycle.initialize(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/llmcompressor/core/lifecycle.py", line 105, in initialize
data = mod.initialize(state=self.state, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/llmcompressor/modifiers/stage.py", line 83, in initialize
modifier.initialize(state, **kwargs)
File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/llmcompressor/modifiers/modifier.py", line 90, in initialize
self.initialized_ = self.on_initialize(state=state, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/llmcompressor/modifiers/awq/base.py", line 123, in on_initialize
self._calibrate(state.model, state.data.calib)
File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/llmcompressor/modifiers/awq/base.py", line 262, in _calibrate
run_calibration_forward(
File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/llmcompressor/modifiers/utils/pytorch_helpers.py", line 82, in run_calibration_forward
forward_fn(batch, module=model)
File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/llmcompressor/pytorch/utils/helpers.py", line 154, in tensors_module_forward
return module(**tensors)
^^^^^^^^^^^^^^^^^
File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/accelerate/hooks.py", line 176, in new_forward
output = module._old_forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/transformers/utils/generic.py", line 965, in wrapper
output = func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/transformers/models/gemma3/modeling_gemma3.py", line 942, in forward
outputs: BaseModelOutputWithPast = self.model(
^^^^^^^^^^^
File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/transformers/utils/generic.py", line 965, in wrapper
output = func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/transformers/models/gemma3/modeling_gemma3.py", line 722, in forward
layer_outputs = decoder_layer(
^^^^^^^^^^^^^^
File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/accelerate/hooks.py", line 176, in new_forward
output = module._old_forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/transformers/models/gemma3/modeling_gemma3.py", line 420, in forward
hidden_states, self_attn_weights = self.self_attn(
^^^^^^^^^^^^^^^
File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/accelerate/hooks.py", line 176, in new_forward
output = module._old_forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/transformers/models/gemma3/modeling_gemma3.py", line 342, in forward
attn_output, attn_weights = attention_interface(
^^^^^^^^^^^^^^^^^^^^
File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/transformers/integrations/sdpa_attention.py", line 30, in sdpa_attention_forward
key = repeat_kv(key, module.num_key_value_groups)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ignace/anaconda3/envs/llmcompressor/lib/python3.11/site-packages/transformers/integrations/sdpa_attention.py", line 11, in repeat_kv
batch, num_key_value_heads, slen, head_dim = hidden_states.shape
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: too many values to unpack (expected 4)
Since Gemma 3 is multimodel, I added the
"re:.*embed_tokens", "re:multi_modal_projector.*", "re:vision_tower.*"]
part that I have come across in previous issues but this did not help.
What am I doing wrong here?
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 20
Just reinstalled and followed the updated awq guide and it’s working. Thanks guys!
@Gman3214 Probably the former. As described by the guide.
Hi @ignaceHelsen! Can you let me know what your model’s particular base model is? Running your script locally gives different results
Your error suggests that the gemma tokenizer is auto-batching the samples, you may want to add a data_collator to remove the extra batching dimension