bitsandbytes: AttributeError: 'Parameter' object has no attribute 'compress_statistics'
System Info
/usr/local/lib/python3.11/dist-packages/huggingface_hub/utils/_auth.py:94: UserWarning:
The secret HF_TOKEN does not exist in your Colab secrets.
To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.
You will be able to reuse this secret in all of your notebooks.
Please note that authentication is recommended but still optional to access public models or datasets.
warnings.warn(
config.json: 100%
601/601 [00:00<00:00, 68.7kB/s]
model.safetensors.index.json: 100%
23.9k/23.9k [00:00<00:00, 2.59MB/s]
Downloading shards: 100%
3/3 [01:07<00:00, 22.15s/it]
model-00001-of-00003.safetensors: 100%
4.95G/4.95G [00:23<00:00, 204MB/s]
model-00002-of-00003.safetensors: 100%
5.00G/5.00G [00:23<00:00, 225MB/s]
model-00003-of-00003.safetensors: 100%
4.55G/4.55G [00:20<00:00, 226MB/s]
Loading checkpoint shards: 100%
3/3 [00:05<00:00, 1.72s/it]
generation_config.json: 100%
116/116 [00:00<00:00, 13.8kB/s]
tokenizer_config.json: 100%
141k/141k [00:00<00:00, 8.10MB/s]
tokenizer.model: 100%
587k/587k [00:00<00:00, 63.7MB/s]
tokenizer.json: 100%
1.96M/1.96M [00:00<00:00, 23.1MB/s]
special_tokens_map.json: 100%
414/414 [00:00<00:00, 54.6kB/s]
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-7-7722530b5801> in <cell line: 0>()
35
36 model = prepare_model_for_kbit_training(model)
---> 37 model = get_peft_model(model, peft_config)
8 frames
/usr/local/lib/python3.11/dist-packages/peft/tuners/lora/bnb.py in dispatch_bnb_4bit(target, adapter_name, **kwargs)
554 {
555 "compute_dtype": target_base_layer.compute_dtype,
--> 556 "compress_statistics": target_base_layer.weight.compress_statistics,
557 "quant_type": target_base_layer.weight.quant_type,
558 }
AttributeError: 'Parameter' object has no attribute 'compress_statistics'
Reproduction
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
model_id = "mistralai/Mistral-7B-Instruct-v0.3"
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_use_double_quant=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16
)
model = AutoModelForCausalLM.from_pretrained(
model_id,
quantization_config=bnb_config,
use_cache=False,
device_map="auto",
)
model.config.pretraining_tp = 1
tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"
from peft import LoraConfig, prepare_model_for_kbit_training, get_peft_model
peft_config = LoraConfig(
lora_alpha=16,
lora_dropout=0.1,
r=64,
bias="none",
task_type="CAUSAL_LM",
)
model = prepare_model_for_kbit_training(model)
model = get_peft_model(model, peft_config)
Expected behavior
I run this code. It was ok an hour ago then suddenly i got this error
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 3
- Comments: 15
!pip install -q peft>=0.6.0 bitsandbytes>=0.41.0
same error.
bitsandbytes 0.45.5 peft 0.15.1
Got the same error. I had a working script on the previous
transformersversion and now updated to the main branch and it stopped. I am on the latest bnb version and latest peft versions.