peft: RuntimeError: self and mat2 must have the same dtype

i got this error when i run the following code

import transformers
from datasets import load_dataset
data = load_dataset('csv',data_files='/content/fyp.csv')
data = data.map(lambda samples: tokenizer(samples['completion']), batched=True)

trainer = transformers.Trainer(
    model=model, 
    train_dataset=data['train'],
    args=transformers.TrainingArguments(
        per_device_train_batch_size=4, 
        gradient_accumulation_steps=4,
        warmup_steps=100, 
        max_steps=200, 
        learning_rate=2e-4, 
        fp16=True,
        logging_steps=1, 
        output_dir='outputs'
    ),
    data_collator=transformers.DataCollatorForLanguageModeling(tokenizer, mlm=False)
)
model.config.use_cache = False  # silence the warnings. Please re-enable for inference!
trainer.train()

RuntimeError Traceback (most recent call last) <ipython-input-25-7f15b87d1ebc> in <module> 20 ) 21 model.config.use_cache = False # silence the warnings. Please re-enable for inference! —> 22 trainer.train()

32 frames /usr/local/lib/python3.8/dist-packages/peft/tuners/lora.py in forward(self, x) 446 return F.linear(x, transpose(self.weight, self.fan_in_fan_out), bias=self.bias) 447 else: –> 448 result = F.linear(x, transpose(self.weight, self.fan_in_fan_out), bias=self.bias) 449 if self.r > 0: 450 after_A = self.lora_A(self.lora_dropout(x))

RuntimeError: self and mat2 must have the same dtype

About this issue

Original URL
State: closed
Created a year ago
Comments: 16 (2 by maintainers)

Most upvoted comments

I am also having issues with this, trying to train llama-13b-4bit through text-generation-webui.

Training 'llama' model using (q, v) projections
Trainable params: 26,214,400 (1.3496 %), All params: 1,942,410,240 (Model: 1,916,195,840)
2023-07-24 16:31:22 INFO:Log file 'train_dataset_sample.json' created in the 'logs' directory.
wandb: Tracking run with wandb version 0.15.5
wandb: W&B syncing is set to `offline` in this directory.  
wandb: Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
Exception in thread Thread-3 (threaded_run):
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/home/nuclaer/gitrepos/text-generation-webui/modules/training.py", line 665, in threaded_run
    trainer.train()
  File "/home/nuclaer/.local/lib/python3.10/site-packages/transformers/trainer.py", line 1539, in train
    return inner_training_loop(
  File "/home/nuclaer/.local/lib/python3.10/site-packages/transformers/trainer.py", line 1809, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/home/nuclaer/.local/lib/python3.10/site-packages/transformers/trainer.py", line 2654, in training_step
    loss = self.compute_loss(model, inputs)
  File "/home/nuclaer/.local/lib/python3.10/site-packages/transformers/trainer.py", line 2679, in compute_loss
    outputs = model(**inputs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/accelerate/utils/operations.py", line 581, in forward
    return model_forward(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/accelerate/utils/operations.py", line 569, in __call__
    return convert_to_fp32(self.model_forward(*args, **kwargs))
  File "/usr/local/lib/python3.10/dist-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/peft/peft_model.py", line 786, in forward
    return self.base_model(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/auto_gptq/modeling/_base.py", line 433, in forward
    return self.model(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/nuclaer/.local/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 806, in forward
    outputs = self.model(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/nuclaer/.local/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 693, in forward
    layer_outputs = decoder_layer(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/nuclaer/.local/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 408, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/nuclaer/.local/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 305, in forward
    query_states = self.q_proj(hidden_states)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/peft/tuners/lora.py", line 668, in forward
    result = F.linear(x, transpose(self.weight, self.fan_in_fan_out), bias=self.bias)
RuntimeError: self and mat2 must have the same dtype
2023-07-24 16:31:24 INFO:Training complete, saving...
2023-07-24 16:31:24 INFO:Training complete!

Interestingly, text-generation-webui claims the training is completed. Anyway, it seems that the source of peft/tuners/lora.py has changed quite a bit since the bulk of this conversation, and it’s not obvious to me how to fix it. I’m new to these repositories. As far as I can tell, the problem originally mentioned in this thread is in regards to 8-bit training. But perhaps the fix was never made for 4-bit?

Here’s some information about my system and installations: Output of uname -a: Linux nuclaer-iridium 5.15.0-76-generic #83-Ubuntu SMP Thu Jun 15 19:16:32 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux Nvidia driver version: 515 Cuda version: 11.7 Graphics cards: a GTX 1070 8GB and a RTX 3060 12GB Peft version: peft-0.4.0 Commit hash for text-generation-webui: 3ef49397bbbf93cc12ab21d83d9a40a83cf8d68e I have monkeypatch installed to allow 4bit training with AutoGPTQ.

Has anyone gotten 4-bit training to work with this recently? Is there something I’m missing?

NuclearManD on Jul 24, 2023