axolotl: `AttributeError: 'function' object has no attribute 'func'`

Edit: See here for one solution: https://github.com/OpenAccess-AI-Collective/axolotl/issues/195#issuecomment-1603189889

I’m noticing crash in latest git commit.

safe: 01248253a3e8aedba6d473469dc839cd368bfe3c crash: f31a338cbbdcd76a5af35e400eb9e0e8cae36b72

Command: accelerate launch scripts/finetune.py examples/openllama-3b/config.yml (point to proper config depend on commit)

INFO:root:Starting trainer...
Traceback (most recent call last):
  File "/workspace/scripts/finetune.py", line 321, in <module>
    fire.Fire(train)
  File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/workspace/scripts/finetune.py", line 308, in train
    trainer.train(resume_from_checkpoint=resume_from_checkpoint)
  File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/transformers/trainer.py", line 1643, in train
    return inner_training_loop(
  File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/transformers/trainer.py", line 1750, in _inner_training_loop
    model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
  File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/accelerate/accelerator.py", line 1182, in prepare
    result = tuple(
  File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/accelerate/accelerator.py", line 1183, in <genexpr>
    self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
  File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/accelerate/accelerator.py", line 1022, in _prepare_one
    return self.prepare_model(obj, device_placement=device_placement)
  File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/accelerate/accelerator.py", line 1308, in prepare_model
    model.forward = MethodType(torch.cuda.amp.autocast(dtype=torch.float16)(model.forward.__func__), model)
AttributeError: 'function' object has no attribute '__func__'

About this issue

Original URL
State: closed
Created a year ago
Comments: 22

Most upvoted comments

I came across this again when I needed to use newer version of transformers. The thing that fixed it now is using latest accelerate.

transformers==4.31.0
accelerate==0.21.0

NanoCode012 on Jul 20, 2023

@anshsarkar @Luxios22 I managed to get my script working by downgrading to transformers==4.29.2, seems like there were some changes from v4.30.0 onwards that introduced this issue. I opened an issue on the HF/transformers repo if you want to track it https://github.com/huggingface/transformers/issues/24431

edit: actually, that downgrade probably won’t work for you guys if you’re manually creating the accelerator object…so it seems like maybe it’s a problem with accelerate instead. regardless, hopefully the HF guys will look into it

StevenSong on Jun 22, 2023

axolotl: `AttributeError: 'function' object has no attribute '__func__'`

About this issue

Most upvoted comments

axolotl: `AttributeError: 'function' object has no attribute 'func'`