transformers: AttributeError: 'InternLMTokenizer' object has no attribute 'sp_model'

System Info

transformers version: 4.34.0.dev0
Platform: Linux-5.10.194-1-MANJARO-x86_64-with-glibc2.38
Python version: 3.9.12
Huggingface_hub version: 0.16.4
Safetensors version: 0.3.1
Accelerate version: 0.21.0
Accelerate config: not found
PyTorch version (GPU?): 2.0.1+cu117 (True)
Tensorflow version (GPU?): 2.10.0 (True)
Flax version (CPU?/GPU?/TPU?): 0.6.9 (cpu)
Jax version: 0.4.8
JaxLib version: 0.4.7
Using GPU in script?: Yes
Using distributed or parallel set-up in script?: No

Who can help?

@abhishekkrthakur

Information

The official example scripts
My own modified scripts

Tasks

An officially supported task in the examples folder (such as GLUE/SQuAD, …)
My own task or dataset (give details below)

Reproduction

Update autotrain-advanced and run autotrain-setup
Try to fine tune the new internlm-20b with peft in 8 bit
Using the command line tool

Expected behavior

Fine tune the model.

About this issue

Original URL
State: closed
Created 9 months ago
Comments: 20 (2 by maintainers)

Most upvoted comments

Hey! You are probably using main which introduced #23909. The InternLM repo relies on "transformers_version": "4.33.1", as you can see from the config.json! Don’t worry, internLM will be add to transformers super soon see #26302

ArthurZucker on Sep 22, 2023

transformers_version": "4.33.1

pip install transformers_version==4.33.1 fixed my issue.

ybdesire on Oct 27, 2023

Same for baichuan2 model: AttributeError: ‘BaichuanTokenizer’ object has no attribute ‘sp_model’

lucasjinreal on Oct 17, 2023

check init function of InternLMTokenizer, then, move super init code to the end of the function.

chenyr99 on Sep 22, 2023

Hi @lucasjinreal - in all cases this is caused by the custom modelling code not being compatible with the latest versions of transformers. You should file an issue on the model repos and tell them to rearrange the tokenizer __init__ so that self.sp_model is created before calling super().__init__()!

Rocketknight1 on Oct 17, 2023

This should be fixed by https://github.com/InternLM/InternLM/pull/419

SingL3 on Nov 10, 2023

Hi @jph00! We’re working on making proper library ports of InternLM (by converting them to LLaMA checkpoints, the model their code is based on). These should be available in a couple of days.

If you really want to get it working on main right now, though, the underlying problem is caused by the base tokenizer __init__() looking for self.sp_model before the child class has created that attribute. Moving the call to super().__init__() to a line after the creation of self.sp_model in tokenization_internlm.py should resolve the issue.

Rocketknight1 on Sep 25, 2023