transformers: Inference API: Can't load tokenizer using from_pretrained, please update its configuration: No such file or directory (os error 2)

Hi,

Having the same issue, Steps followed :

• Trained the model (t5-base) using custom PyTorch (no Trainer).
• Loading model and tokenizer locally works fine using T5Tokenizer (not using AutoTokenizer ).
• Pushed the model to HuggingFace hub using model.push_to_hub() and tokenizer.push_to_hub()

Behavior :

• Loaded tokenizer from hub using AutoTokenizer doesn’t work.
• Loading using T5Tokenizer also from hub works.
• Looking at the files directory in the hub, only seeing tokenizer_config.json !
• Interface API gives the error : Can’t load tokenizer using from_pretrained, please update its configuration: No such file or directory (os error 2)

tokenizer_config.json is necessary for some additional information in the tokenizer. Original gpt2 repo might be different, but there’s some code for legacy models to make sure everything works smoothly for those.

The path within that file is indeed something to look into but it should work nonetheless.

@Narsil I downloaded the tokenizer.json file from the original gpt2-medium checkpoint from the hub and I added it to my model’s repo and it works now. However, this file is not produced automatically by the ‘save_pretrained()’ method of the hugginface GPT2LMHeadModel class, or the AutoTokenizer class . When loading a tokenizer manually using the AutoTokenizer class in Google Colab, this ‘tokenizer.json’ file isn’t necessary (it loads correctly given just the files from AutoTokenizer.save_pretrained() method). Was my solution of adding the tokenizer.json correct, or will it cause any hidden errors?

@Narsil Haha I understand. Yes I can confirm it is working well

@shiffman ,

I am not sure what are the steps for push_to_hub to upload the tokenizer.

tokenizer.push_to_hub("sgugger/my-awesome-model")

Might be necessary

@sgugger can you confirm ?