transformers: Some MarianMT models broken and output garbage

System Info

• transformers version: 4.30.2
• Platform: macOS-13.4.1-arm64-arm-64bit
• Python version: 3.11.4
• Huggingface_hub version: 0.16.3
• Safetensors version: 0.3.1
• PyTorch version (GPU?): 2.1.0.dev20230413 (False)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: no
• Using distributed or parallel set-up in script?: no

Who can help?

@ArthurZucker @younesbelkada @gante

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, …)
• My own task or dataset (give details below)

Reproduction

Some of the MarianMT models I use like Helsinki-NLP/opus-mt-tc-big-fi-en are currently broken. This bug is reproduced in the Huggingface Hub inference widget: https://huggingface.co/Helsinki-NLP/opus-mt-tc-big-fi-en?text=Kissa+kävelee+kadulla You can see that it gives a garbage translation.

This model worked just last week! I also see that the model hasn’t been updated in the Hub for over a month, so the bug must be in the transformers library.

Simple code to reproduce locally:

>>> tokenizer = transformers.AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-tc-big-fi-en")
>>> model = transformers.AutoModelForSeq2SeqLM.from_pretrained("Helsinki-NLP/opus-mt-tc-big-fi-en")
>>> model.generate(**tokenizer("kissa kävelee kadulla", return_tensors="pt"))
tensor([[57829, 16542, 16542, 16542, 16542, 16542, 16542, 16542, 16542, 16542,
         16542, 16542, 16542, 16542, 16542, 16542, 16542, 16542, 16542, 16542,
         16542, 16542, 16542, 16542, 16542, 16542, 16542, 16542, 16542, 16542,
         16542, 16542, 16542, 16542, 16542, 16542, 16542, 16542, 16542, 16542,
         16542, 16542, 16542, 16542, 16542, 16542, 16542, 19074, 19074, 19074,
         19074, 19074, 19074, 19074, 19074, 19074, 19074, 19074, 19074, 19074,
         19074, 19074, 19074, 19074, 19074, 19074, 19074, 19074, 19074, 19074,
         19074, 19074, 19074, 19074, 19074, 19074, 19074, 19074, 19074, 19074,
         19074, 19074, 19074, 19074, 19074, 19074, 19074, 19074, 19074, 19074,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825, 11825,
         11825, 41756]])

Expected behavior

The model should give a proper translation.