transformers: EncoderDecoder does not automatically create decoder_attention_mask to match decoder_input_ids

System Info

- `transformers` version: 4.31.0
- Platform: Linux-4.15.0-192-generic-x86_64-with-glibc2.27
- Python version: 3.11.4
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.21.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

Who can help?

@ArthurZucker @NielsRogge

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, …)
• My own task or dataset (give details below)

Reproduction

I’m using a pretrained BERT model to make a bert2bert model using an EncoderDecoderModel. According to the documentation and a deprecation warning in the source code, it says that you no longer need to pass in decoder_input_ids as they’ll be automatically generated using labels. In the docs specifically, it also goes on to say that the default behavior of decoder_attention_mask is to automatically generate it based on padded tokens in decoder_input_ids, so you don’t need to pass the decoder attention mask either, as expected.

However, when trying to just pass input_ids + attention_mask for the encoder and labels, I get a warning that says something to the effect of “we strongly recommend passing an attention mask”. If I explicitly pass input_ids, attention_mask, decoder_input_ids, decoder_attention_mask, and labels, the warning goes away. Looking at the implementation of creating the decoder_input_ids from labels, it does indeed seem to skip the generation of decoder_attention_mask and simply passes through the value from the arguments, in this case None: https://github.com/huggingface/transformers/blob/e42587f596181396e1c4b63660abf0c736b10dae/src/transformers/models/encoder_decoder/modeling_encoder_decoder.py#L619-L637

You can recreate the warning in the notebook that Patrick made for the blog (https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/Leveraging_Pre_trained_Checkpoints_for_Encoder_Decoder_Models.ipynb#scrollTo=yoN2q0hZUbXN&line=11&uniqifier=1). Specifically, in the process_data_to_model_inputs function, you can just comment out the lines which explicitly set decoder_input_ids and decoder_attention_mask.

Expected behavior

I’d expect that if you can just pass labels to the forward call of EncoderDecoder and it will create decoder_input_ids, it would also create decoder_attention_mask. The fix is probably a few lines:

        if (labels is not None) and (decoder_input_ids is None and decoder_inputs_embeds is None):
            decoder_input_ids = shift_tokens_right(
                labels, self.config.pad_token_id, self.config.decoder_start_token_id
            )
            if decoder_attention_mask is not None:
                raise Exception # some error for passing 1/2 of decoder input_id/attn_mask?
            decoder_attention_mask = torch.where(decoder_input_ids == self.config.pad_token_id, 0, 1)