transformers: encode_plus not returning attention_mask and not padding

Hey! For me setting pad_to_max_length results in an error thrown. Just tried it out with the master branch but this resulted in the same error The code I’m executing:

titles = [['allround developer', 'Visual Studio Code'],
 ['allround developer', 'IntelliJ IDEA / PyCharm'],
 ['allround developer', 'Version Control']]
enc_titles = [[tokenizer.encode_plus(title[0], max_length=13, pad_to_max_length=True), tokenizer.encode_plus(title[1], max_length=13, pad_to_max_length=True)] for title in titles]

The error that I am getting:

<ipython-input-213-349f66a39abe> in <module>
      4 # titles = [' '.join(title) for title in titles]
      5 print(titles)
----> 6 enc_titles = [[tokenizer.encode_plus(title[0], max_length=4, pad_to_max_length=True), tokenizer.encode_plus(title[1], max_length=4)] for title in titles]

<ipython-input-213-349f66a39abe> in <listcomp>(.0)
      4 # titles = [' '.join(title) for title in titles]
      5 print(titles)
----> 6 enc_titles = [[tokenizer.encode_plus(title[0], max_length=4, pad_to_max_length=True), tokenizer.encode_plus(title[1], max_length=4)] for title in titles]

/usr/local/lib/python3.7/site-packages/transformers/tokenization_utils.py in encode_plus(self, text, text_pair, add_special_tokens, max_length, stride, truncation_strategy, return_tensors, return_token_type_ids, return_overflowing_tokens, return_special_tokens_mask, **kwargs)
    816                 If there are overflowing tokens, those will be added to the returned dictionary
    817             stride: if set to a number along with max_length, the overflowing tokens returned will contain some tokens
--> 818                 from the main sequence returned. The value of this argument defines the number of additional tokens.
    819             truncation_strategy: string selected in the following options:
    820                 - 'longest_first' (default) Iteratively reduce the inputs sequence until the input is under max_length

/usr/local/lib/python3.7/site-packages/transformers/tokenization_utils.py in get_input_ids(text)
    808                 the `tokenize` method) or a list of integers (tokenized string ids using the `convert_tokens_to_ids`
    809                 method)
--> 810             text_pair: Optional second sequence to be encoded. This can be a string, a list of strings (tokenized
    811                 string using the `tokenize` method) or a list of integers (tokenized string ids using the
    812                 `convert_tokens_to_ids` method)

/usr/local/lib/python3.7/site-packages/transformers/tokenization_utils.py in tokenize(self, text, **kwargs)
    657                 sub_text = sub_text.strip()
    658                 if i == 0 and not sub_text:
--> 659                     result += [tok]
    660                 elif i == len(split_text) - 1:
    661                     if sub_text:

/usr/local/lib/python3.7/site-packages/transformers/tokenization_utils.py in split_on_tokens(tok_list, text)
    654             result = []
    655             split_text = text.split(tok)
--> 656             for i, sub_text in enumerate(split_text):
    657                 sub_text = sub_text.strip()
    658                 if i == 0 and not sub_text:

/usr/local/lib/python3.7/site-packages/transformers/tokenization_utils.py in <genexpr>(.0)
    654             result = []
    655             split_text = text.split(tok)
--> 656             for i, sub_text in enumerate(split_text):
    657                 sub_text = sub_text.strip()
    658                 if i == 0 and not sub_text:

TypeError: _tokenize() got an unexpected keyword argument 'pad_to_max_length'```

Aha, great. I couldn’t wait because I needed it for a shared task, but nice to see it’s taking form. Almost there!

@BramVanroy Thanks for your comment! It made me try it out in just a plain Python file instead of a Jupyter notebook and it worked… 😄

Hm, you’re right. I think it was (again) an issue with the notebook that I was testing this time, where some values from previous cells were used or something like that.

Thanks for the fix!

Now that we’re at the topic, though, it might be nice to have a convenience method for batch processing? Something along these lines where pad_to_batch_length pads up to the max batch length (rather than max_seq_length of the model) to save computation/memory.

def enocde_batch_plus(batch, batch_pair=None, pad_to_batch_length=False, return_tensors=None, **kwargs):
    def merge_dicts(list_of_ds):
        # there's probably a better way of doing this
        d = defaultdict(list)
        for _d in list_of_ds:
            for _k, _v in _d.items():
                d[_k].append(_v)

        return dict(d)

    encoded_inputs = []
    batch_pair = [None] * len(batch) if batch_pair is None else batch_pair
    for firs_sent, second_sent in zip(batch, batch_pair):
        encoded_inputs.append(tokenizer.encode_plus(firs_sent,
                                          second_sent,
                                          **kwargs))

    encoded_inputs = merge_dicts(encoded_inputs)

    if pad_to_batch_length:
        max_batch_len = max([len(l) for l in encoded_inputs['input_ids']])
        # pad up to max_batch_len, similar to how it's done ine in prepare_for_model()

    if return_tensors:
        # convert to tensors, similar to how it's done in prepare_model()
        pass

    return encoded_inputs

Hi, thanks for raising this issue!

When running this code on the master branch, I do get the attention mask as output, but only when removing the return_tensors argument. When running with this argument, it crashes because a list is being concatenated to a tensor. I’m fixing this in #2148.

It’s weird that you didn’t get an error when running this line. On which commit are you based? encode and encode_plus take kwargs arguments so it wouldn’t raise an error if one of your arguments (pad_to_max_length) was not supposed to be there (e.g. if running on an old version of transformers).

pad_to_max_length is a boolean flag: if set to True with no max_length specified, it will pad the sequence up to the maximum sequence length the model can handle. If a max_length is specified, it will pad the sequence up to that number.