tokenizers: tokenizer is slow after adding new tokens

@manugarri I believe that you should consider the following:

1. How will you define the scope of the new tokens. To did that I follow this article
2. Then, you should add the new tokens to the model.

Regardless the way you will choose to get to the new tokens list, the below steps is what I did, but be aware that I’m not sure it is the way to go. On the other hand, it works for me, and when I say “works” it means that I was able to train the model using MLM, and use the results to run pipeline to guess the masked word. Maybe I should run the MLM with a number of epochs to get a good set of weights to the new vocab, but that is another story.

A. To add new tokens I’ve just merge the list generated by following the article. Here is the code I’ve used:

old_vocab = [k for k,v in tokenizer.get_vocab().items()]
new_vocab = [token for token in new_tokens]
idx_old_vocab_list = list()
same_tokens_list = list()
different_tokens_list = list()

for idx_new,w in enumerate(new_vocab):
    try:
        idx_old = old_vocab.index(w)
    except:
        idx_old = -1
    if idx_old>=0:
        idx_old_vocab_list.append(idx_old)
        same_tokens_list.append((w,idx_new))
    else:
        different_tokens_list.append((w,idx_new))

B. Then, I added the new vocab to the tokenizer and resize the matrix:

## add new tokens to the existing vocabulary (only those not already presents)
logger.info("[ BEFORE ] tokenizer vocab size:", len(tokenizer))
added_tokens = tokenizer.add_tokens([i[0] for i in different_tokens_list])
logger.info("[ AFTER ] tokenizer vocab size:", len(tokenizer))
logger.info('added_tokens:',added_tokens)

# resize the embeddings matrix of the model
model.resize_token_embeddings(len(tokenizer))

model.save_pretrained('./cn-v1')
tokenizer.save_pretrained('./cn-v1')

C. Here is the catch. After save_pretrained, you will find a added_tokens.json in the folder. You will also see that the vocab.txt remain the same.

When you go to use the model with the new tokens it will explode the time as you are seeing. I believe it happens because the tokenizer tries to use the added_tokens.json.

What I did, and once again I quote that I don’t know if it is the correct way, was the following:

1. I went into the added_tokens.json and:

import os
import json
with open('added_tokens.json') as json_file:
    added_tokens = json.load(json_file)
    sorted_tokens = dict(sorted(added_tokens.items(), key=lambda item: item[1]))
    for tk in sorted_tokens.keys():
        print(tk)

1. I manually copy the printed results and pasted right after the last line of the vocab.txt. You would like to append the results using a python code.
2. Then I removed the added_tokens.json file.

After all this I was able to use the tokenizer:

CN_MODEL = 'cn-v1'

config = AutoConfig.from_pretrained(CN_MODEL)

tokenizer = AutoTokenizer.from_pretrained(CN_MODEL)
model = AutoModelForMaskedLM.from_pretrained(
  CN_MODEL,
  config=config
)
tokenizer('teste para diabetes e tamnbém para haloperidol')

{'input_ids': [101, 3515, 221, 30809, 122, 316, 22287, 22285, 22295, 312, 221, 607, 326, 1840, 243, 22290, 102], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}

The token with id 30809 is a new token.

The base model I’ve used was the bert-base-uncased.

Hope it can help.

Nice example @rdemorais, when you mention:

But, in the end, I'm still not sure that it was really needed to add more tokens... which leads me to that meme:

At my company we are adding additional tokens that dont really have any semantic meaning (they are group identifiers), and Ive noticed model performance at sequence classification improving significantly after adding those ‘custom’ tokens.

Hello @raphaelsty thank you for spending time over this matter. I believe it is a good way to generate discussion over a topic. Everybody benefits.

After struggling over this problematic, I moved on creating the actual model I was looking for. I figure out that breaking every word into tokens and only the ones supposed to be added as new tokens are actually added, helped the downstream task to behave as it was meant for.

I believe that it is not the job of the vocab.txt to hold business logic, for instance, 'star wars episode vi: return of the jedi' is task dependent, but ['star', 'wars', 'episode', 'vi', ':', 'return', 'of', 'the', 'jedi'] is not. You can use the words from the second approach to go training another model and leverage previous work.

In the final results, I’ve managed to create something like this:

Translating: no new syncope episodes. maintains dyspnea on minimal exertion, under spontaneous ventilation with support of o2 per cn at 2l/min with a good respiratory pattern

The words AUSENTE and PRESENTE means Absent and Present. It is Assertion Detection built using the same approach we are talking about.

The thing is. Note that I was able to get compound terms by training the NER accordingly. The MLM with new tokens is the underline tech.

But, in the end, I’m still not sure that it was really needed to add more tokens… which leads me to that meme:

I found a workaround by manually adding the new tokens to the vocab.json file and updating the steps to generate these new tokens in the merges.txt file, following the BytePairEncoding format. It works for me, it’s efficient. Is there any better way to do it?

@manugarri can you provide a test script demoing the slowness ? in the test provided by @davidnarganes there’s a slowdown but it’s definitely not on the same level as the “slow” one.

Edit: and all references to unique_no_split_tokens as mentioned in previous answers is definitely “slow”

@rdemorais it worked, after all, thanks!

@rdemorais it worked!

@lacls the vocab.txt is just a list of tokens like that:

loquiação
esplenomegalia
fisiologicas
ortopedia
gastrite
bx
t.
inflamatório
branda

Your new token list is supposed to be added in the end of the file.

Nevertheless, try to confirm if you gonna need every symbol from the line added to the vocab.txt. The way you wrote the code, everything will be there including dots, stop words and so on.

@rdemorais i think that is exactly what is happening. will give a try to your “hack” 😃

I’m also looking forward to hear about it. I’ve added new tokens and now the time process 1.5 GB of documents jumped from 5 min to 8 h.

update

I’ve added new tokens by using add_tokens and, after that, model.resize_token_embeddings(len(tokenizer)). Also I saved both tokenizer and model.

The difference is that I deleted the added_tokens.json file, and manually append the new tokens to the vocab file.

I don’t know if what I did is the way to go, but it is working now.