transformers: Tokenizers throwing warning "The current process just got forked, Disabling parallelism to avoid deadlocks.. To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false)"

This is happening whenever you use multiprocessing (Often used by data loaders). The way to disable this warning is to set the TOKENIZERS_PARALLELISM environment variable to the value that makes more sense for you. By default, we disable the parallelism to avoid any hidden deadlock that would be hard to debug, but you might be totally fine while keeping it enabled in your specific use-case.

You can try to set it to true, and if your process seems to be stuck, doing nothing, then you should use false.

We’ll improve this message to help avoid any confusion (Cf https://github.com/huggingface/tokenizers/issues/328)

I may be a rookie, but it seems like it would be useful to indicate that this is an environment variable in the warning message.

I suspect this may be caused by loading data. In my case, it happens when my dataloader starts working.

Despite the documentation saying that use_fast defaults to False, adding use_fast=False so that it’s AutoTokenizer.from_pretrained(model_name, use_fast=False) removed this warning for me. If I just use AutoTokenizer.from_pretrained(model_name), the warning pops up again.

This might help you: https://stackoverflow.com/questions/62691279/how-to-disable-tokenizers-parallelism-true-false-warning

After testing, it is found that when the data in a dataloader is processed by the token, and the datalodaer jumps out before it is finished, this warning will be triggered; I give a code example:

# for example, following code will trigger the warning
for texts in train_dataloader:
    _ = tokenizer.batch_encode_plus(texts)
    # loader has not been traversed
    # but texts are used
    break 
for texts in test_dataloader:
    # warning ...
    pass or break

# and following code will not trigger the warning
for texts in train_dataloader:
    # loader has not been traversed
    # but texts are not used
    break 
for texts in test_dataloader:
    # No warning 
    pass or break

You are totally right! In the latest version 3.0.2, the warning message should be a lot better, and it will trigger only when necessary.

You must be using a tokenizer before using multiprocessing. When your process gets forked, you see this message because it detects that a fork is happening and that some kind of parallelism was used before.

I want to know if we can ignore this warning. What bad effects will it have? Will it affect the training results? Or is it just a little slower? If the environment variables are changed according to the above solution, what is the cost of doing so?

@hzphzp there is an explanation in SO https://stackoverflow.com/questions/62691279/how-to-disable-tokenizers-parallelism-true-false-warning/72926996#72926996

I want to know if we can ignore this warning. What bad effects will it have? Will it affect the training results? Or is it just a little slower? If the environment variables are changed according to the above solution, what is the cost of doing so?