transformers: Word offsets of some fast tokenizers are not compatible with token classification pipeline label aggregation

System Info

• transformers version: 4.21.0.dev0
• Platform: macOS-12.4-x86_64-i386-64bit
• Python version: 3.9.13
• Huggingface_hub version: 0.8.1
• PyTorch version (GPU?): 1.11.0 (False)
• Tensorflow version (GPU?): 2.9.1 (False)
• Flax version (CPU?/GPU?/TPU?): 0.5.2 (cpu)
• Jax version: 0.3.6
• JaxLib version: 0.3.5
• Using GPU in script?: N
• Using distributed or parallel set-up in script?: N

Who can help?

Tagging @Narsil for pipelines and @SaulLu for tokenization. Let me know if I should tag anyone for specific models, but it’s not really a model issue, except in terms of tokenization.

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, …)
• My own task or dataset (give details below)

Reproduction

I noticed this issue with a DeBERTa model, but it also affects some others. The high level issue is that some tokenizers include leading spaces in the offset indices, some exclude them, and some are configurable with trim_offsets. When offsets include leading spaces (equivalent to trim_offsets==False), the pipeline word heuristic doesn’t work. The result is aggregating all tokens in the sequence to one label. Simple example:

model_name = "brandon25/deberta-base-finetuned-ner"
model = AutoModelForTokenClassification.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

ner_aggregate = pipeline("ner", model=model, tokenizer=tokenizer, ignore_labels=[], aggregation_strategy="max")
ner_aggregate("We're from New York")

Result:

[{'entity_group': 'O', 'score': 0.9999778, 'word': " We're from New York", 'start': 0, 'end': 19}]

Expected behavior

Expected result, something like:

[{'entity_group': 'O', 'score': 0.9999778, 'word': " We're from", 'start': 0, 'end': 10}, {'entity_group': 'O', 'score': 0.9xxx, 'word': "New York", 'start': 11, 'end': 19}]

If you’d like to see actual output, here’s a colab notebook with relevant models for comparison.

This affects at least these:

• DeBERTa V1
• DeBERTa V2/3
• GPT2 (tested because DebertaTokenizerFast is a subclass of GPT2TokenizerFast)
• Depending on config, Roberta (and any other tokenizer that honors trim_offsets==False)

The easiest solution would be to update the heuristic. Here is a change that works for preceding space in sequence (like current heuristic) or leading space in token. I can turn into a PR if desired.

I know a lot of the default configuration matches reference implementations or published research, so I’m not sure where inconsistencies between tokenizers are desired behavior. I did notice, for example, that some sentencepiece tokenizers include leading spaces in offset indices (DeBERTa V2/3), and some don’t (Albert, XLNet). I looked at the converter config and the rust code (which is pretty opaque to me), but it’s not obvious to me why the offsets are different. Do you know, @SaulLu? Is that expected?

I am comparing different architectures to replace a production Bert model and was evaluating models fine tuned on an internal dataset when I ran into this. I have my manager’s blessing to spend some time on this (and already have! 😂), so I’m happy to work on a PR or help out how I can.