transformers.js: [Bug] whisper transcription quality dropped between 2.0.1 and 2.2.0 release

Great! Thanks so much for helping to investigate. This is the commit https://github.com/xenova/transformers.js/commit/ec00d4f540e8c1f298beadc4a71b05613568f6bc which changed the default quantization parameters in the conversion script.

This change was necessary for many text-only models, which showed significantly worse performance without these settings (both in JS and in python). This issue was first noticed a few weeks ago when GH actions started failing. Here’s some example code and output to show the problem:

from optimum.onnxruntime import ORTModelForTokenClassification
import onnxruntime
from transformers import pipeline, AutoTokenizer

model_path = './models/Davlan/distilbert-base-multilingual-cased-ner-hrl'

session_options = onnxruntime.SessionOptions()

tokenizer = AutoTokenizer.from_pretrained(model_path, use_fast=True)

model_ort = ORTModelForTokenClassification.from_pretrained(
    model_path + '/onnx',
    file_name='model.onnx',
    use_io_binding=True,
    session_options=session_options
)

ort_pipeline = pipeline(
    task="token-classification",
    model=model_ort,
    tokenizer=tokenizer,
)

ARTICLE = "The Golden State Warriors are an American professional basketball team based in San Francisco."

out = ort_pipeline(ARTICLE)

print(f'{out=}')

With reduce_range=True, per_channel=True (correct)

[{'entity': 'B-ORG', 'score': 0.9998536, 'index': 2, 'word': 'Golden', 'start': 4, 'end': 10}, {'entity': 'I-ORG', 'score': 0.99986124, 'index': 3, 'word': 
'State', 'start': 11, 'end': 16}, {'entity': 'I-ORG', 'score': 0.9998661, 'index': 4, 'word': 'Warriors', 'start': 17, 'end': 25}, {'entity': 'B-LOC', 'score': 
0.9997049, 'index': 13, 'word': 'San', 'start': 80, 'end': 83}, {'entity': 'I-LOC', 'score': 0.9987282, 'index': 14, 'word': 'Francisco', 'start': 84, 'end': 93}]

With reduce_range=False, per_channel=False (incorrect):

[{'entity': 'B-ORG', 'score': 0.99973637, 'index': 2, 'word': 'Golden', 'start': 4, 'end': 10}, {'entity': 'I-ORG', 'score': 0.99928397, 'index': 3, 'word': 'State', 'start': 11, 'end': 16}]

This is due to “saturation issues” when using int8 for weights: https://docs.openvino.ai/2022.3/pot_saturation_issue.html

However, as you point out, it looks like this is not necessary for whisper models. I will do some more testing, but it might make sense to revert the model to use reduce_range=False, per_channel=False.

Thanks for the quick fix @xenova , I can confirm models from the latest commit https://huggingface.co/Xenova/whisper-base.en/commit/86134155f8ad5593996868d4544b3d49ea0b1163 provide solid transcriptions.

Bingo! When enforcing model revision https://huggingface.co/Xenova/whisper-base.en/commit/95502fc2ffd132c6859cf58a66f4977c3c6abac2

I am getting good the good results for both 2.0.1 and 2.2.0

import { env, pipeline } from "https://cdn.jsdelivr.net/npm/@xenova/transformers@2.0.1/dist/transformers.min.js";

env.allowLocalModels = false;
env.backends.onnx.wasm.wasmPaths = 'https://cdn.jsdelivr.net/npm/@xenova/transformers@2.0.1/dist/';

const file = "tos.pcm";
const model = "Xenova/whisper-base.en";
const revision = "95502fc2ffd132c6859cf58a66f4977c3c6abac2";

const buffer = new Float32Array(await (await fetch(file)).arrayBuffer());
const pipe = await pipeline("automatic-speech-recognition", model, {revision});
const result = await pipe(buffer, {
	chunk_length_s: 30,
	stride_length_s: 5,
	return_timestamps: true});

let content = "2.0.1 result\n";
for(let {text, timestamp} of result.chunks)
	content += `${timestamp[0]} -> ${timestamp[1]} ${text}\n`;
console.log(content);

0 -> 7  We have main engine start.
7 -> 10  4, 3, 2, 1.
10 -> 13  Yeah, whoa!
23 -> 27.08  You're a jerk, Tom. Look, Celia, we have to follow our passions.
27.08 -> 30.240000000000002  You have your robotics, and I just wanna be awesome in space.
30.24 -> 34.519999999999996  Why don't you just admit that you're freaked out by my robot hand?
34.52 -> 37.46  I'm not freaked out, but it's...
37.46 -> 39.36  All right, fine. I'm freaked out.
39.36 -> 42.36  I'm having nightmares that I'm being chased by these giant robotic claws.
42.36 -> 45.18  Oh, what in our tongue? We're done.
50.16 -> 53.78  - Robots memory, synced, and locked.

So I wonder if the new onnx files / conversion process can be somehow tuned to get good speed and good transcriptions at the same time?

The models’ onnx files were updated around 3 weeks ago: https://huggingface.co/Xenova/whisper-tiny.en/tree/main/onnx to be in line with the conversion process used by the rest of the models (which resulted in performance improvements for other seq2seq models). Specifically, this is due to the different quantization parameters used.

If you have a look at attached scripts you will notice there is no revision specified for pipeline. That makes me believe these are the very same models in use