Quieter on silence, better on speech: reducing Whisper hallucination

Whisper is still the speech recogniser most production systems run. One permissively licensed checkpoint covers 99 languages, copes when a speaker switches language mid-sentence, and is downloaded roughly 4.6 million times a month. It is also the teacher: Distil-Whisper, Kotoba-Whisper and NVIDIA's Granary corpus are all built on its transcripts, and Canary-1B-v2 and Parakeet-TDT-0.6B-v3 are trained on Granary.

It transcribes speech very well. What it does badly is say nothing. On 42 clips of pure room tone, whisper-large-v3 writes words on 61.9% of them and writes something, even a bare full stop, on all 42. In a voice agent that matters more than an ordinary recognition error, because a sentence nobody said becomes a turn the agent answers.

Why the usual fixes do not reach it

The common remedies sit outside the model. A voice activity detector in front of it, a threshold on the no-speech probability, a blocklist of known phrases applied afterwards. All of them help, and none of them change the weights. Serving stacks such as vLLM do not implement Whisper's own decoding loop, so most of those thresholds are not there to set.

Distillation does not fix it either. Pseudo-labelling pipelines filter the teacher's hallucinations out of the student's training set. The corpus comes out cleaner, but the deleted clips are exactly the ones that show the failure, so the student never sees one and inherits the behaviour anyway.

The direct remedy is to train on non-speech audio so the model learns to stay quiet. Canary-1B-v2 does this, and so do Mesolitica's open Malaysian Whisper fine-tunes. It works: Malaysian-turbo-v3 is the best checkpoint in the paper on every non-speech arm, at 16.7% on silence. It also recovers 41.1% of genuinely spoken phrases against 69.8% for the base model, and deletes 58.0% of clips containing repeated speech. Training on silence alone teaches the model to suppress text, not to tell silence from speech.

What the paper builds

The missing half is audio where the phrases a model invents are genuinely spoken, labelled with their transcript. The paper builds three things to supply it.

• A benchmark that scores both halves. 11,852 clips over eight arms, three of them with the empty string as the correct answer, plus 6,267 test clips of the phrases spoken for real, 8,296 clips of real audio that made a production model fail, and FLEURS in 58 languages as a multilingual guard. A model that says nothing is perfect on the non-speech arms and fails the rest, so it cannot win.
• A lexicon of what Whisper invents. 40,891 phrases in 100 languages, merged from four public sources. One of them is the list NVIDIA's Granary pipeline uses to delete hallucinated pseudo-labels. The paper uses the same lists to generate training data.
• Spoken positives. The frequent phrases are synthesised in their own languages and in several voices, with the text-to-speech system chosen by a recognition round trip rather than by reputation. Every clip passes a quality gate. The result is 29,112 clips in 83 languages.

The model then sees the same text two ways: something to suppress over silence, and something to transcribe when it is really said.

Results

The paper fine-tunes whisper-large-v3 66 times. LoRA at five ranks, on attention alone or on attention and the feed-forward layers, at three learning rates, plus three full fine-tunes. Every configuration is run twice, identical except for the 22,845 synthetic positives.

Across the 33 matched pairs, adding the positives lowers word emission on real voice-free audio in 31 and raises phrase recovery in 32. Among the 30 pairs that hold English accuracy, it lowers word emission in all 30.

From Tables 2, 5, 6 and 16 of the paper. Greedy decoding, no forced language, no temperature fallback. The silence arm is 42 clips, so each clip is 2.4 points and 2.4% is one clip.

Three findings stood out beyond the headline.

• Negatives alone reproduce the mute model. Without the positives, the quietest run in the paper emits words on 7.1% of real voice-free audio, recovers 29.2% of spoken phrases and triples LibriSpeech word error rate.
• Training loss ranks these models backwards. The mix without positives is 77% empty targets, which are cheap to predict, so it trains to a lower loss and is the worse model on every measure that matters.
• Where the adapters sit matters more than their size. Adapters on attention alone barely move off the base model at any rank. Adding the two feed-forward matrices at rank 8, about a quarter of the parameters of attention alone at rank 64, does better.

Limitations

• Multilingual cost. Every fine-tune is worse than the base model on FLEURS. The positives reduce the damage but do not remove it. The best reaches 0.381 against 0.305, and each run is only 1,000 steps on a Malay-weighted mix.
• Mostly synthetic positives. 52% of the positive pool is synthesised speech and 23% is read prose. A model tuned on it may learn something about the generator as well as the phrases.
• No telephony condition. Everything is 16 kHz. Narrowband 8 kHz call audio was not tested.
• Thin coverage in places. Chinese and Tamil positives are close to absent, at 11 and 4 clips.
• Judge bias. The synthetic corpus was filtered with whisper-large-v3, which flatters that checkpoint against large-v2 and turbo.

The paper also leaves decoder-side settings untested, and applying the winning configuration to whisper-large-v3-turbo, the checkpoint most people serve, is the next step. The benchmark, lexicon, synthetic corpus and wild audio are public.