Where words begin and end: a forced aligner trained on human boundaries

A forced aligner takes a recording and its transcript and returns where each word starts and ends. A lot of a speech stack depends on those times. Text-to-speech corpora are cut at them, end-of-turn detectors learn from the pauses between them, and subtitles, pronunciation scoring and data filtering take them as given.

In our stack one aligner supplied all of them: MMS-300M with CTC decoding, run through the defaults of the ctc-forced-aligner package. Those defaults were wrong in a way that spread. Across 32.3 million words of Malaysian corpora aligned with them, almost every word boundary carried a gap, and roughly half carried one of at least 0.1 s, long enough to count as a pause. Checked against a voice activity detector, 85% to 92% of those pauses sat inside speech. The aligner invented a pause between almost every pair of words.

Why CTC aligners start words late

A CTC model emits a character or a blank at every frame, and training only asks that the characters come out in the right order. The model learns to fire each character on a frame or two and give every other frame to the blank. On clean TIMIT, a median 67% of the frames inside a human-marked word are blank on the forced path. The first character lands a median 33 ms after the word starts, and the last ends a median 26 ms before it ends.

Any decoder that reads word edges off those spikes inherits the offset. The report runs four decoders over the same MMS-300M emissions, and their errors range from 29.2 to 46.8 ms. The decoder matters as much as the model, and none of them can recover where the word actually was.

What exists, and how precise it is

The report surveys 72 aligners and 38 evaluation sets, then benchmarks nine configurations at their documented defaults against human boundaries: TIMIT read speech, Buckeye conversations, and Switchboard conversations recorded over real landlines. TIMIT also runs through six telephone conditions built from real codec binaries, with each codec's delay measured and removed. That step matters. MP3 at 8 kHz arrives 138 ms late, more than the mean error of any aligner tested.

• MFA is the most precise existing aligner. On clean TIMIT it misses by 19.1 ms on average, with 68.2% of boundaries within 20 ms. It needs a pronunciation dictionary and acoustic model per language, and has neither for Malay or Indonesian.
• The multilingual aligners sit near 30 ms. Qwen3-ForcedAligner reaches 30.1 ms on clean speech but 52.7 ms once the audio is band-limited to 8 kHz.
• Telephone speech was unmeasured. We found no aligner of any family evaluated against human boundaries on 8 kHz or codec-processed audio.

What the report builds

The report's aligner fine-tunes Whisper with LoRA to predict each word's start and end as timestamp tokens. It trains only on boundaries placed by people: 47 hours in 87 languages. Five changes moved the error, each measured against the one before.

1. Timestamp tokens instead of a CTC path. On clean TIMIT the error falls from 46.8 ms for the MMS-300M aligner in our stack to 10.4 ms.
2. Telephone augmentation. Half the training clips pass through simulated telephone channels, which brings TIMIT through those codecs from 17.1 to 11.2 ms at no cost on clean speech.
3. Timestamp slots after the word. The decoder reads a word before it times it. Switchboard goes from 23.2 to 18.4 ms.
4. Transcript noise in training. With 10% of training words substituted, deleted or inserted, a wrong word in the transcript no longer throws its neighbours off.
5. A call-centre degrader. Training audio degraded the way real calls are, in place of the six fixed channels.

Two choices held up every step: training only on boundaries people placed, and training on many languages, which cut the error on held-out languages by 31% to 53% against English alone.

Results

The released models are the first seed of the recipe on the public Whisper large-v3 and large-v3-turbo checkpoints, fixed before any test score was seen.

From Section 8 of the report. Published rows are quoted from FA-Bench. Our models trained on TIMIT's training speakers, so they know its labelling convention; MFA's English model did not.

On Switchboard's real landline calls, the released models score 16.0 and 16.7 ms, against 36.5 ms for MFA. They are also more precise than MFA's own Korean and Dutch models on held-out speakers.

Limitations

• English ground truth in the benchmark. All three human-labelled sets are English, so what the benchmark says about other languages is inference.
• No external test. The trained models are tested on held-out speakers of corpora they trained on, and the Korean and Dutch test sets are small.
• Mostly simulated phone lines. Apart from Switchboard, the codecs are real and the line is not. Real networks add packet loss, transcoding chains and gain control.
• Defaults only. Every existing aligner ran at its documented defaults. A tuned configuration, MFA's especially, may rank differently.