Apple ML Research proposes compact models to fix speech recognition errors

In a paper titled "Revisiting ASR Error Correction with Specialized Models," Apple ML Research researchers propose replacing bulky large language models with compact seq2seq models to correct errors in automatic speech recognition (ASR).

Why LLMs are not ideal for fixing ASR errors

Most existing ASR error correction methods rely on text models that know nothing about the error patterns specific to speech recognition. Large language models have recently been applied to the task, but they have two serious drawbacks: they add response latency, and they are prone to hallucinations, meaning they may "correct" a text so that it no longer matches what was actually said in the audio.

How the proposed solution works

The authors went back to compact seq2seq models trained on ASR errors from real and synthetic audio recordings. To scale up the training data, they build synthetic corpora through a "speech synthesis → speech recognition" cascade (TTS, then ASR again): a speech synthesizer first reads the text aloud, and the resulting audio is then run through a recognizer, which outputs realistic errors for training the corrector model.

• Method: compact seq2seq models instead of large LLMs for ASR error correction
• Synthetic training data is generated by a TTS → ASR cascade
• The key quality factor is matching the diversity of errors to the real distribution, not just data volume
• A separate decoding approach is proposed: correction-first decoding

The study's key finding is that the amount of synthetic data matters less than how closely the distribution of generated errors matches real speech recognition errors. The authors also propose a separate decoding strategy, correction-first decoding, in which the error correction step is moved to the start of the processing pipeline.

What this means

The work reflects a countertrend in the voice interface industry: rather than tackling ASR errors by plugging in ever larger LLMs, Apple is exploring narrowly specialized compact models. They are cheaper at inference and less prone to hallucinations, which is critical for voice assistants running in real time on devices with limited resources.