Apple ML Research tackles ASR accuracy loss on long audio recordings

Researchers at Apple ML Research have published a paper on Segmental Attention Decoding, a method built from four architectural modifications that remove a fundamental incompatibility between speech recognition neural networks and long audio recordings. The problem has been known for a while, but until now there has been no systematic fix at the architecture level.

What is wrong with AED models

Attention-based encoder-decoder (AED) models are the leading class of architectures in automatic speech recognition (ASR). Systems of this type underpin most modern voice assistants and transcription tools.

When trained on short, segmented utterances, models implicitly learn the absolute positions of audio frames: because the acoustic context at segment boundaries is limited, the network learns to guess where it is in the sequence. None of this is spelled out explicitly. It emerges as a side effect of the training regime.

With long recordings, these cues disappear. The cross-attention mechanism is permutation-invariant: keys and values carry no information about frame order. Without explicit positional markers, the model cannot tell which frame comes first. The result is transcription errors, skipped passages and shuffled speaker turns.

Key facts from the paper:

• The method is called Segmental Attention Decoding (SAD)
• The authors are Apple ML Research researchers, 2026
• It is applied to existing AED architectures as four modifications
• The first change is explicit absolute positional encodings in cross-attention at every decoding step
• The solution does not require retraining models from scratch

How the method works

The name captures the idea: decoding is still done in segments (segmental), but through an attention mechanism (attention decoding) with explicit positional signals.

The first and key modification injects absolute positional encodings directly into cross-attention at every decoding step. This removes the permutation ambiguity: the model no longer guesses the order from indirect acoustic patterns, but receives the exact coordinates of each frame.

The other three changes are not fully disclosed in the published summary, but they aim to stabilize the attention mechanism when working with long acoustic context. Together, the four modifications are meant to let AED models read long sequences correctly without losing order.

Why practitioners should care

Production ASR systems today work around the problem with forced preprocessing: long audio is cut into manageable chunks, each is transcribed separately, and the results are stitched together. This works, but it creates artifacts at the seams, which are especially painful in dialogues, interviews and meeting recordings, where transitions between speakers are critical.

Segmental Attention Decoding aims to eliminate this preprocessing. The model should handle long recordings in their entirety, which opens a new level of quality for transcribing podcasts, court hearings and medical consultations without losses at the seams.

Apple develops its own ASR systems for Siri, Live Captions in iOS/macOS and transcription tools. Publishing the research openly signals that the method is mature and, probably, that Apple wants the academic community to verify it independently and develop it further.

What this means

Segmental Attention Decoding targets a specific, well-known engineering pain point: AED models trained on short recordings scale poorly to long ones. If the proposed modifications prove effective in full experiments, the method has a chance of becoming a standard component of ASR pipelines, from voice assistants to enterprise transcription systems.