Speech neural networks scale like text models, Apple study finds

Apple ML Research has shown that speech neural networks scale as predictably as text models, thanks to a new metric called pJSD, although speech systems still trail text systems in quality.

Why speech models lag behind text models

Language models that work directly with sound, with no intermediate text representation (so-called speech-only SLMs), still fall well short of text-based and multimodal systems. It looks like a paradox: voice interfaces are in demand and audio data is plentiful, yet progress is slow.

The dominant approach, discrete autoregressive SLMs (AR SLMs), first converts the continuous audio signal into discrete tokens, much as text models work with words. The problem is that speech is structured fundamentally differently from written language: discretization inevitably loses acoustic details that carry meaning. To compensate for these losses, AR models need enormous compute and huge training datasets, which makes them expensive and hard to scale.

How continuous diffusion gets around the limits

Continuous diffusion (CD) takes a radically different approach: it works directly with continuous representations of the audio signal and skips discretization entirely. This removes information loss at the input and could make training more efficient.

A key question remained, however: do CD models obey scaling laws? These laws underpin modern large language models, predicting how quality grows with the number of parameters and the amount of data. Without predictable scaling, committing major resources to an architecture is risky: there is no guarantee the investment will pay off.

Apple examined both types of metrics. Key findings:

• CD SLMs show clear scaling laws for validation loss
• The pJSD metric (linguistic quality) falls predictably as compute grows
• The researchers identified an optimal ratio of tokens to parameters, a speech analogue of the Chinchilla law for text LLMs
• The scaling behavior of CD models closely matches that of AR models

Why a new metric, pJSD, is needed

Standard metrics, above all loss, measure how accurately a model reproduces the training distribution, but they say little about how meaningful the model's speech is from a linguistic standpoint. Apple introduced phoneme Jensen-Shannon divergence (pJSD).

The metric computes the divergence between the distribution of phonemes in speech generated by the model and in real human speech. The closer the value is to zero, the more "human-like" the speech is in a linguistic sense. Crucially, pJSD applies to both architectures, AR and CD, so fundamentally different approaches can be compared fairly on a single scale.

"To quantitatively assess the linguistic quality of speech language models, we introduce pJSD, the

Jensen-Shannon divergence at the phoneme level," the Apple ML Research paper states.

What this means

Apple's study removes a key uncertainty in speech AI: continuous diffusion scales predictably and is comparable to the autoregressive approach. That gives researchers and companies a rationale for long-term investment in CD architecture. Next-generation speech AI systems could work without intermediate text, which could make voice interaction more natural and computationally efficient.

What is a speech neural network?

It is a language model that works directly with sound, with no intermediate text representation: a speech language model (speech-only SLM).