SEAM teaches TTS new words without audio and cuts data needs by 75%

A team of researchers has proposed SEAM (Spectral Embedding Alignment and Merging), a way to add new tokens to conditional TTS models without a single audio recording at the initialization stage. Across three languages and two synthesis architectures, the method matches baseline performance or comes close to it while using up to four times less training data. Sample complexity drops by 75% compared with random initialization and subtoken averaging (mean pooling).

Why a new word breaks speech synthesis

A conditional TTS model relies on a token vocabulary. Each token has an embedding, and the decoder uses these vectors to work out how the output should sound. As long as the vocabulary covers the language and its lexicon, everything works. The problem appears when you need to add new tokens, for example for a low-resource language, a dialect, names, or technical terms. The model has never seen such a token, so its vector has to come from somewhere.

The standard approach is to initialize the vector randomly or average the embeddings of its component subtokens, then fine-tune the model on labeled "text–audio" pairs. For languages with small corpora, these pairs are the main bottleneck. They are scarce and expensive, and nobody wants to record a speaker for every vocabulary extension.

Averaging has a deeper weakness as well. Speech is sequential: "ka" and "ak" are made of the same units but sound different. Mean pooling loses this order, and the resulting vector comes out blurred.

A spectral filter and ridge regression instead of blind fine-tuning

SEAM works in two steps. First, the new embedding is built from known subtokens, but not by simple averaging: a spectral filter is applied that preserves temporal, that is, phonetic, order. Then the vector has to be "seated" in the space the pretrained model is used to. For this, the authors use ridge regression with a closed-form solution. That is an analytical formula, not a separate gradient descent loop.

This is also the main practical advantage: no audio is needed at the initialization stage. The model starts with meaningful vectors that lie on the right condition manifold, so fine-tuning does not begin from scratch. The authors describe the solution as lightweight, reproducible, and theoretically grounded.

Key parameters of the work:

• Task: expanding the vocabulary of conditional TTS models for low-resource and cross-lingual scenarios.
• Audio-free initialization: embeddings for new tokens are built from known subtokens.
• A spectral filter preserves the order of phonetic elements.
• Closed-form ridge regression aligns the vectors with the conditional manifold of the pretrained model.
• Evaluation: three languages and two TTS architectures.
• Result: baseline-level quality or close to it with up to 4 times less data.
• Sample complexity reduced by 75% relative to random init and mean pooling.

Who will benefit first

A fourfold data saving matters not to market leaders with huge corpora, but to those who have almost no recordings. These include teams building voice-over for rare languages, local voice interfaces, and assistants for specialized domains with nonstandard vocabulary, such as medicine or legal terminology.

For engineers, the method is also interesting because it breaks nothing in the pipeline. SEAM slots in as an initialization step before ordinary fine-tuning, with no new architecture and no new dataset. The closed-form alignment formula is simple to reproduce and computationally cheap, which means you can test it on your own model in a short time.

The practical takeaway: if you are expanding a TTS vocabulary and spending budget on collecting paired recordings, first compare your current baseline with subtoken-based initialization using spectral filtering. Even if you do not fully match the numbers from the paper, the potential savings on annotation and recording are substantial. The authors' results were measured on three languages and two architectures, so transfer to other models needs to be verified separately, and the first thing to try is the language where you are currently most short of audio.