CoSTA Compresses Step-Audio 2 mini Speech Tokens with BPE: 10K–320K Added Units

Spoken dialogue models generate their responses as long sequences of discrete speech tokens, and every token costs one autoregressive step. Researchers at the Institute of Automation of the Chinese Academy of Sciences and the School of Artificial Intelligence at the University of CAS have proposed CoSTA, a way to cut the number of these steps without touching either the original speech tokenizer or the decoder that turns tokens into audio. To do this, they add between 10K and 320K BPE units to the Step-Audio 2 mini vocabulary.

Each unit replaces a frequently occurring pair or chain of adjacent speech tokens. The model is trained to output these compressed units, and before synthesis they are deterministically expanded back into the original tokens.

Why speech tokens compress well

In most spoken dialogue models, the same token space serves both generation and synthesis. This makes sequences long: responses come more slowly, text-speech alignment gets harder, and the generation limit is used up faster. That limits the length of the conversation and of the spoken response.

The alternative is tokenizers with a lower token rate. But cutting the rate too aggressively hurts quality, and swapping the tokenizer in a finished model may require retraining the tokenizer, its paired decoder, and the model itself. Generation and synthesis also remain tied to a single representation.

The authors noticed that speech tokens contain many repeated local patterns, which makes them suitable for BPE (byte pair encoding): frequent adjacent pairs are merged into new units. The compression is reversible: expansion restores the original sequence without loss. The researchers tested the effect on eight tokenizer configurations using Chinese conversational speech. The BPE vocabulary was trained on the training split and then fixed for the held-out set. The more units added, the shorter the sequence, and the relationship is roughly log-linear. It holds across different data sources and languages.

How the pretrained model is adapted

CoSTA consists of three post-training steps. First, the vocabulary is expanded, and the embeddings and LM-head rows of the new units are initialized with the mean of the rows of the original tokens that make up each unit. This gives a more stable starting point than random initialization, and the parameters of the original vocabulary stay unchanged.

Next comes pronunciation adaptation on compressed TTS data. Step-Audio 2 mini interleaves text and speech tokens, so BPE is applied only to the speech tokens and the text tokens are left alone. Training is staged: in the first stage, on 50% of the data, only the embeddings and the LM head are tuned while all other parameters are frozen, which calibrates the new vocabulary rows. In the second stage, adaptation extends to the rest of the model. The final step is dialogue adaptation on compressed speech-to-speech data.

At inference, only a lightweight deterministic BPE expansion is inserted in front of the unchanged speech decoder. The original tokenizer and decoder stay the same, so compatibility with the existing pipeline is preserved.

What the experiments showed

The method was tested on two spoken dialogue model architectures. On Step-Audio 2 mini, CoSTA increases the generation speed of the expanded speech tokens in Chinese and English while preserving response quality in automatic and human evaluations. This leads to lower latency for a complete response, higher throughput, and longer interactions under a limited generation budget. The exact speedup factors are not given in the available portion of the text.

Vocabulary size sets a practical trade-off between compression ratio, decoding efficiency, and quality, so it has to be tuned. Experiments on the other architecture showed that the approach transfers and is compatible with low-frame-rate tokenization: the two ways of shortening sequences can be combined.

The code is published on GitHub in the yingzhuodeng/CoSTA repository.