Kriptonit Taught GigaAM to Understand Mixed Russian-English Speech: WER Down 13.6%

A familiar scene: on a call, someone says "let's look at the pull request and do the deploy by evening." A human notices nothing odd. A speech recognition system does: some models turn the English words into a jumble of Latin letters, others pick similar-sounding Russian words, and still others silently drop the piece they don't understand. The reason is that ASR usually determines the language from the first utterances and then sticks with it.

The AI lab at Kriptonit worked on this problem. They came up with a fine-tuning recipe that adds code-switching support to a model, meaning switching languages in the middle of a conversation. The recipe works for Whisper, GigaAM and other ASR models.

What exactly was done

The team took Sber's GigaAM Multilingual as the base. The MIT license allows modifying the model and using it in commercial products, so the result can be put to work without legal headaches.

The main difficulty is that there are almost no ready-made datasets where Russian and English speech are mixed within a single phrase. So the team built the data itself, roughly 10,000 hours of audio. They used two techniques:

• stitching together fragments in two languages from real monolingual recordings (YODAS, Sova and other corpora);
• TTS synthesis of sentences with partial translation, where part of the phrase is voiced in Russian and part in English.

Separately, they added KenLM, an open-source library for fast n-gram language models. It works at the decoding stage: the acoustic model says what a sound resembles, while the language model assesses how plausible a sequence of words is. According to the authors, KenLM reduced the error rate when alphabets get mixed within a single word. A familiar bug: the Russian word for "deploy" turns into something like a hybrid with Latin letters inside the Cyrillic word.

The results

On code-switching benchmarks, WER dropped by 13.6% on average. WER shows the share of incorrectly recognized words, so the model now makes noticeably fewer errors on mixed speech. There was also a check on monolingual speech: the authors verified that quality there did not drop after fine-tuning.

There are intermediate findings as well. Even ordinary monolingual corpora improve quality when fine-tuning an open model. Synthetic data with monolingual speech also lowers WER, but falls short of the optimum. What is needed is a mix of languages within phrases plus a language model at decoding.

What to do with this

If you are building meeting transcription, support call analysis or voice input for an international audience, there are several things you can take from this work:

• Build your own code-switch test set from your own recordings. General benchmarks will not show how the model behaves on your jargon.
• Synthetic data can be made cheaply: stitching real clips and TTS with partial translation. It is a workable way to get data where none exists.
• Always run regression tests on monolingual speech. Fine-tuning for mixed languages should not break what already worked.
• Consider an n-gram language model at decoding. KenLM is cheap and removes some of the absurd word hybrids.

Limitations

The experiment was conducted only on the Russian-English pair. The authors claim the recipe is universal, but it has not yet been verified for other languages. The 13.6% figure is an average across CS benchmarks; the material gives no absolute WER values or breakdown by dataset, so you will have to compare against your own task yourself. In addition, the process is not quick: on a cluster of four NVIDIA L40S GPUs with 48 GB of memory each, fine-tuning takes from one to three weeks, depending on the data volume.

Next, the team wants to work on Language Diarization, that is, automatically splitting a recording into chunks by language and routing them without fine-tuning the ASR. Also planned is a multilingual model based on Whisper V3 Large. If these plans work out, the approach will become noticeably easier to adopt. For now, it is a good practical recipe for those whose audio regularly features English in the middle of a Russian phrase.