An LLM for Speech Recognition: Adapting Nvidia's Architecture to Russian

Automatic speech recognition (ASR) has undergone a fundamental paradigm shift in recent years. Classic architectures that relied on separate acoustic and language components are giving way to hybrid solutions in which a large language model (LLM) acts as the intelligent decoder. Nvidia has led this movement with its SALM architecture, and engineers on the speech technology team at Kontur set out to adapt this SOTA model to the specifics of spontaneous Russian speech in real business conditions. Recognition accuracy has already come close to the limit of human perception, so competition has moved to deep contextual understanding and to handling the difficult audio conditions of telephony and video conferencing.

For a long time, ASR systems improved by refining acoustic models such as CTC or RNN-T. Then transformers and GPT-like models arrived, and it became clear that transcription quality could improve dramatically if the system did more than match sounds to letters and actually understood what was said. The release of Canary-Qwen-2.

5B, built on the Speech Augmented Language Model (SALM) architecture, became an important milestone for the open-source community. The architecture does not simply convert audio to text. It uses the linguistic capabilities of an LLM to correct errors, predict word endings, and handle complex syntax.

For Russian, with its rich morphology and free word order, this approach opens up possibilities that were previously out of reach for traditional systems.

Technically, SALM seamlessly connects a speech encoder to a language model through a special trainable adapter, or projector. The encoder extracts high-level features from the audio signal and turns the sound wave into a sequence of vectors, which are then fed into the LLM as soft prompts. In Canary-Qwen-2.

5B, the Qwen language model serves as the system's "brain": it is trained to interpret these features and generate coherent text. The Russian developers faced a serious challenge: how to adapt this power to spontaneous speech, which is full of interjections, repetitions, and broken phrases. The situation was made harder by a problem typical of the industry, a shortage of tens of thousands of hours of perfectly hand-labeled data.

Solving it required fine-tuning on domain-specific samples from telephony and corporate meetings, where teams fight for every percentage point of Word Error Rate (WER) reduction amid background noise and unstable connections.

Putting LLM decoders into speech recognition pipelines has deep consequences for the whole business analytics industry. High-quality transcription is not an end in itself but the necessary foundation for downstream data processing. Accurate recognition is critical for automatically generating meeting minutes in video conferencing services such as Kontur.Talk, or for in-depth analysis of communication quality in sales departments. When a system understands context, it makes far fewer semantic errors that could distort the substance of business agreements during later text summarization. At the same time, models with billions of parameters raise new optimization questions for engineers. High compute requirements force teams to balance exceptional accuracy against system speed so that millions of calls can be processed in real time.

The experience of adapting Nvidia's cutting-edge architecture for the Russian market confirms that the future of speech technology is inseparable from deep integration of audio and language models. Using an LLM as the decoder makes it possible to break through the "glass ceiling" of accuracy that long held back automatic minute-taking and analytics systems. The successful deployment of Canary in the infrastructure of a major technology company shows that localizing global innovations for specific language and domain tasks can deliver world-class results. Today the competitive advantage goes to those who can do more than train neural networks: they can also fine-tune highly complex architectures with precision for the realities of live human speech.