Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseInsight · On RadarHabr AI→ original

Whisper Handles 454 IT Terms Better Than GigaAM, Even Though GigaAM Makes Fewer Word Errors

Voice tasks for AI agents break on terminology: "nash prod" ("our prod") turns into "na shprot" ("on sprats"), and the agent goes looking in the wrong code. The author compared the speech recognition models GigaAM, T-one and Whisper on 1000 recordings (20.3 hours) containing 454 terms. GigaAM has a lower WER, i.e. a lower share of wrong words, but Whisper spells IT terms more accurately. So terms need to be measured separately, on your own recordings.

AI-processed from Habr AI; edited by Hamidun News
Whisper Handles 454 IT Terms Better Than GigaAM, Even Though GigaAM Makes Fewer Word Errors
Source: Habr AI. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceHabr AI ↗
Editor-in-Chief
CPU-only test7 speakers, 18 thousand word forms, 454 of them terms
Key metric #2An identical WER hides "tmux"→"Teams" and a dropped "don't run"
Key metric #3GigaAM and T-one are fast on CPU; Whisper is noticeably slower without a GPU

Imagine you dictate a task to an AI agent and say "nash prod" ("our prod"), but the automatic speech recognition (ASR) system writes "na shprot" ("on sprats"). The agent dutifully goes looking for something about canned fish, burns tokens, and may make the wrong commit. The author of a "dark software factory", where he does most of his development by voice, set out to test which speech recognition models handle technical speech best: the Russian GigaAM, T-one and OpenAI Whisper. The result is unexpected. By WER (word error rate, the share of words that must be replaced, deleted or inserted to obtain the reference text), GigaAM wins. But Whisper renders IT terms more accurately.

How the measurement was done

The test set consists of 1000 voice recordings totaling 20.3 hours, with 7 speakers. Most of the recordings are the author's own, but he added technical podcasts from open sources so the benchmark would not be tied to a single voice. The speech is familiar to any developer: conversational Russian mixed with anglicisms, filler words and strong language.

From roughly 18 thousand unique word forms, the author picked out 454 terms and labeled each one in the audio. All models ran on CPU only (AMD Ryzen 5 3600, about 63 GB of memory), with no graphics card. There was a single run, and quality was scored against the reference labeling.

There are several metrics. WER and CER show the share of errors by word and by character; lower is better. For terms, the author counted an exact match with the required spelling, plus a softer variant in which both "backend" and its Cyrillic transliteration count as correct. This matters because some models write only in Cyrillic and will not output an English name even if they heard it correctly. RTF was also measured: recognition time divided by audio duration.

What came out

On CPU speed, GigaAM and T-one are well ahead, while Whisper without an accelerator is slow. But for a factory where tasks don't need an answer within seconds, the author treats RTF as a reference field, not a selection criterion.

GigaAM won on overall WER, and it is a strong model for Russian. On terms, however, it loses noticeably to Whisper. That leaves a fork in the road: one model writes ordinary text cleanly, while the other does a better job conveying the names of services, protocols and libraries. The post itself contains no table with the numbers, so we do not give exact gaps here.

Why WER is misleading

Every operation in WER costs one point, so two transcriptions can have the same error percentage and very different consequences. "Open the session tmuks" instead of tmux, "Open the Teams session", and a lost "don't" in "don't run automatically" all produce the same number. The first error is harmless, while the last one can launch a command nobody asked to launch.

On top of that, in long recordings a short, critical term dissolves into the overall denominator. WER does not ignore short words; their contribution to the final figure is simply small.

The author splits errors into two groups. Tolerable ones: "dashbord" (Cyrillic spelling), "dashboard", "dashik" (slang). When searching code, the agent may take a few extra steps and still reach its goal. Dangerous ones: when a plausible but wrong word appears. "Precision" turned into "presidzhen", from which the intent is still recoverable. But "recall" turned into "prikol" ("joke"/"cool thing") is not: "prikol" raises no suspicion and has several meanings. "Nash prod" turning into "na shprot" is even worse.

Where such errors come from

The model receives not letters but a log-mel spectrogram, a map of how the sound changed over time. There are no spaces on it, so word boundaries have to be guessed. The final "d" in "prod" is devoiced to "t", fast speech glues words together, and a coffee machine is humming nearby.

The ways of assembling text also differ. GigaAM CTC and T-one use CTC: at each step the model scores labels independently, including a special blank ("output nothing"), and then repeats are collapsed. If the needed letter is not on the chosen path, the collapse will not bring it back. In GigaAM RNNT, the next label also depends on those already emitted. Whisper writes token by token from a BPE vocabulary: " backend" takes one token, while its Cyrillic transliteration takes four. But more tokens do not mean more errors: "recall" takes one token, "prikol" two.

Beam search keeps several candidates but does not make the model any smarter. If a wrong string is more probable to the model than the right one, widening the beam will not fix it. Only one string comes out anyway, with no information about which candidates were discarded.

What to do about it

For those building voice interfaces to agents, the takeaway is simple: don't choose an ASR on WER alone. Build your own list of terms, label them in the audio, and measure them separately, including exact spelling. The author has already tried post-processing without changing the models, but the details are not given in the available fragment of the post.

The limitations are worth remembering too: a single run, most of the voice data comes from one person, and the accuracy of intent transfer was not measured directly in this benchmark. Still, as a template for your own measurement, the example is a good one.

Story

Key Questions Answered

How the measurement was done?

The test set consists of 1000 voice recordings totaling 20.3 hours, with 7 speakers. Most of the recordings are the author's own, but he added technical podcasts from open sources so the benchmark would not be tied to a single voice. The speech is familiar to any developer: conversational Russian mixed with anglicisms, filler words and strong language. From roughly 18 thousand unique word forms, the author picked out 454 terms and labeled each one in the audio.

What came out?

On CPU speed, GigaAM and T-one are well ahead, while Whisper without an accelerator is slow. But for a factory where tasks don't need an answer within seconds, the author treats RTF as a reference field, not a selection criterion. GigaAM won on overall WER, and it is a strong model for Russian. On terms, however, it loses noticeably to Whisper. That leaves a fork in the road: one model writes ordinary text cleanly, while the other does a better job conveying the names of services, protocols and libraries.

Why WER is misleading?

Every operation in WER costs one point, so two transcriptions can have the same error percentage and very different consequences. "Open the session tmuks" instead of tmux, "Open the Teams session", and a lost "don't" in "don't run automatically" all produce the same number. The first error is harmless, while the last one can launch a command nobody asked to launch. On top of that, in long recordings a short, critical term dissolves into the overall denominator.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…