Habr Author Cleans Up Dictation "Ums" and "Likes" With Local Whisper and an LLM

A writer who types dozens of messages, emails, and prompts for neural networks every day has switched to dictation: talking is faster than typing. But the method has a price. Whisper can transcribe a sentence without a single error, and it will still read badly, because live speech written down word for word is full of filler words ("um," "like"), repetitions, and false starts. The author of the Habr post solves this with a combination of Whisper, running on his own computer, and a small middleman that hands the transcript to a language model for editing.

An important caveat: the available part of the post describes only the problem statement. The concrete implementation of the middleman, the choice of model, and the prompts are behind the cut, and we do not retell them. Below is what follows from the stated architecture, plus general considerations about how such setups work in practice.

Recognition accuracy and text readability are different metrics

Whisper is an open speech recognition model from OpenAI, released in September 2022 under the MIT license. It was trained on roughly 680 thousand hours of labeled audio and supports about a hundred languages, including Russian. The metric for such a model is the share of incorrectly recognized words. If Whisper captured everything that was said, there are no errors.

The problem is that people do not speak the way they write. Speech contains filler words, self-corrections ("on Wednesday, I mean, no, Thursday"), and unfinished constructions. A verbatim transcript faithfully preserves all of it. For an email, a prompt for a neural network, or a book chapter, such text has to be fixed by hand, and part of the gain from dictation disappears.

Why you need a middleman

The setup from the headline looks like this: speech is turned into text locally, then a small middleman program passes it to an LLM with an instruction to remove the excess and return a clean version. Recognition and editing are separated, and each step has its own task.

What is known about the system from the available part of the post:

• recognition is done by Whisper running on the author's computer;
• a small middleman sits between recognition and the language model;
• the LLM's task: remove "um," "like," and other noise while preserving the meaning;
• use case: messages, emails, and prompts for neural networks;
• the author dictates most of his daily text.

Where such setups break

The main risk of this kind of post-processing is that the language model starts editing instead of cleaning. It may smooth out the author's tone, replace a word with a more "correct" one, or add something that was never said. For a writer this is critical: style lives in exactly such small details. So the instruction to the model should explicitly forbid adding facts and paraphrasing, and the original transcript is worth keeping next to the cleaned version.

The second issue is privacy. A local Whisper means the audio does not go to the cloud. But if the middleman calls an external LLM, the text goes there, and it may contain names, work details, and book drafts. The setup is fully self-contained only with a local language model.

What to try yourself

If you want to replicate the approach, check recognition first. For running on an ordinary computer there is whisper.cpp, which works on a CPU, and faster-whisper, built on CTranslate2. Pick the model size to match your hardware: smaller versions are faster but make more mistakes in Russian.

Next, build a minimal middleman: receive the text, add a short instruction, send it to the LLM, return the result. Start with a single rule: "remove filler words and repetitions, add nothing and do not change the wording." Run a dozen of your own real dictations and compare the output with the original line by line. Only after that expand the prompt: punctuation, paragraphs, formatting for an email or chat. For the author's full implementation, see the original article on Habr.