Voice bots built on LLMs turn to Jev, a fast classifier, for decisions in 200–500 ms

A voice bot built on a large language model (LLM) speaks fluently, but on live calls it gets small things wrong. It finds an open slot and then fails to book the client. It interrupts a person who is just searching for a word.

The team at Voximplant, a Russian platform for voice automation and phone bots, has described an engineering fix. A fast classifier model such as Jev from TypeSafe takes the simple decisions away from the LLM. Jev entered early access in September 2026.

Its input is a chunk of transcript, the bot's last utterance and the current scenario step, plus a few questions. It does not generate text. What comes back is a typed structure: a choice from a set of options, a score on a short scale, or the probability of "yes" to a direct question.

Latency measurements vary. In one test, 35 questions took about 170 ms on a warm connection; in another, the result was 332–892 ms. TypeSafe claims 70–500 ms.

Why one model can't handle the whole conversation

From 2022 to 2024, Voximplant built its own classifier, Avatar. It extracted the intent from a phrase using a configurable catalog and flagged whether the phrase contained structured data: dates, names, numbers. Then LLMs arrived, and it seemed the classifier was no longer needed.

Practice quickly disproved that. Mid-sized models could not handle a long starting prompt and began to hallucinate after several topic changes. Expensive models either broke the project's economics or answered too slowly for a phone call.

Some models were unavailable in certain regions. According to Voximplant, which runs several hundred commercial projects a year, a single model misses the moment to call a tool, overflows the context window and burns extra tokens. So bots are now assembled from several parts.

When Jev came out, Voximplant was already replacing Avatar with a setup built around a small, fast model.

Jev is not the only model in this class. Others have appeared alongside it:

• Laya: a free open-source analogue under the Apache 2.0 license.
• Clef from Cloudflare: an open model for very fast decisions inside software systems.
• CMF from Cortiq: an open model weighing 304.5 MB that runs in milliseconds without generating tokens.
• Bespoke Nimble from Bespoke Labs: an open recipe based on Qwen3.5-9B with LoRA fine-tuning and constrained decoding.
• Kev from @jaredpalmer: it started as Kev-0.5B, built on Qwen2.5-0.5B for local runs on laptops, and is now a family of models.
• CLM-8B from Stanford and Nvidia: it caches repeatable agent actions and, by the authors' measurements, is faster than Jev.

Five tasks handed to the classifier

Topic routing. Piotr Zaczek put a classifier at the entry point of a voice menu. It picks a small agent with a short prompt and a single set of tools for one topic. The topic-change decision took 0.5 s instead of almost 5 s when a large model made it, and fewer tokens of the expensive model were spent.

Action verification. Giorgi Lomidze kept Gemini Live as the speech model in Pocos, a phone assistant for salons and studios. Every tool call it makes is first checked by Jev with questions such as "did the caller confirm the action?" and "did the caller say goodbye?" The thresholds are as follows. Agreement counts as strong at 0.9 and above and as weak below 0.1. A farewell counts as strong at 0.75 and above (and only if the person has not started a new topic). The principle: the LLM proposes, Jev scores, business logic decides.

Semantic end of turn. Acoustic detectors catch the pause: at Voximplant these are Silero VAD, Pipecat Smart Turn and the Hush noise suppressor. Jev adds a check on the text. If the detector says "end" but Jev sees a broken-off thought, the bot stays silent. The delay is about 250 ms: roughly 50 ms for speech recognition and 200 ms for classification. Supreet Tare of Taritas asks one question after 350 ms of silence and gets an answer in 0.4 s. After that, the code waits 1.7 s when Jev is confident the speaker has finished (0.75 and above), 2.8 s when it is unsure, and 4.5 s if the phrase broke off on "and," "but" or a comma.

Interruptions. Jay Albo, working on the Agora platform, gets four probabilities from a single call (about 300 ms): whether the person has finished the thought, whether they are asking the bot to wait, whether the utterance is addressed to the bot, and whether they are interrupting the bot or just saying "uh-huh."

Form filling. Jason Stiles built a line for a fictional clinic on Twilio ConversationRelay with Deepgram and ElevenLabs. Each turn costs one request with 30–35 questions. The code itself cuts candidates out of the transcript (numbers for the year of birth, short phrases for the name), and Jev picks the right one. It cannot invent a date; any error always traces back to what the caller said. The author of the review disputes this approach: there are more efficient ways to extract data.

Expensive answer or cheap one

Deepesh Jayal of Evalgent divides answers into three classes: a ready-made canned reply with no model, a fast model for a short fact, and a heavy model for ambiguous cases and those where a mistake is costly. On every turn, Jev decides whether an expensive answer is needed. The rules are:

• Low confidence moves the turn up one class.
• Questions with a high cost of error do not go to the cheap class, even if the classifier is confident.
• The class is not downgraded mid-call: the savings are small, and the expensive model's context can be corrupted.

The order for building the cascade

Voximplant suggests this sequence: first end of turn and interruptions, then topic routing, then form filling, then verification of state-changing actions and selection of the model class. Confidence thresholds will have to be tuned for each project. For engineers, the takeaway is practical. Anything that can be reduced to a yes/no question or a choice from a list is cheaper and more reliable to hand to a fast model. The large LLM stays where a live answer is needed.