JioStar launched a voice assistant in 15+ languages in 8 weeks, and the bet was on evals

With more than 500 million users and up to 72.5 million viewers watching live sport at the same time, "it worked in the demo" means nothing. The assistant has to answer fast, not make things up, and not bankrupt the company on inference. JioStar Chief Architect Vijay Sheshadri told the story of how his team got there at the OpenAI summit in Singapore. The interview was published on the OpenAI blog, but it contains more engineering lessons than advertising.

What they built and why ordinary recommendations fell short

JioHotstar added a conversational helper built on the OpenAI API. A viewer says what they want to watch and gets the answer in visual form. Sheshadri calls this a voice-visual interface: choosing content is essentially visual, but describing a mood or a taste is easier by voice than by clicking.

An example from the interview: "I'm sad, show me something that will cheer me up." A recommender system built on watch history will not understand what kind of sadness this is or what usually helps this particular person. In a dialogue, you can ask a clarifying question.

The queries are very diverse:

• entertainment: "it's Friday, surprise me" or a description of a movie without its title;
• sports: a run of sixes, a fall of wickets, a match where a player scored a given number of runs;
• expert: ball-by-ball statistics.

All of this went into a single interface, in 15+ languages, in about eight weeks. The deadline was a major cricket tournament in February. More than 60% of users prefer voice to text.

How they built it: boundaries, evals and a "judge for the judge"

From the start, the team treated the task as a triangle of accuracy, latency and cost. Every engineering decision moved all three vertices at once.

The order of work is interesting. First, the team decided what the assistant must not answer. A voice interface lets people ask anything, so guardrails came as the first step, not "after launch."

Then came evals. The team took a small seed set of questions and expanded it with the help of models. But they did not start with a prompt. They started with human expertise: for sports statistics, they brought in editors from sports magazines, who know what real fans will ask.

For answer quality, they used a maker-checker scheme. One model decides how to answer the question, and another checks the answer as an LLM judge, with the opposite objective. Sheshadri immediately asks the next question: who checks the judge? The validator itself becomes something that needs evaluating, and this work cannot be skipped.

Codex sped up the generation of eval cases and the updating of the system. But according to Sheshadri, fast assembly does not remove the need for judgment and validation. A small team of data scientists and engineers did the work, with OpenAI forward-deployed engineers sitting alongside them.

What to take from this

Sheshadri's main lesson: "evals, evals and more evals." Newcomers, he says, underestimate them even when they think they are ready.

The second lesson concerns production. You need a feedback loop in which user behavior and failures turn into new eval cases. Sheshadri admits that this flywheel is very hard to set up. In hindsight, he would have invested earlier in monitoring model effectiveness: P99 latency matters, but it does not show whether an answer helps the user.

The third lesson is about management. Measure outcomes, not activity: the number of lines of generated code says nothing about value. For a strong engineer, Codex multiplies output, but without the skill to verify production systems, the risk is multiplied too. Sheshadri puts it this way: you are holding a rocket, and the keys must go to the right people.

If you are building an assistant for a catalog, support or search, the checklist is short:

1. Define what the system will not answer.
2. Build a starter eval set together with domain experts.
3. Put a second LLM in place to check answers, and separately evaluate that checker itself.
4. In production, track the quality of answers, not just latency.

Limitations

This is an interview, not a technical breakdown, and it was published by a vendor. It has no figures for accuracy, latency or cost per request, no comparison with the previous search, and no data on the effect on engagement. How exactly the judge is built and how it differs from the generator is not disclosed either. So the material does not work as proof of effectiveness. As a list of engineering priorities for launching an assistant in a mass-market product, it is quite useful.