Live Nov 18 · Cohort 3Nov 18, 19:00 MSK — free hands-on AI agents live workshop · Live Cohort 3 starts Nov 23Join Nov 18 Live →Cohort 3 (Nov 23)
ReleaseInsight · On RadarHabr AI→ original

45% of AI Reports Scored a 3 or Lower: How Atom Traced the Causes to Its Prompts

At Atom, 45% of AI reports on work meetings received a 3 or lower from the LLM judge in Langfuse. The team manually reviewed 56 such reports alongside their transcripts. It turned out that the prompt caused 45% of failures, and another 36% came from an unsuitable meeting type. Strict rules against fabrication made the model lose real tasks.

AI-processed from Habr AI; edited by Hamidun News
45% of AI Reports Scored a 3 or Lower: How Atom Traced the Causes to Its Prompts
Source: Habr AI. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceHabr AI ↗
Editor-in-Chief
Value metrics matter more than visitsvalue actions per user rose from 3.4 to 6.8
Key metric #2Of the poor scores, 7% were errors by the LLM judge itself
Key metric #3One AI report costs less than 100 rubles

When the AI LAB team at Atom first looked at the automated scores for AI reports on work meetings, the picture was unpleasant: 45% received a 3 or lower. To understand whether that was a lot, they first had to learn to measure who was using the service at all and what they were getting out of it.

Usage Metrics First

They started with an admin-panel tab listing everyone who had visited the Transcriber. It showed interest, but not what users did next. Then they added visitors, active and new users, and the number of projects and transcribed files. Next they defined "value actions": downloading a document, getting a transcript, viewing an AI report for more than a minute, giving feedback, and receiving a chat answer about a meeting. Everything else, including short visits, is treated as incidental.

After that they added DAU, WAU, MAU and "stickiness" (DAU to MAU). The main signal came from the average number of value actions per active user: from April to August it grew from 3.4 to 6.8, even though the audience grew several times over. This means growth was not driven by curiosity alone. The share of visitors who performed at least one action stays above 80%. Stickiness of 10–25% is normal for a tool like this: people come after meetings, and the rhythm is closer to weekly.

Separately, the team introduced counterbalance metrics: the number of transcription errors and the share of unsuccessful calls. Without them, a dashboard shows only a pleasant picture. For the cost layer, they track the price per request, p50 and p95 latency, and availability relative to the SLO. One report costs less than 100 rubles, and an hour-long meeting is processed in about 15 minutes.

What the Review of Bad Scores Showed

Report quality is rated by a separate judge model in Langfuse on a five-point scale. Research meetings, whose structure was designed together with UX researchers, consistently get 4s and 5s. The others fare worse: of 374 reports on work meetings from July 3 to September 22, 168 received a 3 or lower. The team manually read 56 of them alongside the transcripts and wrote a reference version for each using a stronger model.

The causes were distributed as follows: the summarization prompt, 45%; the meeting was not really a "work" meeting at all (a demo, a status sync), 36%; poor source data, 12%; an error by the judge itself, 7%. Most often the main conclusion of the meeting was lost (28 of 56), junk such as "share screen" ended up in the task list, and sections contradicted each other.

An interesting effect: when the rules were tightened so the model would not invent assignments, it began discarding real ones. Reports appeared saying "no active tasks" even though a deadline had been assigned. The rule "better to write nothing than to make something up" without a counterweight simply shifts the error to the other side.

What to Take Away

First, agree in advance on what counts as a useful action for each product. There is no single definition of an active user: if code review runs automatically, is someone who clicked nothing still active?

Second, do not trust the LLM judge's average score. Read at least fifty of the worst cases in full and sort the causes into categories. Some of the "bad" scores will turn out to be a problem with the document type or the judge, not the prompt.

Third, version your prompts and check changes against reference examples; otherwise, fixing one defect will quietly create another.

Story

Key Questions Answered

What are the key details on Usage Metrics First?

They started with an admin-panel tab listing everyone who had visited the Transcriber. It showed interest, but not what users did next. Then they added visitors, active and new users, and the number of projects and transcribed files. Next they defined "value actions": downloading a document, getting a transcript, viewing an AI report for more than a minute, giving feedback, and receiving a chat answer about a meeting. Everything else, including short visits, is treated as incidental.

What the Review of Bad Scores Showed?

Report quality is rated by a separate judge model in Langfuse on a five-point scale. Research meetings, whose structure was designed together with UX researchers, consistently get 4s and 5s. The others fare worse: of 374 reports on work meetings from July 3 to September 22, 168 received a 3 or lower. The team manually read 56 of them alongside the transcripts and wrote a reference version for each using a stronger model.

What to Take Away?

First, agree in advance on what counts as a useful action for each product. There is no single definition of an active user: if code review runs automatically, is someone who clicked nothing still active? Second, do not trust the LLM judge's average score. Read at least fifty of the worst cases in full and sort the causes into categories. Some of the "bad" scores will turn out to be a problem with the document type or the judge, not the prompt.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…