Avito built a bot for metric reviews: the AI writes the text, plain code does the math

Every morning, an analyst opens the financial dashboard, checks whether the data has refreshed, looks at revenue and, if a number jumps, starts working out where and why. It takes half an hour a day and involves no magic, but the checks cannot be skipped. Avito BI developers Aigul Akhmetova and Valeria Smirnova decided to hand this routine to a bot. The result is Big10 Pulse, which lives in the corporate Mattermost. Their most interesting takeaway is not what you can hand to a language model, but what you can't.

Why a single alert isn't enough

A chart shows that a metric has changed, but not why. Before forming any hypothesis, an analyst has to confirm that the day's data loaded completely, that the deviation goes beyond normal fluctuation, that the comparison baseline is the right one and which segment made the main contribution. If you jump straight to an explanation, it is easy to mistake an under-loaded source for a business decline, or a holiday effect for an incident. The chart looks convincing either way, and that is the problem.

So the team did not build a universal chat for reports. They narrowed the task to a first-pass review: check data readiness, separate signal from noise, break the change down into factors and suggest where to look next.

What code does and what the model does

The most down-to-earth step comes first: the bot checks the last update date of the sources. Until the data is ready, the calculation does not run. The model plays no part here. It can smoothly explain even a load failure, and in that situation the team needs not a coherent text but no text at all. Each send is logged, so users do not receive duplicates.

Next, the bot looks for an anomaly. It compares the value with the figure from the previous week and with the seven-day average. The bounds depend on the day type and the history of the specific metric, and a separate rule suppresses the expected rebound after holidays. Ordinary code does all of this, because on identical data the system must make an identical decision. The team chose not to ask a model to "look at the chart and say whether it looks like an anomaly."

Then the bot breaks the total difference down by business area and revenue type. It ranks them not by the largest percentage but by contribution. For example, a total of minus 100 conventional units may consist of minus 80, minus 30 and plus 10, which makes it clear where to start. A small category can grow by tens of percent and change nothing in the total.

The model comes in only at the very end. It receives a short set of ready facts: relative changes are calculated in advance, and Python fills in the absolute values. The model is not given the whole report, because the more incidental connections it sees, the easier it is for it to invent a plausible explanation. The team's formula: the less freedom the model has, the more useful the answer.

Checking the text before it goes out

Even with correct inputs, the model likes to strengthen its conclusions. The calculation says the main negative contribution came from area A. The text then reads: "revenue fell because of a technical error in area A." The model knew nothing about an error, but it sounds confident.

So each note goes through two checks. The first filters out placeholders, template replies and messages without data. The second compares the text with the original set of facts and looks for numbers and claims that were not in it. The key point is that the checking component sees the same facts as the generator. Otherwise one probabilistic model is simply judging how convincing another one is, not whether it is accurate.

The bot distinguishes three things: the anomaly (the metric went outside its bounds), the contribution (where it happened) and the cause (why). A human usually confirms the cause, sometimes with the help of additional sources. If these stages are mixed up, the system starts presenting hypotheses as established facts.

What else had to be solved

A subscription stores the slice, language and report type, and is configured with buttons in Mattermost. If several people subscribe to the same slice, the calculation is reused. After the digest, a user can ask a follow-up question, and the answer is checked again. Access rights do not change: the bot will not show anything the user would not see in the source system. The team monitors usage and failures in Grafana and treats feedback from managers and executives as part of the product.

Results and limitations

During testing there were 9 subscribers, and the team estimated the savings at roughly 99 person-hours per month (9 people × 30 minutes × 22 working days). Later, another 55 users subscribed on their own. This is a calculated estimate, not a timed measurement. A more telling change is in the workflow: instead of spending 30–60 minutes checking the report, a person spends one or two minutes reading the finished review.

The bot does not replace the analyst and does not find the root cause of every movement. It waits for correct data, spots deviations and leads a person to the first testable hypothesis. The article gives no figures on the effect on decision quality or on the accuracy of the notes.

What to take away

If you are building a similar AI assistant for metrics, start with the boundary of responsibility. Data readiness, thresholds, seasonality and contribution decomposition should live in deterministic code. Give the model only the wording, based on a small set of verified facts. Give the validator the same facts as the generator, and account for access rights from day one. The team's most important decision was not to plug in a language model, but to stop it in time.