How to Stop Guessing and Start Measuring AI Agent Quality

Every team that has shipped an AI agent to production eventually runs into the same nightmare. A user writes to support: "your bot is talking nonsense." The developer opens the logs, looks at the prompt and the response, and can't tell what went wrong. Is it a regression from yesterday's commit? A side effect of a model change? Or just an unlucky edge case that has always been there? The Bitrix24 team went through every stage of this process with its AI agent Marta, and is now sharing lessons that anyone working with large language models in production should take on board.

Marta is an AI assistant inside the Bitrix24 ecosystem that works with the CRM, manages tasks and answers user questions. In the early days everything looked simple: open a chat, ask a question, eyeball the answer. That is classic manual testing, and it works as long as the agent can do ten things. But once Marta's functionality grew, the approach started to come apart. One engineer physically cannot run two hundred scenarios after every prompt edit. And in modern AI product development, prompt edits are not the exception but daily routine.

The problem the team describes is systemic. Prompt engineering is inherently unstable: the smallest change in how an instruction is worded can unpredictably affect the model's behavior across dozens of different contexts. Add periodic model version changes from providers, updates to system prompts and a growing set of tools available to the agent, and you get a combinatorial explosion of potential failure points. Without automated quality control, a team is effectively working blind, reacting to problems after the fact instead of preventing them.

Bitrix24's solution was to build a full benchmark system. The idea is to formalize expectations of the AI agent as a set of test scenarios with measurable success criteria. These are not unit tests in the classical sense: a language model's answers are non-deterministic, so checking them for an exact match is pointless. Instead, the system uses metrics that assess relevance, completeness, correctness of tool calls and consistency with the intended communication tone. In essence, the team is building an automated analog of expert review that can run after every change.

Getting from idea to a working system took about six months. That is an honest figure, and it says a lot about the maturity of tooling in this area. Off-the-shelf solutions that can be plugged into an arbitrary AI agent barely exist. Each team has to decide for itself which metrics reflect the quality of its own product, how to generate and keep test datasets up to date, how to interpret results and how to build benchmarks into a CI/CD pipeline. Bitrix24 stresses that its approach is not tied to a specific stack, which is arguably the most valuable part of its experience.

Marta's story reflects a broader industry trend. As AI agents move from experiments to business-critical tools, reliability requirements are growing exponentially. Companies such as Anthropic, OpenAI and Google invest in model evaluation systems at the platform level, but at the level of individual products, responsibility for quality still rests with development teams. The problem is compounded by the fact that users quickly lose trust in an AI assistant after a few bad answers, and regaining that trust is much harder than losing it.

The cultural shift behind this transition deserves separate attention. Manual testing of AI agents is not merely an inefficient practice; it creates a false sense of control. An engineer who has checked twenty of two hundred scenarios tends to assume the system works correctly, when in fact they have tested only ten percent of the surface. Automated benchmarks do not remove uncertainty entirely, but they make it visible and measurable. And what can be measured can be improved.

Bitrix24's experience is a signal for the entire Russian-speaking AI development industry. The era when an AI agent could go to production on the strength of "seems to work" is ending. What lies ahead is an era of metrics, benchmarks and continuous quality control. The teams that master these practices first will gain a decisive advantage in the fight for user trust.