How to Measure an AI Agent's Work in QA: The Story of One Benchmark

When an AI agent works on testing tasks, one question gets painful: is it actually better than it was yesterday? You can't answer that casually. You need numbers. Mikhail Fedorov, a developer of QA Assist, ran into this problem head-on. QA Assist is a system of 11 AI agents that together cover the whole testing cycle, from decomposing requirements to producing ready-made autotests. But how do you tell whether the system improved after the latest update? Eyeballing it is unreliable.

Why eyeballing doesn't work

Subjective assessment can go wrong anywhere: the agent found 5 bugs yesterday and 7 today, but can you be sure the system itself improved, and that the test set didn't simply change? Different model versions, different prompts, different LLM temperature settings all affect the result. Without a systematic benchmark, it is hard to work out what actually helps.

Fedorov solved the problem radically: he built a separate benchmark project in which the agent works under the same rules, on the same requirements, with the same edge cases.

What the benchmark can do

• Compare different versions of the agent on the same dataset
• Test the impact of individual pipeline improvements (prompt engineering, changes to the decomposition logic)
• Experiment with models: GPT-5.5 vs Claude vs others
• Track progress over time, with visualization of improvements
• Generate a full report on the percentage of bugs found, missed bugs and false positives

Important: a benchmark does not mean a "perfect test set." It means a controlled test set with as few variables as possible, where every run is reproducible.

Artifacts from a single run

For each run, the agent produces a complete package: extracted requirements and their decomposition, test scenarios with steps, ready-to-run autotest code, a coverage and gaps report, and a log of accepted and rejected decisions. All artifacts are stored in a public repo, so you can see how the agent reasons across different examples. This helps not only with tracking progress but also with debugging: when the agent makes a mistake, you can see at which pipeline step and why.

What this means

For developers of QA tools, benchmarks are becoming mandatory, because they are the only way to be honest with yourself about the quality of your work. Open access to Fedorov's project shows that this kind of transparency is possible. Other teams working with AI agents in testing now know what to do from the very start.