Yandex QA Team Hands Checklists and Autotests to an AI Agent on a 15,000 ₽ Monthly Budget

Every QA team has a routine nobody likes: open a ticket, work out what changed in the PR, sketch a checklist, write test cases, automate them, and then fix the tests that fail in CI. The testing team for Yandex's Main Page decided to hand a significant part of this work to an agent. They described on Habr how it works, what it costs, and where they hit snags.

From an IDE plugin to a Tracker event

The first version lived in Roo Code for VS Code. An engineer opened the editor, picked a mode, and typed "build a checklist for the task". The agent then went to Tracker on its own, read the PR, and produced the result. It worked, but only people who already know how to automate used it. Everyone else would have had to set up the environment and check access, which means swapping one manual job for another.

So the team moved the agent to an event-driven setup. When a task moves to the "Ready for testing" status, Tracker sends an HTTP POST to CI, CI spins up a container with the agent, and the tester opens the ticket to find a checklist already there. There is also a second trigger: a code word in a comment, handy for regenerating after the PR is updated. Autotests follow a similar scheme, but the agent first asks the engineer for confirmation and only then assembles the PR. This guards against spam.

Subagents instead of one big prompt

The first version was a monolith: one agent read the task, analyzed the PR, and wrote and published the checklist. It was slow and expensive because the entire context went along with every request. Now the process is split into steps: context collection, PR analysis, generation, and publishing. Validated JSON is passed between them.

The payoff is practical. Each step sees only the context it needs, each has its own logs, and when something fails, a single job restarts rather than the whole pipeline.

Three steps weren't enough for autotests. For security reasons, the agent is not allowed to use a live browser, so the locators in its first draft came out imprecise. The fix: run the test in CI (no LLM involved at that point), give the agent the report, let it correct the locators itself, and run the test again. The result is six steps, two of which are ordinary runs without a model.

There was also a small issue that annoyed many people: the PR is created by a robot, not by an engineer. So that a person can fine-tune the locators right away, the author's login is carried through the entire chain, and at the end the robot adds that person as a co-author and as an owner of the branch.

Skills and the memory bank

The agent's knowledge lives in two places, both in the repository. A skill is a step-by-step instruction, written as for a junior: what to do, which templates to use, how to check the result. For example, a test with mocks goes into integration, one without mocks goes into e2e, and checks that aren't in the case are not invented. The team has separate skills for context collection, PR analysis, exporting cases from the TMS, writing a test, code review, and fixing failures. The narrower the skill, the cheaper and more stable the step.

The memory bank answers the question "how do we do things here": a set of Markdown files covering the stack, fixtures, antipatterns, and typical failures. The rule is simple: write down only what will be useful in future tasks, with no details from a single ticket and no duplicates. The bank also costs tokens, so a short, up-to-date guide beats two hundred pages of outdated documents.

What it cost

The team of five was allocated 15,000 ₽ a month. Spending by month was 11,600, 9,300, 14,600, and 15,000 ₽. By the fourth month, the limit was used up completely. For that reason the team now tracks the cost of each step separately: it logs input and output tokens and watches where the money goes. The end of the conclusions section is cut off in the source text, so the details about request volume remain unknown.

What to take away

The scheme is not tied to Yandex or to a specific model. To reproduce it, you need CI with an external trigger, a robot account, and an agent runtime. Jira Automation or Linear Webhooks can play the role of Tracker, GitHub Actions or GitLab CI can play the role of the tasklet, and Claude Code, OpenAI Agents SDK, or LangGraph can handle orchestration.

The article gives five rules: pin the code revision, keep instructions next to the code, split the work into steps with JSON, count tokens per step, and strictly limit the agent's access.

The limitations are stated plainly too. The agent produces a draft, not a finished solution. A human checks complex scenarios, edge cases, and business logic, and also decides whether to merge the PR. Without access to the real page, the tests go through an extra self-repair cycle. It is an inexpensive way to prepare a foundation for the tester, but not a replacement for the tester.