Cloud.ru QA Engineer Cuts an Autotest Task from Half a Day to 1–1.5 Hours with OpenSpec

The agent wrote the tests, the code is clean, and pytest shows green. Can you close the task? Not necessarily. A Cloud.ru QA engineer described how, after several such cases, he put together a process built on four agents and specifications. He now spends more time on requirements and risks than on the code itself.

A green test that proves nothing

The project has a feature for publishing a documentation structure to Git. If a section disappears from the structure on a repeat publication, the corresponding file must be deleted from the repository. The agent wrote a test, test_update_structure_removes_orphan_doc_file. It publishes a structure with the sections overview and setup, then publishes it again without setup, clones the repository and checks that the setup file is gone.

It looks reasonable: the test checks the real state of Git, not just the API response. But before the deletion, the agent never confirmed that the file had been there. It verified the document's presence through get_repowiki_docs(), and that method may return metadata without looking inside the repository itself. If the file never appeared on the first publication, the test would still pass: after the second publication it is absent too. The test checks the end state but not the actual transition from "the file existed" to "the file is gone."

The developer agent's self-check didn't catch this: it analyzed the test's logic but didn't run any experiments. The problem was found by another agent, the reviewer. After that, a step with repowiki_repo_clone() as clone_before was added to the test, which checks that the file exists before the repeat publication. The same class of error shows up in simple cases too: an agent checks for HTTP 400 but doesn't look at the structure of the error body.

How the process works

The author opens Claude Code and gives the main agent, the orchestrator, a Jira ticket number. The rules are described in a CLAUDE.md file in the repository: which agents exist, what each is responsible for, how to move between stages, and when human confirmation is needed. There are currently four agents:

• test-planner researches the task and the existing tests and creates an OpenSpec change. It contains proposal.md (why), design.md (approach and constraints) and tasks.md (concrete steps). The plan then goes through openspec validate --strict.
• test-qa-writer implements the tests according to tasks.md and marks completed items.
• test-reviewer checks the implementation against the specification and architectural agreements. It doesn't run the tests itself; it works only with code and artifacts.
• docs-writer updates the documentation.

The human steps in at two points. The first is after the plan: he checks whether the agent understood the task correctly, whether any cases were missed, and whether the solution fits the architecture of the test project. Disputed business-logic questions he clarifies with developers. The second is after the review: the orchestrator brings back its conclusion, and the human decides whether to rework or archive.

The change retains qa-report.md and review.md. As a result, the task keeps a history: what was planned, what was done and what comments came up.

Checking that the tests aren't nonsense

Besides the review, the author does real runs and checks test sensitivity. For example, he deliberately changes the expected value in an assert and sees whether the test fails. This isn't full-fledged mutation testing of the service, but a cheap way to make sure the assertions actually check something.

There is also an honest caveat. The developer and the reviewer may run on the same model. Separating the roles doesn't make the check fully independent, but it changes the vantage point and the evaluation criteria. It was exactly this setup that caught the story with the setup file.

What to take from this

The author estimates the gain this way: instead of half a working day, a task takes about an hour to an hour and a half, and the agents do part of the work while he is busy with something else. The estimate is approximate, and it is the experience of one person on one project. For small tasks he himself allows a simplified process.

Practical takeaways for teams that delegate tests to agents:

• Include a check of the initial state in the acceptance criteria, not just the final one. Any test of the form "after action X something changed" needs a step confirming that this wasn't the case before X.
• Don't mistake the executor's self-check for a review. A separate pass with a different role and a different prompt catches what the code's author doesn't see.
• Keep specifications and review conclusions in the repository next to the code.
• Keep a human at two decision points: approving the plan and accepting the result.

The original article goes on to examine a case involving CRUD for Git tags: there were three tests for GET /tags that checked only the name field, and there was no coverage for POST and DELETE. But the main takeaway is already visible from the description of the process. Agents take over implementation, while a human remains responsible for the requirements and the final quality.