MAYAK Built a 58-Block QA Pipeline That Caught 9 of 10 Traps in a Spec

The author of the MAYAK project has built a pipeline that takes a technical specification all the way to test execution through 58 specialized blocks. On a synthetic spec with ten planted defects, the automation caught nine. A human found the tenth, but with the help of a questionnaire the machine had drawn up. In 90 seconds the system produced 26 questions for stakeholders and flagged 20 gray areas, and in 26 minutes it wrote 37 BDD scenarios.

Documentation before buttons

The project's name sets the order of work: sort out the requirements first, then write and run the tests. The chain looks like this: "spec → canonicalization → test design → execution." Judging by the name of the stage, canonicalization converts scattered requirements text into a single structured form. Test design is then built on top of it. The logic is clear: tests derived from a vague spec inherit its vagueness and only disguise it in a tidy format.

The project belongs to a growing class of systems in which an LLM works not through a single prompt but through a long pipeline of narrow steps. Here there are 58 such steps.

Trust gates instead of guessing

The main problem with language models in test design is their tendency to fill in missing information with plausible specifics. The spec says "the page should open quickly." The model readily produces "no more than 2 seconds" and presents it as an acceptance criterion. The MAYAK author closes this hole with two rules:

• every acceptance criterion is grounded in a quote from the source document: no quote, no criterion;
• a gray area becomes a question for a stakeholder, not a guessed value.

The third detail is economic. Repeat runs are free thanks to the cache. With 58 blocks, that matters: the spec gets revised as questions are answered, and without a cache every iteration would be paid for again.

What the run showed

The test used a synthetic spec into which the author had planted ten "diseases" in advance. The results from the announcement:

• the automation caught 9 of the 10 traps;
• a human found the tenth, relying on the machine's questionnaire;
• 26 questions for stakeholders were generated in 90 seconds;
• 20 gray areas were identified;
• 37 BDD scenarios were ready in 26 minutes.

The numbers need caveats. The author devised the traps himself, the document is synthetic, and there was a single run. This is a proof of feasibility, not a benchmark. Real specs are usually longer, more contradictory, and in places written by people who do not know what they want. The author himself promises to explain where the platform lies and what remains unfinished. Those sections are in the full text on Habr; the announcement has no details, so the scale of the errors cannot yet be assessed.

The fact about the tenth trap is interesting in its own right. The system did not catch it on its own, but its questionnaire helped a human do so. In practice this is not a weakness but a sound setup: the machine narrows the search area, and the human makes the decision.

What to take away

You can borrow several techniques from the project without copying it wholesale:

• Run your own pipeline on a document with pre-planted defects and count the share found. Without such a reference, talk about quality remains opinion.
• Require the model to cite a source for every acceptance criterion, and automatically discard anything without one.
• Measure not only the number of generated scenarios but also the share of questions that stakeholders actually answered. A good questionnaire is useless if nobody answers it.
• Cache intermediate block results so that editing one paragraph of the spec does not restart the entire pipeline.
• Keep a human on the final review. Nine out of ten on synthetic data is not yet a guarantee on a production document.

If you have accumulated an archive of specs with problems that were identified after the fact, it is a ready-made test set for this approach.