Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseInsight · On RadarHabr AI→ original

QA Engineer Cuts Autotest Development From 2 Weeks to 2 Days Using an LLM

Writing autotests by hand for a script with dozens of flags is slow, and it's easy to miss a dangerous combination. A QA engineer at T1 Innotech gave an LLM the help text for a DAG sync script in Airflow, a conflict matrix, and failure logs, and got a test framework back. Development of the test suite shrank from 2 weeks to 2 days, coverage rose from 60% to 95%, and the number of scenarios grew to 600 from 200. The condition: first remove contradictions from the input data.

AI-processed from Habr AI; edited by Hamidun News
QA Engineer Cuts Autotest Development From 2 Weeks to 2 Days Using an LLM
Source: Habr AI. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceHabr AI ↗
Editor-in-Chief
Key metric #1Before the request, the author manually removed contradictions and outdated examples from the documentation.
Key metric #2A prompt with an expert-role persona produced more edge cases than a short request did.
Key metric #3The flat JSON had to be converted into a tree for Jenkins reports.

Generating tests with an LLM no longer surprises anyone. What's more interesting is how it works on a real task with imperfect documentation. Roman Polyakov, a test engineer at T1 Innotech, described his case on Habr: autotests for a DAG sync script in Airflow. He wrote them in two days instead of two weeks. Below is what he actually did and where the approach breaks down.

The task: flag combinations that are easy to miss

The script accepts a set of keys: --file, --delete, --dry-run, --exclude, and others. Some combinations are valid, and some conflict. For example, --file with --delete, or --dry-run with --exclude, must be tested separately. Coding every combination by hand takes a long time, and there is a high risk of missing a dangerous pairing.

The difficulty was not only the number of combinations. The documentation contradicted itself. One section allowed the *.py mask for --exclude, while another allowed only absolute paths. The command examples in the guide did not match the description, and the logs still contained error messages that had already been fixed in a recent version of Airflow.

How the author worked with the model

The first step turned out to be the least automated. Polyakov manually "cleaned" the data: he identified the canonical rules, discarded outdated examples, and assembled a single reference of flags. In his words, without this the model would generate test scenarios just as contradictory as the source materials.

Next, he built a request in four parts: the script's help text, the conflict matrix, command examples, and logs of failed runs. He set the output format up front: JSON compatible with CI/CD. He assigned the model the role of a senior expert who knows all the "forbidden pairings" of flags, and asked it to cover every valid combination and every dangerous one. A short request like "generate tests for the script," the author observed, produced a shallower result: the model missed edge cases.

The model returned a framework: positive scenarios (--file with --dry-run as a safe trial run), negative ones (conflicting keys), boundary cases (empty paths, nonexistent extensions in --exclude), and stubs for checking the deploy.log journal.

This cannot be called ready to use. The author reworked the script by hand:

  • added log parsing, so that he checked not only return codes but also the messages in deploy.log;
  • simulated network failures with the tc utility: bandwidth limiting and packet loss;
  • merged similar positive scenarios so as not to inflate run time;
  • wrote a converter, because the "flat" JSON from the model did not suit Jenkins, and built a tree from it with the metrics success_rate, total_time, and failed_scenarios.

The process ran in cycles: request, framework, analysis, rework, run, log analysis, prompt revision. Each round reduced manual labor but required time for verification.

What came out of it and how to interpret it

The author reports these results:

  • test development: from 2 weeks to 2 days;
  • scenario coverage: from 60% to 95%, including boundary and extreme cases;
  • number of scenarios: 600 versus 200 with manual work, at the same run duration;
  • time spent analyzing reports: 30% less;
  • no failures in production after the rollout.

The article includes other examples. For the "Counterparty" page, the model produced 80 scenarios in five minutes, and the team reworked 15% of them. For a 150-page banking specification, the author cites 120 scenarios instead of 20 written by hand, and a 30–40% faster start to testing.

These figures should be read with caution. They come from a single project and the author's self-report, not from a controlled comparison. The article does not say which model was used or how many tokens and how much money went into the iterations. The claim of "no failures in production" cannot be attributed entirely to AI: other changes also affect it. The remaining sections of the article, such as those on adaptive testing and prioritization, are, by the author's own admission, based on colleagues' experience and tool demonstrations rather than on his own deployment.

Practical takeaways

First: the quality of the input data matters more than prompt cleverness. Contradictory documentation produces contradictory tests, so it has to be consolidated into a single reference first.

Second: role and goal in the prompt do work. Framing the task as "find the forbidden pairings and don't break the pipeline" pushed the model to look for negative and boundary cases instead of repeating examples from the help text.

Third: ask for an output format your CI understands right away. But plan time for a converter: the structure of the model's response rarely matches what Jenkins or another system expects.

Fourth, the limitations the author himself describes. Documentation longer than 50–100 pages is better split into chunks or handled with RAG, otherwise the model loses the middle of the text. For sensitive logs, consider local models. The model's output remains a draft: 10–20% of the scenarios will still need human rework.

Story

Key Questions Answered

What are the key details on The task: flag combinations that are easy to miss?

The script accepts a set of keys: --file, --delete, --dry-run, --exclude, and others. Some combinations are valid, and some conflict. For example, --file with --delete, or --dry-run with --exclude, must be tested separately. Coding every combination by hand takes a long time, and there is a high risk of missing a dangerous pairing. The difficulty was not only the number of combinations. The documentation contradicted itself. One section allowed the .py mask for --exclude, while another allowed only absolute paths.

How the author worked with the model?

The first step turned out to be the least automated. Polyakov manually "cleaned" the data: he identified the canonical rules, discarded outdated examples, and assembled a single reference of flags. In his words, without this the model would generate test scenarios just as contradictory as the source materials. Next, he built a request in four parts: the script's help text, the conflict matrix, command examples, and logs of failed runs. He set the output format up front: JSON compatible with CI/CD.

What came out of it and how to interpret it?

The author reports these results: test development: from 2 weeks to 2 days; scenario coverage: from 60% to 95%, including boundary and extreme cases; number of scenarios: 600 versus 200 with manual work, at the same run duration; time spent analyzing reports: 30% less; no failures in production after the rollout. The article includes other examples. For the "Counterparty" page, the model produced 80 scenarios in five minutes, and the team reworked 15% of them.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…