AI agent on GPT-5.5 audits 2600 TravelLine autotests in two weeks

TravelLine tester Ilsur Mingazov spent two weeks auditing the autotests of the company's hotel platform together with an AI agent. About 2600 tests out of a base of 6500 E2E scenarios went through a semantic review, while run statistics were collected across the whole base. The trigger was a planned migration of the tests from Java to .NET. The team decided that moving the tests "as is" would drag 15 years of accumulated technical debt into the new stack. As a result, about 450 obsolete tests were deleted from the screenshot testing section, and tasks for integration tests were created for several sections.

The agent's environment: code, MCP, and model choice

A list of scenario names from the TMS is not enough for this kind of work, so the agent was given full context. The working environment looked like this:

• OpenCode — the environment the agent runs in.
• Local repositories — clones of the product and of the autotest project. The agent walks the folders itself, finds methods and classes, and matches calls, without anyone copying anything into the context by hand.
• TestIT via MCP — the team first tried direct API requests, then switched to an MCP server. The agent calls ready-made tools, and the server fetches entities, metadata, and test runs.
• GPT-5.5 — chosen from the OpenCode catalog. The lightweight Flash versions were ruled out right away: when matching architecture and complex logic, they lost details. GPT-5.5 held a long prompt with the rules of the test pyramid, called MCP tools accurately, and did not invent links between tests and code.
• System prompt and response contract — criteria for splitting tests into E2E and integration, plus a strict markdown report format with no filler.

The first thing to be tested was the agent itself. A request to review a section returned an incomplete list. The reasoning logs showed that the agent relied on test suites, while the tester expected a tree of sections and subsections. A screenshot of the tree and a detailed explanation of where to take tests from fixed this. The team then wrote the workflow into the agent's instructions.

First iteration: migrate, move down, or leave alone

In skill.md, every test had to receive one of three labels:

• MIGRATE_TO_NET — a current scenario that is ported to the .NET framework one-to-one;
• SHIFT_TO_INTEGRATION — business logic or an API that should go to backend developers for integration tests;
• OBSOLETE_OR_BROKEN — the functionality was removed or changed, the test is broken, duplicates another test, or is redundant as a UI check; it is not migrated.

The agent assigned critical happy paths, end-to-end user journeys, subsystem interactions, navigation, and display to E2E. It assigned calculations, validation of data combinations, saving of settings, and limits to integration, that is, everything where the browser only slows the run down.

The Transfers section is a clear example. Of 18 E2E tests, the agent matched the scenarios against existing integration tests and found that 15 of them duplicate lower-level checks, but do so at a higher cost and with a greater risk of flaky failures. The result: 3 tests in MIGRATE_TO_NET, 0 in SHIFT_TO_INTEGRATION, 15 in OBSOLETE_OR_BROKEN. A direct port would have brought 15 unnecessary scenarios to .NET: longer run times, more failures to triage, and more code to maintain.

Second iteration: P85 for failures and duration

Code does not show how often a test fails or how long it actually runs. So for each of the ~6500 tests, the team requested up to 100 of the latest results from TestIT and grouped them by section. Two metrics were calculated:

• fail rate — the share of Failed results out of the total, for example 10 out of 100 gives 10%;
• P85 duration — computed over runs with Passed status; failed runs were excluded so that timeouts would not distort the picture.

Within each section, the team calculated P85 thresholds for both metrics and got four groups: frequent failures only, long duration only, the intersection, and everything else. The intersection was reviewed first. A long test is not always a bad one, though, and frequent failures may be explained by the product or the environment. The statistics shaped a review queue; they did not deliver a verdict.

In the reports section, the agent counted 245 tests, 24 051 runs, and 1598 failures, a fail rate of 6.6%. 29 tests landed in the problem zone on failures alone, 28 on duration alone, and 9 on both criteria. For these nine, the agent prepared a report in DOCX and Markdown (passed / failed / P85 passed in seconds):

• changing order parameters — 87 / 11 / 131.0;
• cancelling an operation with partial recalculation — 86 / 11 / 102.0;
• filtering data by parameters — 88 / 10 / 308.0;
• checking the time zone in reports — 78 / 10 / 170.9;
• checking periodicity in exports — 75 / 11 / 162.0;
• confirming an entity's status — 74 / 11 / 144.1;
• finalizing order checkout — 75 / 9 / 169.8;
• sending transactional notifications — 70 / 19 / 240.1;
• processing a payment and issuing a refund — 63 / 33 / 253.7.

What other teams can take from this

The human role in the process stayed the same: set the criteria, check the agent's conclusions, and argue over hard cases with developers. The agent took over finding files, reading code, matching checks, and collecting statistics. The outcome was not a report but a set of tasks. The team removed duplicates, deleted about 450 screenshot tests, created tasks for integration tests and for porting verified scenarios, and put the remaining sections in a queue for discussion.

For teams with a large test base, the lesson is practical. The agent needs access to the code and to the test management system via MCP, not an export of test names. Decision labels are more useful than descriptions, and a first check of the agent itself should be built into the plan. Tests that are both slow and unstable are sensible to review first. A configured agent with analysis rules stays on the team and can triage failures, look for existing coverage, and assess new scenarios without rebuilding the context each time.