Rescale built an AI agent for failed tests in 5 days: release waits cut by 20%
Every morning, a Rescale on-call engineer went through the automated tests that had failed overnight and decided which team should take each failure. While the tests stayed red, the production release was blocked. Now the first pass is done by an AI agent running on Claude, built in five days using the AI-DLC methodology. Its verdicts match human judgment in 68% of cases (it started at 44%), and the wait for a release caused by failure triage dropped by 20%. The routine sorting can be handed to the agent, while a person still reviews the result.
AI-processed from AWS Machine Learning Blog; edited by Hamidun News
Every engineering team knows this ritual. Automated tests fail overnight, the on-call engineer opens the logs in the morning, works out the cause and hands the task to the right team. Then the cycle repeats.
At Rescale, a platform for engineering calculations and simulations, these tests also block the weekly release: until the failures are triaged, new code does not go to production. Now an AI agent does the initial triage. It was built in five days, from defining the task to running on a staging environment, and it reached the production environment a few weeks later.
The results in numbers: the agent's verdicts match the engineers' verdicts in 68% of cases (it was 44% at launch), and the time that the release sits idle because of failure triage dropped by 20%.
What the agent does
The agent runs as a step in the existing CI/CD pipeline on Argo Workflows in Amazon EKS. When the nightly set of Playwright tests, with several hundred cases, produces failures, the pipeline starts a pod with the agent. It then does four things:
- sends each failure to the Claude Opus 4.8 model through Amazon Bedrock and gets back one of four causes: an infrastructure problem, a bug in the code, a flaky test, or an environment problem;
- pulls similar past failures from the knowledge base;
- uses the Model Context Protocol (MCP) to query pod logs in CloudWatch as confirmation of the verdict;
- produces a report in JSON and HTML with files, functions and recommended fixes.
All of this takes seconds. The agent does not fix anything on its own: an engineer reviews the report first. Previously, engineers had to manually go from the report to the logs, the failure history and the source code. Now the agent does this first pass, and a person checks it.
How it was built: AI-DLC and group sessions
AI-DLC (AI-Driven Development Lifecycle) is an open methodology from AWS. In it, AI drafts plans, writes code and proposes architecture, but at decision points it asks questions rather than guessing. Decisions stay with people. The work happens in group sessions: first the whole team formulates the requirements together, then they build while AI writes the code.
The initial brief was a single sentence: a system is needed that classifies failures and recommends fixes. The AI asked more than 20 clarifying questions. The output was a requirements document that separates the boundary of the first phase from the second, and four work blocks: the knowledge base, the agent itself, integration with Argo, and a quality evaluation set. Automatic pull requests were postponed.
During the work, there were two architectural turns. The agent was first meant to be placed in Amazon Bedrock AgentCore, but it was moved to EKS, where Argo already runs. No new environment had to be maintained, and the run starts directly from the pipeline. Logs were first taken from a third-party service, then switched to CloudWatch. The MCP-based scheme stayed the same; only the source changed. Because the requirements and decisions lived in shared documents, they were simply updated in place.
The depth of the design work depended on the complexity of each block. The Terraform-based knowledge base was up within hours. The agent received the full set: architecture review, interface contracts, 12 design questions and iterative code review. It is written with the Strands SDK and can analyze several failures in parallel. On the fifth day, the team assembled a PyTest evaluation framework with live and stub modes and showed the CTO how the agent correctly classifies both an infrastructure failure and a real bug with a recommendation pointing to a specific file.
Results and limitations
The figures are modest and honest. Agreement with human classification rose from 44% to 68%, and the team continues to improve the reference set of examples and the prompts. Roughly every third verdict still differs from the engineer's opinion, so a human remains in the loop. The 20% reduction in idle time is visible directly in release speed: these tests are the gate to production.
For practitioners, there are three lessons. First: start with a routine that is expensive and has a clear success criterion, such as the initial sorting of failures. Second: build the evaluation set from day one. Thanks to it, Rescale has a 44% baseline and can see where quality is heading. Third: the value of AI-DLC lies not in the code itself but in a validated specification. It is easier for an agent to write code for decisions that are already agreed than to invent the architecture and the code at the same time.
Rescale is now considering an internal AI-DLC center of excellence and is incorporating the methodology into the development of new simulation products. It is being applied, among other things, to a catalog of HPC applications, container vulnerability tracking and cloud coding agents. Caveat: this is a partner text from AWS, and the figures have not been independently verified.
- Nov 21, 2026Amazon SageMaker HyperPod Now Saves Live LLM Requests to S3 at Three Levels
- Nov 21, 2026Cloudflare Open-Sources Kitesurf: A Windowless Browser for Agents That Uses Up to 7x Less CPU and Memory
- Nov 21, 2026OpenAI Disbands Robotics Team Again: GPT-6 Astra Completed 74.8% of Tasks in Simulation Versus 51%
- Nov 21, 2026AI Warehouse Clerk Broke a Hidden Rule: DeepSeek V4 Pro Failed in 18% of Cases, Gemini 3.8 Flash in 4.8%
Key Questions Answered
What the agent does?
The agent runs as a step in the existing CI/CD pipeline on Argo Workflows in Amazon EKS. When the nightly set of Playwright tests, with several hundred cases, produces failures, the pipeline starts a pod with the agent. It then does four things: sends each failure to the Claude Opus 4.8 model through Amazon Bedrock and gets back one of four causes: an infrastructure problem, a bug in the code, a flaky test, or an environment problem; pulls similar past failures from the knowledge base; uses the…
How it was built: AI-DLC and group sessions?
AI-DLC (AI-Driven Development Lifecycle) is an open methodology from AWS. In it, AI drafts plans, writes code and proposes architecture, but at decision points it asks questions rather than guessing. Decisions stay with people. The work happens in group sessions: first the whole team formulates the requirements together, then they build while AI writes the code. The initial brief was a single sentence: a system is needed that classifies failures and recommends fixes. The AI asked more than 20 clarifying questions.
What are the key details on Results and limitations?
The figures are modest and honest. Agreement with human classification rose from 44% to 68%, and the team continues to improve the reference set of examples and the prompts. Roughly every third verdict still differs from the engineer's opinion, so a human remains in the loop. The 20% reduction in idle time is visible directly in release speed: these tests are the gate to production. For practitioners, there are three lessons.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.