AWS Strands Evals automatically finds why AI agents fail and suggests fixes

AWS has released Strands Evals, a tool for automatically diagnosing AI agent failures. Standard eval frameworks record that a run failed. Strands Evals goes further: it identifies the root cause and proposes a specific fix in the system prompt, the tool definition, or the orchestration logic.

The problem: agents break silently

AI agents are complex multi-step systems. They call tools, hit external APIs, interpret the results, and decide on their own what to do next. When something goes wrong, debugging turns into detective work. The agent gave a wrong answer, but where exactly did it break? In the system prompt? In a tool definition? In the planning logic? In how it handled a response from an external API?

Traditional eval frameworks measure the final outcome ("the agent failed the test") but don't explain why. Developers have to comb through hundreds of lines of logs by hand, form hypotheses, and experiment by trial and error. For complex agents with chains of 10–15 steps, that kind of debugging can stretch over days.

Strands Evals tackles this by adding automatic diagnostics with concrete recommendations to the eval process.

What Strands Evals can do

The tool relies on detector functions that analyze real agent execution traces and return a structured diagnosis. Key capabilities:

• Failure categorization: classifies the error type, including wrong tool selection, instruction-following violations, hallucination, planning logic errors, and incorrect response handling
• Confidence scores: each category gets a confidence score so developers know how reliable the diagnosis is
• Causal chains: the system shows which root cause led to which downstream symptoms across the agent's sequence of actions
• Fix recommendations: the detector states exactly what to change and where: in the system prompt, the tool definition, or the orchestration logic

"Structured output with categorized failures, causal chains, and fix recommendations — for every test run," is how the AWS ML

Blog team describes the tool.

How to plug it into an eval pipeline

Strands Evals fits into an existing evaluation process without a deep infrastructure overhaul. Developers call the detector functions on each test run and get an automatic diagnosis alongside the usual quality metrics: accuracy, F1, and task completion rate.

The practical payoff is most visible with large test suites. Instead of picking through hundreds of traces by hand, the team sees an aggregated picture. If 70% of "wrong tool selection" errors cluster around a particular type of request, the problem most likely lies in the system prompt, not in the tool's implementation. Those are fundamentally different fixes with different costs: editing a few lines of a prompt versus refactoring tool code.

For teams running regression tests on agents, the tool also tracks trends: which error classes grow after a prompt update and which ones disappear.

Context: the Strands SDK

Strands is an open-source agent SDK from AWS, announced in 2025. It lets developers build agent systems on Amazon Bedrock models and other LLMs, supports multi-agent patterns, and integrates with tools through the standard MCP. Strands Evals is the framework's eval component, which AWS is actively developing.

As agent systems move from prototypes into production, diagnostic tools are becoming a critical part of the DevOps stack for AI. An agent that misreads a task or picks the wrong tool may not just give a wrong answer; it may also take an unwanted action with real-world consequences.

What it means

Strands Evals closes one of the key gaps in running production AI agents: the distance between "something went wrong" and "here's exactly what broke and how to fix it." For teams building agent systems on AWS, it shortens the debugging cycle, reduces reliance on manual log analysis, and makes evals a full part of the CI/CD pipeline.