AWS Strands Evals automatically identifies root causes of AI agent failures and suggests fixes

AWS has released Strands Evals — a tool for automatic diagnosis of AI agent failures. Unlike standard eval frameworks, it doesn't just record the fact of failure, but determines the root cause and suggests specific fixes — in the system prompt, tool definition, or orchestration logic.

Problem: agents break silently

AI agents are complex multi-step systems. They call tools, access external APIs, interpret results, and independently make decisions about the next step. When something goes wrong, debugging becomes a detective investigation: the agent gave an incorrect answer — but where exactly did it break?

In the system prompt? In the tool definition? In the planning logic?

In how an external API response is processed? Traditional eval frameworks record the final quality — "agent failed the test" — but don't explain why. Developers have to manually review hundreds of lines of logs, build hypotheses, and experiment through trial and error.

With complex agents with chains of 10–15 steps, this debugging can stretch over days. Strands Evals solves this problem by adding automatic diagnosis to the eval process with specific recommendations.

What Strands Evals can do

The tool works through detector functions that analyze real agent execution traces and return a structured diagnosis. Key capabilities:

• Failure categorization — classifies the type of error: wrong tool selection, instruction-following failure, hallucination, planning logic error, incorrect response handling
• Confidence scores — each category receives a confidence score so the developer understands the reliability of the diagnosis
• Causal chains — the system shows what root cause led to what downstream symptoms in the agent's action chain
• Fix recommendations — the detector directly indicates what needs to be fixed and where: in the system prompt, in the tool definition, or in the orchestration logic

"Structured output with categorized failures, causal chains, and fix recommendations — for each test run," — the AWS ML

Blog team describes the tool.

How to integrate into eval-pipeline

Strands Evals integrates into the existing assessment process without deep infrastructure overhaul. The developer calls detector functions on each test run and gets automatic diagnosis alongside normal quality metrics — accuracy, F1, task completion rate. The practical value is particularly noticeable when working with large test suites.

Instead of manually parsing hundreds of traces, the team sees an aggregated picture. If 70% of "wrong tool selection" class errors concentrate around queries of a certain type — the problem is likely in the system prompt, not in the tool implementation itself. These are fundamentally different fixes with different effort levels: editing a few lines of prompt versus refactoring tool code.

For teams doing regression testing of agents, the tool also allows tracking dynamics: which error classes grow after a prompt update and which disappear.

Context: Strands SDK

Strands is an open-source agent SDK from AWS announced in 2025. It lets you build agent systems based on Amazon Bedrock models and other LLMs, supports multi-agent patterns, and integrates tools through the standard MCP. Strands Evals is the eval component of this framework, which AWS is actively developing. As agent systems move from prototypes to production, diagnostic tools are becoming a critical part of the DevOps stack for AI. An agent that misinterprets a task or selects the wrong tool can not just give an incorrect answer, but execute an undesirable action with real consequences.

What does this mean

Strands Evals closes one of the key gaps in working with production AI agents: the disconnect between "something went wrong" and "here's exactly what and how to fix it." For teams building agent systems on AWS, this reduces the debug cycle, decreases reliance on manual log analysis, and makes the eval process a full part of the CI/CD pipeline.