AWS lays out five patterns for evaluating deep AI agents

AWS and LangSmith have published a detailed guide to evaluating deep AI agents: autonomous systems that solve multistep tasks on their own, reasoning and making decisions along the way.

Five evaluation criteria

AWS's main conclusion is that evaluating an agent on a single parameter is wrong. A comprehensive approach is needed. The company proposes five areas of checks, each covering a different aspect of the agent's work:

• Result correctness: did the agent give the right final answer to the user's question
• Solution trajectory: which path the agent chose, whether the steps are logical, and whether the reasoning contains obvious errors
• Tool management: which APIs, services and databases the agent called, and whether it used them efficiently
• Safety and compliance: whether the agent followed access policies and stayed within the bounds of permitted actions
• Decision transparency: whether a developer can understand the logic behind each of the agent's decisions

In early prototypes, the focus is on correctness and logic. In a production system, especially a critical one, priority shifts to security, monitoring and the ability to explain each of the agent's decisions.

Offline testing and live monitoring

AWS described a two-level approach: control before launch and control after.

The first level is offline testing during development. You write tests in pytest, give the agent predefined inputs and check whether it returns the right answer. It is classic unit testing, but for AI systems: a set of questions, expected results, a match check.

LangSmith adds call trace tracking. While the agent runs, the tool records every step: which sub-questions the agent asked itself, which services it called, and how it moved from one step to the next. If the result is wrong, you can see exactly where the error occurred and fix it.

The second level kicks in after deployment to production. When the agent works with real users, LangSmith keeps watching. The system tracks metrics in real time: request response time, error rate, task success rate and the duration of each intermediate step. If metrics start to degrade, an alert fires automatically.

A text-to-SQL agent as a full example

AWS built a demo agent that translates plain language into SQL queries against databases. The user writes: "Show the top 10 customers by sales volume this quarter." The agent parses the request, builds the SQL command, runs it against the database and returns a results table.

This example covers all five evaluation criteria in full: correctness of the final result, the logic of the steps, choosing the right tools (which tables to query), safety (not going beyond the available data) and the ability to understand why the agent produced this particular SQL command.

The agent is deployed on Amazon Bedrock, a managed cloud service for working with large language models. Bedrock handles infrastructure scaling, fault tolerance and compliance with security standards. The developer concentrates on the agent's logic, while Bedrock provides reliability and performance.

What this means

Until now, evaluating complex AI systems was an art: you launch an agent, look at the result and guess why one thing or another happened. AWS and LangSmith bring engineering discipline. When you can see the full trace of an agent's decisions and check it step by step, you can not only catch an error but also prevent it during development.

For large, critical systems, where an agent manages payments, access to confidential data or makes important business decisions, this moves from the "nice to have" category to the "must have" category.