AWS shows how to evaluate AI agents in 3 modes: from debugging to production

A customer asked for a refund on an order and instructions on canceling a subscription. The agent processed the refund, politely signed off, and forgot about the subscription. Response speed looked good, and so did the tone. But the customer's task remained unfinished. In a new article, AWS explains how to find these failures using Amazon Bedrock AgentCore Evaluations: assess individual agent actions and the outcome of the entire conversation, then use those checks before release and on live traffic.

This is a practical walkthrough of the service's capabilities. Its value lies in connecting agent evaluation to the specific work a business pays for. The article itself provides no basis for claiming that the checks described improve quality by any particular percentage or reliably prevent errors.

First, define what counts as completed work

AWS suggests starting with the business outcome. For support, that might mean resolving an issue in a single interaction. For a research agent, it could mean a complete, sourced answer grounded in the documents it found. That outcome then needs to be broken down into observable actions: the agent fulfilled every request, retained important details from the conversation, chose the right tool, and passed it the correct parameters.

Back to the refund example. Three independent checks are useful here. GoalSuccessRate assesses whether the goal was achieved across the entire session. ConversationCompleteness from DeepEval helps flag the forgotten subscription question. PIILeakage checks whether the agent disclosed personal data. A successful refund alone says nothing about the other two outcomes.

The scope of evaluation matters too. The correctness of tool-call arguments can be assessed at the level of an individual action. Determining whether the agent answered every question requires the whole conversation. Looking only at the final response leaves some errors out of view.

One set of checks for three modes

AgentCore offers built-in AWS evaluators, managed checks from DeepEval and AutoEval, and custom options. The built-in evaluators cover response quality, task completion, tool use, and other behaviors. In managed mode for third-party checks, the service hosts the code, selects the evaluator model, and runs requests on its own infrastructure. Custom options let teams change the model, instructions, and evaluation criteria.

There are three ways to run them. During debugging, a developer submits a recorded session to the API and receives an evaluation with an explanation. Before release, the team runs a fixed set of conversations to see the effects of changing a prompt or model. In production, the service evaluates a specified share of live traffic. The same evaluator identifiers are used in all three modes.

Multiple criteria can also be evaluated in a single model call. For example, teams can separately check issue resolution, compliance with refund rules, and professional tone. Results are returned for each criterion, without an overall aggregate score. This helps establish independent release requirements: an improvement in tone should not offset a policy violation.

Combining checks makes sense when they need the same data and the same model. If one criterion requires a reference answer while another can be assessed from a live conversation, keeping them separate is more practical.

Where automated evaluation reaches its limits

Some checks require a known correct answer or an expected sequence of tool calls. According to AWS, these checks are available in batch mode: live traffic usually has no reference answer. That means pre-release test sets and production monitoring serve different purposes, even when they use a shared catalog.

The evaluator model itself is another limitation. Its score needs to be checked against what the team considers success. For refunds, it is useful to compare the conversation evaluation with the actual transaction outcome in the order management system. A good explanation from the model does not confirm that the money was actually refunded.

A practical starting point for a team is to choose two or three significant failure types and collect conversations that demonstrate them. Then check whether the evaluators detect those failures, and only afterward enable traffic monitoring. AWS provides no cost or savings calculations in this article. Teams will need to assess the evaluation budget and how much to trust the results using their own sessions.