AWS open-sources Agent-EvalKit: a six-phase framework for systematically evaluating AI agents

AWS has released Agent-EvalKit, an open-source tool (Apache 2.0) for systematically evaluating AI agents. The framework integrates with Claude Code, Kiro CLI and Kilo Code and takes an agent through six sequential evaluation phases.

Why agent evaluation matters

Building an AI agent is not hard. Knowing how well it actually works is another matter. An agent can return plausible answers while calling unnecessary tools, burning many times more tokens than needed, or skipping critical steps in its chain of reasoning. Standard metrics such as accuracy fall short here: an agent is a dynamic system, and what matters is not only the final result but the entire path to it. The tool log, the order of calls and the intermediate decisions all affect how reliable the agent is in production. That is why the AWS team built dedicated evaluation infrastructure.

The six phases

The framework runs the agent through six stages in order:

• Task preparation — building a set of test cases with inputs, context and reference answers
• Agent run — executing the tasks in a controlled environment with full trace recording
• Trajectory evaluation — checking whether the agent called the right tools in the right order
• Final answer evaluation — comparing the result against the reference for content, structure and accuracy
• Safety analysis — checking for undesirable behavior and for straying beyond the assigned task
• Report generation — aggregating metrics and producing an overall score broken down by category

Each phase can be configured separately: you can run only trajectory evaluation, only the final report, or the full cycle.

Example: a travel agent

As a demonstration, AWS shows an agent written with the Strands Agents SDK and running on Amazon Bedrock. The agent takes a user request, for example "Plan a seven-day trip to Tokyo with a $2000 budget", searches for flights and hotels through external tools, analyzes attractions and returns a final itinerary.

Agent-EvalKit checks this agent through all six phases. It verifies that the flight search tool was called before the hotel search, that the final answer contains specific dates and prices, and that the agent stayed within budget and did not invent nonexistent flights. This kind of check surfaces errors that go unnoticed in ordinary manual testing.

Integration with AI assistants

The key difference between Agent-EvalKit and similar tools is its deep integration with AI coding assistants. Claude Code, Kiro CLI and Kilo Code can launch an evaluation directly inside the developer's working environment, with no need to switch to a separate platform or set up a separate pipeline.

The framework is distributed under the Apache 2.0 license. The source code is open on GitHub, and the documentation includes ready-made examples for several popular AI frameworks.

"We wanted to build evaluation infrastructure that developers can plug in within minutes, without building it from scratch," the authors write on the AWS

Machine Learning blog.

What this means

A standardized evaluation tool is an important step toward production use of AI agents. Without a way to systematically measure an agent's performance on real tasks, it is hard to justify deploying it in critical business processes. Agent-EvalKit offers a concrete methodology in place of manual testing.