AWS Open-Sources UAEF: A Free Test Suite for AI Agents Covering 36 Metrics

It's a familiar story: you tweak one line in the system prompt, the agent gets better at your favorite test question, and you ship the change. A week later it turns out the agent has started mixing up arguments when calling an internal API, and answers some requests with too much confidence. Nobody noticed, because the agent was checked by eye. AWS has released UAEF, a free open-source tool aimed at exactly this scenario.

What problem UAEF solves

Ordinary code is checked with tests: you change a function, run the suite, and see red. Agents don't work that way. The result depends on the model, the prompt, the set of tools, and even the order of steps the agent happened to choose this time. Swap in a newer or cheaper model, and quality improves in some places while quietly dropping in others.

According to the announcement, UAEF evaluates an agent across 36 dimensions. They cover a wide range: from how the agent uses tools to safety. The key feature is that the tool compares states and shows whether things got worse after the latest prompt edit or model replacement. This is exactly the kind of regression that users usually find first.

What is known so far

The primary source offers few details, so we rely only on what is confirmed:

• the tool is free and open, released by Amazon Web Services;
• there are 36 checks, covering both quality of work and safety;
• it works with several agent frameworks, meaning you don't have to rewrite your agent for one specific stack;
• it signals when a prompt or model change has made the agent worse.

The last point matters more than the rest. Many teams don't need a pretty report; they need a simple answer to the question: can we ship this version? If the tool gives that answer automatically, it can realistically be built into your process.

How to apply it in practice

Imagine a support agent that looks up an order, checks delivery status, and issues a refund if needed. Right now it runs on one model. You want to move to another one that is a third cheaper. Without a regression suite, you compare a dozen conversations by hand and hope for the best.

With a tool like UAEF, the logic is different:

1. Record the current version of the agent as the baseline.
2. Run the same evaluation on the new model or new prompt.
3. See which metrics dropped, for example tool-calling correctness or safety.
4. Decide whether the savings justify the drop you found.

If you already have CI, it makes sense to put this check before the release of any prompt change, not just before a model swap. Prompts break behavior just as often.

For a product manager, this provides a language for discussing quality: instead of "it seems better," you can say "better on these dimensions, worse on those." For a founder, it offers a way to avoid being locked into a single model vendor, because switching providers stops being a leap into the unknown.

What's still missing

There are caveats. The available description doesn't reveal exactly how the metrics are calculated, which frameworks are supported out of the box, whether you need your own reference scenarios, and how much a run costs once calls to evaluator models are counted. You'll have to check this in the repository and documentation.

Another caution: 36 general-purpose metrics won't replace tests tailored to your business process. They catch common failures well, but they don't know that in your company a refund above a certain amount must go to manual approval. You'll still have to write such scenarios yourself.

Still, the direction is right. Agent evaluation remains a weak spot for most teams, and a free tool from a major cloud vendor that works across different frameworks lowers the barrier to entry. The practical step is simple: take your most important agent, run it through UAEF, and save the result as a reference point before the next prompt change.