AWS releases UAEF: one set of 36 metrics for agents on any framework

You built an agent on LangGraph, it works, and the metrics are set up. Then the team decides to move to Strands or to offload part of the logic to AgentCore, and it turns out that scoring, results storage and reports are all tailored to the trace format of the old framework. The evaluation has to be rebuilt from scratch, and the simple question "did the agent get better after the prompt change?" goes unanswered.

AWS proposes breaking this coupling. UAEF (Universal Agent Evaluation Framework) is an open-source project under the Apache 2.0 license (repository awslabs/uaef). It separates "how the framework recorded the agent run" from "how we evaluate quality." According to AWS, 12 organizations from telecom, healthcare and finance have already used it.

How it works

The input is two things: an agent and a dataset. The dataset holds queries and, optionally, reference answers. To check tool calls, the reference must also include the expected calls: which tools, in what order and with which arguments.

The pipeline then works as follows:

1. An adapter reads the native execution trace (messages, tool calls, handoffs between agents) and converts it into the unified AgentTrace format. You can feed in a ready-made trace, from logs or Langfuse. Alternatively, UAEF itself invokes the agent by ARN or HTTP endpoint on each dataset input.
2. The trace is checked for personal data (emails, phone numbers, card numbers) before being saved.
3. The engine computes metrics. There are 36 of them, organized into 7 dimensions: tool calling, response quality, responsible AI, performance, multi-turn dialogue, multi-agent coordination and reasoning.
4. The result is recorded as an experiment: metadata in DynamoDB, full results along with the judge's rationales in S3.

The key idea is the order of operations: normalization first, then evaluation. That is why an agent on one framework can be fairly compared with an agent on another on a single scale. Each metric returns a number from 0 to 1, where 1 is the best value.

Deterministic metrics and the LLM judge

There are two types of metrics. Deterministic ones are computed directly from the trace and the reference and are reproducible exactly. The rest work through LLM-as-a-judge: a model on Amazon Bedrock evaluates what is hard to formalize, such as relevance or safety. You can add your own metrics and plug in RAGAS and DeepEval.

The authors separately describe protection of the judge. The agent's response goes into the judge's prompt, which makes it untrusted input. Otherwise, an agent able to influence the judge could influence its own scores. Criteria are sent separately from the content, the content is escaped, and the judge must quote a verbatim fragment of the evaluated text. If the quote is not found in the text, the score is discarded. AWS also candidly warns that judge scores are a development signal, not a security mechanism. Runtime protection requires Bedrock Guardrails.

Version comparison and regressions

This is the most practical part. For each metric shared by two runs, UAEF computes the absolute and percentage change and labels it a regression, improvement or no change. The default threshold is 5%. Regression severity has three levels: mild (5–10%), moderate (10–20%) and severe (over 20%). Regressions are grouped by dimension.

Suppose you tweaked the system prompt. The comparison might show that response quality did not change, while tool selection accuracy dropped by 12%, a moderate regression. Without a breakdown by dimension, you would see only an average score that barely moved.

A configuration can be attached to a run: framework, model, tools, prompt, parameters and even the Git commit. One run can be designated as the baseline. Then every score is tied to the change that produced it.

Reports work on three levels: a results summary in the web interface, a statistical report and LLM insights. The statistical report is computed from scores only, without an LLM, with confidence intervals and outliers, so on the same data it is always identical. Insights, meaning hypotheses about causes and priorities for fixes, are generated by a model on Bedrock through the Python library.

What to do with it

First: don't switch on all 36 metrics at once. In its playbook, AWS advises starting from the use case. A RAG assistant needs relevance, accuracy and faithfulness from RAGAS, while an agent with tools needs tool selection, call order and parameter quality. Extra metrics cost time and Bedrock calls.

Second: UAEF can be embedded in CI/CD through the Python library, and for less technical colleagues there is a deployable service with a low-code interface. The engine is the same in both cases.

The limitations are also obvious. The tool is tightly bound to AWS: the judge runs on Bedrock, and storage is DynamoDB and S3. A 5% threshold on a small dataset is easily mistaken for noise, so look at the confidence intervals in the report. And don't confuse quality evaluation with protection: security requires separate guardrails.

For teams that run agents on several frameworks or plan to migrate, this is a ready-made foundation for agent regression tests that they would otherwise have to write themselves.