Raft shows companies how to evaluate AI agents before putting them into workflows

Raft has published a practical breakdown of how companies can verify the reliability of AI agents before trusting them with real business processes. The article's main point is simple: an agent should not be trusted on the strength of a demo or an average success rate. It needs to be run regularly through evals with clear criteria.

Why trust is not enough

As agentic systems move from experiments into working scenarios, businesses ask a perfectly rational question: what happens if the agent makes mistakes, breaks the rules or starts behaving strangely? With a person, you can investigate an incident, change incentives and introduce controls. With AI, that does not work. A model has no incentives of its own to behave "correctly", so trust in it cannot rest on impressions, vendor promises or a single successful pilot.

The authors suggest treating trust as repeatability of results. If a system receives similar inputs many times and consistently produces the expected result, it can be given that class of tasks. If every action has to be rechecked by hand, the value of automation disappears quickly. So evals are not extra analytics here but the basic mechanism for admitting an agent to work.

How to build an eval set

The starting point is a ground truth set: a collection of real or near-real cases in which inputs are linked to an expected result. Such a set is usually assembled from historical tasks the team has already handled manually. The article stresses that evals do not need thousands of examples, as fine-tuning does. What matters more is that each case is unambiguous: two independent experts should give the same answer on whether the agent passed or not.

A typical eval set consists of several layers:

• tasks with specific inputs and a success criterion
• a test run of the agent with its final result
• one or more graders for different aspects of quality
• a transcript of steps: tool calls, intermediate actions and the logic of the route

As an example, Raft describes an online store's support agent that handles refunds. One case tests a simple refund within 30 days, another a rejection of a request outside the policy, and a third an ambiguous situation in which the agent can neither refund automatically nor simply refuse without clarification. This design shows something important: you need to evaluate not only the final answer but also the behavior along the way. Sometimes the best outcome is not an action but a correct escalation to a human.

Three approaches can be mixed for the checks themselves. Deterministic graders fit where exact signals matter, such as the refund amount or the call to the right tool. LLM judges are handy for assessing the tone, completeness and clarity of a reply. Humans are needed at the start to build reference data and calibrate the automated graders; otherwise the system will quickly begin measuring what is convenient rather than what really matters to the business.

Which metrics to watch

The article puts particular emphasis on the fact that agentic systems are non-deterministic. Strictly checking every step is therefore pointless: the same good result can be reached by different routes. But the route still matters, because it consumes time, tokens and tool access, and it can also violate internal policies. A good eval must answer two questions at once: is the outcome correct, and was the path to it reasonable?

A 95% pass rate sounds great, until the errors sit in false positives.

That is why a pass rate alone is not enough. For binary decisions it helps to look at the confusion matrix, precision, recall and F1, because different types of errors cost the business differently. An agent that approves refunds too easily creates one category of risk; an agent that rejects legitimate requests en masse creates a completely different one.

On top of this, the authors remind readers of typical traps: Goodhart's law, an eval set going stale, and the illusion of a "green" dashboard, where the metric looks fine while real user complaints grow.

What this means

For companies that want to deploy AI agents in support, operations or development, the main takeaway is one: build the verification system first, and only then scale automation. The winners are not the teams whose agent looks smarter in a demo, but those who understand the cost of its errors, can measure quality by scenario and regularly update their evals along with the product.