How to Evaluate Libraries for AI Agents: transformers as a Case Study, and Traces Instead of Answers

Developers of libraries for AI agents face a problem: blindly checking an agent's final answer does not show how efficiently it got there. An agent can return the correct string and still take the most expensive route to it, reading unnecessary code, guessing API signatures, failing on errors and rebuilding the solution from zero.

Why a correct answer is not enough

For a library developer, an agent's flawed path is no longer a cosmetic issue but a signal of a new layer of quality in the tool itself. If an agent regularly spends extra steps guessing at APIs or reading code it does not need, that points to documentation, function signatures or library structure that are not clear enough for agent use, even when the end result is formally correct.

• The walkthrough is built on the transformers library
• Not only the final answer is evaluated, but the agent's path to it
• Key metrics: call traces, token consumption, execution time, errors
• Behavioral markers are recorded separately, for example repeated API guessing

What the evaluation harness includes

The evaluation harness, built using transformers as the example, collects full traces of the agent's work: which tool calls it made, how many tokens it spent, how long the solution took and which errors occurred along the way. Behavioral markers are tracked separately: repeating patterns showing that the agent is unsure about the API and works by trial and error instead of applying the right tool immediately.

What this means

As libraries and SDKs are increasingly used not directly by people but by AI agents, tool quality starts to be measured not only by how correct the documentation is for a human, but by how predictably and cheaply an agent can work with it. Evaluation based on traces and behavioral markers is becoming the new standard for developers who want their library to be convenient for agents in particular.