AgentLens: Code-Agent Evaluation Shifts From Outcomes to Process

Researchers unveiled AgentLens, a benchmark for evaluating code agents, in July 2026. Unlike standard tests that reduce the result to a simple pass/fail, AgentLens analyzes the agent's entire trajectory: how it follows instructions, uses the tools available to it, checks its own code, recovers from errors and communicates with the user.

Why older approaches fall short

A typical code-agent benchmark is extremely simple: a task is given, and the agent either solves it or does not. The result boils down to a single binary value: 1 or 0, pass or fail. That is convenient for quickly ranking models, but it does not reflect the real user experience at all.

People who work with code agents in production see far more. They interact with the agent throughout its entire process, follow its reasoning, see the mistakes it makes, and watch it recover and change strategy. They ask it clarifying questions, ask it to rewrite code and point out bugs.

Standard benchmarks capture none of this. They do not explain why the agent chose a particular approach, which intermediate steps seemed logical to it, or where it got stuck. The final result may be technically correct, yet the path to it may be long, inefficient and full of failed attempts.

How AgentLens works

AgentLens combines several evaluation approaches into a single system:

• Formal verification — objective checks where they are possible (code syntax, logic, unit-test results)
• LLM-written trajectory reviews — specially trained models analyze the agent's whole path, assessing every step and decision
• Side-by-side comparisons — the system shows a visual comparison between agent versions, highlighting differences in behavior

The output of each run is not just a score from 1 to 10 but a detailed natural-language explanation. The system describes why the agent received that particular grade, which decisions were logical and correct, where it went wrong, how it got out of its errors, and whether it used tools efficiently.

Where AgentLens is used

The benchmark has proved useful for more than ranking different models. Developers use it for three main purposes:

• Behavior diagnostics — a deep understanding of where an agent systematically errs, which classes of tasks are hardest for it, and which tools it underuses
• Comparing versions of their own agent — tracking whether quality improved between releases or whether a regression appeared in some respect
• Automated monitoring — nightly test pipelines catch problems before they reach live systems

What it means

AgentLens reflects a broader trend in the AI community: code-agent evaluation is moving from simple metrics to a comprehensive analysis of the entire working process. Instead of a binary yes/no, the system now gives a full picture: which decisions the agent made, why, what worked and what did not.

This helps developers not only improve final results but also better understand how their AI systems reason, which types of tasks come easily to them and which cause problems.

The benchmark is released as open source on GitHub, which lets researchers and developers worldwide use it for their own projects.