NVIDIA Leads First Industry Benchmark for AI Agents AA-AgentPerf

NVIDIA has for the first time taken the leading position in the AA-AgentPerf benchmark — the first open multi-vendor standard measuring the performance of inference systems in real agent coding tasks. Its appearance changes the conversation about inference performance: now there is an objective industry tool instead of incomparable vendor claims.

Why Old Benchmarks Don't Work

AI agents change not only what systems do, but how they load infrastructure. Standard performance tests measure the speed of response to a single query — tokens per second or time to first token. For a chatbot this is sufficient. For an agent — it is fundamentally not.

When an agent solves a coding task, it goes through dozens of iterations: writes a function, calls a tool to run the code, reads the error output, analyzes it, rewrites it — and again in a circle until the task is solved. Each step creates a separate request to the inference system. The total latency of the entire trajectory critically affects agent productivity, and synthetic single-query tests simply cannot measure it.

Before AA-AgentPerf appeared, companies deploying agent systems in production were forced to rely on incomparable internal metrics from vendors. Artificial Analysis decided to close this gap and released the first open standard for the entire industry.

How AA-AgentPerf Works

AA-AgentPerf (Artificial Analysis AgentPerf) — the first open multi-vendor benchmark in the industry, specially developed for agent workloads. Instead of synthetic requests, it profiles complete task execution trajectories, as close as possible to real agent coding — from initial task statement to final result. The benchmark evaluates a complex set of parameters critical specifically for agent scenarios:

• Latency of the first token in multi-step interactions
• Throughput during long agent trajectories
• Stability of performance under parallel requests
• Efficiency of tool interaction and code execution
• Total time to solve realistic coding tasks

The openness of the standard is fundamentally important: any vendor can test their system and publish reproducible results. This shifts the conversation about inference performance from marketing to engineering.

NVIDIA's Position and What Stands Behind It

NVIDIA demonstrated leading performance across key metrics of the new benchmark. Behind this result are years of company investment in optimization specifically for agent scenarios. The NIM microservices architecture and the optimized TensorRT-LLM stack were designed with the understanding that agent workloads require consistently low latency for the entire sequence of interactions, not just for a single response.

"AI agents fundamentally changed the complexity of inference loads," — NVIDIA

Developer Blog.

It is also worth noting that NVIDIA has participated in AA-AgentPerf from the very first release of the benchmark. This signals to the market: the company is confident in the competitiveness of its infrastructure in open comparison with other vendors.

What This Means

The first agent benchmark redefines the concept of "high-performance inference system": now what matters is not the speed of a single response, but the efficiency of the entire agent chain from task to result. For engineering teams building agent systems in production, AA-AgentPerf becomes the first tool for justified infrastructure selection. For vendors — an incentive to optimize for real scenarios, not synthetics.