Epoch AI: AI Chip Capacity for Agents Grew 4.6x in a Year
Epoch AI calculated that over one year, the capacity of all AI chips sold to serve agents grew 4.6x, while peak compute grew only 2.6x. An agent has to hold 262 thousand tokens of context in cache, so memory and chip-to-chip interconnect matter more. Blackwell, which accounts for about 73% of the capacity, is far ahead on both. When choosing hardware for agents, compare memory and interconnect rather than headline FLOPS.
AI-processed from TG: @ai_machinelearning_big_data (Machinelearning); edited by Hamidun News
When a team picks hardware for AI agents, it usually looks at the peak teraflops in the spec sheet. A new Epoch AI analysis shows that for agentic workloads, this number badly distorts the picture. The research organization estimated how the entire fleet of sold AI chips changed from the second quarter of 2025 to the second quarter of 2026. Capacity to serve agentic inference grew 4.6x. Peak compute in 8-bit operations grew only 2.6x.
Given the uncertainty in shipment and hardware-mix data, Epoch gives a 90% interval: growth of between 4.2x and 5.1x. In other words, even under the most conservative estimate, the fleet more than quadrupled its agentic capacity.
Why agentic capacity is growing faster than teraflops
Epoch measures it with its own index, IPI. The index shows how many requests a chip can serve at the same time in a typical agentic task. In such a task, the model keeps 262 thousand tokens of context in cache: step history, documents, tool outputs. On top of that, it processes about a thousand new tokens per request.
Imagine a librarian with a huge stack of papers on his desk for each reader. His reading speed hardly matters. What matters is how many stacks fit on the desk, how fast he can pull out the right one, and how easily he can hand it to a colleague. Chips work the same way: memory capacity and bandwidth, interconnect between chips, and the ability to compute at reduced precision decide the outcome. Raw arithmetic takes a back seat.
That explains the large gaps between generations. By Epoch's index, GB300 is 25 times more powerful than H100 and 8 times more powerful than H200. On peak 8-bit operations, GB300 beats H100 by only 2.5x. By mid-2026, Nvidia's Blackwell and Blackwell Ultra chips provided about 73% of the fleet's total agentic capacity. Yet they accounted for only half of its peak compute.
What the index does not account for
The estimate has an honest limitation. IPI is built on a calculated model of the hardware's theoretical limits and does not account for the maturity of the software stack. Epoch tested it on the Kimi K3 model: real GB300 throughput came out roughly half the calculated figure. So the numbers should be read as a ceiling that software still has to grow into. How quickly it gets there depends on the specific frameworks and the teams tuning them.
Also, this is an estimate of agentic workloads with long context specifically. For short chat requests or for model training, the proportions between chips may look different.
Growth will slow
Epoch does not expect the fleet to keep growing at this pace. Chips sold in the second quarter of 2026 added 3.3 times more agentic capacity than those sold in the second quarter of 2025. That is noticeably less than the growth of the whole fleet over the same year.
The reason is that many of the biggest improvements came with the move from Hopper to Blackwell. In Epoch's model, per-GPU capacity grows 8x from H200 to GB300. From GB300 to VR200, the next-generation platform with Rubin GPUs, the gain is only 2.2x. The leap that Blackwell's memory and interconnect delivered will be harder to repeat.
What to do in practice
First: when planning infrastructure for agents, compare chips by memory capacity and speed, chip-to-chip interconnect, and low-precision support. Headline FLOPS are fine for estimating training, but they describe long context poorly.
Second: budget for software headroom. If real throughput is half the calculated figure, the choice of framework and how it is tuned can affect per-request cost as much as the choice of chip generation.
Third: do not assume that cheap agentic capacity will keep growing at the same pace. Waiting for the next generations will bring a smaller gain than the move from Hopper to Blackwell. Teams that have hit the wall on inference cost would do better to optimize context and cache now, rather than wait for Rubin.
- Nov 20, 2026Looping Layers Boosted Nvidia Dory 4B's "Thinking Without Text" by 18 Points, but Its Overall Score by Only 1.5
- Nov 20, 2026Nvidia keeps its open-source course and $12.9 billion Hugging Face deal after Huang's exit
- Nov 20, 2026Anthropic, Maker of Claude, Raises $81.3 Billion in IPO at a $1.5 Trillion Valuation
- Nov 20, 2026Anthropic raises $81.3 billion in IPO, beating SpaceX's $75 billion record
Key Questions Answered
Why agentic capacity is growing faster than teraflops?
Epoch measures it with its own index, IPI. The index shows how many requests a chip can serve at the same time in a typical agentic task. In such a task, the model keeps 262 thousand tokens of context in cache: step history, documents, tool outputs. On top of that, it processes about a thousand new tokens per request. Imagine a librarian with a huge stack of papers on his desk for each reader. His reading speed hardly matters.
What the index does not account for?
The estimate has an honest limitation. IPI is built on a calculated model of the hardware's theoretical limits and does not account for the maturity of the software stack. Epoch tested it on the Kimi K3 model: real GB300 throughput came out roughly half the calculated figure. So the numbers should be read as a ceiling that software still has to grow into. How quickly it gets there depends on the specific frameworks and the teams tuning them.
What are the key details on Growth will slow?
Epoch does not expect the fleet to keep growing at this pace. Chips sold in the second quarter of 2026 added 3.3 times more agentic capacity than those sold in the second quarter of 2025. That is noticeably less than the growth of the whole fleet over the same year. The reason is that many of the biggest improvements came with the move from Hopper to Blackwell. In Epoch's model, per-GPU capacity grows 8x from H200 to GB300. From GB300 to VR200, the next-generation platform with Rubin GPUs, the gain is only 2.2x.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.