ARC-AGI-3 team accelerated its agent from 500 to 3,400 tokens per second

In the ARC-AGI-3 competition, one participant finished sixth and came away with a lesson that applies far beyond game-playing agents. When the compute budget is fixed at nine hours, the cleverest prompt does not necessarily win. Sometimes the deciding factor is how many complete attempts the agent can make—and how much quality is sacrificed to achieve that speed.

The author started with roughly ten parallel game trajectories and throughput of about 500 tokens per second. After a month of work, the server delivered around 3,400 tokens per second while serving 110 trajectories simultaneously on a GPU with 96 GB of memory. Yet speed alone did not guarantee a better result: some optimizations subtly degraded the model's answers, and performance plateaued after several hours of play.

Why prompts barely helped

The participant first tried the usual set of ideas: two agents with a judge, different ways of representing the board, a dedicated game-mechanics "photographer," a programmatic world model, a swarm of advisers, and an agent offering a fresh perspective. Almost every variant produced results within the noise margin or made them worse.

The baseline agent could already receive both an image and an ASCII representation of the board, run Python for analysis, and select an action. Additional roles increased token consumption but did not remove the main constraint: the agent needed more independent attempts to complete 110 games. Under competition conditions, another elegant layer of reasoning proved less useful than several more completed trajectories.

This is a good test for any agentic system. If a new design looks smarter but does not increase the share of tasks completed successfully on a repeatable benchmark, it is not yet an improvement—only an expensive hypothesis.

How to fit 110 trajectories on one card

Most GPU memory was occupied by the model weights and its 512 experts. The game trajectories' state was therefore moved into system RAM, the KV cache was compressed to a 4-bit format, and the number of available experts was initially reduced to 400. This made it possible to increase parallelism sharply.

Batching incoming messages delivered another gain. When several games return new observations to the model at the same time, processing them together is more efficient: reading the input context parallelizes well, whereas response generation still proceeds token by token. A scheduler grouped several requests for this processing, reducing the number of costly launches.

There were smaller speedups as well: fine-tuning the draft model for the agent's context, eliminating unnecessary state recomputation after context truncation, and moving individual operations from Triton to CUDA. Each measure added only a few percent, but together they multiplied throughput several times over.

The cost appeared later. The version running at 3,400 tokens per second sometimes scored lower than the slower one. The reduced expert set and other approximations rarely changed the next token, but small divergences accumulated over a long game trajectory.

Speed matters up to the saturation point

The most practical finding was hot-switching between modes. For the first six hours, the agent ran quickly with 400 experts and launched the maximum number of trajectories. Once the easy games were complete and about 70 difficult ones remained, the system enabled all 512 experts and switched to higher-quality generation. This mode raised the score from roughly 50 to 54 points.

Beyond a certain point, more tokens barely helped. The agent quickly cleared the easy levels and then became stuck on the difficult ones: the difference between 600,000 and 900,000 tokens per game was small. This is the practical limit of scaling attempts without improving the solver itself.

The takeaway for developers is simple: do not measure only tokens per second. You need at least three charts—throughput, solution quality, and results over time. First optimize the number of useful attempts, then find the saturation point and restore quality in the final stretch. And always test every speedup on long trajectories: a microscopic error that is invisible in a single response can cost the entire task.