CyberArena Pits 12 AI Agents Against Each Other in 466 Real-Time Attack-and-Defense Matches

The CyberArena team has published a benchmark in which two AI agents sit in isolated sandboxes and simultaneously hack the opponent's service while patching their own. In total, 12 frontier LLM agents played 466 matches across 40 tasks. According to the authors, the arena still ranks models by strength in the range where single-agent tests against a static target have already hit a ceiling. The work appeared in Hugging Face Daily Papers.

Why static tests stopped working

Most existing cybersecurity benchmarks test an agent in a vacuum. There is a vulnerable target, it does not change, and the agent has to break into it. Such tests saturate quickly: models reach high scores, and the differences between them blur. They also miss the main point. In real operations, defenses change every minute, and the attacker adapts to those changes.

CyberArena is built on a symmetric design. Two agents act under identical rules, and each plays both roles at once: it attacks its opponent and defends its own service. A match runs for several rounds while the services keep changing. The static target becomes a semi-open problem that measures the speed and accuracy of three actions: finding a vulnerability, exploiting it and closing it with a patch.

To keep the benchmark from going stale as fast as its predecessors, the authors built in an automated agentic pipeline. It generates new environments, checks them and validates them on its own. Pairwise duels pose a separate problem: their outcome is hard to reduce to a single number. To address this, the researchers introduced a dynamic model. It converts the course of a match into unified multidimensional scores for offense, defense and domain skills.

What the 466 matches showed

The scale of the experiment in numbers:

• 12 frontier LLM agents;
• 40 tasks (challenges);
• 466 matches;
• scoring along three axes: offense, defense, domain skills;
• format: symmetric, parallel, multi-round, in isolated sandboxes.

The most interesting part of the paper is the agents' behavior. The authors write that the trajectories read like a fight, not an exam. Weak defenders close one hole and miss the next. They also confirm fixes using stale data: they declare a service patched without checking that it is still vulnerable right now. For live defense, this is a typical and costly mistake.

Strong agents behave differently. They think a move ahead and leave backdoors that they will use in later rounds. In one case, an agent wrote its own packet sniffer to observe its opponent's actions. Nobody asked it to build a reconnaissance tool: it decided it could not succeed without one.

There are caveats too. Judging by the abstract, 466 matches across 12 models give few games per pair. If every model played every other, that averages about seven matches per pair. How stable the rankings are will become clear from the full text and the methodology of the dynamic model. Real networks are also more complex than sandboxes with a known set of services.

What teams building security agents should do

First, do not rely on results from static task sets. An agent that passes CTF-style tests is not necessarily able to hold a defense when the opponent changes tactics. Dynamic scenarios with feedback are closer to reality.

Second, add mandatory re-verification to the defensive agent's pipeline. Every patch should be confirmed by a fresh exploitation attempt, not by a past log. According to the authors' description, this is exactly where weak defenders break down.

Third, account for dual use. An agent that can plant backdoors and write a sniffer on its own requires strict isolation, restricted network permissions and full logging of its actions. For researchers, CyberArena can become a shared testbed: the authors position it as an extensible experimental platform for future work on cyber agents.