LLM Agents Spent 20 Days Building an AI for Hearthstone Battlegrounds That Beats Top-10 Players

Researchers at MCG-NJU (Changlian Ma and Limin Wang) set out to test what LLM agents are actually better at: playing or building. The answer turned out to be unambiguous. Their system, AutoTavern, spent 20 days assembling a full Hearthstone Battlegrounds simulator and training an agent, TavernAgent, on top of it — an agent that then went on to outplay the server's ten strongest human players on the live ladder. Its rating climbed from 6,000 to around 15,000 MMR, and over its last 20 matches the bot averaged a 1.60 finishing position with an 80% win rate.

Battlegrounds is a brutal environment for AI: eight players, randomized shops and combat, hidden information, and nearly a thousand interacting cards. That's exactly why the authors chose it as a proving ground for "research" agents capable of carrying a complex problem all the way from raw data to a system that holds its own against humans.

A research factory, not a player

AutoTavern is built as a self-improving research system. Specialized LLM agents share a common typed research memory, every claim passes through verification gates, and live matches close the feedback loop. As a result, the agents improve the simulator, the opponents, and the policy — and refine their own research process along the way. A human sets goals and establishes rules for ambiguous mechanics, acting as an arbiter rather than a developer.

The first result was TavernGym, which the authors describe as the first accurate, reproducible replica of the live game with its full content set. On top of it, the agents shipped 23 versions of TavernAgent. It's trained via PPO against opponents calibrated on real matches, and it plays complete ladder matches entirely on its own.

The numbers against humans and against LLMs

By the public leaderboard metric, TavernAgent posts an average finishing position of 1.60 and an 80% first-place rate over its last 20 recorded games. The server's ten highest-rated players post 2.34 and 59%, respectively. The bot wins on every single metric.

To benchmark against language models, the authors built TavernBench: 960 tasks across 40 scenarios. The key results:

• TavernAgent solves 91% of gameplay tasks, versus no more than 63% for the best frontier LLMs;
• TavernAgent's solve time is 0.03 seconds, versus up to 26 seconds for LLMs;
• within the one-second budget of a live move, TavernAgent retains 90% of its quality, while no LLM exceeds 26%;
• ladder rating: from 6,000 to roughly 15,000;
• average finishing position: 1.60 versus 2.34 for the server's top 10;
• first-place rate: 80% versus 59%;
• scope of work: 20 days, 23 agent versions.

The authors state their conclusion bluntly: LLM agents are far better at building than at playing. A language model that needs tens of seconds to think through a move loses to a small, specialized policy — one that it, or its fellow agents, helped create.

The takeaway

The main practical lesson: don't force an LLM to make every in-game decision. It's far more effective to use it as an engineer — one that builds the environment, verifies the rules, trains a compact model, and iteratively improves it against real data. The same logic applies to any task with hard latency constraints: trading, logistics, robotics, simulation.

Second: verification gates and shared typed memory look like a working recipe for long autonomous research cycles. Without claim verification, agents quickly accumulate errors, and here an accurate simulator became the foundation for everything that followed.

Third: the TavernGym code and the TavernBench benchmark are open-sourced on GitHub (MCG-NJU/AutoTavern). That gives the community a reproducible testbed for evaluating agentic systems on a game defined by hidden information and randomness. It's worth keeping in mind that the results pertain to a single game and are self-reported by the authors, and that a human in the loop still resolved disputed mechanics. Still, going from zero to beating the server's best players in 20 days hints at where the automation of applied research is headed.