LLM agents built a Hearthstone Battlegrounds AI in 20 days that beats the top 10 human players

Researchers at MCG-NJU (Changlian Ma and Limin Wang) tested what LLM agents do better: play or build. The answer was unambiguous. Their system, AutoTavern, assembled a full Hearthstone Battlegrounds simulator in 20 days and used it to train an agent, TavernAgent, which on the live ladder outperformed the ten strongest players on the server. Its rating rose from 6000 to about 15 000 MMR, and over its last 20 games the bot averaged a 1.60 placement and 80% wins.

Battlegrounds is a hard environment for AI: eight players, random shops and combats, hidden information and nearly a thousand interacting cards. That is why the authors chose it as a test for "research" agents, meaning agents that can take a complex problem from raw data to a system that holds its own against humans.

A research factory instead of a player

AutoTavern is designed as a self-improving research system. Specialized LLM agents share a common typed research memory, every claim passes through a verification gate, and live games close the feedback loop. Based on the results, the agents improve the simulator, the opponents and the policy, as well as their own research process. The human sets goals and rules for ambiguous mechanics, acting as an arbiter rather than a developer.

The first result was TavernGym, which the authors describe as the first exact, reproducible copy of the live game with its full set of content. On top of it, the agents shipped 23 versions of TavernAgent. It is trained with PPO against opponents calibrated on real games, and it plays full ladder matches entirely on its own.

The numbers against humans and against LLMs

On the public leaderboard metric, TavernAgent posts an average placement of 1.60 and 80% first places over the last 20 records. The ten highest-rated players on the server post 2.34 and 59% on the same measures. The bot wins on every statistic.

To compare it with language models, the authors built TavernBench: 960 tasks across 40 scenarios. The main results:

• TavernAgent solves 91% of play tasks, while the best frontier LLMs solve no more than 63%;
• TavernAgent takes 0.03 s to solve a task, versus up to 26 s for LLMs;
• within one second, the budget a live move allows, TavernAgent keeps 90% of its quality, while no LLM exceeds 26%;
• ladder rating: from 6000 to about 15 000;
• average placement: 1.60 versus 2.34 for the server's top 10;
• share of first places: 80% versus 59%;
• scale of work: 20 days, 23 agent versions.

The authors state their conclusion bluntly: LLM agents are much better at building than at playing. A language model that needs tens of seconds to think through a move loses to a small specialized policy created by that model itself or by its peers.

What to take away

The main practical lesson: don't force an LLM to make every game decision. It's more effective to use it as an engineer that builds the environment, checks the rules, trains a compact model and iteratively improves it on real data. The same applies to any task with tight latency constraints: trading, logistics, robotics, simulations.

Second, verification gates and a shared typed memory look like a workable recipe for long autonomous research cycles. Without claim checking, agents quickly accumulate errors, and here the exact simulator became the foundation for everything else.

Third, the TavernGym code and the TavernBench benchmark are open on GitHub (MCG-NJU/AutoTavern). This gives the community a reproducible testbed for evaluating agent systems on a game with hidden information and randomness. Keep in mind that the results cover a single game and are reported by the authors themselves, and that the human in the loop still settled disputed questions about mechanics. Still, going from zero to a level above the server's best players in 20 days shows where applied research automation is heading.