SpireBench Puts 20 LLMs Through a Full Run of Slay the Spire: Experts Still Ahead

The authors of SpireBench tested 20 large language models on the full commercial version of the roguelike deck-builder Slay the Spire (StS) at its highest difficulty. Frontier models beat heuristic baselines and reach the level of the average human player, but they do not match experts. The gap grows with the length of the run. The best models win the first act as often as experts do, then begin to fall behind.

Why a long game is needed

Most agentic benchmarks are short and episodic. The model solves a task in a few steps, gets a score, and that is the end of it. Strategic reasoning over a long horizon remains poorly studied. A second problem is that it is hard to break down the result of a long task into three components: stored knowledge, reasoning, and luck.

StS suits this kind of test. Early decisions, such as which cards to add to the deck, determine success many hours of play later. A mistake in the first act shows up in the third. The authors claim their harness is the first of its kind to give the model the full original game rather than a simplified clone. They compare it with existing harnesses.

Where agents lose points

The act-by-act breakdown shows how the gap develops. Over a short stretch, strong models hold their own against experts, but with each successive act, accumulated strategic errors cost them more and more of their chances of winning.

Strong models differ from weak ones in how they play each individual fight. All 20, however, misjudge risk. The models protect their health, even though in games like this it is better to spend it to strengthen the deck. Caution looks reasonable in each individual decision, but over the long run it lowers the chances of winning.

Two more observations concern knowledge and memory:

• The authors systematically renamed all in-game terms. The result showed that part of the models' success rests on knowledge of the game itself acquired during training.
• Between runs, the models write notes to themselves. These notes bring no improvement. The memories often contain errors and are overly cautious.

Key results

• Sample: 20 LLMs, StS at highest difficulty, the full game.
• Frontier models outperform heuristic baselines and match the average human.
• A large gap to experts remains, and it grows with the length of the run.
• In Act 1, the best models are not behind experts in win rate.
• What separates strong models from weak ones is the quality of play in individual fights.
• A shared flaw: underestimating the value of spending health to gain power.
• Renaming terms reveals dependence on pretrained knowledge of the game.
• Self-written memory between runs gives no gain.

What agent developers should take away

The benchmark is not saturated: even the best models have room to improve, which is rare among today's tests. The authors describe it as a tool that shows where and why an agent goes wrong, rather than simply producing a final score.

Several practical conclusions follow from the results. First, evaluate an agent on early, middle, and late stages separately, because the overall win rate hides where degradation begins. Second, renaming terms is a cheap way to check whether a model is solving a task or recalling it from training data. Third, long-term memory mechanisms need to be tested for real gains. If an agent writes conclusions to itself and does not get better, the notes may entrench erroneous and overly cautious habits. Finally, risk calibration should be tested separately: a model that avoids losses may lose in situations where it needs to invest resources in future growth.

These conclusions are based on the preprint's abstract. Model names, exact win rates, and the methodology for comparison with humans appear only in the full paper.