StoreBench: LLM Agents Run an Online Store and Lose to a Script — 49% vs. 97%

Researchers have published StoreBench on arXiv, an environment in which an LLM agent runs a mid-sized online clothing store on a production-grade commercial backend. Seven frontier models were run through 11 scenarios lasting 30–45 days and through a full simulated year. DeepSeek-V4-Pro posts the best result: it clears the threshold in 49% of task × seed cells. The scripted smart-triage policy manages 97%. Human experts, using the same tools and budget, scored 0.708 on the composite score, while the models did not exceed 0.700.

A world that doesn't wait for the agent

The authors begin with a complaint about existing agent benchmarks. In most of them, the world is static and only moves when the agent acts. The reward comes as a single final verdict, and the pass mark is set arbitrarily. Meanwhile, RL environments are now the main lever for improving LLMs in post-training, so flaws in evaluation turn directly into flaws in training.

StoreBench works differently. Customers order around the clock, suppliers change prices and drop out, and market shocks arrive with partial warning or none at all. The agent works through the same 29 tools that a live store operator uses. This tests long-horizon planning and economic decision-making under uncertainty.

Key parameters of the environment:

• 29 merchant tools, the same as a human operator has; • a windowed action budget: simulated time is determined by the number of actions taken, so model latency does not affect it; • pass thresholds calibrated against scripted reference policies; • the reward is protected against a catalog of known reward hacks; • every episode replays identically given the same sequence of actions; • 11 scenarios of 30–45 days plus a full simulated year, three world seeds, and the same reasoning effort level throughout.

A heuristic beats the frontier

The most striking number is the gap to a simple script. None of the seven models matched the smart-triage policy on average. DeepSeek-V4-Pro passes 49% of cells; the heuristic passes 97%. Humans beat all the models, but only narrowly: 0.708 against 0.700 on the composite score. The abstract does not say how the composite is calculated. It names only the best model and does not list the other six.

Over a long horizon, the picture changes. On the full simulated year under the Claude Code harness, most models improve noticeably, according to the authors. It is unclear from the abstract what exactly this gain is measured against. A likely takeaway: the scaffolding and the length of the horizon matter at least as much as the model itself.

The training experiment is the most impressive. In a GRPO run, Qwen3.5-27B was trained on just five non-overlapping tasks. Its mean composite score on held-out tasks rose from 0.136 to 0.373, roughly a 2.7x increase. For an environment with an honest, hack-resistant reward, this is a good sign: even a tiny training set transfers to new scenarios.

What to take away

The authors are releasing five example training tasks, ten sample trajectories, and the evaluation and verification tooling. The full environment and the evaluation set are closed so the benchmark does not end up in training corpora. That guards against contamination, but it means the results table cannot yet be independently re-checked. Readers have to rely on the authors' data: the preprint is not peer-reviewed, and only seven models were tested across three world seeds.

Teams building agents for operational tasks can borrow four practices from StoreBench. First, calibrate the success threshold against a simple heuristic rather than intuition: if a script passes 97%, the bar for "the model coped" should be higher. Second, derive simulated time from actions, so a slow model gets neither an unfair advantage nor a penalty. Third, keep a catalog of reward hacks and check the reward against it before RL begins. Fourth, make episode replay deterministic; otherwise, comparing runs is meaningless.

And the main practical takeaway for those commissioning agents: as long as a baseline script beats frontier models on routine operational work, autonomy should be granted in measured doses. A setup that pairs a heuristic with an LLM for non-standard cases looks more sensible than handing the whole store over to an agent.