UserArena: A 64 729-Session Benchmark Tests Whether Shoppers Can Be Simulated

Agentic recommender systems learn to guide a shopper toward a goal through a chain of context-dependent decisions. Testing such policies on real people is expensive and risky, so the industry needs user simulators. RecGPT-Team has introduced UserArena, a benchmark that tests how faithfully a simulator reproduces the behavior of real shoppers over long sessions. The dataset contains 64 729 trajectories from 11 010 users, 566 102 products and 13 action types. Alongside the benchmark, the authors propose a training method, On-Policy Trajectory Distillation (OPTD), and show that it leads among the simulators they compared.

Why Next-Click Accuracy Guarantees Nothing

Most existing resources for recommender systems rarely contain long, detailed trajectories of real users. Even more rarely do they let researchers evaluate a whole trajectory and check offline results against live A/B tests. As a result, simulators have gotten used to competing on next-action prediction accuracy. But a person in a store does not guess a single step. They search, compare, come back, set items aside, buy or leave. A simulator can predict every local click flawlessly and still produce a pattern of behavior that resembles no real shopper.

UserArena tackles this problem from two sides. The trajectories come with an interactive environment in which the simulator acts as the user and the recommendation policy reacts to its steps. In addition, offline results can be compared with the outcomes of a controlled A/B test from production. For a developer, this is a rare chance to find out whether a simulator can be trusted when choosing between two policies.

Five Levels of Evaluation

Evaluation in UserArena is built as a ladder from simple to hard:

• Output validity. The simulator produces correct actions in a permitted format.
• Local decisions. How accurately the simulator picks the next action at a specific point in a session.
• Sequential behavior. Whether the chain of steps is realistic over the whole session.
• Personalization. Whether the behavior of different users differs in the same way as in real data.
• Outcomes. Whether the final results match those observed in production.

This breakdown shows where exactly a simulator breaks. A model can pass the first levels with confidence and fail on personalization or outcomes. A single aggregate metric will not reveal that.

OPTD:

The Teacher Guides Token by Token, the Student Acts on Its Own

To improve accuracy, the authors propose On-Policy Trajectory Distillation. The method works on-policy: the student simulator generates the responses itself, and the teacher scores them with dense token-level annotations. The key element is trajectory-level intent. It gives the teacher the overall goal of the session, so the guidance at every token stays consistent with what the user is trying to achieve over a long horizon.

The logic is straightforward. If a simulator is trained only on pre-recorded trajectories, it handles its own mistakes poorly: once it takes a wrong turn, it ends up in states it never saw during training. Training on its own rollouts with dense guidance closes that gap. And accounting for intent keeps locally plausible responses from drifting away from the overall goal.

Among the simulators evaluated, OPTD achieved the best overall accuracy under the UserArena protocol. More important, in offline simulation it came closer than the others to reproducing the policy effect recorded in the real A/B test. That metric is the closest to the practical question of whether simulation can replace some live experiments.

What You Can Use Right Now

The code and data are published in the RecGPT-Team/UserArena repository on GitHub, which also hosts a project page. Teams building agents for recommendations and shopping can use the dataset as a testbed for comparing policies offline before launching costly A/B tests. The five-level protocol is also worth applying to your own simulators: it quickly shows at which stage a simulator stops resembling real users. The OPTD approach itself applies wherever a model has to be trained on long sequences of actions with a clear overall intent.

The limitation is evident from the setup: the benchmark is built on data from a single e-commerce environment. How well the findings carry over to other markets and product categories will be shown by independent replications. The paper, by Jujia Zhao, Jiakai Tang, Jing Yu, Wen Chen, Jian Wu, Han Zhu, Fei Sun, Suzan Verberne and Zhaochun Ren, is published on arXiv (2610.30932) and ranked first in Hugging Face's daily papers.