CAS Institute of Software and Kuaishou Build First Benchmark for LLM User Simulation

The Institute of Software of the Chinese Academy of Sciences (CAS) and Kuaishou introduced a benchmark that tests whether a large language model can convincingly play a real human user. The authors call it the first of its kind: it relies on real data and covers the full chain of behavior, not an individual action. The work has been accepted at NeurIPS 2026. The available excerpt of the Jiqizhixin material does not include the dataset size, the list of tested models or final figures, so this article covers what is claimed and what is worth checking in the full paper.

Why user simulation has become a bottleneck

Human users are expensive. A live A/B test in a recommender system costs traffic, time and risky experiments on a real audience. So the industry is trying to replace some of the humans with models: run a new ranking strategy on a virtual audience, estimate ad conversion before launch, simulate how the public will react to a product change.

All of these scenarios rest on one assumption: the simulated user behaves like a real one. If it does not, offline evaluation is useless. Worse, it is misleading, because it looks convincing while its conclusions are wrong.

The benchmark's authors point to the main difficulty. In real life, a purchase can result from something a person happened to see several days earlier. A decision builds up from a long history of contacts, not from the last screen. A model that guesses the next click well does not necessarily reproduce that logic.

What "full chain" means

Judging by the title and description, the benchmark evaluates behavior as a sequence of interconnected events in a real environment, not as a set of separate predictions. For a platform on Kuaishou's scale, where users watch short videos, interact with content and make purchases, this is a natural setup. The chain here includes viewing, reaction and a delayed decision.

Kuaishou's involvement matters for a practical reason. Platforms have real logs of user behavior, while academic groups usually do not. Most earlier simulation work was built on synthetic personas or small public datasets, where it is easy to get a good-looking result under weak validation. A benchmark built on data from a live platform raises the bar.

What is known from the source:

• authors: CAS Institute of Software and Kuaishou;
• venue: NeurIPS 2026;
• status: billed as the first simulation benchmark with real data and a full chain of behavior;
• scenarios: offline testing of recommendation strategies, ad conversion forecasting, social simulation, evaluation of human-system interaction;
• not disclosed in the excerpt: data volume, list of models, metrics and final results.

How to read such benchmarks in practice

Until the full paper is at hand, there are several questions worth asking of any user-simulation benchmark.

First: what is the model compared against. The correct baseline is not random choice but simple statistical baselines on the same logs. If they stay close to the LLM, the expensive model is buying not realism but nicely written text.

Second: over what horizon is quality measured. Guessing the next action is far easier than reproducing behavior over days and dozens of events. If the benchmark really covers a long chain, this is where the most noticeable model failures should be expected.

Third: how is diversity evaluated. Users differ, while LLMs tend to average behavior into a "typical" person. A simulation in which everyone behaves equally sensibly is poorly suited to testing recommendations.

For teams building recommendation and advertising systems, the takeaway is simple. You can already use LLM-based users as a filter to weed out clearly weak hypotheses. Relying on them for production rollouts without checking against real A/B tests is premature. If its data and protocol are released, such a benchmark would become a convenient tool for testing your own simulators. It is worth waiting for the paper and the repository: the details of data collection and metric choice will show how well-founded the claim to being first really is.