Patronus AI Releases GlobeBench: The Best LLM Simulates RL Environments Only 49.6% of the Time

Anyone who has tried training an agent with reinforcement learning knows: the hardest part is not the algorithm, but the environment. You need a website you can safely break, a code sandbox, a simulator of a store or CRM where the agent can make mistakes thousands of times. Building such environments by hand is slow and expensive. That is why the idea is getting louder: let the language model itself play the environment. Patronus AI decided to test how realistic this actually is, and released the GlobeBench benchmark.

What is inside the benchmark

Not much is known: the set contains 1,632 tasks that test whether an LLM can simulate an RL environment well enough to train agents on it. 20 models were tested. How exactly scores are calculated and which domains the tasks are drawn from is not disclosed in the post itself; details are moved to a thread via a link, so it is worth going to the primary source for specifics.

Let us explain the idea with a simple example. An agent is learning to book tickets. In an ordinary environment there is a real (or pre-written) service: the agent clicks "select flight" and the service responds according to strict rules. In an "LLM-based environment" a model sits in place of the service: the agent writes an action, and the model composes a response: what changed, what the reward is, whether the episode has ended. If the model gets confused about the rules, forgets the state, or produces an impossible outcome, the agent learns from garbage and masters not the task but the simulator's bugs.

What the numbers mean

The best result belongs to GPT-6 Astra: 49.6%. That is, even the strongest model is wrong in roughly every second case. The other 19 models fell short of 30%. The gap between first place and the rest is huge, and this is the main signal: being a reliable simulator is a separate skill that does not necessarily grow on its own along with a model's general quality. This is our interpretation; the authors do not state it so directly in the portion of the post provided.

For real training this is not enough. Simulator errors accumulate: in a long episode, the probability that the world has remained consistent drops quickly. An agent, meanwhile, tends to find and exploit any loophole, which means it will find the environment's inaccuracies first.

What to do in practice

If you are building agents, the takeaway is fairly down-to-earth.

• Do not replace real or deterministic environments with LLM simulation by default. Where rules can be written in code, write them in code.

• If you cannot do without an LLM environment, first measure its plausibility on a separate set of checks. GlobeBench works as a ready-made starting point, and it is open.

• Compare models specifically as simulators: the winner of general rankings may turn out to be a poor "world".

• Keep part of the evaluation on a real environment to catch divergence between simulation and reality in time.

Limitations

The post is an announcement, not a paper. We do not see the methodology, confidence intervals, or how the "correctness" of an answer was assessed. The benchmark was released by a company that itself works on AI evaluation, so independent verification would not hurt. Moreover, 49.6% is a measurement on a specific set, not a ceiling for the approach. With fine-tuning, state checks, and a hybrid architecture where the logic lives in code and the LLM merely describes what is happening, results could be noticeably better. But as a baseline, the figure is honest and useful.

Bottom line: the idea of "training agents in a world invented by an LLM" currently runs up against the quality of the world itself. It is good that the industry now has an open way to measure this, and the debate can now be about percentages rather than gut feelings.