GlobeBench: The Best LLM Correctly Predicts Environment Responses in Only 49.57% of Tasks

Researchers have introduced GlobeBench, the first benchmark that tries to put text-based world models in a single comparison table. These are systems that predict how an environment's state will change after an agent's action. The set contains 1632 tasks, five domains and nineteen simulation skills. The team ran 20 models on it, from general-purpose LLMs to specialized text simulators. The leader, GPT-6 Astra, solved 49.57% of tasks, which means it did not even reach half. Models struggle most with cascading effects: when an agent's action should affect dependent objects, the average success rate across all 20 models is 5.1%.

What the benchmark tests and how

The idea behind text-based world models is simple. Instead of using an expensive, brittle real environment for reinforcement learning, a language model itself generates what happens after the agent's next step. This makes it cheaper to scale environments, but until now there was no common way to compare such systems with each other. GlobeBench fills that gap.

Each task gives the model the agent's full trajectory and a specific action. The model must predict the environment's response. To do that, it has to remember earlier state changes, follow the environment's rules, and understand how the action affects both related and unrelated objects. The last point matters: a model that "fills in" extra changes is wrong just as much as one that misses required ones.

The prediction is compared with a reference obtained by actually executing the action. The evaluation is hybrid: deterministic "gates" filter out formally incorrect answers, and a semantic judge assesses the rest.

Where models break down

The main finding concerns the scale of an action's consequences. Accuracy falls as an action touches a larger part of the environment. Predicting that one object changed status is one thing. Tracing how that change reaches every element that depends on it is another. Here, models fail almost across the board.

• Tasks: 1632, domains: 5, simulation skills: 19.
• Models compared: 20, including general-purpose LLMs and specialized text world models.
• Best result: GPT-6 Astra, 49.57% of tasks solved.
• Tasks requiring changes to propagate to dependent objects: 5.1% average success.
• Tasks solved by no one: 631, about 38.7% of the set.

The last figure says a lot. The models' strengths differ from task to task, so an ensemble or routing between them should give some gain: for 1001 tasks, at least one model did get it right. But almost 40% of the set lies beyond the capabilities of the entire list of participants. That is a ceiling for this class of systems, not a flaw of any single model.

What teams building simulators should do

If you train agents on synthetic environments where an LLM generates the state transitions, you now have concrete reference points. You can expect acceptable quality on simple one-step changes, but you cannot rely on chains of dependencies: at 5.1% success, such a simulation will feed the agent wrong consequences of its actions. The agent will learn the simulator's mistakes, not the environment.

In practice, this means three things. First, wherever possible, check critical transitions against executable logic, and leave the model for the parts of the environment that are hard to formalize. Second, test your own simulator separately on tasks involving change propagation, not only on overall average accuracy, which hides the failure. Third, look at the spread between models. Since their strengths differ, it makes sense to compare candidates on your own domain rather than choose based on a single leaderboard.

For researchers, GlobeBench provides a common baseline and a set of 631 unsolved tasks that is well suited to measuring real progress. For now, an LLM-based world model remains a useful but unreliable tool, especially in complex, interconnected environments.