OfficeTown Simulates 23 Organizations and Trains Agents for Office Work on 406 Tasks

A team of researchers has introduced OfficeTown, a generative simulation of professional organizations that produces both tasks for agents and the context around them. From 23 simulated organizations across eight professional domains, the team assembled OfficeTown Bench: 406 tasks and 4 836 evaluation criteria, adjusted for feedback from domain experts. The small Qwen3.5-9B model showed a noticeable improvement on external benchmarks, including GDPval, after training on this data.

An Organization Instead of a Set of Prompts

Most agent benchmarks are built the same way: an isolated task, some files and a check of the result. Real office work looks different. A report relies on decisions made three weeks ago, on an email from a lawyer, on a manager's edits to the previous version. OfficeTown tries to reproduce exactly this accumulated environment.

The input is a natural-language description of an organization: what it does and what its processes are. The framework then builds out the team. Employees coordinate work, delegate limited assignments to assistant agents and review the results. Artifacts, decisions and open questions persist and affect later tasks. Roles and permissions determine what each participant can see and what they are allowed to do, so an agent works under realistic access restrictions rather than with the full dataset.

The assignments, their context and the reviews are then selected and turned into executable resources for evaluation and training. Here, tasks are a product of the simulation's progress, not the result of manual annotation.

What's Inside the Benchmark

OfficeTown Bench is compact but dense in criteria:

• 406 tasks;
• 4 836 evaluation criteria (by our estimate, about 12 per task);
• 23 simulated organizations (roughly 18 tasks per organization on average);
• eight professional domains;
• domain expert feedback, incorporated during assembly.

Evaluation of frontier models showed that the room for improvement remains significant. The abstract gives no specific scores, so how far the leaders are from the ceiling can only be judged from the authors' own wording. For a benchmark, this is usually a good sign: it is not saturated from day one.

The Data Works as Training Material

The most practical result concerns training. The authors applied supervised fine-tuning and on-policy distillation on OfficeTown data to Qwen3.5-9B. The nine-billion-parameter model improved noticeably on external benchmarks of professional work, including GDPval. In other words, the simulation generates material that transfers beyond its own test set. That is the main argument against the concern that the model simply learns the style of one particular synthetic environment.

The authors also tested the execution architecture separately. The principal–assistant setup, in which a main agent delegates limited assignments to helpers and accepts the results, scored higher than the single-agent and multi-agent baselines used in the comparison. An important caveat: this applies to the baselines that were tested, not to all possible multi-agent designs. Still, a hierarchy with review looks like a sensible default for office tasks.

What You Can Take Away for Your Own Work

Teams building agents for documents, reporting or analytics should borrow three things. First, test the agent on chains of related assignments with accumulated history, not on one-off requests. Second, build access permissions into the evaluation: an agent that sees too much or cannot see what it needs behaves differently. Third, consider synthetic organizations as a source of fine-tuning data for small models, if running a frontier model for every operation is too expensive.

The limitations are also clear. The organizations are simulated, not taken from real companies, and quality depends on the generator. The public description gives no numerical gains, so it is too early to draw conclusions about the scale of the effect until the full paper and code have been checked.