Tsinghua Team Releases OSWorld-Science: Testing AI Agents in Scientific Software

A Tsinghua team has introduced OSWorld-Science. The benchmark tests whether an AI agent can do more than explain a scientific task: can it actually carry the task out in a professional program? The work ranges from analyzing nuclear magnetic resonance spectra to processing pathology images, running statistical calculations, and doing physical simulation. The publicly available announcement gives neither the number of tasks nor results for any specific models. So the main question, how big a gap the leaderboard will show, remains open for now.

Where Text Answers End

Modern language models handle the first part of scientific work well. They explain a method, list the steps, and write an analysis script. What comes next takes up the lion's share of time in labs: opening the right program, loading the data, setting parameters, navigating menus, checking an intermediate result, and getting an output that can go into a paper.

This is the stretch the OSWorld-Science authors take as the subject of their test. Tasks such as processing NMR data or annotating histology images come up in research every day. General erudition is not enough for them. The agent has to understand the subject area and work carefully with an interface where a mistake in a single parameter changes the entire result.

What We Know About the Benchmark

• Developer: a research team at Tsinghua University.
• Format: the agent controls the scientific software itself rather than describing a solution in words.
• Task areas: NMR analysis, pathology image processing, statistics, physical simulation.
• Requirements for the agent: domain knowledge plus reliable execution of a sequence of operations.
• Not disclosed in the available excerpt: the number of tasks, the list of programs, the method for verifying results, model scores, and the human baseline.

Context: From Office Tasks to Lab Tasks

The name nods to OSWorld, a 2024 benchmark for agents that operate a computer. It contained 369 tasks across Ubuntu, Windows, and macOS. Humans solved about 72% of the tasks, while the best model at the time of publication solved just over 12%. Agents have improved markedly since then, but general office and browser scenarios remain noticeably easier than scientific ones.

Professional scientific software has several properties that make it awkward for agents. The interfaces are highly specialized, often outdated, and nonstandard. Few training examples are publicly available. Most importantly, errors there are silent: the program does not show an error message, it produces a plausible but wrong plot or p-value. This is our reading of the task, not a quote from the announcement, but it is exactly why the phrase "a result that can be trusted" looks central. A benchmark that only checks whether the actions were completed is useless here. What matters is how the final file or number is verified.

How to Work With This

For labs, the takeaway is down to earth: it is too early to give agents for scientific software unsupervised autonomy. A sensible setup today looks like this: the agent prepares and runs routine calculations, a human checks the parameters and the result, and the action log is saved for reproducibility.

For agent developers, OSWorld-Science will likely become another demanding test. It shows how well a model holds domain context over a long chain of actions in a real interface.

Before drawing conclusions, it is worth waiting for the paper and the repository and checking four things: how the automatic verification of results works, how many tasks can be solved only through the graphical interface, what score human experts achieved, and how leading commercial and open models perform on it. Without this data, the headline remains a promise rather than a measurement.