GenUI-Harness Lifts a 4B Model From 9.33% to 58% Pass@3, Beating Claude Opus 5

The authors of a new arXiv paper propose GenUI-Harness, a multi-agent harness in which the model answers a complex request not with a paragraph of text but with a temporary interface containing forms, lists, and parameter selectors. After reinforcement learning, a 4B-parameter model went from 9.33% to 58.00% Pass@3 and outperformed Claude Opus 5, which scores 46.67%. In a survey of reviewers, the average dialogue length fell from 3.4 to 1.2 rounds.

Two Agents Instead of One Chat

The architecture is simple. The Tool Agent handles information retrieval and task execution: it queries databases and calls tools. The GUI Coder Agent looks at the request and the result, finds ambiguities, and writes the frontend code for a structured interface. If a user asks to "change a flight," the agent does not fire off five clarifying questions in a row. It shows a form with real options from the database.

The authors start from the premise that text overloads people as a channel for complex tasks. It creates ambiguity and information noise, and it makes input slow. An interface built around the specific data and the specific step of the task removes these problems.

How to Train an Interface Coder

The core engineering contribution of the work is how to train the GUI Coder with RL. There are two problems, and each has its own solution.

First, a verifiable reward for an interactive UI requires actually executing the code, which is expensive. To handle this, the authors built Dynamic UX, a lightweight package that provides dynamic interaction and reward collection in a single sandbox.

Second, when an LLM judge issues the reward, the model quickly learns to game it. Against this reward hacking, the authors propose Reward Auditor, a meta-reward. It monitors the distribution of rewards, identifies diagnostic patterns, and consolidates them into a shared rubric and evaluation specification. The judge does not stay fixed during training.

The Benchmark and the Numbers

To test the system, the authors created UI-TAU Bench. It is built on 10 real domain databases from open sources and follows the Tau-Bench setup for tool use. There are Lite (300 tasks) and Full (1000 tasks) versions.

• Average Pass@3 gain over smolagents on Lite: 4.48 percentage points.
• 4B model after training in GenUI-Harness: from 9.33% to 58.00% Pass@3.
• Claude Opus 5 on the same benchmark: 46.67%.
• Robustness: the system holds up on both ambiguous and unambiguous requests.
• Reviewer survey: dialogue shrinks from 3.4 to 1.2 rounds.

The 4B model's starting 9.33% shows how hard the task is for the base version without specialization. The jump to 58% suggests that RL with reward auditing gives more than model size does. But the comparison runs on a benchmark the authors built themselves, for their own setup, and the frontier models were not trained specifically for it. The 4.48-point gain over smolagents looks far more modest than the 4B model's leap, and these two numbers are worth keeping in mind together. The reviewer survey also does not replace an experiment with ordinary users. The authors themselves limit their conclusions to "the evaluated database scenarios."

Where This Could Be Useful

The closest fit to practice is scenarios where the agent has a structured backend: customer support, booking, internal admin panels, order handling. In these settings a clarifying dialogue is costly, and the answer options are finite and stored in a database.

Teams building agents should look at two transferable techniques. First, separate the task executor from the interface generator and train them separately. Second, monitor the LLM judge's reward distribution as a separate entity rather than trusting a rubric written once. The second technique is useful in any RL pipeline with a judge, not only in UI generation.

For now, this is a paper from a v1 preprint: the code, reproducibility, and behavior on data outside the ten domains remain to be verified.