MultiAgent-OPD Trains a Whole Team of Agents End to End and Scores 38.68% on Seven QA Benchmarks

Researchers at TaoLive AIGC (first author Yuhan Sun) have published MultiAgent-OPD, a method for end-to-end training of hierarchical multi-agent systems. It combines on-policy distillation with reinforcement learning. A team consisting of a Leader, a Searcher and an Answerer runs on a single shared language model. On seven QA datasets, it reaches 38.68% macro avg@4. That is 4.06 pp better than Prompted Critic, a method that uses an LLM judge, and 2.61 pp better than C-MAS + RL, a trained cascaded system.

Why the final answer teaches searchers so little

In a hierarchical system, the Leader splits a task into subtasks and subagents carry them out. The Searcher looks up information in several waves and condenses what it finds into short reports, and the Answerer builds the answer from them. Training all of this jointly is hard because of the credit assignment problem: the final answer is either right or wrong, but that signal doesn't show which Searcher query helped and which one led it astray.

The usual workaround is an LLM judge that scores the subagents' work. But its scores depend on the judge's own ideas of what a good answer looks like. If the judge is wrong or biased, the system dutifully optimizes that error. This is the approach behind Prompted Critic, an M-GRPO-like MAS-RL method.

A teacher that doesn't change

The authors build the training in two stages. First, they train a Teacher in a cascaded Searcher–Answerer pipeline, where answer correctness serves as the signal. Then the teacher is frozen and plugged into the team's joint training.

During team training, the student itself generates search queries and summaries along evolving subtask trajectories. The frozen teacher scores those tokens, which amounts to token-level distillation on the student's own actions (on-policy). The Searcher gets a dense, local process signal: how to search and how to report what it found. The teacher also doesn't depend on the subjective opinion of an external judge, because it was itself trained on answer correctness.

The other roles get different signals:

• Leader learns from the success of the team as a whole (outcome-based RL);
• Answerer learns from the correctness of candidate answers;
• Searcher receives a token-level score from the frozen teacher.

All three roles optimize one shared model, which learns to coordinate task decomposition, multi-wave search and answer generation. At inference, the teacher is not needed; the trained student is enough.

Numbers and comparison

• MultiAgent-OPD: 38.68% macro avg@4 across seven datasets;
• Prompted Critic (LLM judge, MAS-RL in the spirit of M-GRPO): 4.06 pp lower, around 34.6%;
• C-MAS + RL (trained cascaded multi-agent system): 2.61 pp lower, around 36.1%;
• the comparison was run on evaluated checkpoints;
• the team solves more questions within four attempts.

A gain of 2–4 pp in the average across seven benchmarks is noticeable but not dramatic. More telling is that the edge over the cascaded system comes without making inference more complex: the roles share the same weights, and the extra cost falls on training.

Who can use this recipe

For teams building search and research agents, the main takeaway is practical. If you have a Leader that hands out tasks and executors whose work is visible only through the final result, you can first train a teacher on a simple cascaded setup with a verifiable answer. Its frozen scores can then serve as a dense signal for the executors, in place of an expensive and noisy LLM judge.

This reduces dependence on a third-party evaluator and simplifies deployment: production runs a single model rather than an ensemble of several. The limitations are also clear. The method has been tested on QA tasks with search, and other types of subagents, such as code or tool-use agents, would need the scheme to be validated separately. The authors have a project page available, and the paper is published on arXiv under number 2610.23234.