Dream2Discern Predicts Multi-Agent System Failures With Over 90% Accuracy

The authors of Dream2Discern have proposed a way to train a runtime auditor for LLM-based multi-agent systems without the expensive collection of new data. The auditor looks at a prefix of an execution and predicts how the run will end. On the AFTraj-2K and StepShield benchmarks, the method reaches more than 90% trajectory-level weighted accuracy and outperforms competing auditors. It also keeps a balance between prediction accuracy and timeliness.

Where auditors hit a bottleneck

LLM-based multi-agent systems fail in different ways. One agent gets stuck in a loop, another loses context, a third confidently passes wrong data to its neighbor. To stop such a run in time, you need an auditor: a model that estimates the likely outcome from the steps observed so far. It is trained on a corpus of trajectories with confirmed outcomes.

That corpus is always limited. Every new trajectory requires a real run of the system, followed by verification of the result. That costs money, time and manual work. The authors point to a second problem: even if you collect more failures, broader coverage by itself does not tell you which prefixes are actually hard for the current auditor to classify. You get more data, but not necessarily more useful information.

How Dream and Discern work

Dream2Discern splits training into two phases, and both rely on the auditor's own prediction loss.

In the Dream phase, the method selects perturbations to the operators that build the prefix representation inside the auditor. The perturbations are chosen so that the loss on recorded prefixes increases. The result is "hard latent variants": the same trajectories, but shifted toward the boundary where the auditor starts making mistakes.

In the Discern phase, the auditor learns to predict the outcome from these variants, using the original labels. If a prefix ended in failure, its hard variant remains a failure example. No verifier is needed for this.

The variants are regenerated during training. As the auditor gets stronger, the earlier perturbations lose their edge, and Dream finds new weak spots. A limited corpus becomes a renewable source of difficult examples. The authors stress that new trajectories and additional outcome verification are not required.

What the benchmarks showed

The evaluation used AFTraj-2K and StepShield. The abstract supports the following facts:

• more than 90% trajectory-level weighted accuracy on both benchmarks;
• better results than competing failure auditors;
• the method accounts for both accuracy and timeliness of prediction;
• no additional trajectory collection or outcome verification is needed;
• training works on prefixes: the auditor assesses a run before it finishes.

The last pair of points matters. An auditor that diagnoses a run flawlessly after it ends is useless: the money for tokens is already spent, and the agent's action is already done. The value lies in an early signal with sufficient accuracy. That is why the authors single out the balance between accuracy and timeliness. The abstract gives no specific latency figures or per-benchmark breakdown; for those, check the full text of the paper and the project page.

What to do with this in practice

For teams running agentic pipelines in production, the approach is interesting for two reasons. First, it lowers the barrier to entry. If you already have a log of runs with known outcomes, you can get more out of it than ordinary classifier training would give you. Second, it does not require expensive labeling of new material.

Before adopting it, check three things:

• how closely your trajectories resemble those in AFTraj-2K and StepShield;
• how the auditor behaves on early prefixes, not only on long ones;
• what a false alarm costs: stopping a healthy run can be more expensive than missing a failure.

For now, this is a research work with a public project page, not a finished product. But the direction is clear. The reliability of agentic systems increasingly depends on data about their own errors, not on the size of the base model. Dream2Discern shows that part of that data can be extracted from logs you have already accumulated.