SynAct Teaches LLM Agents to Ask for Help Where It Changes the Outcome: +11.7 pp

The SynAct authors propose a new way to allocate expert feedback for LLM agents: not per task, but per individual decision within a trajectory. On four agentic benchmarks and four models from three families, the method improved held-out accuracy in all 16 tested configurations, by an average of 11.7 percentage points. The same signal also proved useful for inference-time routing: SynAct outperformed budget-matched cascade, uncertainty-based and "expert-judge" routers in 15 of 16 cases.

Two reasons an agent fails

The starting idea of the work is simple. An agent running a multi-step workflow breaks down for two different reasons. The first is random slips, which another attempt fixes. The second is knowledge gaps: no matter how many times you retry, the agent will take the same wrong action. The second type needs an expert, but the expert's time is limited. So, as the authors put it, the question is not whether to ask for feedback, but where it can change the outcome.

Existing approaches, as the authors describe them, run into one of two problems. Some do without experts, and the agent reinforces its own mistakes. Others hand out feedback at the level of the whole task and never establish which specific decisions determined anything.

How SynAct chooses where to ask

The framework treats expert attention as the main constraint and works at the level of individual decisions. It runs a task several times and, for each decision, estimates the mutual information between the chosen action and the final outcome. If the outcome does not depend on the choice of action, the information is zero, and there is nothing to ask about. The highest-scoring decisions go to the expert. The expert's corrections are then turned into transferable rules, which the agent applies in operation without access to the expert.

There is a theoretical component as well. With k runs and at least m observations per action, decisions with a constant outcome yield zero mutual information, while decisions that split outcomes yield at least H2(m/k) bits, where H2 is the binary entropy. Therefore any threshold t in the interval (0, H2(m/k)) separates these two classes exactly. For illustration: with k = 8 and m = 2, the lower bound is about 0.81 bits (our calculation; the abstract gives no such example).

This also explains why selection by success rate is blind when all runs fail. Success is then a constant, even though the failed runs end in different results. Mutual information sees those differences.

Key figures from the paper:

• 4 agentic benchmarks, 4 models from 3 families, 16 evaluated configurations;
• held-out accuracy up in all 16, by an average of +11.7 pp;
• inference-time routing: wins over budget-matched routers in 15 of 16 cases;
• one targeted correction delivers more than seven extra non-targeted retries.

Where this is useful

The last point matters in practice. Most teams deploying agents fight errors by brute force: more attempts, more samples, voting. The SynAct results suggest that for knowledge gaps this is wasted compute, and a single pinpoint correction pays off better.

A practical scenario looks like this. Run the tasks several times, compute the link between action and outcome for each step, and hand only the top of the list to an annotator or domain expert. The corrections are then assembled into a rule set for the agent. Expert hours go to a limited number of decisions, and the gain remains after the expert leaves.

There are limitations too. This is a preprint on arXiv, the results have not yet been peer-reviewed, and the abstract has no data on the cost of expert annotation or on how well the rules transfer beyond the four benchmarks. Still, the framing itself, which treats expert attention as a scarce resource, looks like a sensible foundation for agentic systems in production.