HERA teaches AI agents to refuse impossible tasks: refusal accuracy rises from 61.7% to 83.3%

An agent that confidently does something that cannot be done is more dangerous than one that simply makes mistakes. It calls tools, burns budget, and reports success where no solution exists. The authors of HERA, a paper featured in Hugging Face Daily Papers, propose a way to fix this. On held-out tasks, their evolved harness raised refusal accuracy from 61.7% to 83.3%. The share of solvable tasks completed also rose, from 68.3% to 76.7%. The best harness then transferred to 19 other LLMs and added an average of 15.3 percentage points of refusal accuracy.

The problem: agents can't say "I can't"

The paper calls this agentic abstention: the ability of an agent to recognize that a task is impossible in the current environment and stop. Previously, researchers tried to improve reliability in such cases in two ways: fine-tuning the model, or tuning the harness, meaning everything that surrounds the model: instructions, rules for working with tools, and checks. Both approaches optimized for a fixed set of tasks. The result is predictable: they work on traps known in advance, but not on new types of failure.

HERA targets exactly this narrowness. The framework has two parts, and the second matters more than the first.

How HERA works

The first part is a pipeline that automatically builds verifiable pairs of tasks: one solvable and one unsolvable. The unsolvable task is derived from the solvable one through a controlled mutation of the environment. For example, the pipeline removes something from the environment without which a solution is impossible. The pair gives a clean contrast: the agent must handle one version and refuse the other. Verifiability is the key point here: the correct answer is known by construction, so no manual labeling is needed.

The second part is a co-evolution procedure. Failures on earlier tasks drive adaptation of the harness, and the same weak spots become a brief for generating new environments and tasks. The harness improves, the environment gets tougher in the places where the agent broke, and the cycle repeats. Instead of a static benchmark, you get a training ground that adapts to the agent's weaknesses.

The numbers

• Refusal accuracy on held-out tasks: 61.7% → 83.3% (+21.6 pp).
• Completion of solvable tasks: 68.3% → 76.7% (+8.4 pp).
• HERA leads the compared methods on both metrics.
• Transfer to 19 other LLMs: an average of +15.3 pp refusal accuracy, without optimization for any specific model.
• With this harness, small models, by the authors' estimate, come close to more powerful ones at roughly 85% lower cost.

The gain on solvable tasks deserves separate mention. The usual price of caution is overcaution: the agent starts refusing tasks that do have a solution. That did not happen here; both metrics rose at the same time. Our arithmetic: the values move in steps that are multiples of 1.67 pp, which points to a held-out set of about 60 tasks. The sample is small, so differences of a few points between methods should be read with caution. The transfer to 19 models serves as an independent check: a result averaged over that many LLMs depends less on luck with a single test set.

The most practical part of the result is the economics. A harness tuned once transfers to other models, and a small model with a good harness can cover some scenarios in place of an expensive flagship. The 85% savings figure is an estimate, and the authors do not present it as a measured production cost; under real workloads the number will depend on pricing and the task mix.

The paper offers three takeaways for teams building agents. First, include impossible tasks in the test set as a separate class and track a separate metric for them. Second, environment mutation is a cheap way to get such cases: take a working scenario and break the condition the solution depends on. Third, before paying for a stronger model, check whether a reworked harness already solves the problem. Limitations remain: the results come from the authors' specific environments, and robustness to failures that the mutations do not cover has yet to be tested.