RiseHarness Teaches Terminal Agents to Fix Their Own Harness Without Fine-Tuning

A terminal agent can solve a task on the first try and fail it on the second, even though the model, prompt, and configuration stay the same. RiseHarness targets exactly this instability. The framework, from a paper featured in Hugging Face Daily Papers, iteratively rewrites the agent's harness while keeping the model weights completely frozen. The abstract gives no numerical results, so the approach itself is what there is to evaluate. It is a fairly concrete one.

The gap between "can do" and "does reliably"

The premise is simple. In long terminal tasks, every agent action changes the context, and the context shapes the next action. A random error at an early step spreads across the entire trajectory. As a result, the same task in the same configuration succeeds in one run and fails in another. The authors call this the gap between capability and reliable execution.

For this reason, RiseHarness does not chase tasks the model never solves. It focuses on "flipping tasks," where some runs succeed and others fail. The successful run becomes the reference against which the failed one is compared. This makes it easier to tell a systematic harness problem from plain bad luck.

How the loop works

The analysis uses two types of data: agent trajectories and internal runtime-monitor traces. The traces show how the model's actions interact with command execution, feedback, context management, and the decision to finish the task. These are the points that most often break over a long horizon.

To tie observations to real code, the framework uses the Harness Handbook. It maps evidence from the runs to the harness's behavior and its source code. This helps find recurring problems in mechanisms shared across many tasks, rather than patching individual cases.

The next step is more interesting. RiseHarness searches other agent frameworks for relevant implementations and adapts them as design references for a specific model and execution protocol. Candidate changes go through behavioral checks and re-evaluation on tasks. Versions with a measurable gain become the basis for the next iteration. The result is a single harness that combines suitable mechanisms from different projects, while fresh run data guides each subsequent change.

What is known about the method

• Model weights do not change; only the harness is optimized.
• The unit of adaptation is a specific model, not an abstract "agent in general."
• Tasks with mixed outcomes under the same configuration get priority.
• Signal sources: agent trajectories and runtime-monitor traces.
• Changes are accepted only after behavioral checks and repeated runs.
• Evaluation: success and reliability across different models, ablations on execution traces and external research, and transfer to unfamiliar benchmarks with the same model.

The last item matters. Harness improvements are easy to overfit to the set of tasks they were tuned on. Testing on unseen benchmarks with the same model shows whether the method found general interaction mechanisms or just fitted the behavior. The abstract gives no concrete numbers for these tests, so you will need to read the full paper.

What you can take into practice

Teams building coding and terminal agents can take away three things. First, measure stability, not just pass@1: run each task several times and track separately those that produce mixed results. Second, log the harness's internal events alongside the model's actions. Without execution traces, feedback, and completion decisions, diagnosing a failure turns into guesswork. Third, revisit the harness when you switch models: by the authors' logic, mechanisms that worked well with one model are not guaranteed to suit another.

The limitations are also visible. The method requires many repeated runs, which means a compute budget, and it depends on the quality of the monitor. How well the adapted "borrowed" mechanisms coexist with each other will be shown by the ablations in the full paper. Still, the idea itself looks practical: part of agents' unreliability sits in the harness, and it can be fixed through engineering without touching the weights.