Planning added 11.6 pp of solved tasks for a weak model, while a strong one needed only a terminal
AI coding assistants run the same model in different ways: with or without a plan, with a toolkit or with a single terminal. Researchers tested 176 such configurations. For the weak Nemotron-3 30B, a plan added 11.6 pp of solved bugs on SWE-Bench. For the strong 550B model, a lone terminal gave +3.6 pp and cut the cost per task by 53%. The takeaway: the harness has to be tuned to the model.
AI-processed from Habr AI; edited by Hamidun News
You have probably noticed that the same model fixes bugs in one AI coding assistant and stumbles on a simple task in another. The model is not always to blame. Around it runs a software wrapper, the "harness".
It decides whether the agent makes a plan, which tools it uses to touch files and the terminal, and what happens when the conversation stops fitting into the context window. Researchers measured how much this matters in practice. For the weak Nemotron-3 model with 30 billion parameters, a plan added 11.
6 percentage points (pp) of solved tasks on SWE-Bench, a set of 500 real bugs in Python repositories. To illustrate: if the model solved 30% of tasks without a plan, it would solve 41.6% with one.
The strong 550-billion-parameter model, by contrast, did better with a single terminal instead of eight ready-made tools: accuracy rose by 3.6 pp, and the cost per task fell by 53%.
How they tested
Harnesses are usually compared by running finished products on the same tasks. That makes it unclear what exactly caused the difference. Here, the authors took a simple ReAct loop (the model reasons, acts, looks at the result and repeats) and changed only three components.
The first is planning. The model has an update_plan tool: a list of steps in which only one step is current at any moment, and the rest are either done or upcoming. The model sees only the latest version of the plan, so old drafts do not take up context.
The second is the action set. The full version has eight tools: reading files, creating them and editing them in place, browsing directories, searching by name and by code text, a terminal, and fetching web pages. The trimmed version keeps only the terminal.
The third is context management, with five variants. The first does nothing. The others replace old tool results with a short stub (the text is either lost for good or can be restored), summarize the old history with the same model, or combine both approaches. In the combined variant, cheap stubs kick in at a soft threshold of window fill, and the expensive summarization only at a hard one.
The models were Nemotron-3 at 30, 120 and 550 billion parameters, plus Mistral-Medium-3.5-128B from a different model family. The tasks were 500 bugs from SWE-Bench Verified and 89 command-line tasks from Terminal-Bench 2.1. Context windows ranged from 32K to 128K tokens, for 176 configurations in total.
Context, summarization and "restoring" text
Without context management, a 32K-token window lost 78.7% of SWE-Bench tasks on average: the conversation hit the limit, and the run aborted with an error. At 128K, losses were 8.7%. On Terminal-Bench they fell from 61% to 12%. With any form of context management, no tasks were lost. The agent's way of working barely changed: it simply had time to reach the verification step.
Stubs and summarization delivered similar accuracy at different prices. A stub replaces text that already exists, while summarization requires a separate paid model call. That is why the hybrid with two thresholds came out ahead: at comparable accuracy, it was the cheapest in almost every combination.
The idea of letting the model restore a deleted result did not work. In more than half of the configurations that had such a tool, the model never called it. The weakest model used it more actively, more than four times per task on Terminal-Bench with a 32K-token window, but its result was 3.4 pp lower than with stubs that had no restore option. On average, the difference did not reach half a point.
Weak models need hints, strong ones are hindered by them
For Nemotron-3 30B, a plan was a precondition for getting any meaningful result. Without it, the typical SWE-Bench trajectory shrank from 40 turns to 5, and more than two thirds of runs ended without a single code edit. With a plan, such runs made up 27.
8%. In 58.4% of runs without a plan, the model never got past searching for the right place in the code; with a plan, that share was 10.
4%. The result: +11.6 pp on SWE-Bench and +4.
5 pp on Terminal-Bench, even though the work became longer and more expensive. When this model was left with only a terminal, it kept calling tools that do not exist. On Terminal-Bench, two thirds of runs aborted because of this, and accuracy dropped by 15 pp on SWE-Bench and by 10 pp on Terminal-Bench.
For Nemotron-3 550B and Mistral, a plan barely changed accuracy, but the cost on SWE-Bench fell by about 30%. What mostly disappeared were post-edit checks, meaning the model re-verified finished solutions less often. For 550B, a single terminal raised accuracy by 3.6 pp on SWE-Bench and by 5.6 pp on Terminal-Bench, while cutting cost by 53% and 30%. The model packed several operations into one command and edited a single file in pieces less often: 1.5 edits per task instead of 4.6.
For Mistral, everything depended on the task type. On SWE-Bench without the ready-made tools, it lost 23.2 pp, and a third of runs (32.8%) ended without a single edit, versus 1.2% with the full toolset. On Terminal-Bench, a lone terminal, by contrast, gave +6.7 pp: those tasks are solved with commands anyway. Nemotron-3 120B behaved unevenly: there was no stable gain from a plan, and the ready-made tools gave +1.6 pp on SWE-Bench and +4.5 pp on Terminal-Bench.
What to do in practice
- With a small window (up to 64K tokens), enable context management, preferably a hybrid of stubs and summarization.
- Give a weak model a plan and the full toolset.
- For a strong model, test a trimmed harness: no plan and a single terminal. This can save up to a third of the budget without losing quality.
- You can skip building a mechanism for restoring deleted history.
- If tasks depend on repository structure, file tools are useful. If everything is solved through the shell, they can get in the way by adding unnecessary choices.
Limitations
Each configuration was run once, and Terminal-Bench is small, so many differences there are not statistically significant. The plan and the tools were tested only with one context strategy and the largest window. SWE-Bench consists of Python tasks, and model size is only a rough proxy for capability. The conclusion is simple: there is no universal harness. You pick one for a specific pair of model and task type, and you measure it on your own data.
- Nov 12, 2026Eight Countries, OpenAI and Anthropic Promise an AI "Kill Switch" — but Set No Deadlines or Penalties
- Nov 12, 2026ElevenLabs Scribe v3 Live Teaches Voice Agents Not to Interrupt: 25% Fewer Errors
- Nov 12, 2026Russian Prompts in ComfyUI: FLUX.2 and Qwen-Image Lose 4–13 Points, SD-Turbo Loses Almost Everything
- Nov 8, 2026AI visit notes fill 50 fields in 35 seconds in Moscow, while private clinics remain silent
Key Questions Answered
How they tested?
Harnesses are usually compared by running finished products on the same tasks. That makes it unclear what exactly caused the difference. Here, the authors took a simple ReAct loop (the model reasons, acts, looks at the result and repeats) and changed only three components. The first is planning. The model has an update_plan tool: a list of steps in which only one step is current at any moment, and the rest are either done or upcoming.
What are the key details on Context, summarization and "restoring" text?
Without context management, a 32K-token window lost 78.7% of SWE-Bench tasks on average: the conversation hit the limit, and the run aborted with an error. At 128K, losses were 8.7%. On Terminal-Bench they fell from 61% to 12%. With any form of context management, no tasks were lost. The agent's way of working barely changed: it simply had time to reach the verification step. Stubs and summarization delivered similar accuracy at different prices.
What are the key details on Weak models need hints, strong ones are hindered by them?
For Nemotron-3 30B, a plan was a precondition for getting any meaningful result. Without it, the typical SWE-Bench trajectory shrank from 40 turns to 5, and more than two thirds of runs ended without a single code edit. With a plan, such runs made up 27.8%. In 58.4% of runs without a plan, the model never got past searching for the right place in the code; with a plan, that share was 10.4%. The result: +11.6 pp on SWE-Bench and +4.5 pp on Terminal-Bench, even though the work became longer and more expensive.
Build & Ship Production AI Agents Without Boilerplate Code
Start your free 72-hour trial at Hamidun Academy (academy.hamidun.com) — interactive lessons on Claude Code & Cursor, live Tech Radar, and production agent templates.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.