Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseInsight · On RadarLangChain Blog→ original

LangChain moved a coding agent from outside the top 30 to the top 5 on Terminal Bench, without changing the model

LangChain moved its Deep Agents coding agent from outside the top 30 to the top 5 on Terminal Bench 2.0 without changing the model. The team only reworked the harness around it. The baseline of 52.8% on GPT-5.2-Codex rose after three fixes: checking the solution against the tests before submission, giving the agent hints about its environment and time limits, and protection against getting stuck in loops. If an agent makes mistakes, the advice is to analyze its traces and refine the harness first, and only then change the model.

AI-processed from LangChain Blog; edited by Hamidun News
LangChain moved a coding agent from outside the top 30 to the top 5 on Terminal Bench, without changing the model
Source: LangChain Blog. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceLangChain Blog ↗
Editor-in-Chief
Key metric #1Before the agent exits, middleware forces it to check the solution against the task
Key metric #2After several edits to the same file, the agent is advised to reconsider its approach
Reasoningheavier on planning and verification, lighter on implementation

When a coding agent performs poorly, the first instinct is to reach for a stronger model. The LangChain team tested a different path: it kept the model as it was and reworked everything around it. Its Deep Agents agent climbed on Terminal Bench 2.0 from just outside the top 30 to the top 5 of the leaderboard. They changed only the harness: the system prompt, the tools, and the intermediate hooks around model and tool calls.

How they looked for weak spots

Terminal Bench 2.0 is a standard suite for testing programming agents: 89 tasks spanning machine learning, debugging, biology, and other areas. The baseline agent with GPT-5.2-Codex scored 52.8%. That is respectable, but below the top 30.

Next, LangChain built a repeatable improvement loop. All agent actions are written to LangSmith traces along with latencies, token counts, and costs. The trace analysis was packaged as an Agent Skill: it downloads the traces from a run, launches parallel agents that analyze the errors, and a main agent collects their findings into proposed edits. A human can review and discuss the changes, but that is optional. There is also a caveat: fixes tailored to one task can break others, so generalization has to be done carefully. The team promises to publish the skill.

What actually improved the results

Self-verification. The most common failure looked like this: the agent writes a solution, rereads its own code, decides everything is fine, and stops. They added a work order to the prompt: study the task and the code, make a plan that accounts for verification, write tests (including edge cases, not only the happy path), run them, read the full output, and compare it against what the task asked for rather than against the agent's own code. In addition, they built PreCompletionChecklistMiddleware, which intercepts the agent before it finishes and forces it to check the result against the specification once more.

Context about the environment. The agent does not know the directory structure, the installed tools, or the hard timeouts in advance. So at startup it is immediately given a map of the directories and a list of available tools, which reduces lookup errors. The agent is also told that programmatic tests will check the solution, and that file paths from the task must be followed literally. They also added warnings about the time limit, since agents estimate time poorly and fail to reach verification in time.

Escaping loops. Having settled on a plan, the agent sometimes makes 10 or more minor variations of an approach that is known not to work. The middleware counts edits to each file and, after a certain number, adds a prompt to reconsider the strategy. This does not always help: if the model is confident it is right, it may continue down the same path.

Reasoning budget. A maximum budget on every step eats up time and runs into timeouts. LangChain chose a "sandwich" scheme: xhigh for planning, high for implementation, and xhigh again for the final verification. In trial runs there was no noticeable difference between other ways of distributing the budget, so they settled on this one. In a multi-model setup, the same logic suggests assigning planning to a large model and implementation to a small one.

What to take from this

First, traces are a source of feedback. A model often goes the wrong way because it lacks the right tool or instruction, and both the prompts and the tools need to be examined together.

Second, guardrails such as loop detectors and test reminders are a temporary measure aimed at today's model weaknesses. The authors themselves write that as models improve, such crutches will likely become unnecessary.

Third, the harness has to be tuned to the specific model. A test with Claude Opus 4.6 on an early harness version was competitive but trailed Codex; the improvement loop was not run for Claude. General principles (context preparation, emphasis on verification) carry over, but several iterations tuned to your own model and task still pay off.

There are limitations too. The result comes from a single benchmark with hard timeouts, and real-world work usually has no such limits. The time-related prompts and some heuristics will therefore need to be adapted. Next, LangChain plans to explore combinations of several models (Codex, Gemini, Claude), memory for continual learning in agents, and ways to measure the effect of harness edits across different models. Deep Agents is an open-source project.

Story

Key Questions Answered

How they looked for weak spots?

Terminal Bench 2.0 is a standard suite for testing programming agents: 89 tasks spanning machine learning, debugging, biology, and other areas. The baseline agent with GPT-5.2-Codex scored 52.8%. That is respectable, but below the top 30. Next, LangChain built a repeatable improvement loop. All agent actions are written to LangSmith traces along with latencies, token counts, and costs.

What actually improved the results?

Self-verification. The most common failure looked like this: the agent writes a solution, rereads its own code, decides everything is fine, and stops. They added a work order to the prompt: study the task and the code, make a plan that accounts for verification, write tests (including edge cases, not only the happy path), run them, read the full output, and compare it against what the task asked for rather than against the agent's own code.

What to take from this?

First, traces are a source of feedback. A model often goes the wrong way because it lacks the right tool or instruction, and both the prompts and the tools need to be examined together. Second, guardrails such as loop detectors and test reminders are a temporary measure aimed at today's model weaknesses. The authors themselves write that as models improve, such crutches will likely become unnecessary. Third, the harness has to be tuned to the specific model.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Hands-On AI Engineering · Hamidun Academy

Build & Ship Production AI Agents Without Boilerplate Code

Start your free 72-hour trial at Hamidun Academy (academy.hamidun.com) — interactive lessons on Claude Code & Cursor, live Tech Radar, and production agent templates.

Start 72h Free Trial →Vibe-Coding Intensive
What do you think?
Loading comments…