NVIDIA predicts the best coding-agent base model before fine-tuning: its ranking matched the final one at 96%

Imagine you are choosing a model on which to build an AI coding agent. There are several candidates, and fine-tuning each one costs hundreds of thousands of dollars, so you have to choose in advance. The usual tests help little here: some HumanEval and SWE-bench tasks leaked into training data, and a noticeable share of SWE-Bench Pro tasks are written with errors.

NVIDIA researchers proposed a different approach: instead of making the base model solve a task, show it a recording of a strong agent at work and check whether it recognizes the code edit that solves the problem. The ranking of ten models from this test matched the post-fine-tuning ranking at 96%: the Spearman correlation is 0.964, where 1 means identical order and −1 means fully reversed.

For HumanEval this figure is −0.394, meaning the order is mostly reversed. The practical upshot is simple: you can filter out weak candidates before you spend money on fine-tuning.

Why the old tests don't work

HumanEval came out in 2021 alongside Codex: 164 small Python tasks, about as hard as junior-level interview questions. Over time it became clear that the tasks are publicly available and end up in training sets. Berkeley researchers found between 8 and 18% of HumanEval tasks in RedPajama and StarCoder-Data, and a team at Allen AI came across an open dataset that matched it at 70.7%.

The industry then moved to SWE-bench Verified. On 23 February 2026, OpenAI stopped publishing its results: of 138 tasks that its o3 model consistently failed, 59.4% contained errors in the tests or in the task description. In addition, GPT-5.2, Claude Opus 4.5 and Gemini 3 Flash reproduced the reference fix from the task number alone, as if they had seen the answers in advance. OpenAI audited SWE-Bench Pro, which it had recommended as the replacement, in under five months and acknowledged that about 30% of its tasks are broken. In one of them, the description shows an indent of one space while the hidden tests expect two, so following the description faithfully fails the task.

What NVIDIA proposed

Can you simply put a base model in front of a real task? The team tried it: they ran six base models as agents on SWE-bench Verified. Five solved no tasks at all, even though after fine-tuning the same models solve between 39 and 81%. A base model can only continue text. Instead of calling a tool, it often writes the rest of the dialogue itself, even inventing the output of a command.

So in the paper Before They Can Solve (22 authors, 7 October 2026), the model is not asked to solve the task. It is shown a recording of a strong agent at work (GPT-5.6-Sol, Claude Opus 5, Kimi-K3), and only the decisive moment is tested. The recording is replayed step by step, the project's tests are run after each edit, and the researchers find the first edit after which they pass. Then there are three ways to test the model:

• How "natural" the model finds the correct move. The researchers measure BPB, that is, how many bits of information are needed to encode a byte of the correct answer. Lower is better.
• Choosing the next step from four options: one correct, three plausible ones that failed the tests. The options are shuffled across positions to remove the model's fondness for the letter A.
• Completing someone else's work: the model receives the recording up to the decisive step and acts on its own for up to three steps, with several attempts.

The correct answer is always determined by the project's own tests, not by a human or another neural network.

What came out of it, and the catch

The researchers tested ten base models: three Nemotron-3 models, Qwen-3.5-35B-A3B, DeepSeek-V4 Flash and Pro, Kimi-K2, GLM-4.5-Air, Hy3 and Gemma-4-26B. They compared the ranking from these tests with the ranking of the fine-tuned versions on SWE-bench Verified. BPB at the decisive step gave 0.964 (across all steps of the recording it was only 0.915). With recordings from SWE-bench Verified itself, completing someone else's work reached 0.988. For HumanEval the correlation is −0.394: models with the best HumanEval results turned out, on average, to be weaker as agents.

One reason is the narrow spread. On MBPP, the ten models fell within a range of 67–86%, while the final agents ranged from 38.8 to 80.6%. A test like that separates strong candidates from weak ones poorly. To give a sense of the costs: DeepSeek spent about 147,000 H800 GPU-hours on fine-tuning R1, roughly 294 thousand dollars by its own estimate, and that covers only the successful run, without experiments.

NVIDIA's audit also covered Tencent's APTBench: of 1682 questions, 727 (43.2%) turned out to be invalid, and in the error-handling section the figure was 90.5%. The wrong answer options for these questions were generated by language models. A caveat: the audit itself was also carried out by an AI agent working from set criteria, and APTBench's overall result still correlates with the ranking at 0.88–0.90.

The method itself has limitations. In the controlled experiment, only three models were fine-tuned, and their order matched the official one, but the authors themselves call this a continuation of the trend rather than proof. On recordings from other benchmarks, the match was weaker. The method is still at the research stage, and there is no ready-made tool.

The practical takeaway for teams choosing a base for an agent: don't rely on HumanEval and MBPP. Use tests tied to real edits in a repository, and determine the correct answer with the project's tests. The logic is the same as in hiring: a structured interview about real work predicts results better than a general trivia test.