AdaPRA Lifts an 80B Qwen by 24.5 Points on Kaggle-Style Leaderboards

Data science agents can now run the entire machine learning cycle on their own, from loading data to submitting predictions. On human leaderboards, however, they still trail human participants by a wide margin. Researchers at Carnegie Mellon University (Atharva Naik, Vivek Chari, Pol Llado, Daniel Fried, Carolyn Rosé, Benjamin Van Durme, Sam Thomson and Justin Svegliato) propose fixing the process rather than the model. With five attempts per task, their AdaPRA system lifted an 80B Qwen model by 24.5 percentile points on DSPredict-Easy and by 14.8 points on DSPredict-Hard, while also reducing the number of process errors.

Where Agents Lose Points

The authors start with a diagnosis. The agents' weak results are not explained solely by a lack of "intelligence." Most often they go wrong in the workflow itself, and the mistakes are fairly mundane:

• insufficient data exploration: the agent grabs a model before working out what is in the table;
• unreliable validation: the validation scheme produces attractive numbers that do not carry over to the test set;
• bugs in the code that quietly corrupt the pipeline;
• poor use of a fixed budget: in every attempt the agent redoes the same EDA and trains the same baseline models instead of moving forward.

The last point is especially telling. If an agent has five attempts at one task, each new attempt should logically build on the previous ones. In practice, it starts from zero.

An Auditor with a Checklist and Memory Between Attempts

AdaPRA has two parts. The first is a persistent harness for the coding agent, which accumulates knowledge across attempts on the same task. What has already been explored, which baseline models have been trained, and what worked stays in memory and is not recomputed.

The second part is a process reward agent, that is, an auditor agent. Classic process reward models score reasoning steps after the fact, working from text. Here the auditor works differently: it inspects the agent's trajectory and working directory and runs checks, rather than just reading logs. It can verify whether the validation is actually set up correctly, whether there is data leakage, and whether the agent did what it claimed.

The auditor scores the agent against rubrics, which are structured checklists of criteria. The key word is "adaptive": the rubrics are retuned for each specific task rather than staying a universal template. The authors note two effects. Rubrics seeded on easy tasks generalize to hard ones. Additional adaptation to the task improves results further still.

The Numbers and Transferability

Main results with five attempts per task:

• Qwen (80B): +24.5 percentile points on DSPredict-Easy;
• Qwen (80B): +14.8 points on DSPredict-Hard;
• the number of process errors drops on both sets;
• rubrics mined from the runs of one base model improve another, DeepSeek-V3.2;
• AdaPRA outperforms DataPRM, a previously proposed strong PRM for data science.

Transfer between models is arguably the most interesting detail here. If rubrics built from Qwen's errors help DeepSeek-V3.2, they capture typical failures of the task and the approach, not quirks of a particular model. Knowledge about how to do validation and exploration properly can be accumulated as a standalone asset, not tied to any model's weights.

The gain on DSPredict-Hard is half that on Easy, and it is an honest benchmark: with checklists, an agent climbs to the middle of the table noticeably more easily than to the top in hard competitions.

The code is published, and the project has a page at atharva-naik.github.io/AdaPRA, so the results can be reproduced.

For teams building agents for analytics and ML automation, the work offers a clear recipe. Before swapping the base model, look at the trajectories: if the agent repeats EDA in every attempt or checks itself against leaky validation, state persistence between runs and a separate auditor that executes checks should help. Rubrics can be built on easy tasks and reused across different models, so you invest once in such a set of criteria and keep applying it when you switch providers.