SWE-rebench Opens 13 337 Repaired RL Tasks: Qwen3.5-9B Learns From 70% Instead of 55%

Imagine you are teaching a model to fix bugs. You give it a task: fix a function in a real repository, then check the result with tests. But the tests check something entirely different from what the bug description says, or the solution is already sitting in the Docker image.

The model then either always fails or always "succeeds," and either way it has nothing to learn from. The authors of SWE-rebench ran into exactly this problem and released SWE-rebench V2.1 RL: 13 337 free RL environments (that is, ready-made tasks with an environment and a verifier) for training coding agents.

In a test on 100 tasks, the number of tasks on which Qwen3.5-9B actually learns something rose from 55 to 70. If you do RL post-training, this is a way to avoid spending GPU time on junk data.

What's in the dataset

The tasks come from real GitHub repositories and span 20 programming languages. They are packaged in the Harbor format, so you can plug them into your own RL pipeline without manual repackaging. The dataset is openly available at hub.harborframework.com.

The main difference from many open datasets is the verification. Every task was run through coding agents that looked for breakage and fixed it. The average cost of this work was $0.24 per task. Multiplied by the size of the dataset, that comes to about 3.2 thousand dollars for the whole collection. That is our own estimate, not a figure stated by the authors, but the order of magnitude is clear: cleaning the data is cheaper than spending compute on useless runs.

Why broken tasks do harm

In RL, a model solves the same task many times and compares its successful attempts with the failed ones. It is like a coach comparing an athlete's best and worst attempts. If the exercise is impossible to perform, or any movement counts as a pass, there is nothing to compare.

Code tasks suffer from three typical defects:

• the tests don't match the task description, and a correct solution fails them;
• the environment won't start: dependencies fail to install or the build crashes;
• the answer leaks into the Docker image, so the model can find it instead of working it out.

In the first two cases, every attempt fails and there is no difference between them. In the third, every attempt succeeds, and again there is nothing to learn from. In both cases, compute is wasted. According to the authors, open datasets contain many such tasks.

What the experiment showed

The check used the same 100 tasks throughout. Before repair, Qwen3.5-9B learned on 55 of them, meaning the ones where the attempts differ in a meaningful way. After repair, the number rose to 70. That is a gain of 15 tasks, or roughly a quarter more useful material at the same sample size.

The practical takeaway is simple: before an expensive RL run, estimate the share of tasks on which the model gets a signal. If it is low, cleaning the dataset first is cheaper than increasing the number of runs.

Limitations and what to do

The figures 55 and 70 come from a sample of 100 tasks and a single 9B model. The authors do not say in their post how much the final agent quality on benchmarks would improve after full training. We see a rise in the share of useful tasks, but not a direct improvement on SWE-style tests. This is worth checking on your own models and your own infrastructure.

Who this is useful for:

• teams doing RL post-training for coding agents who don't want to build environments from scratch;
• researchers who need a reproducible multi-language dataset, not just Python;
• anyone building their own datasets: the "agent verifies and repairs the task" approach, at $0.24 per task, can be adopted.

The authors also promise new releases for open source. If the next datasets are of the same quality, there will be noticeably more open data for training agents, and the gap with closed labs will narrow, at least in environment preparation.