REAP20 Speeds Up Local Qwen by 42% With No Accuracy Loss on DS1000

When choosing a local LLM for code, it is easy to stage a beautiful tournament and end up with a useless winner. All it takes is borrowing someone else's prompt set, forgetting about the reasoning limit, or comparing models in different modes. The author of the original piece did almost all of this himself, and then spent hundreds of GPU-hours figuring out where the experiment had gone wrong.

The practical value of the story lies precisely in the mistakes. On an RTX 4070 Ti Super with 16 GB of video memory, the author tested the Qwen family on DS1000, a set of a thousand tasks covering Pandas, NumPy, SciPy, Scikit-learn, Matplotlib, PyTorch and TensorFlow. The model's code was inserted into a template and executed, and the result was compared with a reference. This is closer to a data scientist's real work than yet another test of building a landing page or a browser game.

First Run: Speed Without Lengthy Deliberation

In the initial test, reasoning was disabled. REAP20-28B and Qwen3.6-35B-A3B came out on top by number of tasks solved. REAP20 behaved particularly interestingly: at comparable accuracy, it generated answers 42% faster than the original model, even though 20% of its weights had been removed.

For day-to-day work this is a significant result. When a person is waiting for the next fragment of NumPy code, latency matters more than the model's ability to think for several minutes. The author therefore proposed a practical split: a fast MoE model helps in interactive mode, while a slower dense model works overnight on tasks that do not need constant supervision, such as a preliminary code review.

But the result should not be turned into a universal ranking. DS1000 was released in 2023, and some of its tasks may have ended up in the training data of newer models. A single run does not show the spread of results, and different quantizations and decoding settings further get in the way of a fair comparison. Finally, the choice of libraries should match real work: the author cared about NumPy, SciPy, Scikit-learn and PyTorch, so he later cut the set from 1000 to 509 tasks.

The Token Limit Broke the Second Experiment

In the second run, the author enabled thinking but left 4096 tokens for reasoning. This decision flipped the table. REAP20's accuracy more than halved, while the full version of the same model barely changed its result. The reason turned out to be mundane: in roughly two cases out of three, REAP20 hit the ceiling and failed to finish its answer.

Here is a simple example. The model knows how to write a solution for Scikit-learn and builds the right plan, but cuts off before the final code. The grading system gives zero, exactly as it does for a completely wrong answer. If you look only at final accuracy, you might conclude that the model got worse. In reality, the measurement loop is broken.

This is an important lesson for any internal benchmark: record the reason the answer ended, the number of tokens used, the generation time and the share of cut-offs. Results with finish_reason=length should be analyzed separately; otherwise an infrastructure constraint masquerades as model weakness.

What Happened After the Fix

The author began to raise the limit step by step and continue the interrupted reasoning. Qwen3.8-27B in maximum-effort mode ultimately solved 252 of 509 tasks, versus the initial 173. The gain was 79 solutions, but the price was unpleasant: the series of iterations took about 153 hours. In final accuracy, the model only came close to the faster Qwen3.6-MoE without reasoning.

On a professional RTX 6000 Pro, the Qwen3.8-Flash-Next-NVFP4 model scored 58.1% on the full DS1000 in 8.5 minutes. However, this result cannot be carried over to a gaming graphics card: different hardware, weight format and configuration change the economics of running it.

The practical takeaway is boring but useful. First, assemble a small set of your own tasks, and fix library versions and identical run parameters. Then measure accuracy, latency, limit cut-offs and the cost of one successful solution. For a daytime assistant, choose a model by speed at sufficient quality; for overnight jobs, you can allow more time and tokens. And above all, do not treat a weak experiment design with additional hundreds of GPU-hours.