Opus plus DeepSeek Flash solved 67% of tasks versus 81% for Opus alone, and cost more

The popular idea sounds logical: an expensive model like Opus plans the work and checks the result, while a cheap one, such as DeepSeek or Qwen, writes the code. Quality should stay almost the same while the bill drops. Ibragim Badertdinov tested this in practice and got the opposite result: the Opus plus DeepSeek Flash pairing solved 67% of tasks at $1.29 per task, while Opus alone scored 81% at $1.20.

How the test set was built

The author used to build test sets from his own working sessions, but the tasks turned out to be too easy: any setup looks decent on them. This time he took 12 hard tasks from public benchmarks, chosen as the closest to his day-to-day work, mostly hard ML and SWE (software engineering) level. He ran several passes and manually fixed the tasks themselves.

That is a finding in its own right: according to the author, more than half of the benchmark tasks are unfair. The tests either failed a working solution or passed a weak one. If you compare models without this cleanup, part of the difference in rankings turns out to be noise from the checks themselves. The tasks are published in Harbor, so anyone can run them on their own setups. The full results table and discussion are in a thread on X and a post on Reddit.

In total, about seven configurations were run, including the setup from DeepSeek's super agent paper and various combinations of models.

Why the cheap worker did not save money

The main cause is the prompt cache, which is where most of the money goes. While the cheap model writes code, Opus's cache goes stale. When control returns to Opus, the context has to be rebuilt from scratch, and that costs more than all of the cheap model's work. The savings on generation tokens are simply eaten by the cache rebuild. For developers of agent pipelines, the conclusion is practical: count the cost of the whole chain, including the idle time of the expensive model's context, not the price of the worker's tokens.

A different trick, based on the DeepSeek paper, did work. Opus was not forbidden from writing code; it was simply told that its tokens are 20 times more expensive. The result was the same score at almost half the price: $0.71 per task. In other words, the model itself decides how much to delegate and what to write on its own, and it allocates effort better than a rigid role scheme does.

Cheap models on their own beat the combinations

The most unexpected result: inexpensive models with no orchestration at all outperformed every combination. Terra solved 78% of tasks and Sol 75%, both at $0.17 per task. That is seven times cheaper than Opus alone, with results within a few percentage points. The post does not reveal what exactly is behind the names Terra and Sol, so they cannot be compared with specific public models.

The reviewer role was tested separately. In a review pipeline, the reviewer looks at diffs and, if it finds a problem, asks the worker to redo the change. Sol did an excellent job in this role, while Opus was mediocre. An expensive model does not guarantee better oversight: review is a separate skill that does not have to match a model's general level.

Limitations and practical takeaways

The sample is small: 12 tasks, so a difference of a few percentage points may be statistical noise. The tasks were selected for one engineer's profile, hard ML and SWE, and the picture may differ for web development or routine refactoring. The post gives results only for the DeepSeek Flash pairing; figures for Qwen as the worker are not provided.

Still, the guideposts are clear. Before building an "expensive brain, cheap hands" setup, measure the full cost, cache included. If you need to save money, try telling the expensive model what its tokens cost, and look at standalone runs of cheap models: on this set, they offered the best price-to-quality ratio. For review, it makes sense to test not only the flagship but also more modest candidates.