Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseResearch Paper · For R&DHabr AI→ original

Opus rewrote llama.cpp code: Qwen runs 40% faster on a CPU alone

Running a neural network without a graphics card is possible, but the replies come slowly. The author of a Habr post had the Opus AI spend several days running tests and editing the code of the llama.cpp engine. As a result, the Qwen3.8-Flash-Next Gyro model on a single CPU produces 2.72 tokens per second instead of 1.97, roughly 40% faster (a token is a syllable or a short word). That is still slow for live chat, but for background tasks on a GPU-less server, the patches are worth testing.

AI-processed from Habr AI; edited by Hamidun News
Opus rewrote llama.cpp code: Qwen runs 40% faster on a CPU alone
Source: Habr AI. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceHabr AI ↗
Editor-in-Chief
Key metric #1Prompt reading sped up from 4.12 to 11.21 tokens/s (fix37 patch)
Key metric #2The fix23 patch gave ×1.23 on generation, fix37 gave ×1.40 over the original
Key metric #3The patches sit in a merge request on GitLab, and the author asks for benchmark results

Almost any advice on speeding up local neural networks sounds the same: buy a graphics card. Or compress and fine-tune the model itself. But what if a machine has only a processor and RAM? The author of a Habr post decided to test this in practice. He had the Opus AI rework the code of llama.cpp, a popular engine for running models on your own hardware. As a result, Qwen3.8-Flash-Next Gyro without a graphics card produces 2.72 tokens per second instead of the original 1.97. Here a token is roughly a syllable or a short word, so the speed rose by about 40%. The result is modest, but it comes with numbers and open code.

What exactly was done

The test model was Qwen3.8-Flash-Next Gyro in GGUF format. This is the format in which models are run through llama.cpp. The author did not change the model itself: no fine-tuning, no compression. Instead, he went after the engine's code. Opus did the main work. For several days it ran tests on the author's computer, looked for places where the processor wastes time, and proposed changes to the llama.cpp code.

The effort produced several versions of the patch. In a table, the author compares two of them, fix23 and fix37, against the original build.

What the measurements showed

Two things were measured: prompt reading speed (how many tokens the model "reads" per second) and response generation speed. Each cell in the table contains two numbers. The author did not explain what the second number means, so both pairs are given below.

  • Original: prompt 4.12 / 3.75 tokens/s, generation 1.97 / 1.92.
  • fix23: prompt 10.98 / 10.05, generation 2.41 / 2.38, speedup ×1.23.
  • fix37: prompt 11.21 / 14.49, generation 2.72 / 2.73, speedup ×1.40.

Prompt reading sped up more than generation: by the first numbers, from 4.12 to 11.21 tokens/s, almost threefold. For a processor this makes sense. A prompt can be processed in large batches, while response generation proceeds one token at a time and is limited by memory speed, so it is much harder to squeeze gains out of it. The author also attached a chart: the longer the context, the slower the generation. The patches do not remove this dependence.

What to do with this, and the limitations

A speed of 2.7 tokens per second is still low for interactive chat. But for background tasks such as batch text processing, where there is no rush, the gain is already noticeable. These are measurements on a single machine. The post does not say which processor or how much memory the author has. So the same speedup cannot be promised on other hardware or with other models. The author himself asks whether the same tuning will work for other models, and asks readers to send their configurations and the gain after applying the patches.

The patches are in a merge request on GitLab, in the agentionai-group/llama.cpp repository. If you run local models on a server without a GPU, you can build them and compare against standard llama.cpp on your own workload. In addition, this is an example of an AI assistant working for several days on low-level code on real hardware, rather than writing typical CRUD.

There is also an unpleasant detail. The account under which the author ran Opus was blocked. Verification and appeal did not help, and the block is final. Now he asks the community which model, and in which harness (the wrapper that gives the model tools and lets it work autonomously), can work on personal hardware for as long as Claude Code. The question is practical: if a project depends on a single cloud provider, it can be shut down at any moment.

A mini poll under the post shows how narrow this niche is. Three people voted: two ran LLMs on a computer with a discrete graphics card, one without one, and nobody used specialized NPUs. The sample is tiny. Still, it is a reminder that speedups on ordinary processors matter to a limited circle of enthusiasts. For them, a 40% gain is already tangible.

Story

Key Questions Answered

What exactly was done?

The test model was Qwen3.8-Flash-Next Gyro in GGUF format. This is the format in which models are run through llama.cpp. The author did not change the model itself: no fine-tuning, no compression. Instead, he went after the engine's code. Opus did the main work. For several days it ran tests on the author's computer, looked for places where the processor wastes time, and proposed changes to the llama.cpp code. The effort produced several versions of the patch.

What the measurements showed?

Two things were measured: prompt reading speed (how many tokens the model "reads" per second) and response generation speed. Each cell in the table contains two numbers. The author did not explain what the second number means, so both pairs are given below. Original: prompt 4.12 / 3.75 tokens/s, generation 1.97 / 1.92. fix23: prompt 10.98 / 10.05, generation 2.41 / 2.38, speedup ×1.23. fix37: prompt 11.21 / 14.49, generation 2.72 / 2.73, speedup ×1.40.

What to do with this, and the limitations?

A speed of 2.7 tokens per second is still low for interactive chat. But for background tasks such as batch text processing, where there is no rush, the gain is already noticeable. These are measurements on a single machine. The post does not say which processor or how much memory the author has. So the same speedup cannot be promised on other hardware or with other models.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…