Opus rewrote llama.cpp code: Qwen runs 40% faster on a CPU alone
Running a neural network without a graphics card is possible, but the replies come slowly. The author of a Habr post had the Opus AI spend several days running tests and editing the code of the llama.cpp engine. As a result, the Qwen3.8-Flash-Next Gyro model on a single CPU produces 2.72 tokens per second instead of 1.97, roughly 40% faster (a token is a syllable or a short word). That is still slow for live chat, but for background tasks on a GPU-less server, the patches are worth testing.
AI-processed from Habr AI; edited by Hamidun News
Almost any advice on speeding up local neural networks sounds the same: buy a graphics card. Or compress and fine-tune the model itself. But what if a machine has only a processor and RAM? The author of a Habr post decided to test this in practice. He had the Opus AI rework the code of llama.cpp, a popular engine for running models on your own hardware. As a result, Qwen3.8-Flash-Next Gyro without a graphics card produces 2.72 tokens per second instead of the original 1.97. Here a token is roughly a syllable or a short word, so the speed rose by about 40%. The result is modest, but it comes with numbers and open code.
What exactly was done
The test model was Qwen3.8-Flash-Next Gyro in GGUF format. This is the format in which models are run through llama.cpp. The author did not change the model itself: no fine-tuning, no compression. Instead, he went after the engine's code. Opus did the main work. For several days it ran tests on the author's computer, looked for places where the processor wastes time, and proposed changes to the llama.cpp code.
The effort produced several versions of the patch. In a table, the author compares two of them, fix23 and fix37, against the original build.
What the measurements showed
Two things were measured: prompt reading speed (how many tokens the model "reads" per second) and response generation speed. Each cell in the table contains two numbers. The author did not explain what the second number means, so both pairs are given below.
- Original: prompt 4.12 / 3.75 tokens/s, generation 1.97 / 1.92.
- fix23: prompt 10.98 / 10.05, generation 2.41 / 2.38, speedup ×1.23.
- fix37: prompt 11.21 / 14.49, generation 2.72 / 2.73, speedup ×1.40.
Prompt reading sped up more than generation: by the first numbers, from 4.12 to 11.21 tokens/s, almost threefold. For a processor this makes sense. A prompt can be processed in large batches, while response generation proceeds one token at a time and is limited by memory speed, so it is much harder to squeeze gains out of it. The author also attached a chart: the longer the context, the slower the generation. The patches do not remove this dependence.
What to do with this, and the limitations
A speed of 2.7 tokens per second is still low for interactive chat. But for background tasks such as batch text processing, where there is no rush, the gain is already noticeable. These are measurements on a single machine. The post does not say which processor or how much memory the author has. So the same speedup cannot be promised on other hardware or with other models. The author himself asks whether the same tuning will work for other models, and asks readers to send their configurations and the gain after applying the patches.
The patches are in a merge request on GitLab, in the agentionai-group/llama.cpp repository. If you run local models on a server without a GPU, you can build them and compare against standard llama.cpp on your own workload. In addition, this is an example of an AI assistant working for several days on low-level code on real hardware, rather than writing typical CRUD.
There is also an unpleasant detail. The account under which the author ran Opus was blocked. Verification and appeal did not help, and the block is final. Now he asks the community which model, and in which harness (the wrapper that gives the model tools and lets it work autonomously), can work on personal hardware for as long as Claude Code. The question is practical: if a project depends on a single cloud provider, it can be shut down at any moment.
A mini poll under the post shows how narrow this niche is. Three people voted: two ran LLMs on a computer with a discrete graphics card, one without one, and nobody used specialized NPUs. The sample is tiny. Still, it is a reminder that speedups on ordinary processors matter to a limited circle of enthusiasts. For them, a 40% gain is already tangible.
- Nov 21, 2026MiniMax H3 Image in ComfyUI: Three Fixes Remove the "Crunchy" Skin Without Changing the Model
- Nov 21, 2026llama.cpp 1.0: Stable Interface, 35 Functions Removed, and a Cache for Large MoE Models
- Nov 21, 2026Gemma 4 on a mini PC writes 63 tokens per second, Qwen3.8 only 12.8: a test of 14 LLMs
- Nov 21, 2026Jensen Huang Steps Down as NVIDIA CEO After 33 Years, CFO Colette Kress Becomes Interim CEO
Key Questions Answered
What exactly was done?
The test model was Qwen3.8-Flash-Next Gyro in GGUF format. This is the format in which models are run through llama.cpp. The author did not change the model itself: no fine-tuning, no compression. Instead, he went after the engine's code. Opus did the main work. For several days it ran tests on the author's computer, looked for places where the processor wastes time, and proposed changes to the llama.cpp code. The effort produced several versions of the patch.
What the measurements showed?
Two things were measured: prompt reading speed (how many tokens the model "reads" per second) and response generation speed. Each cell in the table contains two numbers. The author did not explain what the second number means, so both pairs are given below. Original: prompt 4.12 / 3.75 tokens/s, generation 1.97 / 1.92. fix23: prompt 10.98 / 10.05, generation 2.41 / 2.38, speedup ×1.23. fix37: prompt 11.21 / 14.49, generation 2.72 / 2.73, speedup ×1.40.
What to do with this, and the limitations?
A speed of 2.7 tokens per second is still low for interactive chat. But for background tasks such as batch text processing, where there is no rush, the gain is already noticeable. These are measurements on a single machine. The post does not say which processor or how much memory the author has. So the same speedup cannot be promised on other hardware or with other models.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.