Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)

Gemma 4 on a mini PC writes 63 tokens per second, Qwen3.8 only 12.8: a test of 14 LLMs

A Habr author spent 86 hours running 14 local language models on a mini PC with a Ryzen AI Max+ 395 chip and 96 GB of memory. The dense Qwen3.8 27B generates 12.8 tokens per second, while Gemma 4 26B A4B, which activates only a subset of its parameters for each token, reaches 63, nearly five times faster. Qwen3.8 reads long documents best of all, and the author kept it for complex tasks. For live chat, pick the fast model; for heavy background work, pick the dense one.

AI-processed from Хабр: Машинное обучение; edited by Hamidun News
Gemma 4 on a mini PC writes 63 tokens per second, Qwen3.8 only 12.8: a test of 14 LLMs
Source: Хабр: Машинное обучение. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceХабр: Машинное обучение ↗
Editor-in-Chief
Key metric #1Windows allowed the integrated GPU a maximum of 48 GB of memory, Ubuntu 80 GiB
Key metric #2Processing a 200K-token context takes the models from 8 to 23 minutes
Key metric #3Gemma 4 26B Uncensored powers the chatbot, Qwen3.8 27B with MTP handles background work

Two local chatbots on the same compact computer can respond with a five-fold difference in speed, even though their model files are nearly the same size. The author of the Habr review ran 14 local neural networks on a GMKtec EVO-X2 mini PC over a week, 86 hours in total. Inside is a Ryzen AI Max+ 395 processor with an integrated Radeon 8060S GPU and 96 GB of memory, which the CPU and GPU share.

He measured speed in tokens per second: a token is a fragment of a word, and the more tokens per second, the faster text appears on screen. The dense Qwen3.8 27B produced 12.

8 tokens per second, while Gemma 4 26B A4B produced 63. If you are considering moving some tasks off the cloud, these numbers can help you choose both the model and the hardware.

What the setup looked like and why it was needed

Software on the machine: Ubuntu 26.04, LM Studio 0.4.28, and the llama.cpp engine on Vulkan. All model layers were loaded onto the GPU, Flash Attention was enabled, and requests were processed one at a time. Almost all models were run in Q4 quantization (weights stored in four bits to take up less memory); the exception was gpt-oss-20b, which used MXFP4. Context, meaning the amount of text the model holds in view, was tested at 64K, 128K, and 256K tokens. Three models support only up to 128K. The author candidly admits an omission: he did not record the KV-cache quantization type, so the conditions cannot be reproduced exactly.

The first practical lesson concerns the operating system. For local LLMs, how much memory can be allocated to the GPU matters. On Windows, the 96 GB configuration could allocate no more than 48 GB to the GPU; on Linux it reached 80 GiB. That is why the mini PC runs Ubuntu. If you are considering a similar platform for large models, check this limit before buying.

The machine came about for two reasons: it was shipped from China, and then there was an attack on a Yandex data center. The author's agents ran in Yandex Cloud, and he was left without his usual tools. He did not calculate payback and will continue to use leading cloud models while he can. The question was practical: which parts of everyday work can be handed entirely to local models.

Speed: file size promises nothing

Twelve models took part in the qualifying round. Four leaders and two newcomers, who were added after the selection, advanced to the final, so 14 models went through the tests in total. The most striking result concerns model architecture. The dense Qwen3.8 27B generates 12.8 tokens per second. The Gemma 4 26B A4B model on the same hardware produces 63, although the files are almost the same size.

The difference is explained by architecture. In a dense model, all parameters work for every token. Gemma 4 26B A4B is an MoE ("mixture of experts") model: for each token only some experts are activated, judging by the name, roughly 4 billion of the 26 billion parameters. It needs the same amount of memory but performs far fewer computations, hence the roughly five-fold difference. For chat, where a person waits for a reply, 12.8 tokens per second feels like slow typing, while 63 feels like a normal conversation.

Long context: reading takes minutes

Window size did not affect speed or scores when the request was the same. Enabling 256K "just in case" carries no penalty in itself. But reading long input is expensive: processing 200K tokens takes the models from 8 to 23 minutes. For an agent that regularly re-reads a large repository or a set of documents, this is a direct constraint on the scenario.

Qwen3.8 performed best on long context. This is the main reason the author kept it, despite its low generation speed.

What stayed in memory

The author keeps two models loaded at once. Gemma 4 26B Uncensored answers in a conversational bot, where quick replies matter. Qwen3.8 27B with MTP takes on harder tasks and background work, where waiting is acceptable but quality and stability on large inputs are required. MTP is a multi-token prediction mode that helps speed up generation.

What to take from the review. First, choose the model for the type of task: a fast MoE for dialogue, a dense one for heavy background processing. Second, budget time for reading long context, not just for generating the answer. Third, check how much memory your operating system gives the GPU.

The limitations are worth remembering too. These are results from one person on one configuration. He refined the methodology as he went, some runs were restarted, and the KV-cache quantization was not recorded. As a guide for a home or small office server, the numbers are useful; as a rigorous benchmark, they are not. The author himself writes that he wants to save readers a few evenings, not deliver a final verdict.

Story

Key Questions Answered

What the setup looked like and why it was needed?

Software on the machine: Ubuntu 26.04, LM Studio 0.4.28, and the llama.cpp engine on Vulkan. All model layers were loaded onto the GPU, Flash Attention was enabled, and requests were processed one at a time. Almost all models were run in Q4 quantization (weights stored in four bits to take up less memory); the exception was gpt-oss-20b, which used MXFP4. Context, meaning the amount of text the model holds in view, was tested at 64K, 128K, and 256K tokens. Three models support only up to 128K.

What are the key details on Speed: file size promises nothing?

Twelve models took part in the qualifying round. Four leaders and two newcomers, who were added after the selection, advanced to the final, so 14 models went through the tests in total. The most striking result concerns model architecture. The dense Qwen3.8 27B generates 12.8 tokens per second. The Gemma 4 26B A4B model on the same hardware produces 63, although the files are almost the same size. The difference is explained by architecture. In a dense model, all parameters work for every token.

What are the key details on Long context: reading takes minutes?

Window size did not affect speed or scores when the request was the same. Enabling 256K "just in case" carries no penalty in itself. But reading long input is expensive: processing 200K tokens takes the models from 8 to 23 minutes. For an agent that regularly re-reads a large repository or a set of documents, this is a direct constraint on the scenario. Qwen3.8 performed best on long context. This is the main reason the author kept it, despite its low generation speed.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…