Compressed Bonsai 2 27B hit 17 tokens/s on a 2017 GPU, while Qwen3 8B hit 32
A developer wanted to run a local model on an old mining card with 8 GB of memory and picked the compressed Bonsai 2 27B, which squeezes 27B parameters from 54 GB down to 5.9 GB. It ran, but a regular Qwen3 8B in the same 8 GB delivered almost the same result: 36 points out of 40 vs 35 for Bonsai, at 32 tokens per second vs 17. Bonsai requires the PrismML fork: Ollama and standard llama.cpp cannot read it. For a home assistant on 8 GB, it is simpler to take an 8B model with no workarounds.
AI-processed from Habr AI; edited by Hamidun News
Imagine you want to set up a local assistant without the cloud, on hardware you already have under your desk: say, an old mining GPU with 8 GB of memory. A typical 27-billion-parameter model weighs tens of gigabytes and will not fit. The studio PrismML released Bonsai 2 27B, squeezed down to 5.
9 GB, and promised near-full quality. The author of a write-up on Habr tested this on a 2017 NVIDIA P104-100 mining card, an i7-3770 CPU without AVX2, 15.5 GB of RAM, and two inexpensive Android phones.
The bottom line for practitioners: it can be run, but a regular Qwen3 8B, which already fits in 8 GB, performs almost as well and twice as fast. Speed here is measured in tokens per second (t/s); a token is a chunk of a word, roughly a few letters.
How 27B fits into 5.9 GB
A regular model stores each weight as a 16-bit number, and 27B in that form weighs about 54 GB. In Bonsai 2, each weight has only three possible values: −1, 0 and +1 (this kind of compression is called ternary). In the PTQ1_0 format that comes to 1.75 bits per weight and 5.9 GB; in the PQ2_0 format, 2.13 bits and 7.2 GB. The developers claim it retains 98.2% of the quality of the original Qwen3.8-27B, and that multiplication is replaced by addition and sign flipping.
Size also matters for speed: to produce a single token, the card reads all the weights, so speed is more often limited by memory bandwidth than by compute.
There is an unpleasant catch. Only the PrismML fork understands ternary tensors. Ollama throws an error when it tries to read such a file. Standard llama.cpp is worse: it loads the old Q2_0 format silently, and then the model answers with gibberish. Before compression, the weights are rotated with a Hadamard matrix to smooth out rare outliers, and the inputs must be rotated the same way at inference time, which regular llama.cpp does not do. Ollama can be made to work with the fork through patches, but they disappear with the first update.
What the measurements showed
On the P104-100, Bonsai PTQ1_0 reads the prompt at 93.8 t/s and generates about 17 t/s. With a 16K-token context, it uses 7967 MB out of 8191, and at 32K the model does not start, so the advertised 262K context remains theoretical for 8 GB.
On an RTX 4090, according to PrismML, speed is 81–91 t/s, so the old card delivers roughly a fifth of that. A regular 2-bit Qwen UD-IQ2_XXS of the same size on this card is six times slower at reading the prompt (15.7 t/s) and 1.
6 times slower at generating (10.8 t/s). Speed depends on kernels written for a specific format, not on the number of bits per weight.
The CPU without a GPU managed 0.6 t/s. The disk affects only startup: 25 seconds from NVMe and 87 from HDD; after that, speed is almost identical, 16.
5 and 17.0 t/s.
On quality, there is almost no difference. On a set of 40 short tasks, Bonsai scored 35 points and Qwen3 8B scored 36. Code, JSON, finding a fact in a text, general knowledge and Russian text are handled almost flawlessly by nearly every model.
Only logic and math diverge, and here the 8B model won somewhat unfairly: the prompt said "answer with a number only," yet it wrote out its calculations step by step anyway. When the author enabled reasoning in Bonsai, it solved all five tasks instead of three, spending 63 tokens. On a set of 20 hard tasks with rare knowledge and 8K and 16K contexts, the 27B model showed no clear advantage either: the 8B scored 8 out of 10 at 31.
9 t/s, Bonsai 7 out of 10 at 16.8 t/s. The 27B wrote fairy tales slightly better, 19 points out of 25 vs 18, but a sample of five stories is small.
What about phones
The TECNO Pouvoir 4 is strictly 32-bit: the fork's Android builds exist only for arm64, and a 32-bit process cannot open a 5.9 GB file because of the 4 GB address limit. The smaller Bonsai 2B and 4B had to be built from source, and they deliver 0.
6 to 1.4 t/s. On the TECNO Pova 5 with 8 GB, the 27B model launched and answered 5 of 6 questions correctly.
But an app on an 8 GB phone gets only about 3–4 GB, so the phone re-reads the weights from flash storage for every token. The result is 0.05 t/s, or about 20 seconds per token.
On a flagship with 12–16 GB, the picture is probably better, but the author did not test this.
What to do in practice
For a local assistant on 8 GB, Qwen3 8B is the simpler choice: standard tools, twice the speed and comparable quality. Bonsai is worth trying if you have a card with 12–16 GB and specifically need the 27B class, or if you are ready to live on a fork. The author's method has limits: one run per task, small test sets and a subjective evaluation of the fairy tales.
- Nov 18, 2026LLMs on a GPU-less VPS: 4B models sort support tickets with 90% accuracy in 3–5 seconds
- Nov 18, 2026Qwen on two DGX Spark units runs 34–59% faster in tokens per second and outpaces GLM
- Nov 18, 2026Language Model on ESP32-S3 Sped Up to 406 Tokens/s — 18x Faster
- Nov 18, 2026After a Failure, Agents Roll Back Their Work in Only 47% of Cases, Though They Solve 84% of Tasks
Key Questions Answered
How 27B fits into 5.9 GB?
A regular model stores each weight as a 16-bit number, and 27B in that form weighs about 54 GB. In Bonsai 2, each weight has only three possible values: −1, 0 and +1 (this kind of compression is called ternary). In the PTQ1_0 format that comes to 1.75 bits per weight and 5.9 GB; in the PQ2_0 format, 2.13 bits and 7.2 GB. The developers claim it retains 98.2% of the quality of the original Qwen3.8-27B, and that multiplication is replaced by addition and sign flipping.
What the measurements showed?
On the P104-100, Bonsai PTQ1_0 reads the prompt at 93.8 t/s and generates about 17 t/s. With a 16K-token context, it uses 7967 MB out of 8191, and at 32K the model does not start, so the advertised 262K context remains theoretical for 8 GB. On an RTX 4090, according to PrismML, speed is 81–91 t/s, so the old card delivers roughly a fifth of that. A regular 2-bit Qwen UD-IQ2_XXS of the same size on this card is six times slower at reading the prompt (15.7 t/s) and 1.6 times slower at generating (10.8 t/s).
What about phones?
The TECNO Pouvoir 4 is strictly 32-bit: the fork's Android builds exist only for arm64, and a 32-bit process cannot open a 5.9 GB file because of the 4 GB address limit. The smaller Bonsai 2B and 4B had to be built from source, and they deliver 0.6 to 1.4 t/s. On the TECNO Pova 5 with 8 GB, the 27B model launched and answered 5 of 6 questions correctly. But an app on an 8 GB phone gets only about 3–4 GB, so the phone re-reads the weights from flash storage for every token.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.