BitNet b1.58: как тернарное квантование позволяет запустить 27B-модель без GPU

A few years ago, the idea of running a 27B-parameter model on an ordinary laptop without a GPU seemed absurd. In 2024–2025, this became a reality — thanks to unconventional quantization: instead of approximating weights, the network is built literally on two or three discrete values.

What is unconventional quantization

Standard quantization reduces weight precision along the scale float32 → INT8 → INT4 → INT2, preserving a continuous numerical range. The unconventional approach is more radical: weights take only the values {-1, 0, 1} — this is ternary quantization. Microsoft Research developed the BitNet family, in which each weight is stored at an average of 1.58 bits.

• BitNet — architecture with 1-bit weights during training, proposed by Microsoft Research in 2023
• BitNet b1.58 — ternary version with weights {-1, 0, 1}, averaging 1.58 bits per parameter, published in March 2024
• Gemma 27B in ternary format occupies 6–7 GB and runs on a CPU-only laptop
• llama.cpp has supported 1-bit GGUF formats since March 2024
• Multiplications are replaced by additions and subtractions — with no load on the CPU's FP units

Why this is not the same as INT4

Classic INT4 quantization is post-training compression: a ready float32 model is "squeezed" after training, and quality inevitably drops, especially at small sizes. Unconventional quantization works differently: the network is trained from the start so that weights settle into discrete positions. This fundamentally changes the nature of the losses.

"A model with weights {-1, 0, 1}, when properly trained, is comparable to its full-precision counterpart in perplexity, while requiring 3–5 times less RAM," states the

Microsoft Research technical report on BitNet b1.58.

According to open benchmarks, the gap with the float16 baseline for models from 7B is 1–3% on most tasks — an acceptable price for fitting the model into the RAM of an ordinary laptop.

What runs on a laptop right now

The most illustrative example is 27B-class models on a MacBook with 32 GB of unified memory or a Windows laptop with a standard processor and no discrete GPU. Generation speed on Apple M2 Pro for a ternary 27B is approximately 10–15 tokens per second — sufficient for most practical tasks.

An important caveat: unconventional quantization works best on models trained from scratch in the corresponding mode. Retroactively "ternarizing" an already finished model — such as Llama 3 — without a noticeable quality drop is difficult: it requires either fine-tuning or distillation through a ternary teacher.

What this means

Ternary and 1-bit quantization shifts the accessibility boundary: full-fledged language models of 27B and above no longer require server-grade hardware. For enterprise scenarios with data privacy requirements, this is especially relevant — local inference without the cloud becomes technically and economically feasible.

Frequently Asked Questions

How to run a ternary model without a GPU?

Ternary and 1-bit models in GGUF format run via llama.cpp on a standard processor. 16–32 GB of RAM is sufficient for models from 7B to 27B — no GPU required.

How does BitNet b1.58 differ from INT4 quantization?

BitNet b1.58 is trained with ternary weights from the very beginning, rather than obtaining them as a result of post-training compression. This delivers more predictable quality and completely eliminates multiplication operations from inference.