BitNet b1.58: How Ternary Quantization Lets a 27B Model Run Without a GPU

A few years ago, the idea of running a 27B-parameter model on an ordinary laptop without a GPU seemed absurd. In 2024–2025 it became reality, thanks to unconventional quantization: instead of approximating the weights, the network is built on just two or three discrete values.

What is unconventional quantization

Standard quantization lowers weight precision along the scale float32 → INT8 → INT4 → INT2, while keeping a continuous numerical range. The unconventional approach is more radical: weights take only the values {-1, 0, 1}. This is ternary quantization. Microsoft Research developed the BitNet family, in which each weight is stored in 1.58 bits on average.

• BitNet — an architecture with 1-bit weights during training, proposed by Microsoft Research in 2023
• BitNet b1.58 — the ternary version with weights {-1, 0, 1}, 1.58 bits per parameter on average, published in March 2024
• Gemma 27B in ternary format takes up 6–7 GB and runs on a CPU-only laptop
• llama.cpp has supported 1-bit GGUF formats since March 2024
• Multiplications are replaced by additions and subtractions, so the CPU's floating-point units are not loaded

Why this is not the same as INT4

Classic INT4 quantization is post-training compression: a finished float32 model is squeezed after training, and quality inevitably drops, especially at small model sizes. Unconventional quantization works differently: the network is trained from the start so that its weights sit at discrete positions. This fundamentally changes the nature of the losses.

"A model with weights {-1, 0, 1}, when trained properly, is comparable to its full-precision counterpart in perplexity while requiring 3–5 times less RAM," the

Microsoft Research technical report on BitNet b1.58 states.

According to open benchmarks, the gap to a float16 baseline for models of 7B and above is 1–3% on most tasks. That is an acceptable price for fitting a model into the RAM of an ordinary laptop.

What runs on a laptop right now

The most telling example is 27B-class models on a MacBook with 32 GB of unified memory, or on a Windows laptop with a regular processor and no discrete GPU. Generation speed for a ternary 27B model on an Apple M2 Pro is about 10–15 tokens per second, enough for most practical tasks.

One important caveat: unconventional quantization works best on models trained from scratch in the corresponding mode. Retroactively "ternarizing" an existing model such as Llama 3 without a noticeable quality drop is hard: it requires either fine-tuning or distillation through a ternary teacher.

What this means

Ternary and 1-bit quantization shift the accessibility boundary: full-scale language models of 27B and above no longer require server hardware. For enterprise scenarios with data confidentiality requirements, this matters especially: local inference without the cloud becomes technically and economically realistic.

Frequently asked questions

How do I run a ternary model without a GPU?

Ternary and 1-bit models in GGUF format run through llama.cpp on an ordinary processor. 16–32 GB of RAM is enough for models from 7B to 27B, and no GPU is required.

How does BitNet b1.58 differ from INT4 quantization?

BitNet b1.58 is trained with ternary weights from the start rather than getting them through post-training compression. This gives more predictable quality and removes multiplication operations from inference entirely.