A Frozen 16 MB Vocabulary Matches a Trained One in Small LLMs

In Gemma 3 270M, the embedding table takes up 168M of the model's 268M parameters, almost two thirds of the model. The authors of a new paper, featured in Hugging Face Daily Papers, asked what exactly this table buys. They replaced it with a frozen one: 512-bit codes, 16 MB, not a single trainable parameter. They trained a Gemma-architecture model body from scratch on 2.05 billion tokens, once with a conventional vocabulary and once with the frozen one. On LAMBADA, HellaSwag and PIQA, the frozen-table version came out better or equal. Only ARC-Easy went to the trainable variant.

Why the vocabulary became the biggest line item

In small models, the vocabulary outweighs everything else. Quantization methods usually leave embeddings in 16-bit, so once the rest of the network is compressed, the table takes up most of the file. The vocabulary itself stops you from shrinking the model further.

The authors took a pretrained embedding table, projected it once with principal component analysis, and binarized it by sign. The result was a set of 512-bit codes totaling 16 MB. This table does not change during training and needs no gradients. The trainable version gains 0.20 nats of validation loss, but it costs 2.6 times more trainable parameters. On practical benchmarks, that difference is barely visible.

A ternary body and laptop speed

Next, the authors combined the frozen vocabulary with a ternary BitNet b1.58 body. The whole model fits in 46 MB and still holds level with the float model on LAMBADA and PIQA. On a laptop CPU, it generates 615–740 tokens per second and uses 62 MB of memory. For comparison, an 8-bit GGUF of the float model in llama.cpp runs five to six times slower and needs seven times more memory.

Key figures from the paper:

• Gemma 3 270M embedding table: 168M of 268M parameters.
• Frozen vocabulary: 512-bit codes, 16 MB, 0 trainable parameters.
• Training data: 2.05 billion tokens, identical for both versions.
• Cost of the trainable table: 2.6 times more trainable parameters for 0.20 nats of loss.
• Full ternary model: 46 MB, 62 MB of memory at runtime.
• Speed on a laptop CPU: 615–740 tokens per second.

Rare words and new tokens

The most interesting results are in the tail of the distribution. On rare words, the frozen table wins by 1.7–2.3 nats. Even more interesting is what happens when the vocabulary is extended: after training, the authors added 10,000 new words. The model with frozen codes lost about 3%, while the model with the trainable table collapsed by 43%. The reason is that new tokens receive codes from the same space the model already works in, rather than a vector the model body is not adapted to.

There are limitations. The experiment covers a single architecture, a small corpus of 2.05 billion tokens, and a model of around 270M parameters. The paper does not show whether the same effect holds at larger scales. The authors stress that they have released the weights, the data and every response from the evaluations, so the numbers can be checked independently.

Where this is useful

For anyone building models for devices without a GPU, the takeaway is simple: treat vocabulary size as its own budget line, not a side effect of choosing a tokenizer. If a frozen table holds quality on LAMBADA, HellaSwag and PIQA, trainable parameters can go to the model body instead. A 46 MB model that runs at hundreds of tokens per second on an ordinary laptop is suitable for offline suggestions, local text processing and embedded assistants. Another advantage is the ability to add terminology after training with almost no loss, which matters for narrow domains such as medicine, law and code. Before deploying, it is worth testing tasks like ARC-Easy, where the trainable table still keeps its advantage.