OPTIC compresses Gemma 4 E4B's per-layer embeddings 100x while keeping 97.8% of quality

Researchers at VILA-Lab have shown that the per-layer embedding (PLE) tables in Gemma 4 E4B can be compressed roughly 100-fold with almost no loss in quality. The OPTIC method retains 97.8% of the uncompressed BF16 model's performance, aggregated across seven benchmarks. The best of six classical quantizers, product quantization, achieves only 66.3% at the same compression ratio. At the same time, on an RTX 4090 GPU, memory usage drops by 29.7%, while runtime increases by no more than 0.40%.

Why embedding tables became a problem of their own

A clear trend is emerging in language models: capacity is growing not just through transformer computation, but through embeddings as well. Per-layer embeddings in Gemma 4 are a prime example of this approach. In the E4B version, they account for 35.2% of all parameters. That's a huge share, and it sits in memory, which is almost always in short supply on-device.

This creates a challenge: preserve the benefits of large embeddings while still fitting the model into a limited deployment budget. At 100x compression, classical methods hit a ceiling. They encode each token–layer vector independently, leaving roughly five bytes per vector. A code that small can't carry much information. On top of that, the relationships between layers go completely unused.

One code per token instead of a separate code for each layer

OPTIC (Output-Path-fitted, Token-shared Indexed Codes) works differently. For each token, it stores a single quantized code tuple shared across all layers. At each layer, a dedicated affine decoder unpacks it, reconstructing the vector specific to that layer. The byte budget is spent once per token and then shared across layers.

The second part of the idea concerns exactly what gets optimized. The codes and decoders aren't fit to the stored vectors themselves, but to the PLE's output path: the vector passes through a gate and an output projection, and compression is evaluated based on the result after those steps. The objective function matches how the model actually uses the embeddings, so accuracy isn't wasted on details that don't affect the output.

The numbers from the paper

• PLE accounts for 35.2% of Gemma 4 E4B's parameters.
• At roughly 100x compression, each token–layer vector is left with about five bytes.
• BF16 E4B at ~100x: OPTIC retains 97.8% of the uncompressed model's quality (aggregated across seven benchmarks). Product quantization, the strongest of six classical baseline methods, achieves 66.3%.
• At 594x, OPTIC retains 90.5%.
• At 1002x, OPTIC retains 84.1%.
• The official E4B release with 4-bit linear weights and 16-bit activations (W4A16): 96.9% at ~100x PLE compression.
• BF16 E4B inference from compact codes at ~100x: 29.7% less GPU memory at the same KV cache size (1 GiB), with slowdown no greater than 0.40%. Measured on an RTX 4090.

The W4A16 result deserves particular attention. Backbone weight quantization and PLE compression address different parts of the problem, and the 96.9% figure shows that they stack: OPTIC doesn't force a choice between the two.

Who this matters for, and when

The main practical takeaway for anyone deploying Gemma 4 E4B on consumer hardware: PLE no longer has to be an unavoidable memory cost. Saving 29.7% of VRAM at the same KV cache size means either running on a smaller GPU or freeing up room for a longer context. The slowdown—up to 0.40%—is essentially unnoticeable.

For developers building on-device AI products, the underlying principle is also worth noting. If architectures keep increasing the share of parameters devoted to embeddings, those are precisely what will need to be compressed. A shared code per token, paired with per-layer decoders and output-path-aware fitting, offers a working template for such models.

The authors (Jiacheng Liu, Yaxin Luo, Xiaohan Zhao, Peng Sun, Xinyi Shang, Jiacheng Cui, Yonina C. Eldar, and Zhiqiang Shen) have published the code and a breakdown of the method on GitHub, in the VILA-Lab/optic repository. A ready-made artifact with 100x PLE compression is available on Hugging Face under the name gemma-4-E4B-it-OPTIC-100x, so the results can be verified on your own tasks. The preprint is available as arXiv 2610.31309.