NVIDIA X-Token: A Distillation Method That Beats GOLD by 3.82 Points

NVIDIA has released X-Token, a method for optimizing small language models. X-Token is a knowledge distillation approach that fixes two structural flaws of the earlier GOLD method and shows a significant improvement on standard benchmarks.

What X-Token is

X-Token stands for Projection-Guided Cross-Tokenizer Knowledge Distillation. Put simply, it is a way to transfer knowledge from a large model to a small one while accounting for their different token vocabularies (the sets of units into which a model splits text). Small models often run on their own tokenizers, the dedicated system that parses text, and earlier distillation ignored this. X-Token solves the problem.

The method introduces an intermediate projection layer that translates representations between two different token spaces. It works like a translator operating at the very foundation of the model. When the large model passes knowledge to the small one, X-Token ensures that information is not lost in translation from one encoding scheme to another.

Results that impress

On the Llama-3.2-1B model, X-Token shows a consistent advantage:

• On general benchmarks: an average improvement of 3.82 points over GOLD
• On math tasks (GSM8k): a jump from 2.56% to 15.54% accuracy
• On the standard MMLU test: a rise from 24.0 to 24.7%

This is not a marginal gain: math accuracy grew 6 times. For a small 1-billion-parameter model, that matters. Every percentage point counts, because at this scale even a small improvement in capability helps the model handle harder tasks.

GOLD's structural flaws

The earlier GOLD method ignored that the tokenizer in a small model may be entirely different. That led to two problems: first, knowledge from the large model lost its meaning when the small model translated it into its own vocabulary; second, distillation could not make effective use of all the small model's capabilities.

X-Token builds a projection between different token spaces into the distillation process. It acts as a bridge between two information encoding systems. This matters especially when the small model is tuned for fast execution on mobile or edge devices and has its own unique tokenizer to save memory.

What this means

Small models are needed everywhere: on phones, in IoT devices, on local servers where there is no cloud access or where latency is critical. X-Token shows that it is possible to take the knowledge of a huge model and effectively "compress" it into a small format, keeping the small model's own vocabulary.

This is a path to AI that works everywhere, not only on cloud compute. And the 6-fold improvement on math is a signal that small models are starting to gain real capabilities for practical tasks. Local AI may soon become the standard rather than the exception.