Nepali Text "Weighed" Three Times More Than English — Tokenizer Fix Couldn't Fully Solve It

Why language models waste tokens on Nepali

A team of researchers figured out why language models like Llama-3 and Qwen3 handle Nepali text so wastefully — and tried to fix it. It turns out the problem isn't in the model's vocabulary at all, but in the very first, most invisible step of text processing: the pre-tokenization split that happens before the tokenizer even starts assembling tokens.

Where the token overhead comes from

Tokenizers like GPT-2's, which underpin Llama-3 and Qwen3, cut text with a regular expression before the main tokenization step. This expression only counts a sequence of Unicode letters as a "word." The trouble is that Nepali uses the Devanagari script, where vowels aren't represented by separate letters but by special marks placed above or below a base letter — so-called combining marks. In Unicode, these marks don't formally count as "letters," so the regular expression slices Nepali words apart at literally every vowel mark.

The result: where an English speaker spends one token, Nepali text on the same models requires 2-3 times more tokens to convey the same amount of meaning. The researchers measured this on the FLORES-200 parallel corpus (identical sentences across different languages) — Qwen3's tokenizer spends noticeably more tokens on a Nepali sentence than on its English translation, and the tokenizer used by Phi-4, OLMo-2, and Granite-4.1 is even worse.

For users, this isn't an abstract metric. More tokens for the same meaning means a more expensive request, a slower response, and less useful content fitting inside the model's context window.

Why the usual vocabulary "tune-up" doesn't help

The standard way to teach a finished model to better handle a new language is so-called merge-based vocabulary extension: new merged tokens are added to the vocabulary and fine-tuned on text in the new language. The problem is that the regular-expression split happens before any merges are applied — and if a word has already been cut apart at its vowel marks, no number of new merges can glue back together pieces separated by that cut. The researchers prove that this approach has a mathematical "ceiling" on possible savings, one that depends only on the regular expression, not on vocabulary size.

They verified this against real-world releases: they found ten previously published vocabulary extensions across six different scripts (for Llama-3, Llama-3.1, Qwen2.5, and Qwen3) — and all ten hit their respective ceilings, squeezing out at least 98% of the possible savings. In other words, the developers of these models had already done everything that could be done with the standard approach — pushing further on vocabulary alone is pointless.

The researchers also tested a loophole: registering Nepali strings as "added tokens" that bypass the regular expression entirely. This does break through the ceiling, but it ruins word-boundary segmentation — a fifth of such tokens end up matching right in the middle of a word, which undermines the meaningfulness of the split.

What "repairing" the regular expression achieved

The researchers rewrote the regular expression itself so that Devanagari vowel marks count as part of the word rather than as separate characters. An important property of their fix: they mathematically proved that for any text without combining marks and without Devanagari characters (that is, for English and code), tokenization remains exactly identical to before — not a single token changes.

With the new regular expression and 32,000 new merged pairs, Nepali text ended up requiring almost as many tokens as English — 54% fewer tokens than the standard vocabulary extension.

But when they actually fine-tuned Llama-3.2-1B and Qwen3-0.6B on a mix of Nepali and English text, the result was unexpected: the original, "unfixed" tokenizer showed better Nepali prediction quality (measured in bits per byte) on an equal budget than either the fixed or the standard extended versions. This was the researchers' main hypothesis — and it didn't hold up.

With an equal number of training steps (rather than bytes), the fixed tokenizer did perform better on Qwen but worse on Llama — meaning the gain was inconsistent across models. Increasing the learning rate for Qwen improved results for all three variants, but didn't change their relative ranking.

What this means in practice

For engineers fine-tuning open models for languages with combining marks (not just Nepali, but also, say, Thai, Arabic, or Bengali), the takeaway cuts both ways. On one hand, the token savings are real and mathematically confirmed — the fixed tokenizer genuinely spends 54% fewer tokens on the same Nepali text, without touching English or code. On the other hand, fewer tokens doesn't automatically mean a better model: the researchers' own tests show the old, "unfixed" tokenizer still wins on quality at an equal compute budget.

One question remains open, which the researchers honestly didn't test: whether the token savings translate into practical gains in generation speed and cost, even if quality is slightly worse. The code, trained tokenizers, and analysis plan are openly available on GitHub and Hugging Face.