Amharic input costs 6x more than English in GPT-5: 78 tokens vs. 13

Imagine you are launching a customer-support chatbot and first calculate the budget using English texts. Then you enter the Ethiopian market, and the same reply to a customer suddenly costs six times more. The cause is the tokenizer: it cuts text into pieces (tokens), and model providers charge for tokens, not for meaning.

The Coalition for Language Equity in AI Research (CLEAR) has published the first study to measure this difference and has named it the tokenizer tax. In GPT-5's tokenizer, the same phrase takes 13 tokens in English, 26 in Malayalam and 78 in Amharic. Models received grades from A to F, and GPT-5 got the worst one: F.

If you build a product for more than English-speaking users, these numbers directly affect your bill.

Where the tax comes from

Tokenizers were trained mostly on English and about a dozen high-resource languages. As a result, English words often remain whole tokens, while text in less-represented languages shatters into small fragments. The meaning is the same, but there are more tokens, and you pay for each one.

Amharic has a particular reason for the gap: each character of the Ethiopic script is split into two tokens of one byte each. As a result, Amharic input comes out roughly six times more expensive than English, with the same content and almost the same number of characters.

The measurements used the FLORES-200 set of phrases, and the analysis covered 203 language-script pairs. The main metric is the ratio of the token count for the most expensive language to that of the cheapest. The researchers also calculated the Gini coefficient, a measure of inequality from economics that is typically used to describe income stratification. The CLEAR scale works like this: up to 3x is an A, up to 6x a B, up to 9x a C, up to 12x a D, and anything higher is an F.

Who got which grade

On the twenty languages most represented on the internet, the models barely differ: the spread ranges from 1.5 to 3.1x. The problem shows up across all 200 languages. There, only the translation model NLLB-200, which was built for 200+ languages from the start, received the top grade, A. Gemini and Gemma got a B. The Chinese stack (Qwen 3, DeepSeek V4.1 Flash and Kimi K3), as well as Fable 5, Opus 4.8 and Llama 4 Scout, got a D. GPT-5 got an F.

Script turned out to matter more than the number of speakers. Languages written in the Latin alphabet are cheaper even when they are not widely spoken: Asturian is absent from Common Crawl data, but it benefits from the Latin script and its kinship with the Romance languages. Burmese, Tibetan, Tifinagh, Khmer, Lao and Ol Chiki fare worst. Large languages pay the tax too: Urdu (288 million speakers), Bengali (266 million) and Punjabi (200 million).

Not just price

The authors point to two more consequences of the same inflated token count. The first is latency: the model has to process more tokens for the same text, so the response takes longer to arrive. The second is usable context: in the same nominal 128 thousand tokens, a low-resource language fits only part of the text that would fit in full in English. Roughly applying the 78-to-13 ratio, an Amharic context window would hold about one-sixth of the English volume. This is our own back-of-the-envelope arithmetic; the report contains no such figure. For RAG and long documents, this means they will have to be cut earlier, and more of the context will be lost.

What to do in practice

CLEAR asks vendors for three things: publish both metrics in model cards, factor the share of languages into the vocabulary of next-generation tokenizers and, since a tokenizer cannot be changed before a new base model is trained, normalize pricing by characters now.

The advice for developers and product managers is simpler:

• don't calculate your budget from an English prototype; run real texts through the tokenizer of the model you need and check the multiplier;
• compare providers not only by price per million tokens but also by how many tokens your language consumes;
• build in a margin for latency and the context window for languages with inefficient tokenization;
• the test code is in the project's open repository, and you can run it on other models.

This is only the first part of the work: it measures token counts on short phrases, not the quality of responses. CLEAR promises to publish a follow-up on equal quality across languages later.