Why GPT Gets Letter Counts Wrong: Tokenization Explained

When you type a prompt into ChatGPT, you see ordinary text made of letters. The model sees something entirely different: a sequence of numbers called tokens. This mismatch produces some strange effects. GPT may fail to work out how many letters are in the word strawberry, or get confused during a simple count.

How LLMs see text

To a neural network, text isn't a set of letters but a sequence of numeric codes. Each token maps to a number, and the model works only with those numbers. It never sees the letters themselves. Imagine reading a book through a system that first converts the words into codes, lets you process the codes, and then converts them back into text.

A common word may be a single token, while a rare word gets split into several pieces. For example, "hello" is encoded as one number, while "strawberry" may become three or four numbers. Russian is even trickier: its rich morphology means words are split less efficiently.

Why this causes problems

The gap between how people see text and how the model sees it leads to a whole range of problems:

• Counting errors: GPT can't simply count letters, because it works with numbers, not characters
• Poor handling of rare words: unusual letter combinations get split into several tokens, and the model sees them as separate chunks
• Language asymmetry: English tokenizes more efficiently than Russian, Chinese, or Arabic
• Context consumption: if a word splits into 3 tokens instead of 1, your prompt takes up more room in the context window
• Unpredictable behavior: the model can act strangely with numbers, codes, and rare names, because they're broken into fragments

The algorithm behind the scenes

Tokenization relies on an algorithm called Byte Pair Encoding (BPE). Here's how it works: first, the text is split so that every character is its own token. The algorithm then finds the most frequent pairs of characters and merges them into a new token. Next, it looks for frequent pairs among the resulting tokens and merges those too. This repeats thousands of times.

The result is a GPT vocabulary of roughly 50 thousand tokens. Common words and word fragments get their own tokens, while rare letter combinations stay fragmented. It isn't perfect, but it's more efficient than encoding every letter separately.

What this means

Understanding tokenization changes how you work with LLMs. If you know the model will get confused counting letters, you can ask it to approach the task differently — for example, by first printing the letters out one by one and then counting them. It's not a cure-all, but it helps you write more reliable prompts.

Knowing how tokenization works also helps when optimizing long prompts: you can predict where the model will "waste" tokens. That matters for anyone working with LLMs at a deep level, from prompt engineers to developers building applications on top of neural networks.