ttok 1.0 Switches to the GPT-5/6 Tokenizer: Seven Models Return 44,794 Tokens

Simon Willison has released ttok 1.0. It is a command-line tool for counting tokens, and it now defaults to the GPT-5/GPT-6 family tokenizer instead of GPT-4. The trigger was a small thing. The author shipped ttok 0.4, upgraded with `uv tool upgrade ttok`, ran a file through the new version, and saw that the tool was counting tokens with the previous generation's scheme. He decided to ship the default fix as a major release. An independent test that Willison cites shows that seven OpenAI models counted the same 44 794 tokens across 31 test files.

Why the default changed

Token counts matter constantly. They are used to estimate the cost of a request, check whether a document fits in the context window, and trim long texts before sending them to a model. ttok does this from the command line: you feed it text and get back a number. If the tool has the GPT-4 tokenizer baked in, its numbers drift from reality. The error is small, but when you budget for thousands of requests, it adds up.

Willison admits that 1.0 is partly a formality: a good excuse to finally ship a round version number. But changing the default behavior does break backward compatibility, so a major version number is appropriate here.

What William Liu's experiment showed

Willison is not certain that GPT-6 uses the same tokenizer as GPT-5. OpenAI has not confirmed it. The tiktoken repository has issue #608, which the author calls "angry": users are demanding clarity. William Liu provided an indirect answer. In a commit to the token-count-compare repository, he described an experiment with seven models:

• GPT-5.5;
• GPT-5.6 in the Sol, Terra and Luna variants;
• GPT-6 in the Astra, Sol and Luna variants.

The results, according to Liu:

• all seven models returned 44 794 tokens;
• the results matched on each of the 31 test files;
• GPT-6 did not change input token counts on this corpus.

On that basis, Willison judged it reasonable to make GPT-5/GPT-6 the default.

The catch

The evidence remains indirect. It is a comparison of input token counters on a single set of 31 files, not a published vocabulary and splitting rules. Matching totals and per-file values on such a corpus point strongly toward a shared tokenizer, but divergence on rare languages, exotic markup or long sequences cannot be ruled out yet. Until OpenAI speaks up, the conclusion remains a working hypothesis.

The difference between input and output tokens also matters. Liu's test covers input counting only. It says nothing about how the models behave during generation.

How to use it now

If you installed ttok through uv, upgrade with `uv tool upgrade ttok`. After that, counting will follow the GPT-5/GPT-6 scheme by default. If you set budgets or context limits based on GPT-4 numbers, recalculate them: the result may change on the same texts. It is worth checking any scripts that relied on the old default. If accuracy is critical for you, compare ttok's estimate against the usage field in API responses on a few real documents. And wait for an answer from OpenAI on issue #608 before treating the question as closed.