Hugging Face's BERT Tokenizer Gets a 50x Speedup by Skipping Needless Work on ASCII

A tokenizer turns a string into the numbers a neural network accepts. In Hugging Face's tokenizers library, this step was needlessly slow for BERT models. The team profiled the code, found the redundant work and removed it. The BERT tokenizer is now roughly 50x faster than in version 0.23. That figure comes from two steps: an 8.6x speedup from the v1.0.0 release preparation, and another 5.7x from the new patch. The patch is still in a pull request. In parallel, classic BPE tokenizers, which power most LLMs, got up to 30x faster.

Why the old BERT still matters

BERT came out in 2018, but its descendants are still everywhere: embeddings, search, RAG, rerankers, classifiers. The three most-downloaded models on the Hugging Face Hub use the BERT normalizer:

• all-MiniLM-L6-v2: 218 million downloads a month; • the ms-marco-MiniLM-L6-v2 cross-encoder: 79 million; • bge-small-en-v1.5: 59 million.

bert-base-uncased itself ranks sixth with 34 million. Among the 10 000 most popular models, BPE accounts for 842 million downloads a month and WordPiece (the BERT algorithm) for 627 million. More than half of BPE downloads go to tokenizers with Unicode normalization, most often NFC, as in Qwen or ModernBERT. These improvements speed that up too.

Where the time went

Normalization maps different encodings of the same character to a single form. For example, "é" can be written as one code point, or as the letter "e" plus a separate accent mark. Without normalization, the model would produce different tokens for them. Uncased BERT has five steps: cleaning control and whitespace characters, adding spaces around Chinese characters, NFD decomposition, stripping diacritics and lowercasing.

A flame graph (a chart where block width shows the share of time a function takes) gave an unambiguous picture. The normalizer consumed 88% of the time, while WordPiece itself, which picks the tokens, took only 8%. About two thirds of the total time went to the binary_search_by function, which checked whether a character is a combining mark. That is a binary search over an array of 1567 characters (6 KB), about 11 comparisons with branches the CPU cannot predict. The search ran for every character, including ordinary Latin "s", "l", "o" and "w".

Three lazy tricks

The starting point was studying the data. Real texts contain a lot of ASCII, characters from one script tend to appear together, and most text is already normalized. That led to three techniques.

• Skipping ASCII with SIMD. In UTF-8, a byte is ASCII if its high bit is zero. The CPU loads 16 bytes, gathers their high bits into a number with one instruction and compares it with zero.

If it is zero, the whole block needs no decoding, no table lookups and not even copying. If it is not, counting zero bits finds the first non-Latin byte. Converting uppercase ASCII letters to lowercase is also done in a batch: an OR operation with 0x20.

• Caching by code point blocks. Cyrillic occupies 256 consecutive code points (U+0400–U+04FF). The first time a character from a block appears, the algorithm immediately computes the answer to "can normalization affect it?"

for all 256 positions. After that, a single memory read is enough. The block takes four 64-byte cache lines and stays in the fastest cache next to the core.

• Lazy copying. If the text did not change, the original string is returned without allocating memory. On the first edit, memory is allocated once, and untouched segments are copied in large chunks.

What worked and where it did not help

On English text, speed rose from 23 to 151 MB/s relative to the main branch, and the normalizer's share of runtime fell from 88% to 13%. Tokens match the previous output on all test corpora. The more ASCII a text contains, the bigger the gain: English and source code sped up the most. Russian, Greek and Arabic contain about 11% ASCII, yet the block cache made them 2.2–3.2x faster. For ModernBERT with NFC, English barely changed (1.06x), because ASCII was already being skipped there, while code, Arabic and Russian became about twice as fast.

The approach has weak spots:

• Chinese text did not speed up: it contains little ASCII, and BERT puts spaces around every Chinese character anyway. • Strings up to 32 bytes may be processed more slowly, because the overhead exceeds the gain. In the worst case (StripAccents on 8 bytes of Chinese), speed drops to 0.35 of the previous level. • For the normalizer alone, slowdowns above the margin of error appeared in 55 of 320 cases, 19 of them on inputs up to 32 bytes. • Cased BERT processes Chinese 8–15% slower on inputs of 64 bytes or more.

The takeaway for engineers is simple. If you run millions of documents through MiniLM or bge for search and RAG, tokenization on CPU can be a noticeable cost, and after the v1.0.0 release it will get much cheaper. If your pipeline has many short or Chinese strings, measure the speed on your own data.