How close is tokenizers v1 to a drop-in replacement?
Each number compares a v1 build with tokenizers 0.23.3, the latest 0.x release. A number only changes when one of its inputs does: a tokenizers release or a change to its code on main, a dependent's release or a change to its tokenizer code. Updated 2026-10-20 09:23:19 UTC.
| Measured on | 1.0.0rc3 | main @ 51afdd138 |
|---|---|---|
| Hub tokenizers the 4,578 of the top 10,000 models by 30-day downloads that ship a tokenizer.json (1,351 distinct files) | 87% of downloads encode and decode text exactly like 0.23.3. 4% fail to load, 9% load but raise on encode, and 45 tokenizers give different output (<1%). Pairs of texts: 87% match. | 87% of downloads encode and decode text exactly like 0.23.3. 4% fail to load, 9% load but raise on encode, and 45 tokenizers give different output (<1%). Pairs of texts: 87% match. |
| transformers #49764 @ 6120eaf2e its tokenizer tests | fails to import ImportError: cannot import name 'AddedToken' from 'tokenizers' | fails to import ImportError: cannot import name 'AddedToken' from 'tokenizers' |
| sentence-transformers 6.1.0 its tokenizer tests, on transformers #49764 @ 6120eaf2e | fails to import ImportError: cannot import name 'AddedToken' from 'tokenizers' | fails to import ImportError: cannot import name 'AddedToken' from 'tokenizers' |
| sentence-transformers main @ 5e8d96f5a its tokenizer tests, on transformers #49764 @ 6120eaf2e | fails to import ImportError: cannot import name 'AddedToken' from 'tokenizers' | fails to import ImportError: cannot import name 'AddedToken' from 'tokenizers' |
| vLLM 0.32.0 its tokenizer tests, on transformers #49764 @ 6120eaf2e | fails to import ImportError: cannot import name 'AddedToken' from 'tokenizers' | fails to import ImportError: cannot import name 'AddedToken' from 'tokenizers' |
| vLLM main @ e4a676cdc its tokenizer tests, on transformers #49764 @ 6120eaf2e | fails to import ImportError: cannot import name 'AddedToken' from 'tokenizers' | fails to import ImportError: cannot import name 'AddedToken' from 'tokenizers' |
Why Hub tokenizers don't match yet
Share of the downloads above affected on main, by cause.
| Downloads | Files | When | Error |
|---|---|---|---|
| 8.8% | 89 | loads, then raises | ValueError: this … pattern needs a system-regex backend; enable the … featuree.g. Qwen/Qwen3.5-4B |
| 1.3% | 56 | fails to load | ValueError: This byte-level vocabulary has no token for bytes <bytes>. Text containing those bytes cannot be encoded.e.g. HuggingFaceTB/SmolLM2-135M |
| 1.0% | 38 | fails to load | ValueError: the slim JSON reader does not support a … pre-tokenizer. Convert the config offline (tk-convert), or build with the … feature for the full reader. …e.g. OpenMed/OpenMed-NER-PharmaDetect-SuperClinical-434M |
| 0.9% | 38 | fails to load | ValueError: Byte fallback … not found in the vocabularye.g. google/siglip2-giant-opt-patch16-384 |
| 0.5% | 19 | fails to load | ValueError: a … pre-tokenizer with … is not supportede.g. w11wo/indonesian-roberta-base-posp-tagger |
| <0.1% | 17 | fails to load | ValueError: a … pre-tokenizer must be the last member of its …e.g. tiiuae/falcon-7b |
Different output, no error
Tokenizers that load on main but return different ids or text than 0.23.3. These are the ones to fix first: nothing warns the user.
| Model | 30-day downloads | First check that differs |
|---|---|---|
| pysentimiento/robertuito-sentiment-analysis | 601,637 | encode_raw[20] |
| Marqo/marqo-fashionSigLIP | 559,783 | decode[16] |
| timm/ViT-B-16-SigLIP | 464,667 | decode[16] |
| huggyllama/llama-7b | 352,796 | decode_skip[0] |
| perplexity-ai/pplx-embed-v1-0.6b | 342,134 | decode[16] |
| vinai/phobert-base | 200,063 | encode_raw[10] |
| NousResearch/Llama-2-7b-hf | 147,100 | decode_skip[0] |
| Nanbeige/Nanbeige4.2-3B | 120,395 | encode[16] |
| ggml-org/stories15M_MOE | 106,953 | decode_skip[0] |
| openai-community/openai-gpt | 103,679 | decode[10] |
| Maykeye/TinyLLama-v0 | 102,983 | decode_skip[0] |
| optimum-intel-internal-testing/tiny-random-bigbird_pegasus | 84,807 | decode[16] |
The 12 most downloaded of 45.
How this is measured
Hub tokenizers. Every distinct tokenizer.json among the 10,000 most downloaded models (gated ones excluded) is loaded by both versions and run through the same encode, pair, batch and decode checks, on text with special tokens, byte fallback, CJK, emoji and whitespace edge cases. Results are weighted by the 30-day downloads of every model that ships the file. Files that 0.23.3 itself cannot load, or encodes differently from one run to the next, are left out.
Dependents. transformers runs its tokenizer tests at #49764 @ 6120eaf2e, a branch whose only change is relaxing its tokenizers pin to accept 1.x. The latest release and main of sentence-transformers and vLLM run their own tokenizer tests on that same transformers. Nothing else is patched. A percentage is the share of the tests passing with 0.23.3 that still pass.