Tokens
Tokenizer
A tokenizer is the frozen vocabulary and merge rules that convert text into tokens. Each provider trains its own, so the same text costs different amounts on different models.
A tokenizer is not a rule you can reason about from first principles — it is a learned artefact of whatever corpus it was trained on. Sequences that appeared frequently earned a single token; anything unusual is rebuilt from fragments.
OpenAI publishes its tokenizers, so counts for GPT models can be computed exactly in a browser. Anthropic, Google and DeepSeek do not publish browser-capable tokenizers, which means any pre-flight count for those models is an estimate.
In practice
A team migrating a 2,000-token system prompt from an OpenAI model to a Claude model measured the same text at about 2,400 tokens on arrival. Nothing about the prompt changed and no price changed; the tokenizer did. Any per-million price comparison made without re-counting on the target tokenizer is off by whatever that gap happens to be.
Common questions
Do all models use the same tokenizer?
No, and this is why price-per-million comparisons mislead. Each provider trains its own vocabulary, so identical text produces different token counts on different models. A 10–20% gap between providers on the same paragraph is normal.
Can I count Claude or Gemini tokens exactly in a browser?
Not currently. Neither publishes a browser-capable tokenizer, so any pre-flight count for those models is an estimate derived from a related encoding. OpenAI publishes theirs, so GPT counts can be exact.
Measure it
- Tokenizer PlaygroundEvery token rendered separately, with its ID. The fastest way to understand why a prompt is expensive.
- o200k_base vs cl100k_base ComparatorBoth encodings, both exact. Matters on any model migration.
- LLM Token CounterReal BPE tokenization, not characters ÷ 4. Shows which counts are exact and which are estimates.