How-to · 6 min · 5 steps
How to count tokens
Count tokens exactly rather than estimating: which tokenizer applies to which model, why characters ÷ 4 misleads, and what to do for models with no public tokenizer.
Tokens are the unit every LLM price, rate limit and context window is denominated in, and the one unit you cannot see in your editor.
Counting them exactly takes about ten seconds. Estimating them takes one second and is wrong in the expensive direction on most real payloads.
Identify which encoding your model uses
For OpenAI: o200k_base for GPT-4o and everything after it, cl100k_base for GPT-4 and GPT-3.5. The difference is ten to twenty percent on the same text, and larger on non-English content.
o200k_base vs cl100k_base ComparatorBoth encodings, both exact. Matters on any model migration.Run the real encoder over your text
Not an approximation. Paste the actual payload — including system prompt and any structured data — and read the count. For OpenAI models this is the exact number the API bills.
LLM Token CounterReal BPE tokenization, not characters ÷ 4. Shows which counts are exact and which are estimates.Check the characters-per-token figure
English prose lands near 4.0. Below 3.0 means code, JSON, identifiers or non-Latin script, all of which cost two to three times more per character than prose. This number tells you whether the payload is worth optimising.
Token HeatmapLine-by-line token bars with a density flag for the expensive parts.For Claude, Gemini and DeepSeek, use a labelled estimate
None of them publish a tokenizer that runs in a browser. Anthropic offers only a server-side endpoint. Treat any browser figure for those models as accurate to within ten to twenty percent, and never as the basis for sizing a prompt against a context window boundary.
Remember the count is your text, not your request
Your bill covers the whole request: system prompt, conversation history, tool definitions, retrieved documents and the model’s output. A 300 token user message inside an agent loop can sit in a 40,000 token request.
Context Window Budget PlannerWindow allocation with a safety margin and the fixed overhead shown.
What goes wrong
Trusting characters ÷ 4 on structured data
It understates minified JSON by thirty to fifty percent. The payloads that dominate real API bills are rarely English prose, which is exactly where the rule of thumb was derived.
Assuming a count transfers between providers
Each provider trains its own vocabulary, and providers change theirs between model generations. Re-measure whenever you switch models.
Frequently asked questions
- How many tokens is 1,000 words?
- About 1,330 for English prose. Closer to 2,600 for source code, and over 3,000 for text in a non-Latin script. The single-number answer people quote is an English-prose answer applied to everything.
- Can I count Claude tokens exactly in a browser?
- No. Anthropic does not publish a browser-capable tokenizer; the only exact method is their server-side token counting endpoint. Any browser figure, here or anywhere else, is an estimate — and a site that does not say so is not being more accurate, only less transparent.
Tools for this
- LLM Token CounterReal BPE tokenization, not characters ÷ 4. Shows which counts are exact and which are estimates.
- Tokenizer PlaygroundEvery token rendered separately, with its ID. The fastest way to understand why a prompt is expensive.
- Words to Tokens ConverterQuick conversion in both directions, with separate ratios per content type instead of one wrong average.
Read next
- How to reduce token usage — Seven changes, ordered by return. Most teams find 40% in the first three.
- How to calculate LLM API costs — Six steps to a number you can take to a planning meeting.