TokenPad

Tokens

Token

A token is an entry in the fixed vocabulary a language model was trained with — the unit it reads, and the unit you are billed in.

Also written: LLM token, what is a token

Before text reaches a model it is converted into a sequence of integers, each an index into a vocabulary of typically 100,000 to 200,000 entries. The model never sees characters or words. Some tokens are whole common words, some are word fragments, and a few are raw bytes for text the vocabulary has no better representation for.

For ordinary English prose one token averages about 0.75 words, or roughly four characters. That ratio is where the ubiquitous "characters ÷ 4" rule comes from, and it holds only for English prose. Source code runs closer to 2.7 characters per token and pretty-printed JSON closer to 2.2.

In practice

Paste the same paragraph into a counter in English and in Spanish and the second is routinely 30–40% longer in tokens for identical meaning, because the vocabulary was trained on an English-dominated corpus. At a million requests a month on a 500-token prompt, that difference alone is roughly 150 million extra tokens billed — for saying exactly the same thing.

Common questions

How many tokens is 1,000 words?

About 1,300 for ordinary English prose, at roughly 0.75 words per token. Code runs denser — closer to 1,800 tokens for the same word count — and non-English text denser still. The ratio is a starting point, not a substitute for counting the actual text.

Is a token the same as a word?

No. Common words are usually one token, but longer or rarer words split into several, and a leading space is part of the token. "Unbelievable" is typically three tokens; "the" with a space before it is one.

Measure it

  • TokenizerA tokenizer is the frozen vocabulary and merge rules that convert text into tokens. Each provider trains its own, so the same text costs different amounts on different models.
  • Byte-pair encoding (BPE)Byte-pair encoding builds a tokenizer vocabulary by repeatedly merging the most frequent adjacent pair of symbols in a corpus until it reaches a target size.
  • Context windowThe context window is the total token budget for a single request, covering the system prompt, conversation history, tool definitions, documents and the response the model generates.

More in Tokens