Context window vs token limit vs max tokens

Three terms, used interchangeably in most conversations and meaning three different things. Confusing them produces a specific and common production bug, so it is worth ten minutes to separate them properly.

The three terms

Context window

The total budget for a single request, covering everything: the system prompt, every message in the conversation, tool definitions, attached documents, retrieved passages, and the response the model is about to generate. If a model advertises a 200,000 token context window, that is the sum of all of it — not the input alone.

Max tokens

A parameter you set, capping how many tokens the model may generate in its reply. It is a ceiling on output only, and it is subtracted from the same window the input occupies. Setting it high does not reserve capacity; it declares an upper bound.

Token limit

Ambiguous, and best avoided. Depending on who is speaking it means the context window, the max tokens parameter, or a rate limit on tokens per minute — an entirely separate quota governing throughput rather than the size of any single request. If someone says "we hit the token limit", ask which one.

How they interact

The relationship is straightforward once stated: input tokens + generated output tokens must fit inside the context window.

The bug this causes is the one worth remembering. Your 190,000 token document fits comfortably inside a 200,000 token window — so the request is accepted. Then the model has 10,000 tokens in which to think and answer, and your request for a detailed analysis produces something truncated and strange. Nothing errored. The capacity simply was not there.

The fix is to decide your output budget first and treat the remainder as your real input capacity. The context window calculator has a field for exactly this: set the reserve to the longest answer you actually need and it shows which models still fit your text.

Which error means what

• "Prompt is too long" on send. Your input alone exceeds the window. Every provider rejects this outright rather than truncating — chunk the input with the text splitter or retrieve less of it.
• Generation stops mid-sentence with no error. Either max_tokens was reached, or input plus max output exceeded the window and the model ran out of room. Newer models accept such requests and stop when they hit the limit, which is considerably harder to notice in production than a rejection. Check the stop reason in the response — it distinguishes them.
• 429 or rate limit errors. Nothing to do with the context window. This is tokens per minute across all your requests, and the fix is backoff or a higher tier, not a shorter prompt.

Handling the stop reason explicitly is worth the twenty lines it takes. Silent truncation that reaches a user is a much worse failure than an exception you caught.

Why filling the window is a bad idea

Million-token windows are now common and the instinct is to use them. Resist it. Retrieval and recall degrade as a window fills — the effect usually called context rot. Anthropic's own documentation makes the point directly: curating what goes into context matters as much as how much space is available.

In practice a tightly retrieved 20,000 token prompt beats a padded 500,000 token one on accuracy, and the cost difference is not subtle. Long context is billed on every turn of a conversation, so a bloated prompt is a per-request tax rather than a one-off charge. Put both versions through the cost calculator and the argument usually settles itself.

The sequence that works: measure what you have with the token counter, decide the output reserve, then retrieve or chunk down to fit comfortably rather than exactly. Headroom is cheap. Discovering you had none is not.