TokenPad

RAG

Context stuffing

Context stuffing is filling a large context window with everything that might be relevant, instead of retrieving what is.

It is tempting because million-token windows exist and retrieval is work. It performs worse on both axes: recall degrades as the window fills, and long context is billed on every turn.

A tightly retrieved 20,000 token prompt regularly beats a padded 500,000 token one on accuracy and always beats it on cost.

In practice

Filling a window because it is available is the most expensive habit in this field. Sending 200,000 tokens to answer a question about one paragraph pays for 199,000 tokens of noise and measurably hurts the answer, because the relevant material now competes for attention. A tightly retrieved 20,000-token prompt usually wins on accuracy and always wins on cost.

Common questions

Why is filling the context window a bad idea?

You pay for every token and the relevant material now competes with the irrelevant. A tightly retrieved 20,000-token prompt usually beats a padded 500,000-token one on accuracy and always on cost.

How do I know if I am over-stuffing?

Estimate what share of what you send is genuinely relevant. Below about a third, the fix is better retrieval rather than a bigger window — and the padding has an annual cost you can calculate.

Measure it

  • Context rotContext rot is the degradation in recall and instruction-following that occurs as a model’s context window fills up.
  • Context windowThe context window is the total token budget for a single request, covering the system prompt, conversation history, tool definitions, documents and the response the model generates.
  • Retrieval-augmented generation (RAG)RAG finds passages relevant to a question and places them in the prompt, so the model answers from supplied material rather than from training data.

More in RAG