RAG
Top-k retrieval
Top-k is how many chunks you retrieve and place in the prompt. It is the single largest cost lever in a retrieval system.
Also written: k value
Eight chunks of three hundred tokens is 2,400 tokens on every request, billed every time. Going from k=8 to k=4 halves the largest line of a RAG bill.
It is also a quality lever in the same direction. Irrelevant context degrades answers, so fewer, better chunks often improves output while reducing cost.
In practice
Top-k keeps a fixed number of candidates regardless of how confident the model is, which is its whole weakness: the same k that sensibly widens an uncertain prediction also forces alternatives into a confident one. Top-p adapts to the distribution instead, which is why it is generally the better default of the two.
Common questions
What top-k value should I use?
Prefer top-p instead. If you must use top-k, values around 40 are a common default, but the fixed count is the weakness — the same k that sensibly widens an uncertain prediction forces alternatives into a confident one.
What does top-k 1 do?
Greedy decoding — always the single most likely token. It is a way to get deterministic output, though setting temperature to zero is the more usual route to the same place.