RAG
Reranking
Reranking scores a wide set of retrieved candidates with a more precise model and keeps only the best few for the prompt.
It adds a per-query fee and removes generation tokens, because you send four good chunks instead of twelve mediocre ones. At typical prices that nets out positive, since generation input costs far more per token than reranking costs per query.
The stronger argument is quality: irrelevant context measurably degrades answers, so cutting from twelve chunks to four often improves output at the same time as reducing input.
In practice
Retrieval optimises for recall, reranking for precision, and the split matters: fetch fifty candidates cheaply, then have a slower model order the top five properly. The cost is an extra model call per query, so the question is whether your failures are "the right chunk was missing" — which reranking cannot fix — or "the right chunk was ninth".
Common questions
When is reranking worth the extra call?
When your failures are "the right chunk was ninth" rather than "the right chunk was missing". Reranking reorders what retrieval found; it cannot recover something retrieval never returned.
How many candidates should I rerank?
Fetch broadly and rerank narrowly — commonly fifty candidates down to five. The point of the split is that retrieval optimises recall cheaply and the reranker optimises precision expensively.