Chunking strategies for RAG that actually retrieve

Most retrieval-augmented systems that answer badly do not have a model problem or an embedding problem. They have a chunking problem. The passage that would have answered the question was split across two chunks, or buried inside one large enough to dilute it.

Why chunking decides retrieval quality

An embedding compresses a whole chunk into a single vector. That vector is an average of everything the chunk contains, so its usefulness depends entirely on the chunk being about one thing.

A chunk covering one coherent idea produces a vector that sits close to queries about that idea. A chunk covering six ideas produces a vector sitting near none of them. No amount of reranking recovers information that the embedding never represented — which is why chunking is upstream of every other retrieval decision and worth more attention than it usually gets.

Choosing a size

Start at 300 tokens. For most prose corpora the useful range is 200 to 500, and the two failure modes point in opposite directions:

• Too small and chunks lose the context that makes them interpretable. A sentence referring to "the second approach" is meaningless without the paragraph introducing the first. Retrieval succeeds and the answer is still wrong.
• Too large and the embedding dilutes. One relevant sentence among two hundred irrelevant ones produces a vector nowhere near the query. Retrieval simply misses.

Content type shifts the range. Reference documentation and FAQs do well at 150–250 because each entry is self-contained. Narrative prose and technical explanation need 400–600 because the argument spans paragraphs. Code is a special case: function boundaries beat token counts, and a chunk should almost always be a whole function.

Size in tokens rather than characters. The embedding model's limit is denominated in tokens, and character-based splitting produces chunks of wildly varying token length — some of which silently exceed the limit and get truncated, with no error and a wrong vector. The text splitter shows the exact token count for every chunk it produces.

Overlap, and why it is not optional

Meaning does not respect chunk boundaries. A sentence split across two chunks appears complete in neither, so neither embedding represents it and it becomes effectively unsearchable — a silent hole in your index.

Ten to twenty percent overlap means any given sentence appears whole in at least one chunk. At 300 tokens that is 30 to 60 tokens of overlap. The cost is duplicated storage and a slightly larger index; compared to a retrieval failure that returns the wrong passage confidently, it is nothing.

Going much beyond twenty percent starts to hurt: near-duplicate chunks crowd out genuinely different results, and your top-five becomes five views of the same paragraph.

Boundaries beat uniformity

The temptation is to slice at exact token indices for perfectly uniform chunks. Resist it. A chunk beginning mid-clause embeds badly and reads badly when it is later shown back to the model as retrieved context.

Split at sentence boundaries as a default, and at paragraph boundaries when paragraphs are self-contained units — documentation, FAQs, structured articles. Accept the variance in chunk size; it costs you nothing that matters. Exact token splitting has one legitimate use, which is a fixed-shape index where deterministic sizing is a hard requirement, and it should not be a default.

Where the source has structure, use it. Markdown headings, HTML sections and code function boundaries are all better split points than anything you can infer from the text, because the author already told you where the ideas begin.

Diagnosing a retrieval failure

When an answer is wrong, find out which half of the pipeline failed before changing anything. Retrieve for the failing query and read the chunks that came back.

• The right passage was not retrieved. A chunking or embedding problem. If the passage is split across a boundary, raise overlap. If it is buried in a large chunk, reduce chunk size.
• The right passage was retrieved but the answer is wrong. Not a chunking problem — it is prompt construction, or too much irrelevant context drowning the relevant chunk. Retrieve fewer, better chunks.
• The passage was retrieved but reads incoherently. A boundary problem. Switch from exact-token to sentence or paragraph splitting.

The cost side

Retrieved chunks usually dominate every other part of the prompt. Eight chunks of 400 tokens is 3,200 tokens on every single request, billed every time. Retrieving three good chunks instead of ten mediocre ones cuts the bill by two-thirds and usually improves the answer, because irrelevant context degrades output as well as costing money.

Check that your retrieved set actually fits alongside the system prompt in the context window calculator, and price it at your volume in the cost calculator. Retrieval quality and retrieval cost are the same optimisation, which is unusual and worth exploiting.