Chunk Size, Overlap, and Retrieval Quality

Two failure directions for chunk size

Having chosen a chunking strategy, you still have to size the chunks, and the exam frames this as a balance between two opposite failures. Too small, and a chunk loses the surrounding context a question needs, so retrieval fetches a fragment that cannot answer the query on its own. Too large, and the chunk packs in so much that its embedding becomes a blurry average of many topics, diluting the relevance signal so the chunk matches queries less precisely. Chunk size is a tuning problem with a failure at each extreme, and the goal is the middle where a chunk is big enough to carry context and small enough to stay relevant.

Overlap is the companion setting that guards the boundaries. Together, size and overlap determine whether the right passage ends up in a retrievable, coherent chunk, or gets fragmented and diluted into uselessness.

Too small loses context; too large dilutes relevance

The small-chunk failure is about missing context. A chunk holding a bare fact without the sentences that frame it may not carry enough for the model to answer, and its embedding may not even match the query, because the surrounding words that connect the fact to the question were cut away. Retrieval brings back a fragment that is technically on-topic but insufficient.

The large-chunk failure is about diluted relevance. An embedding represents a chunk as a single vector; the more distinct topics a chunk contains, the more that vector is an average that matches everything weakly and nothing strongly. A giant chunk covering five subjects will retrieve less precisely than a focused chunk covering one, because the relevant signal is drowned in the irrelevant surroundings. This is why 'make chunks large to capture more context' backfires: past a point, extra size costs precision. Bigger is not automatically better.

Overlap protects boundary-spanning passages

Even a well-sized chunking scheme has boundaries, and a relevant passage can land exactly across one, its first half in one chunk and its second half in the next, so neither chunk holds the whole thing. Overlap fixes this by letting adjacent chunks share a margin of content, so a passage near a boundary appears whole in at least one chunk. Zero overlap leaves boundary-spanning passages vulnerable to being split and lost; a sensible overlap makes it far likelier the complete passage sits inside a single retrievable chunk.

So overlap is cheap insurance against the boundary problem that pure chunk-size tuning cannot solve, because wherever you put a boundary, some passage will straddle it. The two settings work together: size controls context-versus-precision, overlap controls boundary loss.

Tune to query length and fact granularity

The sizing target is not a fixed number but a fit to the workload. Chunk size should be tuned against the typical query length and the granularity of the facts the system needs to retrieve. If queries are short and facts are fine-grained, smaller focused chunks retrieve more precisely; if answering requires broader context, somewhat larger chunks are warranted. The point is that the right size is derived from how the system is actually queried and what it must retrieve, not chosen by a rule of thumb and certainly not maximised on the theory that more context is always better.

What the exam trips candidates on

Two traps. The first is setting a very large chunk size to capture more context without checking that it dilutes retrieval precision, treating size as costless. The second is using zero overlap between chunks and losing information that spans a chunk boundary. The credited answer tunes size to query length and fact granularity and uses overlap to protect boundaries.

Common misreadings to avoid

How this shows up on the exam

Expect a scenario with an oversized chunk, zero overlap, or a 'more context is always better' assumption, and dropping precision or missing boundary facts. The reliable reading is to tune size to query length and fact granularity and to use overlap to protect boundaries. This knowledge point builds on chunking strategy trade-offs, pairs with metadata-enriched indexing, and is central to diagnosing retrieval failures rooted in chunking.