nenadstojkovic.dev

Reviewed 6 min

Chunking: the basics

Both earlier posts in this series mentioned chunking in passing and moved on. This one stays: why a whole document embeds into one blurry vector, how fixed-size chunking with overlap works, the size/overlap tradeoff, and when to reach for something smarter than splitting on word count.

Chunking: the basics

A five-minute read.

The vector databases post mentioned, in one line, that a whole 50-page document embeds into one blurry vector and that splitting it into passages retrieves far better. The embedding models post treated chunk size as a fact of life imposed by context windows. Neither stopped to explain the step itself. This post does.

Why one vector per document doesn’t work

An embedding model collapses however much text you give it into a single fixed-length vector. Feed it a sentence and that vector is a fairly precise summary of one idea. Feed it fifty pages and the model still has to produce one vector, so it averages across everything the document talks about. The result sits somewhere in the middle of the semantic space, close to nothing in particular.

flowchart LR
    subgraph whole["one vector per document"]
        D["50-page document<br/>(intro, methods, results, appendix)"] --> M1["embedding model"] --> V1["one blurry vector"]
    end
flowchart LR
    subgraph chunked["one vector per chunk"]
        C1["intro paragraph"] --> M2["embedding model"] --> V2["vector A"]
        C2["methods section"] --> M2 --> V3["vector B"]
        C3["results section"] --> M2 --> V4["vector C"]
    end

A query about the methods section should land near vector B. It has no chance of landing near a single vector that’s already an average of intro, methods, results, and appendix. Chunking is what makes that match possible at all.

The simplest thing that works: fixed-size windows

Split the text into pieces of roughly N words, and start each next piece a little before the previous one ends, so a sentence sitting on the boundary still appears whole in at least one chunk.

def chunk_text(
    text: str,
    doc_id: str,
    chunk_size: int = 200,
    overlap: int = 40,
) -> list[Chunk]:
    words = text.split()
    step = chunk_size - overlap
    chunks = []
    start = 0
    while start < len(words):
        piece = words[start : start + chunk_size]
        chunks.append(Chunk(text=" ".join(piece), doc_id=doc_id, ...))
        start += step
    return chunks
flowchart LR
    subgraph words["200 words per chunk, 40-word overlap"]
        direction LR
        A["chunk 1: words 0-200"]
        B["chunk 2: words 160-360"]
        C["chunk 3: words 320-520"]
    end
    A -.overlap.- B
    B -.overlap.- C

This is most of what “chunking” means in practice. It’s crude, it doesn’t know what a sentence is, and it works well enough that a large fraction of production retrieval systems never move past it.

Picking chunk_size and overlap

Chunk size too small and each piece stops carrying enough context to mean anything on its own. A chunk that reads “the results were consistent with this” retrieves poorly, because the embedding has almost nothing to grab onto — “this” refers to something two sentences up, in a different chunk.

Chunk size too large and you’re back to the blurry-vector problem, just at a smaller scale. A 2,000-word chunk covering three different subtopics gives you a vector that’s a compromise between all three.

Overlap exists to stop a chunk boundary from landing in the middle of the one sentence that actually answers the query. Without it, a fact that happens to straddle word 200 gets split across two chunks and may not read clearly in either. The cost is direct: 40 words of overlap out of 200 means every word gets embedded roughly 1.2 times, which is more vectors, more storage, and more near-duplicate results competing at query time.

There’s no universal right answer, only a starting point: a few hundred words with 15-25% overlap, then measure retrieval quality on your own queries and adjust. The same “embed a sample, check what actually retrieves” advice from the embedding-models post applies here — chunk size is just as empirical a choice as the model itself.

When fixed-size windows aren’t good enough

Splitting on a raw word count doesn’t know that it just cut a code block in half, or separated a table header from its rows, or sliced a sentence into two chunks despite the overlap. A few steps up in sophistication:

  • Structure-aware splitting. Split on paragraph or heading boundaries first, and only fall back to a raw word-count cut when a single paragraph is still too long. This is what most “recursive” text splitters do: try the biggest natural boundary, then the next-smallest, down to sentences and finally words.
  • Semantic chunking. Embed individual sentences, then cut where consecutive sentences stop being similar to each other, instead of at a fixed word count. More expensive to compute, and it pays off most on documents that shift topic irregularly rather than in even sections.
  • Format-aware splitting. Code gets chunked by function or class, not by line count. Markdown gets chunked by heading, keeping a table or a fenced code block intact rather than letting a fixed window slice through it.

None of these remove the size/overlap tradeoff above — they just choose smarter places to apply it.

What a chunk carries with it

A chunk isn’t just its text. Each one needs enough metadata to be useful once it’s retrieved: which document it came from, its position in that document, and whatever tags or filters the vector-databases post described. doc_id and chunk_index are what let a search result be traced back to “page 12 of this PDF” instead of surfacing 200 words with no origin.

That bookkeeping also matters when a chunk retrieves well but reads confusingly in isolation. A common fix is to return a chunk’s immediate neighbours alongside it, using chunk_index to look up chunk N-1 and N+1, so the result shown to a user (or handed to an LLM in a RAG pipeline) has the surrounding context the embedding itself couldn’t carry.

Two things that bite beginners

More chunks isn’t free. Smaller chunks generally improve retrieval precision, but every chunk is a vector in the index and a row of metadata. At a large enough corpus, chunk size becomes a storage and latency decision as much as a quality one.

Re-chunking means re-indexing, same as re-embedding. Change chunk_size or overlap and every chunk boundary shifts, which means every vector changes, which means the old index is comparing apples to oranges against new queries. There’s no incremental fix — like swapping embedding models, it means rebuilding the index from scratch.