Chunking strategies: splitting documents smartly for RAG

When building a Retrieval-Augmented Generation (RAG) system, the focus is often on the choice of language model or the vector database used. In practice, however, it turns out that the quality of the answers is overwhelmingly determined by a much earlier step in the pipeline: dividing source files into manageable chunks of text, also known as chunking .

When a document is split incorrectly, even the most advanced language model cannot formulate the right answer. Split too coarsely, and specific information drowns in a sea of irrelevant context, needlessly exceeding your context budget. Split too finely, and the core message becomes fragmented, causing the search algorithm to lose the coherence needed to semantically link the question. In this article, we systematically cover the most important chunking methods, their strengths and weaknesses, and how to select the optimal strategy for different document types.

The chunking balancing act: context preservation versus search precision

Every chunking strategy tries to solve a fundamental paradox. On one hand, vector search functionality (dense retrieval) requires compact, focused text segments. An embedding model tries to compress the overall meaning of a passage into a single vector. If such a segment contains ten different topics, the resulting vector becomes diluted and the accuracy of targeted search queries drops.

On the other hand, the final language model (the generator) actually needs sufficient surrounding context to correctly interpret nuances, conditions, and definitions. A sentence like "This exception applies only to employees in pay scale 8 or lower" is effectively useless to an LLM if the preceding paragraph — which defines the actual rule — ended up in a different text fragment.

When designing the splitting strategy, we constantly balance between two risks: context fragmentation (where crucial meaning is lost at the cut line) and retrieval pollution (where too much noise is injected into the prompt). You can find a deeper insight into the practical vector computation in the article on the practical application of vector embeddings, which demonstrates the mathematical consequences of diluted vectors.

Fixed windows with overlap (Fixed-size chunking)

The most basic and widespread approach is splitting text based on a fixed number of tokens or characters, combined with a constant overlap. A common configuration, for example, is a window size of 512 tokens with an overlap of 50 tokens (about 10%).

The mechanism is simple: a counter moves through the text sequence and starts a new block after every 462 new tokens, carrying over the last 50 tokens of the previous block. The overlap acts as a safety net to prevent a sentence from being cut precisely in half and losing its meaning.

# Voorbeeld van vaste vensterverdeling in Python
def fixed_size_chunk(tokens: list[str], chunk_size: int = 512, overlap: int = 50):
    chunks = []
    step = chunk_size - overlap
    for i in range(0, len(tokens), step):
        chunk = tokens[i:i + chunk_size]
        chunks.append(chunk)
        if i + chunk_size >= len(tokens):
            break
    return chunks

Although this method is computationally very cheap and easy to implement, it has serious drawbacks. The algorithm is completely blind to syntax and semantics. It cuts straight through paragraphs, lists, tables, and sometimes even sentence structures. For unstructured plain text without strict hierarchy, it can suffice, but for contracts, manuals, or policy documents, it results in structural quality loss.

Recursive character and sentence splitting

To prevent natural text units from being brutally cut through, recursive splitters are frequently used (such as the RecursiveCharacterTextSplitter). This method tries to split the text step by step using an ordered list of separators, from large to small.

The algorithm first tries to split on double line breaks (paragraph boundaries). If the resulting paragraphs are still larger than the set target length, it falls back to single line breaks. If a line is still too long, it splits on sentence-ending punctuation (periods, exclamation marks, question marks), then on spaces between words, and only as a last resort on individual characters.

# Schematische werking van recursieve scheidingstekens
separators = [
    "\n\n",  # 1. Alineagrens (behoudt thematische eenheid)
    "\n",    # 2. Regelgrens (behoudt lijsten en strofen)
    ". ",    # 3. Zinsgrens (behoudt grammaticale eenheid)
    " ",     # 4. Woordgrens (voorkomt halve woorden)
    ""       # 5. Noodgreep (karakterniveau)
]

Thanks to this hierarchy, logical paragraphs and sentences remain intact as much as possible. The resulting text chunks feel natural and rarely contain broken sentence fragments. Nevertheless, the method remains fundamentally rule-based: the algorithm doesn't "know" whether two consecutive paragraphs belong together in content or introduce an entirely new topic.

Semantic chunking: splitting based on embedding distances

Instead of relying on visual whitespace or static character limits, semantic chunking measures the actual shift in content within a text. Here, the document is first split into individual sentences. An embedding is then calculated for each sentence (or a small sliding window of sentences).

The algorithm calculates the cosine distance between the vectors of consecutive sentences. As long as consecutive sentences are thematically closely related, the distance remains small. As soon as the author switches to a different topic, a sudden jump occurs in the vector distance. When this distance exceeds a predetermined threshold (for example, the 95th percentile of all measured distances), the system inserts a split.

# Conceptuele detectie van semantische grenzen
cosine_distance = 1.0 - cosine_similarity(embedding_a, embedding_b)

if cosine_distance > dynamic_threshold:
    # Betekenisverschuiving gedetecteerd: begin een nieuw tekstblok
    create_new_chunk()
else:
    # Onderwerp loopt door: voeg zin toe aan huidig blok
    append_to_current_chunk()

The strength of semantic chunking lies in its adaptive length: homogeneous passages remain one coherent whole, while compact paragraphs with many content shifts are quickly split up. The weak point is the computational overhead. For a 100-page document, hundreds of individual API calls or local embedding computations must be performed, solely to determine the split points. Anyone using local models for this task can consult the guide on searching documents locally with AI for practical considerations around compute power and throughput.

Document-aware and structural chunking

Many business documents — such as annual reports, software specifications, legal statutes, and Markdown files — already have an explicit formal structure. Document-aware (or layout-aware) chunking makes use of this formatting instead of forcing plain text.

In document-aware splitting, headings (H1, H2, H3) are analyzed to build a tree structure of the document. Each chunk inherits the metadata of its parent sections. A paragraph under "3.1 Exceptions to the return policy" is given that full path structure as contextual metadata in the vector index.

Handling tables and lists

Tables are a notorious bottleneck in RAG pipelines. When a fixed-size splitter cuts through the middle of an HTML or Markdown table, the column headers disappear and numbers become completely detached from their labels. A cell with the value "€45,000" means nothing without the corresponding row ("Senior Developer") and column name ("Maximum salary").

Structural splitters therefore isolate tables as indivisible units. If a table is too large for the embedding window, it is transformed: each row is converted into a self-contained, readable sentence (for example: "Role: Senior Developer | Maximum salary: €45,000 | Scale: 11"). This keeps the relationship between fields intact, regardless of where the split falls.

Advanced patterns: Parent Document and Hierarchical Retrieval

To definitively resolve the dilemma between small search vectors and rich generation context, modern architectures use a decoupled approach: the Parent Document or Small-to-Big pattern.

In this scenario, the source file is processed at two levels simultaneously:

• Parent Chunks: Large sections of, for example, 1,000 to 1,500 tokens that cover a complete topic. These are stored in a traditional document store or relational database.
• Child Chunks: Small fragments of 100 to 200 tokens cut from the parent. Only these small fragments are embedded and stored in the vector database.

During the retrieval phase, the system searches for the best-matching child vector. But instead of sending that small fragment directly to the language model, the application uses a reference ID to fetch the full parent parent document . This way, the search step benefits from the sharp precision of small vectors, while the generating model has access to the full thematic context.

You can read how this setup scales within larger databases and index structures in the article on vector indexing with HNSW and IVF, which covers indexing patterns for large vector collections.

Late Chunking and Contextual Retrieval

A recent innovation in the field is Late Chunking (introduced around 2024–2025). Traditionally, you first split the text and then run each fragment individually through the embedding model. As a result, at the embedding step there is no interaction whatsoever between consecutive paragraphs.

In Late Chunking, the entire document (or a very large section of up to 8,192 tokens) is passed through the transformer encoder in a single pass. Only at the last layer of the network, after all token representations have exchanged information across the entire document via the self-attention mechanism, are the token embeddings grouped and pooled into separate chunk vectors.

The result is remarkable: the vector of an individual paragraph now contains subtle traces of information from the introduction and earlier chapters. References like "this system" or "as mentioned above" carry the meaning of their antecedents with them in vector space, without the chunk needing to be physically larger.

For those working with Dutch-language documents, the choice of underlying model is decisive here; see the overview on choosing the right embedding model for Dutch documents to see which models support long sequences and cross-token interaction.

Comparison of chunking strategies

The table below compares the five main approaches based on computational complexity, context preservation, and ideal use cases.