Chunking is the first decision in building a knowledge base, and it is the one most often made by default. The chunk size and boundaries determine what the retriever can match, what the embedding can represent, and what the answer can quote. Get it wrong and no amount of tuning fixes it: the information is in the base, but the retriever can never find it. This article covers the trade-offs between fixed-size, overlapping, and structure-aware chunking, and how chunking interacts with hybrid BM25-plus-dense retrieval.

What Chunking Actually Controls

Chunking controls three things at once.

Granularity. The chunk is the retrieval unit, so it is also the smallest unit a response can be built from. A retriever cannot quote half a sentence that lives in two chunks, and it cannot combine facts that were split across a boundary as easily as facts that share one.

Boundary quality. Where you cut matters as much as how long the chunks are. A cut in the middle of a sentence produces two chunks that are both slightly wrong — each one contains an incomplete thought that dilutes the embedding and confuses term matching.

Coverage. Chunks must cover the source text without losing information. Overlap and duplication cost index size and skew term statistics; gaps cost retrieval recall.

Every chunking strategy is a compromise among these three. The right strategy depends on the text (prose versus reference material), the retriever (sparse, dense, or hybrid), and the question types the bot must answer.

Fixed-Size and Overlapping Chunks

The simplest strategy is fixed-size chunking: split the text every N characters or tokens. It is easy to implement, predictable in index size, and wrong in the ways you would expect — it cuts sentences in half and mixes unrelated paragraphs into one chunk. Still, it is a useful baseline, and overlap fixes its worst failure mode.

function chunkBySize(text, size, stride) {
  const chunks = [];
  for (let i = 0; i < text.length; i += stride) {
    chunks.push(text.slice(i, i + size));
  }
  return chunks;
}

With size = 300 and stride = 150, every chunk overlaps its neighbors by half. Overlap exists so that a sentence or idea that straddles a boundary still appears intact in at least one chunk. The cost is duplication: overlap inflates the index, and more importantly, it changes sparse statistics — duplicated words appear more often, which raises their term frequencies and shifts BM25 scores. If you use overlap, keep it modest (10-25% of the chunk) and measure whether it actually improves hit rate before committing.

The real problem with fixed-size chunking is that text does not come in uniform lengths. Paragraphs, code blocks, and math formulas have natural boundaries that fixed windows ignore. Fixed chunking is a reasonable fallback for uniform prose; for anything with structure, cut on the structure.

Structure-Aware Chunking and Hybrid Retrieval

Structure-aware chunking cuts at natural boundaries: headings, paragraphs, sentences, and list items. Its advantage is boundary quality — chunks contain complete thoughts, so embeddings are coherent and BM25 matches real text. Its cost is variable chunk sizes, which means you still need a merge-and-split pass to keep chunks within a usable length range.

A practical recipe: split the document into sections by headings; split each section into paragraphs; keep paragraphs that are too long, split them at sentence boundaries; merge very short paragraphs with the next one. The result is a set of chunks that are semantically whole and bounded in length.

For knowledge bases with typed content, the semantic chunk is even more natural. ReLU.chat structures entries as categorized fragments — definition, intuition, example, formula, application. Each fragment is a chunk chosen for what it contains, not for its length. "Give me an example of X" retrieves the example fragment; "define X" retrieves the definition. This is structure-aware chunking taken to its logical end: the chunk type is part of the retrieval signal.

Hybrid retrieval makes chunking harder because the two scorers want different things. Dense retrieval wants coherent semantic units — complete thoughts that embed cleanly. Sparse BM25 wants keyword-dense text — chunks where the query terms actually appear, ideally multiple times, since term frequency and idf drive the score. A chunk that satisfies both is one that is semantically whole and contains the exact vocabulary of the domain: the term, its definition, and its typical neighbors. This is another argument for typed fragments and repeated names. ReLU.chat repeats entry names three times and aliases twice during indexing, so a definition fragment containing the entry name is simultaneously a coherent embedding target and a term-rich BM25 target. When chunks carry the terms users actually type, the sparse side pulls its weight in the 70/30 dense-sparse ensemble fusion instead of relying on the dense side alone.

Choosing a Chunk Size and Evaluating It

There is no universal chunk size, but there is a useful band and a reason for it. Sentence transformer models such as all-MiniLM-L6-v2, which produce the 384-dimensional embeddings ReLU.chat uses, have a fixed context window — text beyond it is truncated, and truncation silently discards meaning. Below the window, longer chunks still dilute the embedding: a 400-word chunk about three topics embeds as a blur of all three, and its similarity to any focused query drops.

In practice, chunks of roughly one to three paragraphs, or 100-300 words, sit in a good spot for hybrid retrieval. Short chunks (under 50 words) lose context — a definition separated from its formula may retrieve but not answer. Very long chunks (over 500 words) defeat both retrievers: the embedding is diluted, and BM25's document-length normalization (b = 0.75 in ReLU.chat's field-weighted BM25) already penalizes long documents by spreading matching terms across more length.

The better approach is to make size a constraint on structure-aware chunks rather than a goal in itself: define chunks by structure, then split or merge only the outliers. Set the boundaries by meaning and let length be a bound, not a target.

Chunking changes are easy to make and hard to evaluate by feel. Build a small evaluation set of realistic questions with the fragment or passage that should be retrieved, then measure: does the correct chunk appear in the top-k, and does the final answer contain the needed facts? Retrieval hit rate alone is not enough — a chunk can be retrieved yet too small to answer, or the answer can splice fragments that contradict each other. Score end-to-end: hit rate, answer completeness, and boundary artifacts such as cut-off sentences quoted into responses. A/B the chunking parameters on the same evaluation set — fixed-size with and without overlap, structure-aware, fragment-based — and compare. Chunking is the cheapest retrieval improvement available, because it does not require new models or new code paths, just a better split of the same text.

Key Takeaway

Chunking decides what a knowledge base retriever can find. Fixed-size chunks are a baseline; overlap repairs boundary cuts but inflates the index and skews term statistics; structure-aware chunking at headings, paragraphs, and sentences produces coherent embeddings and clean BM25 matches. For typed content, chunks aligned with content type — definitions, examples, formulas — serve hybrid retrieval best, and chunking decisions should be evaluated end-to-end on retrieval hit rate and answer completeness.