How to choose a chunk size for RAG

Chunk size is the length of the passages a document is split into before retrieval. It is one of the settings that most affects whether a RAG system finds the right text.

Small or large

• Small passages match a question precisely, but can lose the sentence before or after that gives them meaning.
• Large passages keep context together, but mix several topics, which weakens the match and fills the prompt with unrelated text.

Overlap

Overlap repeats the end of one passage at the start of the next. It protects sentences that fall on a boundary. Ten to twenty percent of the passage size is a common starting point.

How to split

• Fixed size cuts every set number of words. It is simple, but it can cut a sentence in half.
• By sentence keeps sentences whole.
• By heading keeps each section together and carries its heading into every passage. This often helps when a section does not repeat its own topic word.

Test it, do not guess

The best setting depends on the document. Write a few questions you know the answers to, note the words the right passage must contain, and measure how often each setting retrieves it.

The Tune tab of the RAG tool does this for you: it tests fifteen combinations of splitting method and passage size on your document and lists them best first. New to the idea? Start with What is RAG?