How to choose a chunk size for RAG

By TryRAG · Updated

Chunk size is the length of the chunks a document is split into before retrieval. It is one of the settings that most affects whether a RAG system finds the right text, and it is the cheapest one to change. This guide explains the trade-off, gives starting points, and shows how to test a choice instead of guessing.

Small or large

Size also changes cost. Three retrieved chunks of 300 words put about six times as many tokens into every prompt as three chunks of 50 words. The token counter shows what that means in money.

Overlap

Overlap repeats the last few words of one chunk at the start of the next. It protects a sentence that falls on a boundary: cut in two by the split, it still appears whole in one of the chunks. Ten to twenty percent of the chunk size is a common starting point. Try fixing a split answer in the RAG tool.

Overlap is not free. It increases the amount of text stored, and high overlap makes neighbouring chunks nearly identical, so the top results can be several copies of the same sentences. If you see that, lower the overlap or retrieve with some variety.

How to split

The chunking visualizer shows the three methods on your own text, with a count of how many chunks end in the middle of a sentence.

Starting points by document type

These are places to begin, not rules. Two documents of the same type can behave differently.

Signs the size is wrong

Test it, do not guess

The dependable way to choose is to measure. Write five to ten questions you already know the answers to. For each one, note a word or phrase that the right chunk must contain. Then, for each setting you want to compare, check whether the right chunk is among those retrieved and how high it ranks.

The Evaluate tab of the RAG tool does this for you: it tests fifteen combinations of splitting method and chunk size on your document and lists them best first. The guide to testing retrieval explains how to write good test questions.

New to the idea? Start with What is RAG?