How to choose a chunk size for RAG
By TryRAG · Updated
Chunk size is the length of the chunks a document is split into before retrieval. It is one of the settings that most affects whether a RAG system finds the right text, and it is the cheapest one to change. This guide explains the trade-off, gives starting points, and shows how to test a choice instead of guessing.
Small or large
- Small chunks (roughly 20 to 60 words) match a question precisely, because the chunk is about one thing. The risk is lost context: the sentence that contains the answer may depend on the sentence before it, which now sits in a different chunk.
- Large chunks (150 words and more) keep context together. The risk is dilution: a chunk that covers three topics matches each of them weakly, and whatever is retrieved fills the prompt with text that has nothing to do with the question.
Size also changes cost. Three retrieved chunks of 300 words put about six times as many tokens into every prompt as three chunks of 50 words. The token counter shows what that means in money.
Overlap
Overlap repeats the last few words of one chunk at the start of the next. It protects a sentence that falls on a boundary: cut in two by the split, it still appears whole in one of the chunks. Ten to twenty percent of the chunk size is a common starting point. Try fixing a split answer in the RAG tool.
Overlap is not free. It increases the amount of text stored, and high overlap makes neighbouring chunks nearly identical, so the top results can be several copies of the same sentences. If you see that, lower the overlap or retrieve with some variety.
How to split
- Fixed size cuts every set number of words. It is simple and predictable, but it can cut a sentence in half.
- By sentence fills each chunk with whole sentences up to the size limit. Chunks are slightly uneven in length, and nothing ends mid-sentence.
- By heading keeps each section together and carries its heading into every chunk. This helps when a section never repeats its own topic word: a paragraph under "Warranty" may not contain the word "warranty" at all.
The chunking visualizer shows the three methods on your own text, with a count of how many chunks end in the middle of a sentence.
Starting points by document type
- Questions and answers, support articles, policies: small chunks of 40 to 80 words. Each answer is short and self-contained.
- Manuals and documentation with clear headings: split by heading, with a size limit of 100 to 150 words for long sections.
- Reports, essays and other continuous prose: split by sentence, 80 to 150 words, with 10 to 20 percent overlap.
- Forms, tables and lists: small chunks, because each line stands alone. Check that a label and its value have not been separated.
These are places to begin, not rules. Two documents of the same type can behave differently.
Signs the size is wrong
- Too small: the right chunk is retrieved but the answer is incomplete, or the model says the context does not contain the answer although the next chunk does.
- Too large: every question retrieves the same few chunks, scores are all close together, and answers drift onto side topics.
Test it, do not guess
The dependable way to choose is to measure. Write five to ten questions you already know the answers to. For each one, note a word or phrase that the right chunk must contain. Then, for each setting you want to compare, check whether the right chunk is among those retrieved and how high it ranks.
The Evaluate tab of the RAG tool does this for you: it tests fifteen combinations of splitting method and chunk size on your document and lists them best first. The guide to testing retrieval explains how to write good test questions.
New to the idea? Start with What is RAG?