What is RAG?
By TryRAG · Updated
RAG stands for retrieval-augmented generation. It is a way of getting an AI language model to answer from your own documents instead of only from what it learned in training. The idea is simple: before the model answers, find the few chunks of your documents that are most relevant to the question and hand them to the model along with the question.
The problem it solves
A language model does not know your company handbook, your product manual or last week's report. Asked about them, it either says it does not know or, worse, produces a confident answer that is wrong. You could paste the whole document into every question, but that is slow, costs more with every request, and stops working once the documents are longer than the model can read at once.
RAG avoids both problems. The model sees only the parts that matter for this question, and it is told to answer from them.
How it works, step by step
- Split. Each document is cut into short passages, called chunks. A chunk is usually a few sentences to a few paragraphs long.
- Turn into vectors. Each chunk is converted into a list of numbers, called a vector or embedding, that captures what the chunk is about. Chunks about similar things get similar numbers.
- Store. The vectors are kept in an index, often a vector database, so they can be searched quickly.
- Retrieve. When a question arrives it is converted to a vector in the same way. The system compares it with every stored vector and picks the closest few, typically three to ten.
- Build the prompt. The retrieved chunks are placed in a prompt together with the question and an instruction such as "answer using only the context below".
- Generate. The model reads the prompt and writes the answer.
Steps one to three happen once, when documents are added. Steps four to six happen for every question.
A small example
Take a bicycle shop's customer handbook and the question "How many days do I have to return a bike?". Split into chunks of about 45 words, the handbook becomes eight chunks. The question shares the words "return", "bike" and "days" with the chunk about returns, so that chunk scores highest, with a similarity of about 0.43, while chunks that share no words with the question score close to zero. The top three chunks go into the prompt, and the model can now answer "30 days" because the sentence that says so is in front of it.
You can follow exactly this example, with every number shown, in the RAG tool: press "Show me" in the Step by step panel.
What decides the quality
Most poor RAG answers come from retrieval, not from the model. If the right chunk is not among those retrieved, the model cannot answer well however capable it is. Four things have the most effect:
- How the document is split. Chunks that cut a sentence in half or mix two topics are harder to match. See the chunk size guide.
- How chunks are matched. Keyword search misses rephrased questions; semantic search can miss exact codes and names. Many systems combine both, which is called hybrid search.
- How many chunks are retrieved. Too few and the answer may be missing; too many and the prompt fills with unrelated text.
- The instruction in the prompt. Telling the model to say "I don't know" when the context does not contain the answer reduces invented answers.
When to use RAG
- Questions about documents the model has never seen, such as internal policies or product manuals.
- Information that changes often, where retraining a model would be too slow or too expensive.
- Answers that should point back to a source chunk so a person can check them.
When it is not the right tool
- The document is short. If everything fits comfortably in one prompt, send it all and skip retrieval.
- The question needs the whole document. "Summarise this report" or "how many times is X mentioned" cannot be answered from three chunks.
- The task is about style or behaviour, not knowledge. Changing how a model writes is a job for instructions or fine-tuning.
Words you will meet
- Chunk: one chunk of a split document.
- Embedding: the list of numbers that represents a chunk. See what embeddings are.
- Top-k: the number of chunks retrieved, for example top-3.
- Context: the retrieved chunks as they appear in the prompt.
- Hallucination: an answer the model made up that is not supported by the context.
Try RAG on your own document to see each step, then read about common RAG mistakes.