What is RAG?

By TryRAG · Updated

RAG stands for retrieval-augmented generation. It is a way of getting an AI language model to answer from your own documents instead of only from what it learned in training. The idea is simple: before the model answers, find the few chunks of your documents that are most relevant to the question and hand them to the model along with the question.

The problem it solves

A language model does not know your company handbook, your product manual or last week's report. Asked about them, it either says it does not know or, worse, produces a confident answer that is wrong. You could paste the whole document into every question, but that is slow, costs more with every request, and stops working once the documents are longer than the model can read at once.

RAG avoids both problems. The model sees only the parts that matter for this question, and it is told to answer from them.

How it works, step by step

  1. Split. Each document is cut into short passages, called chunks. A chunk is usually a few sentences to a few paragraphs long.
  2. Turn into vectors. Each chunk is converted into a list of numbers, called a vector or embedding, that captures what the chunk is about. Chunks about similar things get similar numbers.
  3. Store. The vectors are kept in an index, often a vector database, so they can be searched quickly.
  4. Retrieve. When a question arrives it is converted to a vector in the same way. The system compares it with every stored vector and picks the closest few, typically three to ten.
  5. Build the prompt. The retrieved chunks are placed in a prompt together with the question and an instruction such as "answer using only the context below".
  6. Generate. The model reads the prompt and writes the answer.

Steps one to three happen once, when documents are added. Steps four to six happen for every question.

A small example

Take a bicycle shop's customer handbook and the question "How many days do I have to return a bike?". Split into chunks of about 45 words, the handbook becomes eight chunks. The question shares the words "return", "bike" and "days" with the chunk about returns, so that chunk scores highest, with a similarity of about 0.43, while chunks that share no words with the question score close to zero. The top three chunks go into the prompt, and the model can now answer "30 days" because the sentence that says so is in front of it.

You can follow exactly this example, with every number shown, in the RAG tool: press "Show me" in the Step by step panel.

What decides the quality

Most poor RAG answers come from retrieval, not from the model. If the right chunk is not among those retrieved, the model cannot answer well however capable it is. Four things have the most effect:

When to use RAG

When it is not the right tool

Words you will meet

Try RAG on your own document to see each step, then read about common RAG mistakes.