What is RAG (retrieval-augmented generation), and when do you need it?
RAG gives a language model your documents at question time instead of baking them into the weights. Here is how the pipeline works, why most RAG failures are retrieval failures, and when a simpler approach wins.
Kurze Antwort
What is retrieval-augmented generation (RAG)?
Retrieval-augmented generation is a pattern where the system searches your own documents for passages relevant to a question, puts those passages into the model's prompt, and asks the model to answer using them. The model's weights never change; the knowledge arrives as context at request time.
Das Wichtigste
- RAG is search plus prompting. If the search step returns the wrong passages, no model can rescue the answer.
- Vector similarity alone is weak on names, codes and exact phrases — hybrid keyword-plus-vector retrieval is the practical default.
- It updates instantly: change the document, and the next answer changes. Fine-tuning cannot do that.
- Citations are the point. Grounded answers that link to the source passage are auditable in a way that raw generation is not.
A language model knows what was in its training data and nothing else. It does not know your contract terms, last quarter's numbers, or the incident report filed yesterday. Retrieval-augmented generation is the standard way to close that gap without retraining anything.
The pipeline, end to end
RAG has two phases. The first happens ahead of time.
Indexing (offline)
- Collect the documents.
- Chunk them into passages — typically a few hundred to a couple of thousand tokens each.
- Compute an embedding for each chunk: a vector of numbers positioning that text in a space where similar meanings sit close together.
- Store the vectors, the original text, and metadata (source, section, date, permissions) in an index.
Answering (per request)
- Take the user's question and embed it the same way.
- Retrieve the closest chunks — often combined with a keyword search.
- Optionally rerank the candidates with a smaller model that scores relevance more accurately than raw vector distance.
- Assemble a prompt: instructions, the retrieved passages, the question.
- Generate the answer, with instructions to cite which passage each claim came from.
That is the entire idea. The cleverness is in the retrieval step, not the generation step.
Most RAG failures are retrieval failures
When a RAG system gives a wrong answer, the instinct is to blame the model. It is almost always the search.
Before changing anything else, run this check: take the failing question, look at the passages that were actually retrieved, and ask whether a careful human could have answered correctly from them. If not, the model was never given a chance.
Common causes, roughly in order of frequency:
- Vocabulary mismatch. The user asks about "termination"; the contract says "cancellation". Pure vector search handles this reasonably well; pure keyword search does not.
- Exact identifiers. The user asks about invoice
INV-2024-8871. Vector search is bad at this — the embedding of an identifier carries almost no meaning. Keyword search finds it instantly. This is the single strongest argument for hybrid retrieval: run both, merge the rankings. - Bad chunk boundaries. The definition is in one chunk, the exception in the next, and only one was retrieved.
- Missing metadata filters. The answer came from a document the user is not allowed to see, or from a superseded version. Filter by permission and validity before similarity, not after.
- Too few results. Retrieving three chunks is efficient and brittle. Retrieve twenty, rerank, keep the best five.
When RAG is the wrong tool
RAG is not free. It adds an index to maintain, a retrieval quality problem to monitor, and latency to every request. Skip it when:
- The corpus is small. If your whole knowledge base is 20 pages, put it in the system prompt and cache it.
- The question is not about documents. Aggregations — "how many orders shipped late last month?" — belong in SQL. Give the model a query tool instead of a vector index.
- The task is style, not fact. Making the model write in your house voice is a prompting or fine-tuning problem.
There is also a middle path worth knowing: agentic retrieval, where the model is given a search tool and issues its own queries, refining them after seeing results. It costs more requests but handles multi-hop questions ("compare the 2024 and 2025 policies") that a single retrieval pass misses.
Making the answers trustworthy
Three practices do most of the work:
Require citations. Ask the model to attach a source identifier to each claim, and render those as links. Users can then check, and you can measure how often the cited passage actually supports the sentence.
Permit refusal. Instruct explicitly: if the retrieved passages do not answer the question, say that you do not know. Without this, a helpful model fills the gap from its general knowledge — which may be right, wrong, or about a different company entirely.
Evaluate retrieval separately. Build a set of question-and-correct-passage pairs and track recall at k. This number tells you whether retrieval is improving, independently of how the answers read. Teams that only eyeball final answers tune blind.
What good looks like
A mature RAG system is boring in a specific way: it answers from your documents, links to them, admits when it cannot find something, and shows the same behaviour tomorrow as today. Getting there is mostly search engineering — chunking, hybrid retrieval, reranking, filters — with the language model doing the last, smallest step.
If you are choosing where to spend a week of work, spend it on retrieval.
Häufige Fragen
- Is RAG better than fine-tuning?
- They solve different problems. RAG supplies facts the model does not have; fine-tuning teaches format, tone or a narrow skill. If your issue is 'it does not know our data', use RAG. If it is 'it does not answer the way we want', consider fine-tuning.
- Does a very large context window make RAG obsolete?
- It reduces the pressure but does not remove the need. Sending an entire corpus on every request is expensive and slow, and accuracy still suffers with very long inputs. Retrieval keeps each request small and relevant.
- Why does my RAG system still hallucinate?
- Usually because retrieval returned nothing useful and the model answered anyway. The fix is an explicit instruction and check: if the retrieved passages do not contain the answer, say so.
- What is chunking and why does it matter?
- Chunking is how documents are split before indexing. Chunks that are too small lose the surrounding context; chunks that are too large dilute the match. Splitting on document structure — sections and headings — usually beats splitting on a fixed character count.