What is Retrieval-Augmented Generation?

The plain definition

A language model knows only what was in its training data, up to its cutoff, and it cannot cite a source. RAG addresses both problems with a straightforward pipeline: take the user's question, retrieve the most relevant passages from your document store, put those passages in the prompt, and ask the model to answer using them.

The payoff is substantial. Answers reflect your current, private, authoritative content. Citations become possible because you know which passages were supplied. Updating knowledge means updating documents rather than retraining a model. And hallucination drops sharply, because the model is asked to summarise provided text instead of recalling from weights.

Retrieval quality is the whole game

Almost every disappointing RAG system fails at retrieval, not generation. If the right passage is not in the context, no model can produce a correct grounded answer — and the model will usually produce a confident wrong one anyway. Before blaming the model, measure retrieval directly: for a set of real questions with known source passages, what fraction of the time does the correct passage appear in the top k?

The levers that actually move that number, roughly in order of impact. Chunking — split on semantic boundaries such as headings and sections rather than fixed character counts, with modest overlap, and prepend document and section titles to each chunk so it is interpretable alone. Hybrid search — combine embedding similarity with keyword matching; pure vector search reliably fails on part numbers, proper nouns, and exact identifiers. Reranking — retrieve 50 candidates cheaply, then reorder with a cross-encoder that scores query and passage jointly; this is often the single largest quality gain available. Query rewriting — expand or decompose the question before retrieving, since users rarely phrase queries the way documents are written.

Also handle the retrieval-fails case explicitly. Instruct the model to say it cannot find the answer when the context does not contain it, and verify that it does. A system that fabricates when retrieval misses is worse than one that admits ignorance.

RAG versus fine-tuning, and the security problem

These solve different problems and are frequently confused. RAG supplies knowledge — facts, documents, current state. Fine-tuning teaches behaviour — format, tone, domain vocabulary, task structure. If the complaint is 'it does not know our products', that is RAG. If the complaint is 'it will not answer in our required format', that is fine-tuning. Production systems often use both.

One security issue deserves specific attention, because it is easy to build in by accident. Retrieved content enters the prompt, and the model cannot reliably distinguish instructions written by you from instructions embedded in a retrieved document. If any part of your corpus is user-supplied or externally sourced, an attacker can plant text that hijacks the model's behaviour — indirect prompt injection. Treat retrieved content as untrusted input: never let it trigger privileged actions, enforce permissions at retrieval time so a user cannot pull documents they should not see, and keep the model's available tools narrowly scoped.