RAG or fine-tuning? Start with neither

They solve different problems

The reason this comparison generates confusion is that both get described as 'customising a model', which obscures the fact that they address unrelated gaps.

RAG supplies knowledge — facts, documents, current state, anything specific to your organisation or to right now. Fine-tuning changes behaviour — output format, tone, domain register, task structure.

So the diagnostic is straightforward once you listen to the actual complaint. 'It doesn't know our products' is a knowledge gap: RAG. 'It won't answer in our required format' is a behaviour gap: fine-tuning. 'It makes things up about our returns policy' is a knowledge gap that fine-tuning will make worse, not better — training on question-answer pairs teaches the model to produce confident answers in that shape without giving it any means of verification.

Why you should start with neither

Before either, try a properly specified prompt with a few worked examples. Prompt engineering costs nothing, iterates in seconds, and resolves a genuinely surprising share of what arrives framed as a training problem.

Most disappointing prompt results come from underspecification rather than from model limitation. State the output format explicitly — or better, use structured output mode. State the length, the audience, what to do with edge cases, and crucially what to do when the answer is unknown. Give two to five examples that cover the boundaries of the task rather than its easy centre. Decompose a prompt that is doing three jobs into three chained calls each doing one.

There is also a maintenance argument for exhausting this option first. A fine-tune is pinned to a base model version and must be re-run when the provider upgrades. A retrieval index needs re-embedding when the embedding model changes. A prompt has neither obligation.

When to add retrieval

Add RAG when the gap is factual and the facts are yours: internal documentation, current pricing, this quarter's policies, a specific customer's history. None of that is in the model's weights, and none of it should be — you want to update it by editing a document, not by retraining.

RAG is also effectively mandatory whenever answers must be verifiable. In legal, clinical, financial, and support contexts, an assertion the user cannot trace to a source is not useful, and retrieval is what makes citation possible at all.

The honest caveat, covered at length in our retrieval debugging piece: RAG systems fail at retrieval far more often than at generation. Budget your effort for chunking strategy, hybrid keyword-plus-vector search, and a reranking stage. That is where quality comes from — not from the choice of generation model.

When to add fine-tuning

Reach for fine-tuning when prompting has plateaued on a behaviour you need reliably, not occasionally. A strict output structure that must hold across thousands of calls. A house voice. A specialised classification judgement where your criteria differ from any general model's defaults. Domain vocabulary the model handles awkwardly.

There is also a purely economic case. If a small fine-tuned model can replace a large general one on a high-volume narrow task, the inference saving compounds every day the product runs. Fine-tuning to reduce cost is often a better-justified project than fine-tuning to increase quality.

On data: curation beats volume here to an unusual degree. A few hundred carefully consistent examples routinely outperform tens of thousands of noisy ones, because the model will faithfully learn any inconsistency in your labels. Every example should demonstrate exactly the behaviour you want, in exactly the format you want.

Where mature systems end up

Both, with a clean division of labour. Retrieval supplies current, private, citable facts. A light fine-tune enforces the output contract and domain register, so you are not spending several hundred prompt tokens per request re-explaining your format — which is a cost saving as well as a consistency gain.

The sequence we recommend: prompt until it plateaus; add retrieval for the knowledge gap; measure; fine-tune last, only for the behavioural residue prompting could not deliver consistently.

Whichever path you take, build the evaluation set first. A held-out set of real inputs with known good outputs is what lets you tell whether any of these changes helped — and without it you are making architectural decisions on impressions. It is also, not incidentally, the artefact that makes it possible to test whether a cheaper model would have been sufficient all along. Our full RAG vs fine-tuning comparison has the side-by-side detail.