Your RAG system isn't bad at generating. It's bad at retrieving.

Measure retrieval before you touch anything else

Almost every underperforming RAG system we are asked to look at has a retrieval problem, not a generation problem. And it is diagnosable in an afternoon.

Build a set of 30 to 50 real questions, and for each one record which passage in your corpus actually contains the answer. Then measure one number: how often does that passage appear in the top k results your retriever returns? This is your top-k hit rate, and it is the ceiling on your whole system. If the right passage reaches the prompt only 60% of the time, no model upgrade fixes the other 40% — the information simply is not there.

The reason this diagnosis gets skipped is that the failure does not look like a retrieval failure. When the context lacks the answer, the model does not say so. It produces a confident, fluent, wrong response, which reads like a model problem. Measuring retrieval separately is what breaks that illusion.

Fix chunking first — it dominates everything

What you embed matters more than which embedding model you use, and chunking is where most quality is won or lost.

The common failure is splitting on a fixed character count. That cuts sentences in half, separates a heading from the text it governs, and produces chunks that are meaningless in isolation. Split on semantic boundaries instead — sections, headings, paragraph groups — with modest overlap so a fact spanning a boundary is not lost.

The second common failure is chunks that cannot stand alone. A retrieved passage reading 'this must be completed within 30 days' is useless without knowing what 'this' refers to. Prepend context to every chunk: the document title, the section heading, and any relevant identifier. This costs a few tokens and it substantially improves both retrieval and the model's ability to use what it retrieves.

On size: too large and the embedding becomes diffuse and matches nothing precisely; too small and it lacks the context to be interpretable. There is no universal answer, which is why you should test two or three configurations against your hit-rate measurement rather than adopting a default.

Then add hybrid search

Pure vector search fails on a specific and predictable class of query: exact identifiers. Part numbers, SKUs, error codes, proper nouns, contract references, drug names. Embeddings capture semantic similarity, and 'RX-4021' is semantically similar to 'RX-4022' — which is exactly wrong when a user needs the first one.

Keyword search, which the field spent several years treating as obsolete, is simply correct for these cases. BM25 matches the literal token and ranks it first.

So run both and fuse the results. Reciprocal rank fusion is a reasonable default and needs no tuning. The improvement is usually immediate and largest on precisely the queries your users complain about most, because exact-identifier lookups are common in technical, legal, medical, and e-commerce corpora.

Then rerank — often the single biggest gain

Embedding retrieval is fast because query and document are encoded independently, which means the model never sees them together. That independence is what makes indexing possible and it costs accuracy.

A cross-encoder reranker takes the query and a candidate passage as a single input and scores their actual relevance. It is far too slow to run across a whole corpus and perfectly fast across 50 candidates. So the pattern is: retrieve 50 cheaply with hybrid search, rerank those 50, keep the top 5 for the prompt.

In most pipelines we have measured, this is the largest single quality improvement available, and it is a bounded change — it does not require re-embedding anything or altering your index.

Then handle the queries themselves

Users do not phrase questions the way documents are written. They ask 'can I get my money back', and the document says 'refund eligibility criteria'. They ask compound questions containing two separate lookups.

Two techniques help. Query rewriting uses a cheap model call to expand or rephrase the question into terms closer to your corpus vocabulary before retrieving. Query decomposition splits a compound question into its component lookups, retrieves for each, and combines the results — which fixes the common case where a two-part question retrieves well for one part and not at all for the other.

In a conversation, also resolve references before retrieving. 'What about the second one?' is unretrievable on its own; rewritten against the conversation history into a standalone question, it works.

Finally, make failure explicit

Retrieval will sometimes miss, and the system's behaviour in that case is a design decision you should make deliberately.

Instruct the model to answer only from the supplied context and to state plainly when the answer is not present — then test that it does. Ask questions whose answers are genuinely absent from your corpus and confirm the system declines rather than improvising. A system that fabricates on a retrieval miss is worse than one that admits ignorance, because it destroys the user's ability to trust any answer.

Requiring citations serves the same purpose from the other direction. If every claim carries the passage it came from, a user can verify in seconds, and an unsupported assertion becomes visible rather than invisible.

One security note while you are in this part of the system: retrieved content enters the prompt, and the model cannot reliably distinguish your instructions from instructions embedded in a document. If any part of your corpus is user-supplied or externally sourced, treat retrieved text as untrusted input — enforce permissions at retrieval time, and never let retrieved content trigger a privileged action.