RAG in production: retrieval, citations and evals that keep answers grounded

TL;DR

• Retrieval decides answer quality, so tune chunking and search on your own documents before you tune the prompt.
• Every answer cites the passages it used, and the system says it does not know when retrieval comes back weak.
• Filter by the user's permissions before the model sees a document, and guard grounding with a regression set that runs on every change.

Retrieval is the product

When a RAG answer is wrong, the model is rarely the first thing to fix. Bad answers often start with the wrong passages, or the right passage cut in half. If the evidence is not in the context, no prompt will put it there.

So measure retrieval on its own. For a set of real questions, note which documents should come back, and check that they do. Retrieval problems show up there before they show up in answers, and they are far cheaper to debug.

Chunk by structure, search by keyword and meaning

Split documents along their structure: headings, sections, list items, and table rows. Fixed-size windows separate definitions from the terms they define. Keep the title and section path with every chunk, so a passage makes sense on its own and can be cited.

Combine keyword search with vector search. Exact matches matter for product names, error codes, and clause numbers, and embeddings can miss them. Semantic search catches paraphrases that keywords miss. A reranker on the merged top results is usually worth trying before a bigger model.

Store the source, the version, and the access rules with each chunk. You will need all three.

Cite every claim

Ask the model to answer only from the supplied passages and to tag each claim with the passage it came from. Then check the tags in code. Every citation must point to a passage that was actually retrieved, and an answer with no citations is not shown as an answer.

Citations do two jobs. Users can verify an answer in one click, and you can trace a bad answer back to the passage, the chunk, and the document that caused it.

Refuse when retrieval is weak

A model given poor context will still write something fluent. Decide before generation whether the evidence is enough: a minimum relevance score, a minimum number of supporting passages, or keyword and vector results that agree. Below the line, say that the documents do not cover the question, and show what was found.

const hits = await search(query, { user, topK: 8 })
const strong = hits.filter((h) => h.score >= MIN_SCORE)
if (strong.length < MIN_HITS) {
  return { answer: null, reason: "not enough evidence" }
}
const draft = await generate(query, strong)
const ids = new Set(strong.map((h) => h.id))
const grounded = draft.citations.length > 0 &&
  draft.citations.every((id) => ids.has(id))
return grounded ? draft : { answer: null, reason: "ungrounded" }

The threshold is a product decision. Set it with the regression set, not by feel, and write down why when it changes.

Only retrieve what the user may read

Access control belongs in retrieval, not in the prompt. Filter by the user's permissions inside the search query, so documents they cannot open never reach the model. Telling the model to ignore restricted text does not work. Once it is in the context, it can leak into the answer.

Sync permissions from the source systems along with the content, and treat a stale permission as a bug. When access to a document is revoked, the index should follow in the same sync, not at the next full rebuild. Test it with two users who should get different answers to the same question.

Guard grounding with a regression set

Build a set of real questions with the documents that should be retrieved and the facts a correct answer contains. Include questions the documents cannot answer, where the right result is a refusal, and questions where two users should get different answers.

Run it on every change to the chunker, the embedding model, the prompt, or the model. Score retrieval, citation validity, refusals, and factual agreement separately, so a failure points at one layer. A new embedding model that lifts one score and drops another is a trade-off to decide, not an upgrade.

Add every grounding bug from production to the set, so it cannot quietly return.

Where this meets a delivery

Taking one RAG assistant from prototype to production, with access-aware retrieval, citations, and a grounding regression set, is what an AI production sprint covers. Once it is live, Managed AI operations keeps the index, the permissions, and the evaluations running as documents, models, and prompts change.

FAQ

How large should chunks be? Large enough to stand alone and small enough to cite precisely. Follow the document structure rather than a fixed size, and let the regression set decide.

Should the model or the code decide when to refuse? Both. Code checks retrieval strength and citation validity before anything is shown, and the prompt tells the model to say when the passages do not answer the question.

Can we filter restricted documents after generation? No. Filter before retrieval results reach the model, because anything in the context can end up in the answer.