Engineering guide · AI
Production RAG Checklist for Business Software
A retrieval-augmented generation demo can answer a few curated questions in an afternoon. A production system must handle stale documents, permissions, ambiguous questions, missing evidence, model changes and users who treat confident language as fact. This checklist focuses on the controls that make a document assistant operable after the demo.
By Anjula Paulus, Founder and software engineer · Published · 4 min read
Key takeaways
- Treat source ownership and freshness as product features.
- Evaluate retrieval separately from answer generation.
- Require citations that resolve to the exact supporting passage.
- Enforce document permissions before retrieval, not after generation.
- Design uncertainty, refusal and human handoff into the normal flow.
01
Define the answer boundary
State which sources are authoritative, which questions the assistant should answer and which must be refused or handed to a person. If policies conflict, define precedence. Without an answer boundary, evaluation becomes subjective and the assistant gradually expands into decisions it was never designed to make.
- Allowed source collections and owners
- Supported question types and user groups
- High-risk topics that require human review
- Freshness requirements for each source
02
Build a reliable ingestion pipeline
Track document identity, version, source, owner, permissions and ingestion time. Preserve headings, tables and useful boundaries instead of flattening every file into arbitrary character counts. Failed or partial ingestion must be visible and retryable.
Deletion matters as much as addition. When a source is withdrawn or access changes, its chunks and cached answers must stop appearing predictably.
03
Measure retrieval before judging the model
If the correct evidence is absent from the retrieved context, a better prompt cannot recover it reliably. Build a question set with expected source passages and measure whether retrieval finds them. Include paraphrases, acronyms, misspellings and questions whose answer does not exist.
Hybrid retrieval, metadata filters and reranking can help, but each addition should be justified against the evaluation set rather than adopted because it is fashionable.
04
Make citations useful and enforceable
A citation should open the document and identify the passage that supports the claim. Check that cited text actually entails the answer; merely retrieving a nearby document is not evidence. For important workflows, reject or flag statements that cannot be tied to a source.
05
Enforce privacy and permissions
Apply the user's access rights during retrieval so restricted chunks never enter the model context. Log source identifiers rather than unnecessary raw content, redact sensitive values where possible and document retention at every provider boundary.
Prompt-injection instructions can exist inside retrieved documents. Treat source text as untrusted data and keep system instructions, tool permissions and downstream actions outside its control.
06
Evaluate answers and operate the system
Use a versioned evaluation set covering correct answers, unanswerable questions, permission boundaries and adversarial inputs. Track retrieval quality, groundedness, citation validity, refusal behavior, latency and cost. Run it when prompts, models, chunking or source pipelines change.
Production monitoring should show ingestion failures, empty retrieval, model/provider errors, latency, token cost and user feedback. Preserve enough trace detail to reproduce a bad answer without storing more sensitive content than needed.
- No answer found and low-confidence paths
- Clear human escalation with context
- Model and prompt versions attached to traces
- Rollback path for model, prompt and index changes
Questions
Common questions about production rag checklist
How do you know whether a RAG system is accurate?
Use a versioned set of real questions with expected evidence. Measure retrieval, grounded answer quality, citation validity and refusal behavior separately, then rerun the set whenever the model, prompt or index changes.
Should a document assistant always answer?
No. It should say when evidence is missing, conflicting or outside its allowed scope and provide a clear path to a person for high-risk or unresolved questions.
How should document permissions work in RAG?
Filter sources using the requesting user's permissions before content reaches the model. Removing restricted text after generation is too late because the model has already received it.