Build guide · AI
Building an AI Chatbot That Answers From Your Company Documents
An assistant that answers questions from your own policies, manuals and contracts is now a week of work to demonstrate and a long way from that to a system people trust. The demo is easy because curated questions hit curated documents. Production is harder because real documents are stale, contradictory, permission-controlled and full of tables, and because a confident wrong answer is worse than no answer. This guide covers the decisions before the build.
By Anjula Paulus, Founder and software engineer · Published · 9 min read
Key takeaways
- Decide what the assistant replaces before choosing how to build it.
- Retrieval quality, not model choice, decides whether answers are correct.
- Permissions must be applied during retrieval, never filtered afterwards.
- Document ownership and freshness are the ongoing cost, not the model bill.
- If your documents contradict each other, fix that first; no assistant resolves it.
01
Terms used in this guide
These terms appear in every vendor conversation and are worth pinning down before the first demo.
- Retrieval-augmented generation (RAG) — fetching relevant passages from your documents and giving them to a language model as the basis for its answer.
- Chunk — a passage a document is split into for retrieval; chunk boundaries decide what evidence can be found.
- Embedding — a numeric representation of text that lets similar meanings be matched without identical wording.
- Vector search — finding passages by meaning rather than keyword; usually combined with keyword search for names and codes.
- Reranking — a second pass that reorders retrieved passages by relevance before the model sees them.
- Grounding — requiring the answer to follow from retrieved evidence rather than from the model's own training.
- Hallucination — a fluent statement with no support in the sources, which is the failure mode users notice last.
- Evaluation set — a versioned list of real questions with their expected evidence, used to measure changes.
- Deflection — the share of questions answered without reaching a person, the usual business metric for a support assistant.
02
Decide what the assistant is for
Three jobs are usually bundled into one request, and they have different accuracy requirements, different users and different measures of success. Separating them before the build decides most of what follows.
Deflecting repeated internal questions, such as policy and procedure queries from staff, tolerates occasional uncertainty because the person asking can escalate cheaply. Answering customers in your name does not. Drafting, where the assistant produces a first version a person edits and signs off, is the most forgiving of the three and frequently the most valuable, because the human review step is built in.
- Internal question answering — moderate accuracy requirement, cheap escalation, measure by deflection and time saved
- Customer-facing answering — high accuracy requirement, reputational exposure, measure by escalation rate and complaint rate
- Drafting with human review — moderate accuracy requirement, measure by edit distance and time to approved output
- Search and discovery — the answer is a document, not a sentence; often the simplest and most useful option
03
The build options side by side
Three routes cover most situations. Each line gives setup effort, control, permission handling, cost shape, who maintains it and where it fits.
- The assistant already in your document suite — setup: days; control over retrieval: none; permissions: inherited from the suite, which is a real advantage; cost: per seat; maintained by the vendor; fits low-risk internal questions over documents already in that suite.
- A hosted RAG platform — setup: weeks; control: configuration but not internals; permissions: as good as its connector for your systems; cost: per seat or per query plus storage; maintained by the vendor with your configuration; fits standard documents and standard sources when data residency is acceptable.
- A custom build — setup: months to production; control: complete, including chunking, retrieval, reranking and refusal behaviour; permissions: whatever your own systems enforce; cost: build effort plus model and infrastructure usage; maintained by you or your supplier; fits unusual documents, strict residency, deep integration or customer-facing use.
- A deliberate non-answer — rewrite and consolidate the documents, add a good search index, and reassess; setup: weeks; cost: staff time; fits any situation where the source documents contradict each other.
04
Where accuracy actually comes from
Teams reach for a larger model when answers are wrong. That is rarely the cause. If the passage containing the answer was not retrieved, no model recovers it, and the better the model the more plausibly it fills the gap. Measure retrieval first and separately: take fifty real questions, record which passage should answer each, and check whether retrieval finds it.
After retrieval, the next largest factor is the source documents themselves. Two policies that disagree, a manual three versions out of date, a table flattened into unreadable text at ingestion: each produces confident wrong answers that look like a model problem and are not. Our production RAG checklist guide covers the operational side of this in more detail, including evaluation, citations and monitoring.
- Build the evaluation set before the assistant, from questions people actually ask
- Measure whether the correct passage is retrieved, separately from whether the answer reads well
- Preserve headings, tables and lists at ingestion instead of splitting on character counts
- Record document version, owner and date so staleness is visible
- Require citations that open the document at the supporting passage
- Make refusal a normal outcome, with a route to a person
05
Permissions and data, decided early
If your documents have access controls, the assistant must apply them during retrieval so restricted passages never enter the model's context. Filtering the answer afterwards is too late, and an assistant that can summarise a document a user may not open is a data breach with a friendly interface.
Settle the data questions before the build rather than during procurement review: which provider processes the text, in which region, whether it is retained, whether it may be used for training, and what is logged. These answers also determine which of the three build options is open to you.
- Permissions resolved per user at query time, not per index
- Provider, region and retention written down for each model call
- Training on your data explicitly disabled in the contract
- Logging that records source identifiers rather than full content where possible
- Treat retrieved document text as untrusted input, since instructions can be hidden inside it
06
What it costs to run
The model bill is usually the smallest line and the one everyone asks about first. The durable costs are the document work and the operational attention, and a project that budgets for the build but not for these produces an assistant that is accurate in month one and quietly wrong by month six.
- Per-question model and retrieval usage, which scales with traffic and with how much context you send
- Storage and reindexing as documents change
- A named owner for each document collection, responsible for freshness
- Periodic evaluation runs when prompts, models or documents change
- Monitoring for ingestion failures, empty retrieval and user-reported bad answers
- The review time of whoever handles escalations
07
A decision rule
Apply these in order and stop at the first that matches.
- Rule 1: if the documents contradict each other or nobody owns them, fix that first; an assistant will repeat the contradiction with more confidence.
- Rule 2: if the documents already live in one suite, the questions are internal and low-risk, and the suite has an assistant, use it before building anything.
- Rule 3: if the sources are standard, residency is unconstrained and the use is internal, trial a hosted platform against your own evaluation set before committing.
- Rule 4: if the assistant will answer customers in your name, require citations, enforce refusal when evidence is missing, and plan a human escalation path from the start.
- Rule 5: if residency, permissions or document structure rule the hosted options out, build custom, and budget for evaluation and document ownership as part of it.
- Rule 6: if the real need is to find the document rather than to be told the answer, build search; it is cheaper, faster and often what people wanted.
08
When not to build one
Some requests for a document assistant are better answered another way, and recognising that early saves a project. The signals below are reliable.
- The questions are mostly about live data in a system, not text in documents
- A wrong answer carries legal or safety consequences and no review step is acceptable
- The documents are a small, stable set that a well-written page could cover
- Nobody will own document freshness after launch
- The volume of questions does not justify the operational attention
Questions
Common questions about company document chatbot
How accurate can a document chatbot be?
Accuracy is a property of your documents and your retrieval, not of the model alone. With clean, owned, well-structured sources and retrieval measured against a real question set, a narrow assistant can be dependable. With contradictory or stale documents, no model produces trustworthy answers.
Should we use an off-the-shelf tool or build a custom assistant?
Start with the assistant in the suite your documents already live in if the use is internal and low-risk. Move to a hosted platform for standard sources, and build custom when data residency, permission handling, unusual document structure or customer-facing accuracy requirements rule the others out.
How do document permissions work with an AI assistant?
The user's access rights must be applied during retrieval so restricted passages never reach the model. Removing content from the answer afterwards does not work, because the model has already read it and can paraphrase it.
What does it cost to run a document assistant?
The model usage is usually the smallest part. The ongoing costs are document ownership and freshness, reindexing, periodic evaluation runs, monitoring and the review time for escalated questions. Budget for those or the assistant degrades quietly.