There are six versions of the expenses policy. One is called FINAL. Another is called FINAL-NEW. Someone has helpfully uploaded both.
An AI knowledge base will not resolve that disagreement by becoming very confident about it.
The short answer: prepare a trusted, permissioned source collection before adding retrieval and answer generation. Test whether answers are supported, current and visible only to the right people. The quality of the knowledge base begins with who owns the documents.
Start with one collection and one audience
Choose a bounded use case: internal support procedures, an approved product library or a specific policy set. Name the intended users and the document owner. Avoid beginning with “all company files”.
Inventory the sources, formats, versions, permissions and review dates. Remove obsolete duplicates from the active collection while preserving records according to policy. Identify documents that must never enter the system.
If two sources conflict, ask the owner to resolve them or define how the answer should present the conflict. Do not make the model the accidental policy authority.
Prepare the source record
| Field | Why it matters |
|---|---|
| Stable document ID | Connects an answer to a specific source |
| Title and owner | Helps readers assess and challenge the content |
| Version and effective date | Separates current policy from old material |
| Access group | Controls who may retrieve it |
| Review date | Creates a maintenance obligation |
| Source location | Lets a reader inspect the authoritative document |
Keep tables, headings and contextual information intact where possible during extraction. A policy exception separated from its condition can become a dangerously incomplete passage.
Build retrieval before polishing the answer
Retrieval-augmented generation, or RAG, supplies relevant source material to a model when it answers. It does not automatically make every answer correct. Microsoft’s RAG overview explains the retrieval and generation pattern.
Test whether the retrieval step finds the right material before spending time adjusting the prose. Use real questions, synonyms, abbreviations and ambiguous wording. Check that the system retrieves the exception as well as the general rule.
A sensible first answer format is a short response, supporting sources and any uncertainty. The reader should be able to open the cited material. A citation that merely points to a large folder is not much help.
Enforce permissions before information reaches the model
Use the user’s authorised identity to constrain retrieval. Do not retrieve everything and ask the model to hide restricted content. Check both the search results and answer output for leakage.
Test with accounts from different teams. Ask about restricted documents directly and indirectly. Check what happens after access is revoked, including cached results and derived indexes. Permission changes need to propagate through the system.
For UK and UAE organisations, map where the source data and derived material are processed. Have the relevant owners determine the applicable data and contractual requirements. A “private” product label is not a complete assessment.
Build a test set with uncomfortable questions
Include questions with a clear answer, questions requiring multiple sources, conflicting policies, obsolete information and questions the collection cannot answer. Keep some examples separate from development.
Score source support, completeness, access compliance and appropriate abstention separately. Also measure usefulness to the person doing the work. An answer can be technically supported and still omit the step they need.
Use OWASP’s GenAI guidance to inform adversarial tests, including instructions embedded in documents. Retrieved text is evidence to interpret, not authority to override the system’s controls.
Give the knowledge base a maintenance owner
Define how documents are approved, updated and removed, how failures are reported and which changes trigger retesting. Track unanswered questions as evidence of a content gap, not automatically a reason to change models.
Monitor cost and response time alongside quality. Start with a manageable collection and expand when ownership and testing work reliably.
A knowledge base is a maintained service, not a one-off upload. If nobody owns the next policy update, the system has already begun drifting towards yesterday’s answer.
Sources and scope
Reviewed 1 November 2026. The examples and worksheets are my suggested working methods, not measured client results or official standards.
