Chatbot knowledge base checklist

A chatbot knowledge base is the material the bot is allowed to use when it answers. The hard part is not uploading files. It is deciding which document controls, who can change it, when it expires, and what the bot says when the answer is missing.

Microsoft describes a knowledge source as content made available to an agent at design time. Its current Copilot Studio documentation supports websites, files, SharePoint, Dataverse, ServiceNow, Salesforce, Azure AI Search, and other connectors. That range is useful, but connecting a source does not make its contents accurate or safe for customers.

Name the source of truth

Start with an inventory. For each source, record the system, owner, audience, access rule, last review date, and the subjects it may answer. A pricing page may control public prices. A CRM may control an individual customer's appointment. An internal policy may explain an escalation path. Those are different jobs.

Do not load five copies of the same policy and hope retrieval chooses the newest one. Mark one controlling version. Archive or exclude the rest. If two departments disagree, stop the affected answer until a person resolves the conflict.

The inventory also needs an explicit boundary. Draft contracts, employee notes, private customer records, and expired promotions do not belong in a public website bot simply because a connector can reach them.

Keep permissions attached to the content

Source permissions must survive retrieval. Microsoft says its Copilot connectors respect source-level permissions. AWS documents permission-aware filtering for some SharePoint knowledge sources, while warning that the application still has to authenticate the user and pass verified identity context.

That distinction matters. Retrieval filtering is not a substitute for login, authorization, or a customer-data policy. Test with at least three identities: a public visitor, an ordinary signed-in user, and an administrator. Each should see only the material allowed for that role.

Record service accounts and connector scopes too. A knowledge base that works only because one employee granted broad personal access will fail when that employee leaves.

Give every article an owner and an expiry rule

Each answerable subject needs a named business owner. Marketing may own public package descriptions. Operations may own hours and service areas. Legal or compliance may own required disclosures. The chatbot team owns retrieval and display, not the underlying policy.

Add a review date or a trigger. Prices should be checked when the price file changes. Hours should be checked when a location changes its schedule. Policies should be checked after a contract, regulation, or vendor change. A generic annual review is too slow for material that can change next week.

AWS explains that changed or deleted source files must be synced before a managed Bedrock knowledge base reflects them. Its documentation also exposes ingestion-job status and warnings. Keep the same operational record even if another platform hides the mechanics: source changed, sync started, sync completed, test passed, owner approved.

Make the answer traceable

Store a stable title, URL or document identifier, version, owner, and effective date with the content. Metadata should help retrieval separate a current public policy from an internal draft with similar language.

For customer-facing answers, show a useful source link when the channel supports it. Microsoft notes that grounded answers can be configured to require an in-text citation, and that an answer may be withheld when the model does not supply one. That is a sensible failure mode for material claims.

Do not treat a citation as proof that the answer is correct. The cited paragraph may be old, ambiguous, or outside the user's situation. Citations make review possible. They do not replace review.

Write refusal and handoff rules

The bot needs a defined response when retrieval returns nothing, sources conflict, the material is expired, or the question falls outside scope. It should say that it does not have an approved answer and offer the next human step. It should not fill the gap from model memory.

Set stronger stops for prices, availability, legal or medical questions, account changes, refunds, safety issues, and complaints. The stop should preserve the user's question and route it with the conversation history. Asking the customer to repeat everything is a broken handoff.

Google Cloud describes grounding against an external search API as a way to use current information from the business's own systems. Current is still not approved. The source system needs its own owner, access rule, and change process before real-time retrieval is safe to expose.

Test the questions people actually ask

Build a test set from support tickets, search terms, sales calls, and failed chatbot conversations. Include the obvious question, misspellings, vague wording, two subjects in one message, old product names, and a request that should be refused.

For each test, record the expected source, required facts, prohibited claims, and handoff condition. Run the set after a source change, connector change, model change, or retrieval configuration change. Microsoft provides a knowledge-source test workflow and status checks for whether a source is ready. Platform status is only the first check. The answer still has to match the approved material.

Review misses as knowledge work. A bad answer may come from missing content, a weak title, stale text, poor chunking, access failure, or retrieval ranking. Rewriting the prompt will not repair a policy that nobody owns.

Keep a change log that can explain an answer

Record the source version, ingestion or sync job, configuration version, test set, reviewer, approval time, and release time. When a customer reports a wrong answer, this is how the team reconstructs what the bot could see at that moment.

NIST's Generative AI Profile calls for documented risks, content provenance, human oversight, incident handling, and ongoing monitoring. A short change log supports all five. It also prevents a quiet source edit from erasing the cause of an incident.

Keep rollback simple. If the new source or index fails its tests, restore the last approved set and route affected questions to a person until the correction is ready.

Use the release checklist

Before launch, confirm that every source has an owner, audience, access rule, version, review trigger, and deletion path. Confirm that conflicting and expired copies are excluded. Run permission tests, grounded-answer tests, refusal tests, citation checks, and the human handoff. Save the results beside the release record.

DATIK's Lead Chatbot is built around approved knowledge and a named human takeover. The ownership terms explain who keeps the accounts, configuration, and records. A free diagnostic can map one real customer question from source to answer, refusal, or human owner before a connector is installed.

Sources and review limits

• NIST AI 600-1: Generative Artificial Intelligence Profile
• Microsoft Copilot Studio: knowledge sources summary
• Microsoft Copilot Studio: test knowledge sources
• Amazon Bedrock: sync a data source
• Amazon Bedrock: document-level access controls
• Google Cloud: grounding with your search API

Sources were checked on October 11, 2026. Vendor documentation describes current platform behavior and may change. Test the exact product, tier, connector, identity model, and channel used by the business.