AI Chatbot Readiness Checklist: 12 Tests Before You Go Live

A chatbot can make a support queue look smaller while customers are still stuck. A conversation that ends without a ticket is not necessarily a resolved enquiry. The customer may have received the wrong answer, been unable to reach a person or given up.

Before comparing AI chatbots for small business, decide whether a shortlisted product can meet your minimum operating conditions. This checklist is for a first public-facing bot on a website or WhatsApp, not a redesign of your entire support operation. Test each intended channel separately. A successful website trial does not establish that the WhatsApp handover works.

For the wider support journey, see our customer support and success blueprint. For human-assisted replies and queue design, use the support backlog playbook. The decision here is narrower: is this specific bot, in this specific scope, safe enough for a controlled public pilot?

1. Define the bot's permitted scope

What it is: A written boundary around what the bot may answer and what must go to a person. Answering a general policy question is different from making a decision about an individual customer's account.

Action: List the permitted topics, excluded topics, channels and any actions the bot may take. Start with answers drawn from approved public information. Keep refunds, account changes and individual eligibility decisions with a person unless they have been separately designed and tested.

Pass: Before seeing the bot's response, a support lead can classify each trial question as answer, admit uncertainty or hand over. The bot can be configured to stay within that scope.

Fail: The vendor cannot restrict the proposed scope, or the team cannot agree who should handle an excluded request. Narrow the scope and test again. Do not rely on a general instruction to be careful.

Definition: Chatbot scope: the questions, information and actions a bot is permitted to handle without a person deciding the outcome.

2. Prove the knowledge is current and accessible

What it is: The approved information the bot will use to answer customers. A bot cannot reliably apply a policy that is missing, disputed or available only in an obsolete version.

Action: For each permitted topic, record the source, owner, approval date and review trigger. During the trial, change a synthetic policy and check that the bot gives the new answer and no longer uses the old one. Record the time and configuration needed to make the change visible.

Pass: The owner can identify the approved source for every scripted answer. The trial bot gives the updated answer and does not reuse the superseded instruction.

Fail: The source is disputed, inaccessible or stale, or cannot be replaced without an unclear manual workaround. Remove that topic from scope or delay the pilot.

Microsoft's Copilot Studio knowledge-testing guidance, updated in 2026, describes testing knowledge sources in its test panel and notes that only sources with a Ready status can be used there. That is a useful product-specific check, not evidence that any particular answer is correct.

Definition: Knowledge base: the approved material a chatbot may use when responding, such as customer help articles and current policy instructions.

3. Run the 10-conversation answer and escalation script

What it is: A repeatable test of whether the bot answers, admits it does not know, or requests human help. A polished demonstration tells you little about how the bot handles difficult or out-of-scope questions if those questions are missing.

Illustrative fixture: A fictional shop permits answers about delivery and returns. Its approved trial source says: delivery takes three working days; unopened items can be returned within 30 days; customers must contact support to request a return. These are invented policy inputs, not legal rules or real business terms. Refund decisions and account lookups are outside the bot's scope.

Pass: All 10 conversations meet their expected result in three separate runs, using the same configuration and a fresh conversation each time. This is an illustrative strict threshold. It can expose repeat failures, but it does not prove reliability in production. Save the transcript and mark each result pass or fail. An unrun test is not a pass.

4. Verify human handover end to end

What it is: The route from a chatbot conversation to an accountable person. A message saying "contact support" is not the same as a request arriving in a monitored queue.

Action: In the trial channel, request a person and inspect the receiving helpdesk or shared queue. Check that the case includes the customer's issue, relevant conversation context and an owner or queue. Use synthetic identities only. Ask the support lead to confirm that an agent can find and act on it.

Pass: The customer sees what happens next, the request reaches the agreed queue with usable context, and the team can identify who owns it. If immediate transfer is unavailable, the customer receives a workable alternative and an accurate expectation rather than a false promise of a live agent.

Fail: The request disappears, lacks ownership or forces the customer to start again without a clear fallback.

Microsoft's handover guidance, updated in 2026, distinguishes an escalation message from a configured transfer to an engagement hub. Check the actual configuration rather than treating an escalation button as proof that the transfer works.

Definition: Human handover: the transfer of an unresolved conversation to a person or monitored queue, with enough context to continue it.

5. Test unavailable agents and failed delivery

What it is: A controlled failure drill for the handover route. It tests what happens when a customer asks for help outside staffed hours or the destination queue is unavailable.

Illustrative scenario: In a test environment, configure the fictional shop's support queue as unavailable and attempt the human-help request. Then simulate a failed case delivery. Do not interrupt a live customer service queue.

Pass: The customer receives the business's approved fallback, such as a monitored contact route and its stated coverage hours. The failed delivery is visible to an owner, who can recover the request safely. No message claims that an agent has received the case when delivery has failed.

Fail: The conversation ends with no route to help, the failure is invisible, or a promised transfer did not happen. Record what you observed, restore the test configuration and rerun the drill after correction.

6. Map the personal data before activating the bot

What it is: A record of what the chatbot collects, where information travels and which parties can access it. Free-text support conversations can contain more detail than the business intended to request.

Action: Ask the vendor to show the data path from the chosen channel through the bot, model provider, logs, helpdesk and any subcontractors. Identify the purpose of each field, the access permissions and whether conversation data is used for model improvement. Test with synthetic data only. Do not paste real customer transcripts into an unapproved trial.

Pass: Your privacy and security decision-maker accepts the documented data flows, permissions, contractual terms and proposed customer notice for the intended use. The bot asks only for information needed at that stage, and there is a documented handling route for sensitive or unexpected information.

The ICO's AI security and data minimisation guidance explains the data minimisation principle and says organisations should assess security risks in AI systems. The ICO notes that this guidance is under review. Check the current guidance and your own circumstances before making a legal decision.

Definition: Data minimisation: limiting personal data processing to information needed for the stated purpose.

7. Verify retention, deletion and access settings

What it is: Evidence of how long conversation records persist, who can see them and how deletion is handled across the systems in the proposed data path. A retention setting in the chat interface alone may not describe what happens in logs or connected systems.

Action: Obtain the vendor's documented settings and contractual answers for transcripts, logs, backups, connected helpdesk records and any subprocessors. Agree an appropriate retention period with your privacy lead. In a sandbox, create a synthetic conversation, restrict access and request deletion. Inspect what can actually be verified and record any exceptions or delays.

Pass: The relevant owner accepts the documented retention and access settings, and the deletion test produces evidence consistent with the agreed process. Any parts that cannot be tested are supported by satisfactory vendor documentation and remain visible as limitations.

Fail: Retention is unknown, permissions are broader than accepted, or deletion behaviour conflicts with your requirements. This blocks launch; passing the test would not, on its own, prove GDPR compliance.

8. Test the helpdesk record and customer context

What it is: How well the bot fits with the helpdesk or CRM your support team already uses. An integration is useful only if the receiving record contains what an agent needs without exposing unnecessary data.

Action: Use a synthetic customer and a test queue. Send a scripted escalation and check the record's channel, issue, conversation context, owner and status. If the intended workflow links to a CRM record, test a correct match, no match and ambiguous match. An ambiguous match should go to a person rather than attach information to the wrong account.

Pass: The support lead finds the correct record, can continue the conversation and can identify a failed or uncertain match. Required permissions are agreed and no unapproved account change is made.

Fail: The integration works only in a vendor demonstration, creates an unusable record or associates the conversation with the wrong customer. For a deeper test of recovery between systems, see how to test a cross-system handoff.

9. Establish what counts as a resolved enquiry

What it is: A measurement rule that distinguishes useful self-service from an abandoned or unresolved conversation. Without one, a lower ticket count can conceal a worse customer experience.

Action: Before a public pilot, agree the eligible enquiry types, the channels included and how you will identify follow-up contacts. Record support contacts across chat, email, phone and other relevant routes, alongside unresolved conversations, complaints, response times and human review findings. Where possible, match repeat contacts without collecting unnecessary identifiers.

Pass: The team can report both completed self-service and subsequent human contacts for the defined scope, with limitations stated. A customer leaving the chat without confirmation is recorded as unknown, not automatically resolved.

Fail: The proposed dashboard counts every closed chat as a successful deflection, or the team cannot observe contacts that move to another channel. Fix the measurement before using ticket reduction to justify the purchase.

Definition: Deflection: an enquiry handled without reaching a human support queue. Deflection alone does not establish that the customer's issue was resolved.

10. Set a controlled pilot and stop rules

What it is: An agreed way to expose a limited audience to the bot after the trial, with a person authorised to stop it. A short scripted test cannot cover every customer question.

Action: Select a limited topic and channel, a review owner and a comparable period or control group where practical. Compare like-for-like support hours and enquiry types, and record changes in volumes and repeat contacts. Review all early pilot conversations if the volume permits. If it does not, reduce exposure until the team can review safely.

Illustrative stop rule: Pause the bot immediately for an unsupported high-impact promise, a privacy incident, a lost human-help request or a failed handover without a working fallback. The owner investigates, corrects and retests before restarting.

Pass: The team can disable the bot, route customers to a working alternative and review the agreed indicators. Fail: There is no owner, no fallback or no practical way to stop exposure. A pilot may show whether support demand changes; it cannot guarantee causation or future performance.

11. Check the commercial and operational commitment

What it is: The people, permissions and vendor commitments needed to run the bot after the trial. A product may pass a demonstration and still be unsuitable if nobody can maintain its sources or investigate failed handovers.

Action: Ask for the proposed channel, integration, support arrangements, usage limits, data terms and the route for exporting or retrieving records if you leave. Confirm the actual quotation and plan with the vendor. Name a knowledge owner, a support owner and someone authorised to disable the bot.

Pass: The decision-makers accept the written proposal and have time allocated for source updates, conversation review, exception handling and vendor escalation. They know which work stays with the business.

Fail: A necessary integration or channel is only promised for later, the required plan is unclear, or ownership depends on an unavailable individual. Do not treat a verbal assurance as trial evidence.

If you still need to judge whether the repetitive work is worth pursuing, the optional Workflow Time Savings Calculator converts your inputs into estimated hours and pounds per year. It does not establish chatbot readiness or customer outcomes.

12. Repeat the tests in the intended channel

What it is: A final check that the actual website or WhatsApp configuration behaves like the trial configuration. A bot's answer may be identical while its customer journey, contact route or integration differs.

Action: Run the 10-conversation script, handover drill and failed-delivery drill in each intended channel using synthetic accounts. Confirm the deployed knowledge version, permissions, customer-facing instructions and receiving queue. Record the product configuration and version or settings tested, plus the date and reviewer.

Pass: Every applicable test passes on the channel proposed for the pilot, and the support owner signs off the evidence. If only the website passes, the decision applies only to the website. WhatsApp remains out of scope until independently tested.

Fail: A channel cannot be tested before purchase or its fallback, knowledge or destination differs from the approved setup. Delay that channel or negotiate a trial that exposes the actual configuration. A pass in one channel is not a transferable guarantee.

Trade-offs, risks and limitations

A strict launch gate costs time. Someone must maintain policy sources, review conversations, inspect vendor terms and respond when an integration or handover fails. Limiting the bot to public information may reduce the number of questions it can handle, but it also limits the consequences of a wrong account-specific answer.

The illustrative 10-conversation script is deliberately small. Passing it three times can expose an obvious failure, but it cannot establish a statistical error rate or cover every phrasing, accessibility need or unusual customer circumstance. Expand the script with your own high-consequence cases before launch. Keep the original cases unchanged so shortlisted products are tested fairly.

Privacy evidence has limits too. A vendor demonstration may show a setting without proving what happens in every log or backup. Record what was observed, what rests on vendor documentation and what needs legal or security review. None of the checklist results is a GDPR compliance guarantee.

When this recommendation does not apply

Do not use this checklist as permission to launch a bot that makes account-specific decisions, carries out refunds or changes customer records. Those actions need their own authorisation, identity and recovery tests. If your proposed trial cannot isolate synthetic data from production, fix that before testing.

The recommendation also fails if the support team cannot provide a workable human route. What happens if your only experienced agent is absent? An escalation path that depends on that person alone is not a dependable pilot condition. Arrange an owner and fallback, or delay.

If the knowledge is disputed or the bot repeatedly gives unsupported answers, consider human-reviewed drafts instead of public replies. That alternative still needs suitable data controls and a reviewer; it is not an automatic privacy exemption. The support backlog playbook covers the broader human-assisted approach rather than repeating it here.

Final review / summary

Use the following illustrative 14-day trial schedule as a buying exercise, not a promise that a vendor or your team can complete it in that time:

• Days 1–3: Agree scope, synthetic fixture, approved sources, privacy questions and named reviewers.
• Days 4–9: Run the scripted conversations and the handover, retention and integration drills. Save transcripts, settings and receiving-record evidence.
• Days 10–14: Correct failures where possible, rerun affected tests and obtain support, privacy and operational sign-off. Leave an untested item marked not run.

Decision rule: All 12 tests must pass for the intended configuration and channel before a limited public pilot. Any failed, unrun or unevidenced test means delay for that scope. A corrected failure must be rerun; a vendor promise alone does not turn it into a pass. After launch, the stop rules remain active and measurement determines whether to continue.

What to explore next

• Discuss a bounded support workflow → AI Automation Services
• Learn about the business → About SIMARA AI
• Discuss your trial evidence → Contact SIMARA AI

Sources & further reading

• Microsoft Learn: Test your agent's knowledge sources, updated 2026.
• Microsoft Learn: Hand off to a live agent, updated 2026.
• ICO: How should we assess security and data minimisation in AI?. The ICO notes that this guidance is under review.