How to test AI chatbot before launch is the question that separates a launch that surprises customers from one that surprises the owner. An agent that answers five friendly questions well in a demo can still fail the sixth real question, invent a price, or trap an angry customer in a loop — and customers meet those failures first. Testing is not a final polish; it is the work that decides whether the agent is ready to meet strangers. This plan takes an afternoon and catches most of what a demo hides.
Build the question bank from real chats
Write thirty to fifty questions, drawn from real customer messages wherever possible: the ten most frequent questions, the ten with money attached (prices, refunds, deposits), the ten with conditions (cutoffs, exceptions, eligibility), and the ten awkward ones (complaints, comparisons, “why should I choose you?”). Include misspellings, shorthand, and vague questions — “how much?” with no product named — because real customers write that way and the agent must handle it by asking a clarifying question rather than guessing a product. A bank built from marketing copy tests nothing; a bank built from real chats tests everything customers will actually do.
Score every answer on four grades
- Correct: the fact, the condition, and the next step all match the source document. This is the only passing grade for money, dates, and policy answers.
- Partial: the fact is right but a condition or next step is missing. Partial answers ship only if the missing piece cannot cause a wrong booking or payment.
- Wrong: any invented detail, any superseded price, any answer from the wrong document. Wrong answers block launch until fixed at the source — document or instruction — not by telling the agent to “try harder.”
- Harmful: a promise the business cannot keep, a rude reply, personal data disclosed, or a customer left in a loop. One harmful answer blocks launch on its own, whatever the rest of the bank scored.
The edge-case battery
Run a fixed battery of hostile-but-ordinary cases: a question with no answer in any document (the agent must say so and offer a human, not invent one); a question about a competitor's price (answer from your own documents or decline, never quote a rival's figures you do not hold); an angry message (acknowledge, remedy, escalate path); a request to speak to a human at every stage of the conversation (the handoff must work from any point, not only from the greeting); and a second question that changes the first (“actually, make it Friday”) to test whether the agent tracks context or restarts blindly. An illustrative example: a veterinary clinic's agent passed every routine question and failed the edge battery on “my dog ate chocolate” — it gave general advice instead of escalating immediately. The fix was an instruction: urgent health situations escalate to a human first, answer second.
Multi-turn threads and regression
Single questions flatter agents; threads expose them. Test five full threads end to end — a booking with a change, a complaint with a resolution, a price question followed by a discount question — and check that context survives, that the agent does not contradict its earlier reply, and that the handoff hands over the thread, not just the customer. Then fix what failed and re-run the entire bank, not just the failed questions: every fix can break a previously passing answer, and only a full re-run proves it did not. Keep the bank and the scores; at the next document change, the same bank becomes the regression test that proves the change was safe.
The sign-off gate
Launch when three conditions hold together: no wrong or harmful answers remain in the bank, every partial answer is a deliberate accept with a named owner, and the handoff has been tested by a real person from a real phone, not only in a dashboard preview. Write the gate down and have the owner sign it — informally is fine, explicitly is what matters. A launch without a gate launches on optimism; a launch with one launches on evidence, and the first week of live chats will confirm which kind it was.
Owner: “We tested five questions and launched.” The sixth real customer asked about a refund, and the bot invented a policy. Thirty real questions, scored honestly, would have caught it in an afternoon.
100 AI replies a month on WhatsApp, no credit card. Plans on the pricing page.
Frequently asked questions
- How do I test AI chatbot before launch?
- Build a question bank from real chats, score every answer on a four-grade rubric, run an edge-case battery, test full multi-turn threads and the human handoff, fix at the source, re-run the whole bank, and launch only through a written sign-off gate.
- How many test questions does a chatbot need?
- Thirty to fifty is a practical bank: the most frequent questions, money questions, conditional questions, and awkward ones, including misspellings and vague questions customers really send.
- What counts as a failing chatbot answer?
- Any invented detail, superseded price, or answer from the wrong document is wrong; any unkeepable promise, rude reply, data disclosure, or customer loop is harmful, and one harmful answer blocks launch on its own.
- Should I test chatbot handoff before launch?
- Yes, from a real phone and from every point in a conversation, not only from the greeting — and confirm the handoff passes the thread to the human, not just the customer.
- Do I re-test everything after fixing chatbot answers?
- Yes. Every fix can break a previously passing answer, so re-run the entire question bank after fixes, and keep the bank as the regression test for future document changes.