When AI Chatbots Invent Store Policies
Shopping chatbots sometimes state delivery times, returns windows, and discounts that do not exist. How stores can catch invented policy answers before customers rely on them.
A customer asks the store chatbot whether returns are free for sixty days. The bot says yes, fluently and confidently. The real policy is thirty days and a fee. The customer relies on the answer, and the dispute that follows is not really with the bot.
Invented policy answers are a known failure mode of generative chatbots grounded loosely in store content. This guide covers where inventions come from, the policy areas to test hardest, and the guardrails that reduce them.
Why policy answers get invented
A generative bot fills gaps with plausible text. Where its source material is thin, contradictory, or spread across old help pages, it blends versions into an answer that sounds like the store and matches no document the store ever published.
Staleness is the common trigger. A holiday returns extension ends, the help page updates, and the bot keeps quoting a cached or summarised version for weeks. The invention is often last seasons truth delivered with this seasons confidence.
Test the money questions on a schedule
Keep a fixed set of test questions covering returns, delivery times and costs, warranties, discounts, price matching, and cancellation, and run them after every policy or help content change. Compare answers word for word with the current policy, not with what sounds right.
Include the awkward variants: sale items, gift purchases, international orders, and items already used or opened. Inventions cluster at the edges, where the real policy has exceptions the bot smooths into a simple yes.
Ground the bot and show the source
The strongest practical guardrail is restricting policy answers to retrieved store documents, with the bot directed to say it does not know and offer a human route when no source covers the question. A bot allowed to improvise policy will, eventually, at scale.
Showing the policy link beside the answer helps customers and disciplines the system. An answer that must point at its source is an answer that can be checked, by the customer in the moment and by the store in its testing.
Handle the disputes the bot creates
When a customer relies on a wrong bot answer, the store faces a choice between its written policy and its published agent statement. Decide the stance before it happens, brief support on it, and log every case with the transcript.
That log is the improvement loop: each dispute names a question to add to the test set and a gap in the source material. Stores that treat bot disputes as support noise keep paying for the same invention monthly.
Related reading
If this topic is useful, these earlier guides go deeper on connected questions: AI Visual Search and the Photos Shoppers Bring, When AI Recommends Substitutes for Out of Stock Items, and How AI Search Summaries Describe Your Returns Policy.
Frequently asked questions
Why do chatbots invent policies instead of saying they do not know?
Because generative systems produce plausible text to fill gaps unless they are explicitly grounded in retrieved documents and instructed to decline. Thin or stale source material makes invention more likely, not less.
Which policy areas should be tested most?
Returns, delivery times and costs, discounts, warranties, and cancellations, including edge cases like sale items and international orders. Inventions cluster where real policies have exceptions.
What is the best guardrail against invented answers?
Retrieval grounding: answers drawn only from current store documents, a source link shown with the answer, and a clear fallback to a human when no source covers the question.
What should a store do when a customer relied on a wrong answer?
Follow a stance decided in advance, briefed to support, and log the transcript. Each case should add a test question and fix a source gap, or the same invention will recur.