OnClarity Insights
How do you stop an AI agent from making things up?
Asked by a support leader at a European consumer marketplace.
6 min read
The short answer
You cannot make a language model incapable of inventing, but you can stop inventions reaching customers. Let the AI answer only from approved, current knowledge and only on topics it has been tested on, make it hand over when it is unsure, test it on real past conversations before every change, and check what customers did next.
Why do AI agents make things up?
In customer service, most invented answers start somewhere you can see:
No article for the question. The knowledge base does not cover it, so the model fills the gap with something plausible.
Two sources that disagree. An old policy article sits next to the new one, or the instructions contradict the knowledge base.
No limit on scope. The agent is allowed to try anything, so it tries.
Pressure to answer. The instructions reward answering and offer no safe way to say “I don’t know”. Long, complicated instructions are harder to follow, which makes it worse.
Before you blame the model, read your own instructions. Some “made up” answers are written into the prompt: a promised wait time, or a check the agent has no way to run.
The second cause has been tested in law. In February 2024, a Canadian tribunal held a company liable for a wrong refund answer from its website chatbot, although the correct policy was on another page of the same site. The company, it said, is responsible for all the information on its website, whether it comes from a static page or a chatbot.
Grounding the model in your own documents helps, but does not finish the job. In a preregistered Stanford study, legal research tools built on retrieval still gave misleading or false answers to 17% to 33% of questions (Stanford RegLab, 2024). So the controls below assume some inventions will get through, and are built to catch them.
What are the five controls?
Each control comes with a test. If you cannot run it, the control is not in place.
Five controls, each with the test that shows it works.
1. Answer only from approved, current knowledge
Give the agent one source of truth: articles with a named owner and a review date. When two disagree, retire the old one, so the fix holds for every customer, not just the next one.
Test: ask 20 questions whose answers changed in the last 90 days. Every answer should match the current article and show which article it came from.
2. Limit what it may answer
Name the topics the agent handles. Everything else goes to a person. Gartner’s advice to service leaders is to “prioritize reliability over reach”: a chatbot that reliably resolves a targeted set of issues builds more confidence than one that tries everything. In its survey, only 27% of customers said they would try a chatbot again after a negative experience (Gartner, September 2026).
Test: ask 20 questions just outside the agreed topics. The agent should hand over on all of them.
3. Make “I don’t know” safe
When the agent has no source, it should say so and hand the customer to a person with the full conversation, so they do not have to explain again. A clean handover beats a confident guess.
Test: ask 10 questions you know have no article. Count the confident answers. The target is zero, and the person who picks up should see everything the customer already said.
4. Test on real past conversations before every change
Build an evaluation set from real conversations: the common ones, the hard ones, the hostile ones and the ones that went wrong. Rerun it whenever an article, an instruction or the model changes. Ten to 20 cases that must pass before a new version goes live is enough to start.
Test: change one article and rerun the set. Nothing that passed before should fail.
5. Check what was said, and what the customer did next
A correct-looking answer can still leave the problem in place. Review answers with no supporting source, and watch whether customers came back about the same issue. Each failure should end in a fixed article, a narrower scope or a new test.
Test: pull last week’s AI-handled conversations where the customer came back within 7 days. For each one, can you say which control should have caught it?
Where would each control catch it?
An illustrative example. A customer asks for a refund after the refund window. The agent says yes. Last month the window was shortened, but the old article is still live next to the new one.
Knowledge: a conflict check flags the two articles, and the old one is retired.
Scope: refund exceptions are on the list of topics that go to a person.
Handover: with two sources disagreeing, the agent hands over instead of picking one.
Testing: a “refund after the policy changed” case fails when the new article is published.
Monitoring: the customer comes back within 7 days, and the answer is traced to the old article.
Any one of these stops the wrong answer reaching the next customer. Retiring the old article stops it for everyone.
Which two numbers should you watch?
Unsupported answers: the share of AI answers with no supporting source in your approved knowledge.
Recontact: the share of AI-handled issues where the same customer comes back about the same thing within 7 days.
The first tells you the agent is inventing. The second tells you whether it mattered to the customer. Closed means it worked.
Who owns it?
Knowledge owners fix articles at the source. CX operations owns scope, testing and the weekly review. And a person, not the model, decides what the right answer should have been.
The honest trade-off
Tighter controls lower the automation rate at first, because more conversations go to people. Evaluation sets take expert time to build and keep current. Reviewing conversations and outcomes costs money. Some wrong answers will still reach customers; the aim is fewer of them, caught faster and fixed once. And for low-stakes questions, too much caution costs customers speed. You are trading some reach for answers customers can trust. Start with the topics where a wrong answer is costly, and widen the scope as the tests pass.
What to do this week
Write down the topics your agent may answer, and send everything else to a person.
Find articles that disagree with each other and retire the old ones.
Ask the agent 10 questions it has no article for, and count the confident answers.
Build a test set of 10 to 20 real past conversations, including ones that went wrong, and run it before every change.
Each week, review AI-handled issues where the customer came back within 7 days.
Questions people ask
What are AI hallucinations?
An AI hallucination is an answer a language model presents with confidence that is false or not supported by its sources. NIST’s Generative AI Profile calls this “confabulation”. In customer service it usually looks like an invented policy, a made-up step or a promise the company cannot keep, delivered in the same tone as a correct answer.
What causes AI hallucinations?
Language models predict likely text, so when the right information is missing they can produce something plausible instead. In customer service the common triggers are gaps in the knowledge base, articles that contradict each other or the agent’s instructions, questions outside the agent’s scope, and instructions that push it to answer rather than hand over.
How do you prevent AI hallucinations?
You cannot remove them completely, but you can stop most of them reaching customers. Limit the agent to approved, current knowledge and named topics, make handing over to a person the default when it is unsure, test every change on real past conversations, and review unsupported answers alongside whether customers came back.
Sources
Gartner, Gartner Survey Finds Only 27% of Customers Would Try a Chatbot Again After a Negative Experience, press release, 2 Sep 2026 (survey of 3,566 B2B and B2C customers, February and March 2026).
Magesh, Surani, Dahl, Suzgun, Manning and Ho, Stanford RegLab, Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, May 2024 (preregistered evaluation).
British Columbia Civil Resolution Tribunal, small claims decision 2024 BCCRT 149, 14 Feb 2024.
National Institute of Standards and Technology, NIST AI 600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, July 2024.
The refund example above is illustrative. General information, not legal advice.
