OnClarity Insights
How accurate does AI need to be before it talks to customers unsupervised?
Asked by a contact centre leader in financial services.
6 min read
The short answer
There is no single safe number. Set the bar per type of contact, by what a wrong answer costs: AI can answer cheap-to-correct questions alone, while anything that moves money or touches regulated advice needs a person to approve or decide. Name who signs off each bar, then check every conversation after launch.
Why is one accuracy number the wrong question?
The person who asked us this works in a financial services contact centre, where customers can dispute what they were told. Their worry was simple: if the AI does not know the answer, does it say so, hand the customer to a person, or reason its way to something that sounds right? In their words, the confidence has to be high, because a wrong answer can turn into a financial dispute.
They are right to worry, and a single accuracy figure will not settle it. One number averages a wrong opening time and a wrong refund as if they cost the same. It is usually measured on a test set someone else chose, not on your customers’ questions. And it says nothing about what happens when the AI is unsure.
The cost of being wrong is not spread evenly. The US consumer finance regulator put it plainly in its 2023 review of chatbots in banking: “When a person’s financial life is at risk, the consequences of being wrong can be grave.” A wrong answer about opening hours costs a second contact. A wrong answer about a fee, a refund or a dispute costs money, trust and sometimes a regulator’s attention.
So the useful question is not “how accurate is it?” It is “how accurate does it need to be for this type of contact, and who decides?”
How do you set the bar per type of contact?
Sort your contact types by what a wrong answer costs, then decide how much the AI does alone at each level. Four levels cover most teams:
| Contact type | Examples | What a wrong answer costs | What the AI does | The bar |
|---|---|---|---|---|
| Information | Opening hours, delivery status, how-to questions | A follow-up contact | Answers alone; hands over when unsure | High on your own past conversations |
| Account changes | Update an address, change a plan, rebook | A reversible mistake and an annoyed customer | Acts after the customer confirms; hands over below its confidence threshold | Higher, with confirmation before every action |
| Money | Refunds, fees, credits, payment disputes | A financial loss or a dispute | Prepares the case; a person approves before money moves | Near perfect, or a person signs |
| Regulated or sensitive | Advice, complaints, fraud, security, customers in vulnerable circumstances | Regulatory and personal harm | Hands over to a person with the full conversation and a summary | The AI does not decide |
The higher the cost of a wrong answer, the less the AI does alone.
This is the shape of the answer, not a fixed rule. Your own levels depend on your products, your customers and your regulator. The point is to stop asking one question of the whole operation.
Who signs off each bar?
A bar without an owner is a hope. The CX leader can sign off the information level. Account changes need the operations owner of those systems. Money needs finance, because a wrong refund is a loss on their numbers. Regulated and sensitive contacts need risk and compliance.
Regulators already expect this. In its review of AI in financial services in October 2026, the UK Financial Conduct Authority wrote that “Where firms are deploying AI-supported tools, they remain responsible for consumer outcomes.” As good practice, it described firms “using risk scoring or tiering, to calibrate their governance requirements”, and one firm that required the accountable senior manager and the business areas to approve a playbook before signing off an AI tool.
Executives fund three things: more revenue, lower cost or less risk. A per-level bar with a named owner is how the AI line shows the third one.
What has to be in place before a level goes live?
Proof on your own conversations. Test each contact type on real past conversations, including the hard and hostile ones, before it goes live. A result on someone else’s test set is not proof that it works in your environment.
A confidence threshold. Below it, the AI hands over instead of guessing.
Approval for sensitive actions. Nothing that moves money happens without a person saying yes.
Handover with full context. The person who picks up sees everything the customer already said, so the customer never explains twice.
How you stop the AI inventing answers in the first place is a separate job, covered in How do you stop an AI agent from making things up?
How do you know the bar is holding?
Check every conversation after launch, not a sample. For each contact type, watch wrong answers, handovers, customers who come back about the same issue within 7 days, and complaints or disputes. Closed means it worked: a correct-sounding answer that brings the customer back did not fix anything.
Then move contact types between levels on the evidence. Start on the safe level, show it holds, and widen from there. Pull a contact type back the week it slips.
One warning from the same FCA review: human review can become “a ‘box-ticking’ exercise, rather than meaningful oversight.” Approval only protects you if the person approving has the time, the context and the authority to say no.
The honest trade-off
Setting a bar per contact type is slower than quoting one number. The automation rate starts lower, because money and regulated contacts stay with people or need approval. Testing on your own conversations takes expert time, and so does reviewing what the AI did every week. Some contact types may stay with people for a long time, and that is a choice, not a failure. And for cheap-to-correct questions, waiting for perfection costs customers speed. You are trading some early reach for a launch your finance and risk teams will sign, and one you can widen without going back to the start.
What to do this week
List your top 20 contact types by volume.
Score each one for what a wrong answer would cost: a follow-up, a reversible mistake, money, or regulatory harm.
Place each one on a level and decide what the AI does there: answer, act after confirmation, prepare for approval, or hand over.
Name the person who signs off each level, and get the first signature this month.
Pick the three highest-volume contact types on the information level and test them on real past conversations before go-live.
Questions people ask
How accurate are AI chatbots in customer service?
It depends on the question and on the setup, so a single figure tells you little. Accuracy on simple information questions can be high, while account actions, money and regulated topics are harder and costlier to get wrong. Test on your own past conversations, per type of contact, and judge each against what a wrong answer would cost.
What is human in the loop AI?
Human in the loop means a person reviews or approves what the AI does before it reaches the customer or changes something, such as a refund. It suits high-stakes contacts. For low-stakes ones, people can stay on the loop instead: the AI acts alone and people govern it by checking conversations and outcomes.
How do you improve AI accuracy in customer service?
Limit the AI to approved, current knowledge, narrow what it may answer, set a confidence threshold below which it hands over, and test every change on real past conversations. Then review conversations after launch and fix the knowledge or scope behind each wrong answer, so the same mistake does not reach the next customer.
Sources
Consumer Financial Protection Bureau, Chatbots in consumer finance (issue spotlight), June 2023.
Financial Conduct Authority, AI in financial services: good practice and areas for improvement, 19 Oct 2026 (updated 20 Oct 2026).
The four levels and owners above are a starting framework to adapt, not a standard. General information, not legal advice.
