How do you measure an AI agent's performance in customer service?
Short answer
· 6 min read
Why do containment and deflection rates overstate how well an AI agent is performing?
Containment and deflection rates overstate performance because both count a conversation as a win the moment it avoids a human agent, regardless of whether the customer’s problem was actually solved. Containment rate is the share of conversations that close inside the automated channel without escalating to a person. Deflection is broader: the share of all incoming contacts, including self-service hits on a help article, that never reach a queue at all. Neither tracks what happens to the customer afterward.
Verified resolution is the alternative: the issue is marked closed and the same customer does not contact support again about the same issue within a window the team sets. There is no single industry-standard length for that window. Published practice ranges from about 24 hours up to seven days depending on the vendor, so define the window explicitly before reporting a resolution number. A customer who abandons a chat, or gets an answer that sounds complete but is wrong, still counts as contained today. If that customer emails tomorrow about the same issue, it gets logged as a new, unrelated conversation unless someone deliberately links the two. Containment rate vs resolution rate comes down to whether anyone joins conversations by customer and issue tag to check for repeat contact. Manual QA review compounds the problem: most contact centers review only a small share of conversations by hand, far too thin a sample to catch a repeat-contact pattern on its own. The fix is to pick a window, join by customer and issue, and recompute containment net of repeats before reporting AI agent resolution rate.
What belongs on an AI agent QA scorecard?
An AI agent QA scorecard needs rows that separate real resolution from an answer that merely sounded complete.
Metric | Definition | How to measure | Common trap |
|---|---|---|---|
Verified resolution rate | Issue closed, no repeat contact within the team’s window | Join conversations by customer and issue, divide verified closures by total | Counting an abandoned chat as resolved |
Repeat contact rate | Same customer returns about the same issue | Inverse of the resolution join | Tags too coarse to link the repeat contact |
Correct-answer rate | Share of AI answers factually correct against approved knowledge | QA review against a written rubric, evidence quoted for each score | Reviewing only clean-looking conversations |
Policy compliance rate | Share of answers within refund, disclosure, and data rules | QA review scored against policy, separate from tone | Scoring tone instead of policy |
Share of conversations passed to a human agent | Handovers as a percentage of total conversations | Treating a low handover number as the goal | |
Handover quality | Share of handovers with full context, no repeated explanation | QA review of the transfer moment itself | Measuring transfer speed, not usability |
CSAT or effort on AI-handled conversations | Customer-reported satisfaction or effort after an AI interaction | Survey all AI-handled conversations, resolved or not | Surveying only cases marked resolved |
Total cost per verified resolution | Platform cost plus review cost plus escalation cost, divided by verified resolutions | Leaving out QA and escalation time |
Read this as one outcome metric sitting above seven guardrails; none of them should be averaged into a single blended score. A guardrail moving the wrong way, such as handover rate dropping while policy compliance also drops, signals that resolution gains are coming from hidden risk rather than real improvement.
How do you measure AI agent performance before and after go-live?
AI agent testing before go-live uses a fixed test set and a written rubric. The same rubric then carries into live monitoring, so quality is checked the same way before and after launch.
Before go-live:
Build a test set from real past conversations across every intent, including long-tail questions and known edge cases.
Write the rubric first, with one worked example each of correct, incorrect, and out-of-policy, so two reviewers reach the same verdict.
Score with at least two reviewers, calibrate on a shared set of 20 to 30 conversations, and fix any rubric line where they disagreed.
Set a launch threshold per intent, not one blended score, and launch only the intents that clear it.
After go-live:
Review every conversation, or a large random sample, against the same rubric, with evidence quoted for each score.
Alert when correct-answer rate or policy compliance drops below threshold over a rolling window, so a single bad day doesn’t trigger a false alarm.
Add every production failure back into the test set as a named regression case.
Re-run the full test set after any knowledge base or prompt change, before it reaches customers.
For customer service AI metrics reported upward, put verified resolution rate in front of leadership as the single outcome number, with correct-answer rate, policy compliance, handover quality, and cost per resolution trended weekly underneath it. A weekly trend catches drift that a monthly average hides. A rising hallucination count inside QA review is usually the first sign that correct-answer rate is about to slip.
How does OnClarity help with this?
OnClarity (www.onclarity.com) is built around this scorecard. Containment alone is not treated as the goal. Its AI agents answer only from approved company knowledge, with validators and confidence thresholds that stop a response from going out when the agent is uncertain. Low-confidence and high-risk cases hand over to a human with full context, which is the handover-quality row above.
AI Agent QA scores 100% of conversations, AI and human, against the team’s own rubric, with evidence quoted for each score, supporting correct-answer rate and policy compliance without sampling. Every AI and human action is logged and reviewable. OnClarity holds SOC 2 Type II certification and is GDPR and HIPAA Ready, Saudi PDPL aligned.
Frequently asked questions
What is a good resolution rate for an AI customer service agent?
There is no single industry benchmark for AI agent resolution rate, and any number quoted without context is close to meaningless. Set a baseline from the human-handled equivalent for each intent, then measure the AI agent’s movement against that baseline. Define the repeat-contact window the team is using before quoting or comparing any resolution figure.
How large a sample do you need to review AI conversations?
Review all of them if automated QA coverage is available, since full coverage removes the sampling question entirely. Without that, use a random sample large enough to split by intent, stratified across call types instead of being cherry-picked from easy or already-clean conversations, with reviewers calibrated on shared transcripts before scoring independently.
Should a high handover rate be treated as a failure?
No. A handover triggered by low confidence or a high-risk case is correct behavior. Driving handover rate toward zero just pushes risky or wrong answers through instead of escalating them. Judge the AI agent by handover quality and the resolution outcome after transfer, instead of by the raw handover rate itself.
Key terms
Containment rate
Containment rate is the share of automated conversations that end without escalation to a human. It is the primary efficiency measure for AI support, but a high rate only indicates success when paired with satisfaction and repeat-contact data confirming customers actually got their issue resolved.
Deflection
Deflection is steering a contact toward a self-service or automated resolution before it reaches an agent queue. Effective deflection resolves the underlying question; unsuccessful deflection merely delays the contact, so teams measure it by whether the customer returns rather than by how many contacts were diverted.
Hallucination
A hallucination is a fluent but unsupported answer produced when a model is not grounded in real data. In customer service the risk is specific: an invented policy, price, or delivery date reads as authoritative to the customer and creates a commitment the business never made.
Resolution rate
Resolution rate is the share of contacts closed with the customer’s issue actually solved, as distinct from contacts merely closed. The gap between the two is where reopened tickets and repeat contacts live, so the measure is only trustworthy when reopens are tracked against it.
