How do you measure an AI agent's performance in customer service?

Short answer

How to measure AI agent performance starts with verified resolution, not containment. Track correct-answer rate, policy compliance, handover rate and quality, CSAT or effort on AI-handled conversations, and cost per resolution. Report AI agent resolution rate as the single outcome metric, with the rest as guardrails.

How to measure AI agent performance starts with verified resolution, not containment. Track correct-answer rate, policy compliance, handover rate and quality, CSAT or effort on AI-handled conversations, and cost per resolution. Report AI agent resolution rate as the single outcome metric, with the rest as guardrails.

· 6 min read

Why do containment and deflection rates overstate how well an AI agent is performing?

Containment and deflection rates overstate performance because both count a conversation as a win the moment it avoids a human agent, regardless of whether the customer’s problem was actually solved. Containment rate is the share of conversations that close inside the automated channel without escalating to a person. Deflection is broader: the share of all incoming contacts, including self-service hits on a help article, that never reach a queue at all. Neither tracks what happens to the customer afterward.

Verified resolution is the alternative: the issue is marked closed and the same customer does not contact support again about the same issue within a window the team sets. There is no single industry-standard length for that window. Published practice ranges from about 24 hours up to seven days depending on the vendor, so define the window explicitly before reporting a resolution number. A customer who abandons a chat, or gets an answer that sounds complete but is wrong, still counts as contained today. If that customer emails tomorrow about the same issue, it gets logged as a new, unrelated conversation unless someone deliberately links the two. Containment rate vs resolution rate comes down to whether anyone joins conversations by customer and issue tag to check for repeat contact. Manual QA review compounds the problem: most contact centers review only a small share of conversations by hand, far too thin a sample to catch a repeat-contact pattern on its own. The fix is to pick a window, join by customer and issue, and recompute containment net of repeats before reporting AI agent resolution rate.

What belongs on an AI agent QA scorecard?

An AI agent QA scorecard needs rows that separate real resolution from an answer that merely sounded complete.

Metric

Definition

How to measure

Common trap

Verified resolution rate

Issue closed, no repeat contact within the team’s window

Join conversations by customer and issue, divide verified closures by total

Counting an abandoned chat as resolved

Repeat contact rate

Same customer returns about the same issue

Inverse of the resolution join

Tags too coarse to link the repeat contact

Correct-answer rate

Share of AI answers factually correct against approved knowledge

QA review against a written rubric, evidence quoted for each score

Reviewing only clean-looking conversations

Policy compliance rate

Share of answers within refund, disclosure, and data rules

QA review scored against policy, separate from tone

Scoring tone instead of policy

Handover rate

Share of conversations passed to a human agent

Handovers as a percentage of total conversations

Treating a low handover number as the goal

Handover quality

Share of handovers with full context, no repeated explanation

QA review of the transfer moment itself

Measuring transfer speed, not usability

CSAT or effort on AI-handled conversations

Customer-reported satisfaction or effort after an AI interaction

Survey all AI-handled conversations, resolved or not

Surveying only cases marked resolved

Cost per resolution

Total cost per verified resolution

Platform cost plus review cost plus escalation cost, divided by verified resolutions

Leaving out QA and escalation time

Read this as one outcome metric sitting above seven guardrails; none of them should be averaged into a single blended score. A guardrail moving the wrong way, such as handover rate dropping while policy compliance also drops, signals that resolution gains are coming from hidden risk rather than real improvement.

How do you measure AI agent performance before and after go-live?

AI agent testing before go-live uses a fixed test set and a written rubric. The same rubric then carries into live monitoring, so quality is checked the same way before and after launch.

Before go-live:

  1. Build a test set from real past conversations across every intent, including long-tail questions and known edge cases.

  2. Write the rubric first, with one worked example each of correct, incorrect, and out-of-policy, so two reviewers reach the same verdict.

  3. Score with at least two reviewers, calibrate on a shared set of 20 to 30 conversations, and fix any rubric line where they disagreed.

  4. Set a launch threshold per intent, not one blended score, and launch only the intents that clear it.

After go-live:

  1. Review every conversation, or a large random sample, against the same rubric, with evidence quoted for each score.

  2. Alert when correct-answer rate or policy compliance drops below threshold over a rolling window, so a single bad day doesn’t trigger a false alarm.

  3. Add every production failure back into the test set as a named regression case.

  4. Re-run the full test set after any knowledge base or prompt change, before it reaches customers.

For customer service AI metrics reported upward, put verified resolution rate in front of leadership as the single outcome number, with correct-answer rate, policy compliance, handover quality, and cost per resolution trended weekly underneath it. A weekly trend catches drift that a monthly average hides. A rising hallucination count inside QA review is usually the first sign that correct-answer rate is about to slip.

How does OnClarity help with this?

OnClarity (www.onclarity.com) is built around this scorecard. Containment alone is not treated as the goal. Its AI agents answer only from approved company knowledge, with validators and confidence thresholds that stop a response from going out when the agent is uncertain. Low-confidence and high-risk cases hand over to a human with full context, which is the handover-quality row above.

AI Agent QA scores 100% of conversations, AI and human, against the team’s own rubric, with evidence quoted for each score, supporting correct-answer rate and policy compliance without sampling. Every AI and human action is logged and reviewable. OnClarity holds SOC 2 Type II certification and is GDPR and HIPAA Ready, Saudi PDPL aligned.

Frequently asked questions

What is a good resolution rate for an AI customer service agent?

There is no single industry benchmark for AI agent resolution rate, and any number quoted without context is close to meaningless. Set a baseline from the human-handled equivalent for each intent, then measure the AI agent’s movement against that baseline. Define the repeat-contact window the team is using before quoting or comparing any resolution figure.

How large a sample do you need to review AI conversations?

Review all of them if automated QA coverage is available, since full coverage removes the sampling question entirely. Without that, use a random sample large enough to split by intent, stratified across call types instead of being cherry-picked from easy or already-clean conversations, with reviewers calibrated on shared transcripts before scoring independently.

Should a high handover rate be treated as a failure?

No. A handover triggered by low confidence or a high-risk case is correct behavior. Driving handover rate toward zero just pushes risky or wrong answers through instead of escalating them. Judge the AI agent by handover quality and the resolution outcome after transfer, instead of by the raw handover rate itself.

Support teams are no longer a cost centre.

See how AI Support Agent, Agent Assist, and VoC Analyst work on your own ticket volume.

Support teams are no longer a cost centre.

See how AI Support Agent, Agent Assist, and VoC Analyst work on your own ticket volume.

Support teams are no longer a cost centre.

See how AI Support Agent, Agent Assist, and VoC Analyst work on your own ticket volume.