Model drift

An evaluation set is a fixed sample of real conversations used to score an AI agent’s quality before and after changes. Holding the sample constant is what makes results comparable, so evaluation sets are versioned and refreshed only deliberately when the underlying contact mix genuinely shifts.

Quality assurance is the structured review of conversations against a scorecard, applied to human agents and AI agents alike. Traditional QA samples a handful of conversations per agent each month; automated QA scores every conversation, which changes the exercise from spot-checking into actual measurement.

A knowledge gap is a recurring customer question the knowledge base cannot answer, usually visible as repeat escalations or low-confidence fallbacks on the same intent. Systematically closing gaps is the highest-return maintenance activity available to any team running automated support at scale.

A hallucination is a fluent but unsupported answer produced when a model is not grounded in real data. In customer service the risk is specific: an invented policy, price, or delivery date reads as authoritative to the customer and creates a commitment the business never made.