LLM inference
Related terms
Large language model
A large language model is the model class behind modern conversational AI, summarisation, and reply drafting. Trained on broad text corpora, it predicts language fluently but holds no inherent knowledge of a specific business, which is why grounding and retrieval are required before answering customer questions.
LLM token
A token is the basic unit a large language model, or LLM, reads and writes. A token is typically a word fragment, not a whole word. Both input and output tokens count toward usage. The prompt, any retrieved knowledge-base content, and the model’s reply all add up to the total. Usage-based AI pricing is commonly tied to token volume, so conversation length, system prompts, and retrieved context all affect cost. Example: passing a long call transcript to a model for quality scoring consumes far more input tokens than a short chat exchange does. A ten-minute call transcript might use thousands of tokens before the model even produces a response. Every model also has a fixed context window, which is the maximum number of tokens it can process at once. That limit controls how much transcript or history the model can consider, and it factors directly into latency.
Latency
Latency is the delay between a customer’s message and the first useful response. In live chat and voice it directly shapes perceived quality, and it is one of the few AI quality measures a customer notices immediately, well before judging whether the answer was correct.
Usage-based pricing
Usage-based pricing charges customers for measured consumption of a value metric, such as conversations, minutes, messages, or tokens, rather than a flat per-seat fee. Contracts for tools like an AI support agent or AI voice agent usually combine a committed minimum volume with an overage rate for consumption above that threshold. Some agreements let unused volume roll over. Because support volume can spike sharply (a product recall driving a sudden surge in chat volume, for example), usage-based pricing shifts budget volatility onto the buyer. It differs from outcome-based pricing, which charges only for successfully resolved contacts. Buyers should check overage rates, rollover terms, and true-up timing before they sign.
