How accurate is sentiment analysis?

Short answer

Sentiment analysis accuracy is not one fixed number. It depends on the method, the channel, and the quality of the transcript behind it, and it drops fast on sarcasm, mixed opinions, short replies, and noisy call transcripts. Trained human reviewers only reach 0.81 kappa agreement with each other, which sets the real ceiling for any benchmark.

Sentiment analysis accuracy is not one fixed number. It depends on the method, the channel, and the quality of the transcript behind it, and it drops fast on sarcasm, mixed opinions, short replies, and noisy call transcripts. Trained human reviewers only reach 0.81 kappa agreement with each other, which sets the real ceiling for any benchmark.

· 5 min read

How is sentiment analysis accuracy measured?

Sentiment analysis accuracy is measured by comparing a model’s labels against human-reviewed labels on the same messages. Four measures do most of the work.

Agreement with human labels is the share of messages where the model’s label matches a reviewer’s label. Precision for a class is the share of messages the model called negative, say, that a human also labeled negative. Recall is the share of all human-labeled negative messages the model actually caught. Macro-F1 averages precision and recall per class, then averages across positive, negative, and neutral equally, so strong results on one class cannot hide weak results on another. See how CX teams define and measure customer sentiment.

Published benchmarking on three-way message labelling has put top independent systems near 0.68 average recall on realistic social text, per SemEval-2017 Task 4 (2017). That ceiling tracks human limits: annotation research has found trained reviewers reaching roughly 0.81 agreement, measured as Cohen’s kappa, on detailed opinion labelling, per a 2003 study by Wilson and Wiebe, meaning two careful humans still disagree about one time in five. A model that scores above human inter-annotator agreement on a sample is usually fitting one reviewer’s habits, not reading customers more accurately. Per-message scoring and per-topic scoring also produce different denominators, so the two numbers are not interchangeable. See how net sentiment rolls these labels into a single trend line.

What makes sentiment analysis less accurate, and which methods hold up?

Accuracy drops most when the method cannot match the complexity of the conversation in front of it, not because the model is simply bad.

Accuracy risk

What it looks like

Lexicon

Supervised ML

LLM-based

Sarcasm

“Great, another outage”

Reads “great” as positive

Catches it only with similar training data

Stronger on sarcasm detection sentiment analysis tasks, still inconsistent

Mixed sentiment

Praises the product, complains about billing

Averages into one flat score

Forces one label, loses the split

Can separate aspect-based sentiment analysis into product versus billing

Domain language

“The app crashed”

Misreads domain terms as generic negative words

Learns meaning from labelled examples

Infers context, still misses niche jargon

Short replies

“Sure.”

Defaults to neutral

Low-confidence guess

Often guesses tone without enough text

Poor transcript quality

Garbled words from a noisy call

Scores whatever word survives

Degrades with noise

Degrades similarly, errors compound

Language switching

Mid-message language change

Covers one language only

Needs labelled pairs per language

Handles switching better if trained multilingually

Poor call transcript quality and messy verbatim text explain much of the gap between lab accuracy and production accuracy. Independent irony-detection research has found binary scores near 0.71 F1 that fall to roughly 0.51 once systems must classify irony type, per SemEval-2018 Task 3 (2018), which is the gap mixed and ambiguous messages create for every method above.

Lexicons hold up for narrow, stable vocabularies where every rule must be auditable. Supervised models earn their cost once a team can label a few thousand in-domain examples. LLM-based classification fits mixed and aspect-level sentiment best, provided a worked example set exists to check it against. Learn more in the sentiment analysis glossary entry.

How do we test sentiment accuracy on our own conversations?

A CX team can test sentiment accuracy on its own data in about a week, using five steps.

  1. Sample: pull a random set of recent conversations, stratified by channel, queue, and language.

  2. Label: have two reviewers independently label each conversation as positive, negative, neutral, or mixed, plus the topic involved. Calculate their agreement rate first. This sets the human baseline, since Wilson and Wiebe’s 2003 annotation study found trained reviewers reaching a Cohen’s kappa of about 0.81, and no system should be judged against a bar no human clears.

  3. Compare: score the system against the agreed labels and report precision and recall per class, then macro-F1. One blended accuracy number hides the fact that neutral and mixed classes usually carry most of the error.

  4. Review errors: read every disagreement and tag the cause using the risk table above, such as sarcasm or poor transcript quality. This turns a score into a fix list.

  5. Re-test: re-run the sample after any prompt, rubric, or model change, and draw a fresh sample each quarter so drift in product language shows up early.

Report the result as an accuracy range per channel, not one company-wide figure.

How does OnClarity help teams test sentiment accuracy?

OnClarity (www.onclarity.com) does not change the testing math above, but it changes what a team has to test. OnClarity Voice of Customer adds sentiment and topic labels to every conversation across calls, chats, tickets, reviews, surveys, and social, rather than a sampled subset. Full-population labelling ties sentiment to the causes behind it, so a negative trend points to a product or process issue instead of a bare score. It also gives a team the labelled population needed to draw the stratified sample from step one above, instead of building one from scratch. OnClarity is SOC 2 Type II certified, and GDPR and HIPAA Ready.

Frequently asked questions

Can sentiment analysis detect sarcasm?

Sentiment analysis can catch some sarcasm but misses a meaningful share, especially when the sarcasm depends on context outside the message itself. Independent sarcasm detection sentiment analysis research shows binary detection scoring far higher than fine-grained sarcasm-type classification, and simple lexicon tools fail most often because they score individual words rather than the gap between what is said and what is meant.

What is a realistic sentiment analysis accuracy target for customer conversations?

A realistic target is a range scored separately by channel, not one blended figure, since clean text classification and noisy voice transcripts do not perform the same. Trained human reviewers reach only about 0.81 kappa agreement with each other, per a 2003 annotation study, so a target above that ceiling usually signals overfitting, not better accuracy.

Why does sentiment analysis label so many messages neutral?

Sentiment analysis defaults to neutral when a message lacks clear polarity words or enough text to support a confident call, which happens often with short replies like “sure” or “ok.” Aspect-based sentiment analysis reduces this problem by scoring sentiment per topic within a message, instead of forcing one label onto the whole conversation.

Key terms

Sentiment analysis

Sentiment analysis scores the emotional tone of a conversation to flag risk and measure experience without waiting for a survey response. Applied across every contact rather than a sampled few, it covers the large majority of customers who never complete a satisfaction survey at all.

Verbatim

A verbatim is a customer’s exact words, kept unedited as evidence in voice-of-customer reporting. Verbatims carry the specificity that aggregate scores strip out, which is why they persuade product and executive audiences when a percentage change on a dashboard cannot do so on its own.

Net sentiment

Net sentiment is a single directional score for a topic or time period. It is calculated as positive mentions minus negative mentions, divided by total mentions. Unlike a per-item sentiment score, which rates one piece of feedback, net sentiment aggregates many. Buyers should know two limits. Neutral mentions are handled inconsistently across vendors. And the score is volume-sensitive, so a stable reading can mask rising positive and negative shifts that cancel each other out. Example: a CX team compares net sentiment on checkout-flow feedback before and after a release. OnClarity’s Voice of Customer supports voice of customer analytics by classifying feedback by topic and sentiment, so a net sentiment score can be traced back to the comments driving it.

Word error rate

Word error rate (WER) measures speech recognition accuracy. It is the sum of substitutions, insertions, and deletions, divided by the total words in the reference transcript, expressed as a percentage. Lower is better. Vendor-published WER figures are only meaningful when compared on identical audio. Accents, background noise, telephony codecs, and domain-specific vocabulary all push the error rate higher in production, regardless of benchmark claims. A contact center example: a voicebot transcribes a caller’s accent clearly but consistently mishears a product or brand name. This inflates substitution errors until that term is added to a custom vocabulary list. This voice AI glossary will keep growing as new contact center infrastructure terms and voice AI terminology enter vendor conversations. Bookmark it alongside OnClarity’s main glossary of call center technology definitions and AI contact center glossary entries for the next RFP or vendor call.

Support teams are no longer a cost centre.

See how AI Support Agent, Agent Assist, and VoC Analyst work on your own ticket volume.

Support teams are no longer a cost centre.

See how AI Support Agent, Agent Assist, and VoC Analyst work on your own ticket volume.

Support teams are no longer a cost centre.

See how AI Support Agent, Agent Assist, and VoC Analyst work on your own ticket volume.