How do you analyze open-ended survey responses?

Short answer

Analyzing open-ended survey responses means coding every response, not a sample, against themes that capture sentiment, effort and intent, then tying each theme to your NPS or CSAT score. This guide lays out a nine-step method, from export to a fix someone owns.

Analyzing open-ended survey responses means coding every response, not a sample, against themes that capture sentiment, effort and intent, then tying each theme to your NPS or CSAT score. This guide lays out a nine-step method, from export to a fix someone owns.

· 5 min read

What are the steps to analyze open-ended survey responses?

Getting from a raw export to a fix someone owns takes nine steps.

  1. Frame the decision the analysis must inform, for example “should we change the cancellation flow,” not a vague goal like “understand feedback.”

  2. Clean the export: drop blanks, test rows and duplicates, but keep each verbatim comment’s score, segment and date attached.

  3. Build a starting codebook from your support taxonomy, or let themes emerge from a first read of a few hundred comments, following the familiarization and coding phases of thematic analysis.

  4. Code every response, allowing multiple themes per comment alongside sentiment, effort and intent, since one comment can mention a billing error, frustration and cancellation intent at once.

  5. Quantify theme volume and share across the full coded set, reporting both the count and the percentage with the base size stated.

  6. Tie each theme to the score it sits beside, whether NPS, CSAT or CES, showing the detractor or low-CES share, not just overall frequency.

  7. Find the cause behind the theme rather than the words customers used. “Slow replies” usually traces back to a staffing gap or a broken handoff.

  8. Assign one named owner and a date to each prioritized theme. A person, not a department, commits to a fix.

  9. Re-measure the theme share next cycle and report whether the detractor share dropped after the fix shipped, the only proof the loop closed.

Manual coding, keyword rules, machine learning or an LLM: which method should you use?

Most teams working out how to analyze open ended survey responses end up mixing methods rather than picking one, because nuance and consistency trade off against speed.

Method

Catches

Misses

Time per 1,000 responses

Manual codebook coding

Nuance, sarcasm, edge cases

Consistency over time as codebooks drift

Days of reviewer time

Keyword and rules tagging

High-frequency, literal complaints

Paraphrase, negation, new issues nobody wrote a rule for

Near-instant once built

Supervised ML classifiers

Consistent tagging at volume

Anything outside its training labels

Fast to run, slow to train

LLM theme detection

Emergent themes, paraphrase, multiple themes per comment

Comparability unless definitions are frozen

Much faster than manual coding, run across the full dataset instead of a sample

A practical sequence lets themes emerge first, then freezes a reviewed taxonomy for every cycle after. That change is what lets survey text analytics compare detractor share this quarter against last quarter without the comparison breaking every time someone renames a category. See the definition of text analytics for more detail.

How do you quantify themes and prove the coding is right?

Theme share equals the number of comments carrying a theme divided by the number of comments in that score band, reported separately for detractors and promoters. A theme that appears in, for example, 40 percent of NPS detractor comments (scores 0-6) with a clear cause outranks a theme appearing in more comments overall but split evenly across promoters and detractors.

A common minimum base size is around 30 responses before reporting a percentage at all. Below that, report counts and the underlying verbatim comments instead, and treat small shifts as noise rather than a trend.

Proving the coding is right takes three steps: spot-check a random sample against written theme definitions, have two reviewers independently code the same 50 to 100 responses and compare agreement with Cohen’s kappa (values above roughly 0.60 are typically treated as acceptable), then rewrite any definition the reviewers disagreed on and recode the affected responses.

Survey comments read better next to tickets, chats, calls and reviews on the same topic, because a comment names the break but rarely shows its scale. A support ticket thread or call recording on the same theme shows where it happens and how often, shortening the path from coding open-ended responses to a confirmed root cause.

How does OnClarity help with this?

OnClarity (www.onclarity.com) runs the coding and linkage work above across the full dataset, not a sample. Its Voice of Customer product reads 100% of conversations across calls, chats, tickets, reviews, surveys and social, applying topics and sentiment to every one and tying them back to the causes behind them, so survey comments share a taxonomy with the tickets and calls raised about the same issue.

Research Copilot answers questions in plain language over the full dataset, and alerts flag a theme when it starts trending, ahead of the next reporting cycle.

OnClarity analyzes survey results; it does not send or build surveys. It is SOC 2 Type II certified, GDPR and HIPAA Ready, and Saudi PDPL aligned. Learn more about Voice of Customer.

Frequently asked questions

How many open-ended responses do you need before the percentages mean anything?

A commonly cited minimum is around 30 responses before reporting a percentage, and under 100 responses any percentage should appear alongside the raw count. Below roughly 30, report counts and verbatim comments instead, and treat small shifts in theme share as noise rather than a trend.

Can an LLM code open-ended survey responses reliably on its own?

LLM theme detection can come close to expert human coding on factual, clearly worded responses, but accuracy drops on sarcasm, nuance and attitudinal answers unless paired with human review. Treat LLM output as a fast first pass, then spot-check a sample and compare two reviewers before trusting the theme shares.

Why is a word cloud a poor way to report open-ended survey results?

A word cloud shows word frequency, not meaning: it cannot tell whether “slow” appeared in a complaint or a compliment, carries no statistical weighting, and counts “fish” and “fishes” as separate words. It also cannot show that one comment ties a theme to effort, sentiment and intent at once.

Key terms

Verbatim

A verbatim is a customer’s exact words, kept unedited as evidence in voice-of-customer reporting. Verbatims carry the specificity that aggregate scores strip out, which is why they persuade product and executive audiences when a percentage change on a dashboard cannot do so on its own.

Text analytics

Text analytics automates the process of extracting structure and meaning from unstructured written feedback: support tickets, chat transcripts, survey verbatims, reviews, and social posts. Core techniques include classifying text into predefined topic taxonomies, extracting named entities such as products, features, and locations, modeling topics to surface unlabeled themes, and tagging a sentiment score to gauge tone. The practical output is a count of how often each issue appears and how that frequency trends over time. Results are only as useful as the underlying taxonomy: a vague or inconsistent category structure produces noisy, hard-to-action data. OnClarity’s Voice of Customer applies these techniques across feedback aggregated from 100+ sources.

Root cause analysis

Root cause analysis is the practice of tracing a symptom, such as a spike in complaints, back to the underlying process, policy, or product defect that produced it. The alternative, resolving each ticket in isolation, never stops the symptom from recurring. Established methods include 5 Whys (repeated questioning to reach the originating cause), fishbone (Ishikawa) diagrams (mapping possible causes by category), and Pareto analysis (isolating the small number of causes driving most of the volume). In CX, this starts with clustering tickets by theme, then tracing the largest theme to its source. OnClarity’s Voice of Customer automates this by surfacing root causes directly from feedback. Grubhub uses it to turn feedback into insights that shape its roadmap. The real test of success is recurrence: a fix that addresses the root cause should stop the issue from reappearing.

Support teams are no longer a cost centre.

See how AI Support Agent, Agent Assist, and VoC Analyst work on your own ticket volume.

Support teams are no longer a cost centre.

See how AI Support Agent, Agent Assist, and VoC Analyst work on your own ticket volume.

Support teams are no longer a cost centre.

See how AI Support Agent, Agent Assist, and VoC Analyst work on your own ticket volume.