How to Find All the Qualitative Data You Need for CRO Testing (In Unexpected Places)

This entry is part 5 of 5 in the series Customer Research As An Enhancer For Ecommerce CRO Efforts

Numbers and words — or quantitative and qualitative research, as optimizers think of them — are two of the most important parts of thorough conversion research. The results are the basis for CRO testing.

But while locating conversion improvement opportunities through quantitative data is relatively easy and straightforward (simply compare your numerical results to your expected results!), implementing qualitative feedback for better conversions (CRO testing) just isn’t as simple.

Qualitative data isn’t easily compared, or analyzed, or even visualized, which means there’s a lot of room for error. It takes some serious interpretation skills to turn wordy answers from surveys and interviews into valuable, actionable insights.

But within these wordy answers lie the reasons your customers are or aren’t converting. It’s the most valuable kind of feedback you can collect.

And there are many, many ways to gather this data, each with their own benefits.

Understand your options for gathering qualitative feedback for proper CRO testing

The goal of qualitative research is to capture the “voice of the customer,” which you can do in several ways:

  1. Surveys, polls, and queries
  2. Interviews
  3. User testing
  4. Live chat logs
  5. Support call logs
  6. User reviews of products and services
  7. All other forms of direct feedback (social media posts, blogs, forums)
CRO Testing: Seven main types of qualitative research give you a well-rounded picture of your customer’s deepest pain points, desires, and priorities.
CRO Testing: Seven main types of qualitative research give you a well-rounded picture of your customer’s deepest pain points, desires, and priorities.

All of these methods are useful, but not all are appropriate to use every time. Here’s a rough guide for what method to use when.

1. Surveys, polls, and queries

As a method for getting the most possible responses from your visitors, you can’t beat surveys or their cousins: polls and queries. These methods enable you to pose identical questions to as large an audience as you want.

Depending on which tool you use, you may be able to segment your audience and target only the people who bought something, the people who logged in a specific number of times to your website without buying, and so on.

The results you get from surveys will be as relevant and actionable as the amount of thought you put into the questions. To get answers that truly reflect your customer’s perception of the website, your questions need to allow respondents to express their opinions freely and without being led.

Surveys come in several different varieties, including exit surveys, consumer surveys, pop-up surveys, and Net Promoter Score surveys. For a detailed discussion on how to conduct surveys, check out our guide.

Once you’ve conducted your survey and collected the results, you’ll have the challenge of interpreting them. Ever try to make sense of 200 or more answers? It’s not easy!

To successfully process this amount of text data, start by isolating the specific keywords that best indicate the issue you’re trying to diagnose. Once you do that, count the number of times respondents mention that specific issue. This will give a good indication of the severity of the issue.

CRO Testing: Creating a word cloud is a popular technique for evaluating qualitative research.
CRO Testing: Creating a word cloud is a popular technique for evaluating qualitative research.

You may find that you have to create several categories for the most commonly mentioned issues. If that’s the case, order issues by severity so you can tackle them in order when it comes time to A/B-test solutions.

Now it’s time to reread your results and try to hypothesize the best ways to eliminate the issues identified. Sometimes, if you’re lucky, the customers will point you in the right direction. But don’t count on them do the problem-solving work — that’s your job. As Henry Ford supposedly said:

“If I asked people what they wanted, they’d have said faster horses.”

Whatever hypothesis you create, be sure to test it first before implementing it permanently.

Now, there are some drawbacks to conducting surveys. For starters, they require a significant amount of effort. This is no excuse not to do them, however — just be sure to plan and prepare your surveys to get valid results. Otherwise, all that effort will be for nothing.

Eliminating the issues your customers help you identify through survey responses is one surefire way to increase your conversions… as long as you interpret the answers correctly.

On-site polls

Polls serve a different function from surveys in that they help to uncover the proportion of people who have specific issues with some aspect of your website.

You’ll use a poll once you have already uncovered an issue and need to decide how to approach the solution. If your solution is difficult to implement, for example, polling your visitors may help eliminate less successful variants before you even design them, reducing the stress on your resources.

When running a poll, you should be very confident in the issue you’re addressing in order to avoid putting words in the mouths of the respondents. When running a poll that calls for open-ended answers, phrase the question in a way that leads to longer, more detailed responses.

For instance, don’t set up a poll asking, “Are you leaving without purchasing because you don’t trust our site?” Instead, phrase the question, “What kept you from purchasing today?”

Why poll rather than survey?

Poll results often lend themselves more easily to comparison and interpretation than survey results, which can include multiple answers that may all relate to each other.

Multiple-choice poll responses can be given numerical values, so you can use those to instantly analyze your data. The best way to use number-based polls is to set up questions with responses on a scale of 1 to 5 or 10, where 1 stands for “Strongly Disagree” and 5 or 10 stands for “Strongly Agree”.

2. Prospect or customer interviews

Interviews are an expanded version of surveys. During interviews, you actually talk with your prospects or customers, so you have the opportunity to discuss their responses and get more detailed answers.

Interviews are by far the best way to gain insight into your customers. They can serve many useful purposes, from identifying and exploring issues in detail to finding fodder for future marketing copy and customer success materials.

The only disadvantage to conducting interviews is the amount of time and effort they require. However, when done well, they’re worth their cost, and should be high on the list of priorities for qualitative research.

3. User testing

User testing involves recruiting a group of random users to navigate your website and identify common issues. They are usually given specific tasks to complete, and often, their session visit is recorded.

The results, from users’ verbal feedback to their session recordings, can be of great value on the heuristic side of CRO research. Users might just offer you a new view on your UI, or reveal a problem you hadn’t noticed.

Here’s our detailed guide to user testing, including where to source users and how to interpret their feedback.

4. Live chat logs

If your website maintains a live chat option (and if it doesn’t, it should — but more on that later!), then going through your live chat logs can be an enlightening experience.

Chat logs, especially support logs, can contain a treasure trove of insights. Anything from visitors’ complaints to their product information queries can indicate areas that need to be improved.

The best part is: you’ve already got these resources. Unless you don’t.

If your website does not have a live chat feature, make it the next priority for your developers. They’re relatively inexpensive and easy to implement, and can make a big difference in conversions. In fact, research from Kissmetrics and eDigital Research indicates that conversion often doubles for visitors who use live chats while shopping online.

5. Support call logs & support staff interviews

Recordings of support calls from customers can also provide valuable information that you can use to improve calls to action, fix value propositions, and identify common problems with products or services.

In fact, interviewing support personnel is nearly as useful as conducting customer interviews. Customer support workers are on the front lines with your customers, and interviews with them will frequently yield insights you can’t get any other way.

For example, in regular customer interviews, customers tend to be hesitant to express negative opinions because they don’t want to hurt your feelings. However, they may not be as concerned about the feelings of your customer support staff!

6. User reviews of products & services

This form of direct user feedback is too often ignored, even by large companies. (Only click that link if you’re ready for a horror story featuring toilet paper and terrible support.)

It’s well worth dedicating time to this readily available user feedback. Customers volunteer so much information via their reviews, and you don’t need to organize a survey or conduct interviews to get it.

Ideally, you already have user reviews enabled on your website.

A slight snag: sometimes stores allow reviews, but nobody writes them. If this is the case, you may need to incentivize customers to review you, at least at the beginning. If participation is low, carefully check out the review process and try to make it more prominent, and ask customers via email to review the product they purchased after a few days.

CRO Testing: Reviews are a source of invaluable (and often unfiltered) customer feedback.
CRO Testing: Reviews are a source of invaluable (and often unfiltered) customer feedback.

Some companies use different incentives to compel their consumers to review, such as point-gathering schemes that offer rewards or coupons. Strategies like these don’t cost much to implement and can be very effective.

As with any other form of user feedback, this one is too valuable an opportunity to miss. Gathering consumer reviews is effortless, and once the basic mechanism is set up, the only task remaining is to actually read the reviews.

The best way reviews can be used? To improve your copy. Notice which words your customers use to describe the biggest benefits you offer and the biggest pain points your product solves — then use those same words and phrases in your copy.

7. Other forms of feedback

Today, most (if not all) web users maintain some sort of profile on at least one social network. While you can’t directly track the social network profiles of your customers, it’s possible to track their mentions of your website or the products/services you offer.

Sentiment analysis tools are the quickest way to get this information, and will additionally tell you whether these mentions are generally positive, or negative. You can use this information to keep tabs on how well you’re serving your target audience, to help inform your buyer personas, and to help your writing team “learn customers’ language” for future copy.

You’ll also want to monitor mentions of your brand or service on blogs and forums — particularly professional or niche forums for products/services that do not have a wide target audience.

It’s not the data-gathering that matters for CRO testing

The most challenging part of conducting qualitative research isn’t gathering qualitative data, though that can be a labor and time-intensive process.

It’s interpreting and quantifying that data accurately. Whenever words are involved, there is room for interpretation, which can be flawed. But when we begin to gather lots of voice-of-customer feedback, patterns emerge that can point us in valuable, conversion-linked directions.

When structuring your plan to gather qualitative data, remember to use more than one method, because no single method will give you a well-rounded perspective on what your customers are really thinking, feeling, and experiencing on your site.

And keep in mind: Your goal for conversion optimization is to find out what customers think about your website — not what they think of your products or services. That’s a whole different ballgame.

Look for feedback that indicates areas of your website that cause friction, anxiety, confusion or distraction. These are the worst conversion-killers. Then use that qualitative feedback alongside your quantitative data to form a clear picture of what areas to optimize first, and which hypotheses will stand the best chance of success.

Series Navigation<< Conversion Research: 6 Treasure Troves of Qualitative Research You Can Access Right Now

Published by

Edin is a Senior CRO Consultant. Edin is into Google Analytics and testing (any A/B testing tool really) and likes to write about it. You can follow Edin on Twitter.