- e-Commerce Growth: Understanding A/B Testing
- You Can’t Improve What You Can’t Measure: Why A/B Testing Starts With Analytics
- Know Thy Customer: Why You Need Qualitative Research for Effective CRO
- How to Create Effective A/B Tests from Scratch
- Building a Sustainable Testing Culture
A while ago, we started working with an e-commerce client who wanted us to help optimize their checkout page for conversions.
On the surface, they had everything they thought they needed to A/B test: $5 million in annual recurring revenue stemming from high traffic, plus a solid analytics program and the ability to analyze that data.
They’d been running some tests on their own and doing a good job, carefully measuring results and making sure the conversion sample sizes were large enough to be statistically significant.
When we started working with them, though, they couldn’t solve the mystery of their checkout page.
They’d tested a lot of things – colors on the checkout page, the placement of various elements, etc., and they couldn’t figure out why their data told them people abandoned their checkout process on that page at an alarming rate.
We looked at the tests – they were solid.
We looked at the data and analytics – also solid.
Then we asked, “What do your customers say?”
They hadn’t thought to ask. So we helped them ask.
As it turned out, their trust elements were scaring people away. The company, proud of its commitment to security, had recently added, “We are now using Komodo, the best security there is,” to their “verified by Visa” and SSL certificate info – and it was scaring people away.
“Why would they need the best security? It seemed like maybe there was a reason, like something bad had happened in the past,” one customer told us. Others echoed those sentiments.
We ran the tests based on that idea, and they confirmed – the line about Komodo had to go. We ended up replacing it with just the logo mark instead.
And then we helped the company set up a robust customer research program, the qualitative complement to their already successful quantitative analytics program.
There were two lessons at play here that we see a lot in testing and conversion rate optimization:
- Challenge your assumptions through testing, and
- Put your data into human context, always.
The human brain is only knowable to a point.
We can spend forever crunching datasets and watching people interact with our pages and products, and we still might not have the faintest inkling of what’s going on in their minds as they consider whether or not to buy our product.
Data can tell you a lot, but to get context, you need to talk to people.
How to Approach Qualitative Customer Research
The category “customer research” encompasses a lot of things, but for e-commerce stores, we prefer to think of four major ways you can approach it:
Calling and interviewing customers
This is the most straightforward method and the one that’s been around the longest. Marketers and sales professionals have been utilizing customer calls for decades.
Most often, you’ll do this by recruiting people from your website or because they’re existing customers. That can be done through some kind of automated form or during the follow-up phase via email.
Many e-commerce operators will also compensate the customer for their time with a discount or some other offer. We’d recommend trying a friendly, non-incentivized approach first, but understand that a lot of times incentives can help you get more calls confirmed and spend less time trying to get people to agree.
Regardless of how you recruit or incentivize the customer, you should move the conversation to the phone. Email surveys can be a good tool (see below), but if you’re looking for a candid conversation about your product, an in-person conversation works best. You’ll be able to ask follow-up questions and hear their tone.
You can also supplement phone calls with on-page chat apps like Intercom or Drift. These won’t replace voice or face-to-face interviews, but they can lower the barrier to having a conversation with a real customer.
Some of these tactics will overlap with quantitative analytics and usability testing a bit, but tracking interactions gives you the ability to consider your customer’s actions in the context of your real page.
Rather than having to look at a datastream and figure out what actions the numbers represent, tracking the way customers interact with elements of your page shows you exactly what actions they take – and don’t take.
This can include things like heat maps, analytics, scroll mapping, referral tracking and more.
This one’s easy to lump in with interviewing, but surveying is a lower-barrier way to get customer feedback while still providing you useful information. If you’re having trouble getting enough people to agree to phone calls or other in-person interviews, surveying can fill that gap.
You should take their results separately from your interviews, however, as there won’t be as much nuance since you can’t ask follow-up questions or clarify answers on the spot.
To gather surveys, you can implement a user form on your website – you can even call it your store’s annual survey – and then ask questions about customer demographics, how they use your product, and what they want from your product and experience on your site. The goal here is to understand how your product fits into the context of their lives.
This type of customer research offers you “live” tracking – rather than relying solely on something like a heat map, which shows aggregated activity after the fact, usability testing lets you watch users interacting with your website in real time.
This comes in handy when you’re trying to understand how your customers physically interact with certain elements on the page. It won’t necessarily tell you why, but watching users can give you a lot of insight. It’s especially useful for multi-stage operations like onboarding or checkout sequences, or to see if elements of your page are as intuitive as you think they are.
(It’s worth mentioning that sites like UserTesting ask the tester to narrate their thoughts as they’re navigating your page, giving you insight into actions and thoughts.)
Best Practices for Surveying
Just get started
Sometimes the hardest thing is getting started. We get that, so our advice if you’re thinking about implementing a customer research program is to just start. Try one thing first, just one campaign.
If you have an email list (and you really, really should), then start there. Examine what areas of your e-commerce site you’d like to work on optimizing first, and create a brief survey asking about customers’ experiences in that realm. Then send it to your email list with a friendly introduction asking them to do you a favor.
You can expand to including that survey on your website, and then move toward asking questions through chat. Starting to add a customer research component to your CRO doesn’t mean you have to sign up for all the fanciest tools right away – just ask your customers about your product and their experience on your site.
Craft the right discovery questions
You wouldn’t ask a personal trainer about stock tips or a stockbroker about a workout routine – don’t ask your different audience segments the same questions.
Generally speaking, you have three main types of people you want to reach through your customer research efforts:
- Qualified nos
- Customers who bought moments ago
- Existing customers
And in general, you’d like to find out four things from each of those groups:
- Uncover where customers come from
- Discover appeals
- Understand reservations
- Understand position relative to your competitors
Depending on your goals, you could craft an endless variety of questions to get at those answers, but here are a few you can start with:
- Why did you choose us?
- What do you use us for?
- What value have you gotten out of it lately?
- What new things would you like to see?
- Are there any aspects to our products or shopping experience that you find frustrating, or which you’d be likely to change?
- What’s the one thing that nearly stopped you buying from us?
- What was your biggest fear or concern about using us?
- What was your biggest challenge, frustration or problem in finding the right product online?
- Where exactly did you first find out about us?
- What persuaded you to purchase from us?
- Please list the top three things that persuaded you to use us rather than a competitor.
- Which other options did you consider before choosing our product?
- Which of our competitors, both online and offline, did you consider before choosing our product?
- How were you recommended to us?
- Did you take a look at any of our competitors?
- On a scale from 0 to 10, how likely are you to recommend us to a friend or colleague?
Combining Quantitative and Qualitative Research to Identify Testing Opportunities
Of course, asking people what they’re thinking, even in the most targeted and unbiased of ways, doesn’t always get you information you can use.
People might lie. They might not understand a subconscious thought process. They might misremember. They might tell you what they think you want to hear.
Humans are, well, human.
So relying solely on customer research isn’t a good idea either. The best foundational base combines insights from your analytics and customer research to give you more three-dimensional insights.
We always call this the “foundation” of your testing process, because taken together, your qualitative and quantitative data can tell you what to test.
There’s an infinite number of things you can test on any page at any time. Remember our client from the beginning of the article, the ones wondering what on earth was wrong with their checkout page? They tested so many things, but only the combined analytics and customer research data pointed them toward the right thing.
If you spend the time and make the effort to build out a reliable analytics program and a customer research program, you can rest assured that the tests you develop and run will provide actionable results.
This article is third in a five-part series, The Foundation of A/B Testing for E-Commerce Growth. To read the rest of the series, click here.