You Can’t Improve What You Can’t Measure: Why A/B Testing Starts With Analytics

You can't improve what you can't measure: Why A/B testing starts with analytics

Published by

October 29, 2017

The most common question we get from e-commerce shop owners is “When can I expect to see results from our A/B testing?”

It’s a natural question. You’ve decided to invest in the more sustainable and long-term gains you can get through a conversion rate optimization (CRO) program based on A/B testing. You’re looking beyond just traffic as a revenue growth track, and you’re ready to start seeing results from testing – now.

And we understand – we’re in it for your results, too. We also want to see you get the best possible results from taking the time to set up a strong testing program and create revenue gains through deliberate experience design changes. We balance a desire for long-term revenue growth with short-term “quick wins” to build a pervasive testing culture.

Those are the exact reasons why we’re completely honest with most people and tell them: we don’t know when you’ll see results.

A/B Testing Variations
A/B Testing Variations

The exact amount of time it takes to see results depends on a lot of factors. Oftentimes, e-commerce businesses need to set up the foundations of a testing process from the ground up. It depends on what your goals are, and what “results” means to you.

Most of the time, when we first start working with a client, we don’t start with testing at all. We start with understanding your business and the first step in that process is analytics.

The Relationship Between Analytics, Testing and CRO – or “The Gym Analogy”

Our favorite super-simple definition of CRO comes from the folks at Qualaroo. CRO is (emphasis ours):

…the method of using analytics and user feedback to improve the performance of your website.

And then they simplify it even further. CRO is:

…finding why visitors aren’t converting and fixing it.

We love how both get straight to the point, but we highlighted the words “analytics” and “user feedback” in the first definition because those two things are the key to why some organizations succeed at testing and CRO – and some don’t.

CRO is about figuring out where to close those holes in your funnel and improve your visitor experience. You can figure out what you need to do through targeted A/B testing.

But if you can’t trust the numbers underpinning the whole enterprise are accurate…what’s the point? That’s where analytics come in.

A properly set up analytics program provides the accurate data you need to support a testing and optimization strategy. It forms the foundational bedrock of the whole process.

Let’s use what we call “the gym analogy.”

You may see this pop up throughout our blogs because it’s such a useful analogy for understanding the process of testing and CRO.

You hire a personal trainer because you want to be able to lift 300 pounds. On the first day, you ask, “When will I be able to lift 300 pounds?

If you asked, “When will I be able to lift 300 pounds?” your personal trainer would never say “Five months.”

Your personal trainer would probably explain that the ability to lift 300 pounds comes from a variety of factors, including how often you’re going to work out, your nutrition level, the types of exercises you’re doing, your existing strength level, your musculature and more. They would never just say, “Five months.”

As you begin your workout regime, you’ll see a lot of “small wins” on the way to 300 pounds. One day, you’ll be able to lift 100. Another day, you’ll notice you’ve lost weight and gained muscle. And eventually, through meeting a number of small goals, you’ll be able to lift 300 pounds.

In our analogy, your workout routine and all those great exercise habits represent testing. And the foundational thing you need to lift weights? Weights, and weights that you can trust actually weigh what they say. That’s the analytics part of the analogy – data you can trust to form the foundation of your system.

What Makes a “Proper” Analytics Setup

So what makes an analytics setup trustworthy and accurate? There are a few major elements:

1. The Right Tools

You’re only as good as your tools. Here at Objeqt, we’re big fans of industry standard Google Analytics, and here’s why:

  • It’s free. This is a major one. We’re respectful of our clients’ budgets and a “free, powerful enough” tool always beats a “paid, slightly-more-powerful tool” in our opinion. The fact that anyone can sign up for Google Analytics without having to worry about pricing themselves out of their own testing process gives it a major thumbs up in our book.
  • It’s an industry standard. A lot of people in CRO and e-commerce use Google Analytics. This means once you’re set up with Google Analytics, you have lots of options. There are a ton of resources, so you can work with it yourself. Or you can work with a consultant – and most professional testing and CRO consultants should know their stuff in Google Analytics.
  • It’s easy to use and scale. A lot of tools require a lot of study before you can move past the simple early metrics and really get into the meaty stuff. Not so with Google Analytics. Whether you’re looking at very standard KPIs or really diving in, the tool doesn’t get in your way or require hours of tutorials to become proficient at setting up experiments like this one:
Google Analytics Tracking Code
Google Analytics Tracking Code

For more information on how to install Google Analytics and Google Tag Manager you can read our blog posts on Google Tag Manager.

We also like CrazyEgg or its alternatives such as Mouseflow, or HotJar, the useful tools that tell you about your users’ click and scroll habits using heat maps. In essence, CrazyEgg and other tools track where your visitors click, where they start or stop scrolling, and where they linger, through heat maps like this one:

Sample Heatmap
Sample Heatmap

These types of tools provide useful information for understanding customer behavior. All the tools can show the data in multiple types of heat maps, such as:

  • Click maps, that show where your visitors click on your website
  • Hover maps, that track the movements of the mouse pointer
  • Scroll maps, that track the vertical movement of the screen as visitors read your content
  • Uniquely, MouseFlow offers attention heat map that shows the amount of time visitors spend on each section of the page

Of course, their usefulness does not stop there. Every one of these tools features session recorder, where you can watch the interaction of each individual visitor. Their creators have included a few more options, such as form and funnel tracking or extremely useful session recording. HotJar also offers possibility to create surveys and polls.

Of course, everything these tools do can also be achieved using Google Analytics event tracking in conjunction with Google Tag Manager and Page Analytics. However using heatmaps and session recordings adds an important visual dimension that may help gain insights faster.

2. The Right KPIs

The right suite of analytics tools get you started on the right track, but next you have to set up what you’re measuring with those tools.

The things you can measure are almost infinite, and which ones you select will depend on your ultimate goals, but here’s a list of some you can consider to start with:

  • Cart abandonment rate
  • Cart abandonment point (where in the process)
  • Checkout conversions
  • Traffic to sales pages
  • Unique visitors
  • Returning visitors
  • Scroll depth
  • Length of time on page

Once you’ve determined what you’re going to measure, you can set up your tools and platforms to deliver you data on a regular basis.

3. The Right Analysis

Of course, having the data isn’t the end-all, be-all either. You have to know what to do with the data and how to analyze it to see what it’s telling you.

For instance, if you’re looking at your cart abandonment data and you’re seeing a lot of dropoffs right after you ask customers to fill out a form, you could start thinking of tests designed to find out why. Is your form too long? Does it happen too early in the process? Do you require too many fields? All of this stems from being able to read your data.

The Other Side of Research

Of course, data can only tell you so much. For the best possible research-backed foundation for your testing program, you’ll need to talk to your customers.

How to ask your customers the right questions to get information to complement your data is a topic for another day, but it’s important to remember: once your analytics are set up, your job isn’t done.

This article is second in a five-part series, The Foundation of A/B Testing for E-Commerce Growth. To read the rest of the series, click here.

If you’re enjoying our posts on the philosophy and process of A/B testing, subscribe to our free A/B testing email course!

Series Navigation<< e-Commerce Growth: Understanding A/B TestingKnow Thy Customer: Why You Need Qualitative Research for Effective CRO >>

Published by