e-Commerce Growth: Understanding A/B Testing

e-Commerce Growth: Understanding A/B Testing
This entry is part 1 of 5 in the series The Foundation of A/B Testing

Imagine you had a goal to ride a bicycle a hundred miles.

Of course, you’d want to make sure you knew how many miles you’d gone. You’d want to strive for that first 10, that first 20, the huge milestone of 50. You’d want to make sure you kept putting a lot of effort and energy into riding that bike so you could get to 100 eventually.

But you’d also make sure you were using the right wheels. You’d also check to see that your seat was the best kind for you, and that your pedals were the right type. You’d want a professional to look over your chain, brakes, alignment, and everything else that makes your bike run as efficiently as possible.

Because if you’re putting tons of energy into just hitting distance markers, you might be able to hit those goals faster and more easily with a tuned-up bike, right?

We should think about e-commerce the same way. The goal is always revenue, always more miles, but instead of thinking constantly about traffic and the sheer effort it takes to keep growing traffic base, what if we thought about the efficiency of the store at the same time?

Ultimately, there’s two ways to improve the revenue of your e-commerce store:

  1. More traffic to your properties
  2. More conversions with your existing traffic

We get super focused on growth and traffic, and for early stage e-commerce stores, we encourage that laser focus. Your store flat-out can’t be successful without traffic. Without traffic, improving your conversions won’t accomplish much, and you’ll lack enough data to even run tests.

But once you’re seeing a solid traffic base and you’re wondering what else you can possibly do to improve your numbers, start thinking about your website in a different way. Instead of, “How do I get more people to visit my site?” start thinking, “How do I get more people who are already visiting to take the actions I want?”

We’re talking about the art and science of conversion rate optimization (CRO), of course, and the main mechanism to conduct CRO – A/B testing, or split testing.

At its most basic, A/B testing sets up one change in one variable, and records any difference in your customer behavior to help you determine if you should make the change or not.

So you have the control, which is exactly what you’re already doing, and then the same exact page with one thing changed:

A/B Testing
A/B Testing

We go a lot more into how to set up effective A/B tests in some of our other posts, but it’s important to keep your tests scientific by only changing one element at a time.

For instance, if you’re trying to figure out if adding an interstitial helps or harms your sales page conversion rate, don’t also change the copy and button colors at the same time. You won’t be able to tell which change caused the uptick or drop in conversions.

It takes some patience, but isolate exactly what you want to improve and change one thing at a time.

Why to Divert Some Attention from Growth Toward CRO

Let’s back up for a minute before we jump further into “What is A/B testing?” and answer “Why A/B testing?”

If growth is so important, why does it make sense to stop focusing solely on growth and start paying attention to CRO?

Because when it comes to getting the best results for your effort, it’s hard to beat the incremental changes and potentially huge gains you can get through optimizing based on testing.

For example, let’s say – as the example below shows – that you have 10,000 visitors to your site during a given period. We’ll call it 10k visitors a month, for simplicity.

Shopping Conversion Funnel
Shopping Conversion Funnel

So every month, 10,000 people visit your site. Of those 10k, 60% visit your shopping area, 30% place an item in their cart and 3% make a purchase, leaving you with 54 conversions, or customers who bought.

Imagine that through some small tweaks like adjusting your sales messaging or shopping cart process, you could increase all of those numbers by 2% at each stage.

Now, 62% of visitors make it into the shopping area. 32% of them put something in their cart. 5% buy it. Instead of 54 purchases, you now have 99, or almost double!

And if you could replicate those small gains across every important area of your web property, you’ve doubled your conversions without making any effort to grow your traffic.

That’s why testing matters. You could carry on running your store as usual, working on growing your traffic, and making changes to your website based on your intuition or what you’ve seen other stores pull off successfully – but nothing will work as well as honing in on what you should be doing based on durable testing.

Difference can be visualized like this:

Conversion Rate Optimization Impact
Conversion Rate Optimization Impact

A strong e-commerce operation may still get some conversion jumps without testing, but a statistically strong and properly set up testing program makes that line follow a much more exponential curve, as successful tests build on successful tests.

The Four Building Blocks of A/B Testing

If you’re ready to add optimization to your toolkit for growing your e-commerce business, that’s excellent. Let’s talk broadly about the things you’ll need to succeed.

Testing mindset

This may feel a bit fuzzy, but it’s the most important thing to arm yourself with before you start working on your CRO. At Objeqt, clients come to us at a variety of success levels and for a variety of reasons, but the first thing we work on is testing mindset and approach to testing.

Here are the things we believe about testing:

  • Testing produces the most sustainable lifts of any tactic you could employ
  • Testing should be done consistently and continuously
  • Testing requires patience to generate the best results
  • A successful testing program should combine short-term successes and long-term goals
  • The entire organization should adopt a testing culture where data- and research-driven testing leads to actionable changes
  • Testing significantly minimizes an organization’s risk exposure
  • Conversion improvement is as or more important than traffic for revenue growth

We talk about all those principles in detail in other posts, but those ideas form the basis of our testing philosophy.

If you’re new to testing, or if you’re thinking of testing as a quick and easy “hack” aimed solely at boosting revenue, we’d encourage you to carefully consider your longer-term goals.

Testing requires patience, and your testing philosophy should be geared toward discovering the right adjustments to make to deliver a better customer experience and sustainable gains for your store – not a quick boost and then on to the next thing.

Traffic for statistically sound tests

As a wise person once said (probably): There’s no replacement for traffic. Get it or you can’t test.

It might feel arbitrary and exclusionary, but there’s a reason behind this idea. If you run split tests with too few visitors, your test results might not be statistically sound.

For instance, let’s say you’re running a test about changing your call-to-action button on a landing page. You only have 10 visitors to that page, and 4 pick red and 6 pick green.

On its surface, 60% for green seems pretty conclusive, but when it’s only a difference of two visitors, it’s hard to justify changing all your buttons to green with any confidence. You might run the same test another day and the results could be swapped because there’s just not enough traffic to test.

The Kissmetrics sample size calculator tells us that this test isn’t statistically significant:

Kissmetrics Sample Size Calculator
Kissmetrics Sample Size Calculator

There are a lot of automatic calculators out there to tell you how many conversions you need to run a statistically sound test, including Optimizely, VWO and our personal favorite from Evan Miller.

Each calculator runs a bit differently. If you have any questions about your model or what a calculator is telling you, we’ll always recommend working with an expert contractor. You get the double benefit of putting a specialist in charge of your testing, and you get to go back to what you do best – leading your e-commerce business.

Quantitative research (analytics)

Analytics needs to come before testing.

This can sometimes be a hard sell for store owners who already have enough traffic to begin a testing program – if that’s the case, most of the time the business is doing well and has had some success with traffic growth tactics.

It’s hard to say, “OK, now it’s time to properly set up a system for accurate quantitative research and analytics data gathering, and only then will it make sense to test!”

But this goes back to that crucial testing mindset – why test if you have a weak research foundation that ensures your test results will be useless? Or worse, dangerously incorrect?

We have a few posts that go into way more depth on how to properly set up an analytics program, including this one that we recommend starting with.

Qualitative research (customer research)

Analytics forms only one part of a solid research program. As we like to tell our clients, you can’t improve what you can’t measure – and data only tells you so much.

Complementing your analytics with a comprehensive customer research program yields the kinds of customer insights you’d only dream about otherwise. It’s crucial to try to understand your customer deeply and predict what will be important to them as they experience your website – before you start testing.

You have an infinite number of things you can test, but not all of them will make a difference to your customers. So you can use this deep research to hone in on the important things and only spend your time testing those.

Testing for Sustainable and Continuous Optimization

So now your bike is as tuned up as it’s ever going to be, and you’ve also put in all the effort growing your stamina and physical ability – you’re ready to cycle a hundred miles. Congratulations!

But bike technology changes. Parts wear down. Things fall out of alignment. Your needs change. If you’re ready to go your next hundred miles, you’ll need to continuously tune up your machine, right?

And e-commerce stores are the same way. Testing helps store owners establish a proven process and system for optimizing the most important aspects of the customer experience on a continuous basis.

This article is first in a five-part series, The Foundation of A/B Testing for E-Commerce Growth. To read the rest of the series, click here.

Want to learn more about how you could grow your revenue through effective testing? Let’s talk! Book a free consultation with one of our experts and we’ll identify your exact next steps.


Series NavigationYou Can’t Improve What You Can’t Measure: Why A/B Testing Starts With Analytics >>