Building a Sustainable Testing Culture

Building a Sustainable Testing Culture

Published by

October 29, 2017

Which is more dangerous to your e-commerce store: your data or your intuition?

Answer: If you’re using one without the other, they could both be disastrous.

The case for intuition without data being more dangerous:

A few years ago, a study of 800 Fortune 1000 marketers to review their relationship with data revealed marketers used data to make customer-related decisions on 11 percent of the time. The other 89 percent, they relied on intuition.

Without data, intuition could lead you anywhere. You might blow up a highly successful onboarding process on a hunch. You might change your checkout flow to solve a cart abandonment issue, when all you needed to do was add a few confidence factors. It’s the epitome of flying blind, no matter how good you think your gut might be.

The case for over-reliance on data being more dangerous:

In the same study, marketers revealed on the opposite end of the spectrum, 11 percent of the time they “couldn’t get enough” of the massive data streams we have access to now. They used data for every decision – and made decisions based on data too often. Without a proper testing structure, iterating based on every blip or bump in your data can be just as dangerous as flying blind with only your intuition to guide you.

The truth is, the best way to make decisions for your e-commerce company is through combining data and intuition – and creating a sustainable testing culture to help grow your revenue.

More e-commerce companies should make a commitment to creating a culture of testing. We’re talking about a comprehensive, end-to-end approach to qualitative and quantitative information, functional design and effective split testing, a commitment that stretches from your C-suite to your interns. When everyone in your company approaches problems with a testing mindset, you’ll have a solid foundation for growth.

Why (a) Testing (Culture) Matters

In an excellent Harvard Business Review article (quoted by ConversionXL’s Peep Laja in his article encouraging continuous optimization), Dan Ariely notes that companies want to give weight to intuition or expert opinion because that’s what we’re conditioned to do: drive toward answers.

“When we pay consultants, we get an answer from them and not a list of experiments to conduct. We tend to value answers over questions because answers allow us to take action, while questions mean that we need to keep thinking. Never mind that asking good questions and gathering evidence usually guides us to better answers.” – Dan Ariely, HBR

When confronted with more questions than answers, or answers you can’t be sure of, testing is the risk reduction process you’re looking for.

Even if you’re not testing, you’re going to make changes to your business. That’s the nature of business, after all. You’ll change your web properties, your app, your messaging, your checkout process – you’ll change all of that at some point, because you understand that businesses can’t stagnate. Things move too fast. You need to iterate. You need to stay ahead of complacency.

But what are you changing? And why? Are you relying on intuition without data? Or data without context?

Creating a testing culture that prioritizes optimization should be a natural stage for serious companies.

Testing Culture that Prioritizes Optimization
Testing Culture that Prioritizes Optimization – source: Medium

Organizations need a testing culture and a testing mindset because a testing culture:

  • Removes the risk of changing the wrong things and moving your business in the wrong direction.
  • Ensures you always know what your customer wants through qualitative testing.
  • Gives you a way to contextualize your quantitative data to make real analysis.
  • Helps you become so practiced at the cycle of hypothesizing, testing, adjusting, analyzing and iterating that when you need to make major changes and conduct major tests, you can trust your process and your results.
  • Instills a “challenge everything” mindset in your team, which means no lazy conclusions or wild guesses make it into your business decisions.

Guiding Principles of a True Testing Culture

Effective A/B Testing Checklist
Effective A/B Testing Checklist

Before you start emphasizing optimization and creating your testing culture, make sure you fully understand these guiding principles:

1. Testing isn’t about proving yourself right

Intuition has its place, but you shouldn’t go into your tests with the intention of confirming or disproving your hunches. Keep your agenda clear. Your results and analysis can show you a concrete way forward when you have a lot of options, but be wary of influencing your analysis with preconceived notions.

2. Start at the beginning

As recommended by Optimizely, you’ll want to build testing into everything you do that could involve making a lot of assumptions, especially early on. Optimizely uses the example of your information architecture and content strategy, but this could be applied across a lot of areas in an e-commerce operation.

3. Testing belongs to everyone – encourage collaboration

By design, testing should remove the rule of what ConversionXL’s Peep Laja calls a HiPPO (highest-paid person’s opinion). A testing culture should encourage team members to work together to form tests and give input on analysis.

For instance, a test run by the marketing automation team might turn up results that your customer success team would find extremely valuable, or your sales team might have valuable input for a copywriting split test based on what they know about sales-qualified lead behavior. If your team tests in silos, none of that information would be shared.

Part of collaboration is keeping everyone informed. Transparency about who’s doing what can help avoid duplicate work or incomplete analysis. You can use something like this testing roadmap, similar to a product roadmap, to keep everyone abreast of what experiments are going on.

Collaboration - Testing Culture
Collaboration – Testing Culture – source: Online Behavior

4. Some tests should be about short-term wins

Not every test measures something earth-shattering. Part of a testing culture is infusing the testing ideology into everything you do, so that means sometimes you’ll be testing something small, like an inline CTA v. an image-based CTA. But “smaller” tests don’t need to result in small results – if an experiment over those CTAs gains you one percent more conversions, then you’ve gained a lot for little effort.

Balancing quick wins with longer-term “major” tests (like full homepage tests) also means your team gets more testing experience and can stay nimble to respond to unanticipated changes in your customer base.

5. Embrace functional design

Everything you do to any of your customer-facing properties – including your website and apps – will affect your customer’s experience. Functional design involves finding the balance between optimized design and aesthetically pleasing design. All elements should have a function and exist for a reason, and testing helps you find that reason. Combining the aesthetic sensibilities of design with the functional practicality of CRO helps you create e-commerce experiences that are both enjoyable and optimized.

6. Nothing is “done”

Testing should beget more tests. As we quoted Dan Ariely saying before, “good questions and gathering evidence guides us to better answers.” The more good questions we can pose around the “why” and “how” of our business decisions, the better our answers will become.

Maintain an attitude of cycles. It’s all too easy for to complete a big project like a website redesign, call it “done” and not revisit it again for years. If you maintain the attitude that nothing is “done” and everything needs to be tested on a schedule, you’ll avoid becoming outdated or drifting away from what your customers want.

Focusing on the Long Run

Testing takes time, but the results are worth it. Applying testing to business decisions results in incremental but sustainable gains (Forbes found as much as 241 percent ROI). “Move fast and break things” results in a string of fast moves and broken things, but not necessarily a clearer picture of where you should go and what you should be focused on.

A testing culture breeds the sort of sustainable momentum that builds lasting companies. Employees who ask “why” before “how.” Processes designed to eliminate waste and promote efficiency. A company focused on long-term goals rather than short-term puffs of smoke.

In an industry obsessed with the latest and greatest, companies dedicated to meticulously building testing into their culture can experience the kinds of results others can only wish to see. Companies built on testing culture will build what its customers want, and their customers will respond accordingly.

This article is the last article in a five-part series, The Foundation of A/B Testing for E-Commerce Growth. To read the rest of the series, click here.

Thinking about creating a testing culture within your organization, but not sure where to start? We work with high-traffic e-commerce companies to build testing systems for continuous revenue gains – schedule a free consultation with us to see if we can help.

Series Navigation<< How to Create Effective A/B Tests from Scratch

Published by