Google Analytics Customer Lifetime Value Reports

Google Analytics Customer Lifetime Value ReportsGoogle Analytics Customer Lifetime Value Reports

Published by

Edin is a Senior CRO Consultant. Edin is into Google Analytics and testing (any A/B testing tool really) and likes to write about it. You can follow Edin on Twitter.

February 7, 2018

[Psst, before you read this: Make sure your Google Analytics account is configured properly! Otherwise, you won’t be able to rely on any of your data. Grab your free Google Analytics Audit Checklist here.]

Customer lifetime value & why it’s so important

Customer lifetime value (CLV for short) is a metric that you can use to predict how much a given visitor contributes to the goal of your website. Put another way, CLV helps you know how much revenue to expect from a visitor who converts.

CLV can be calculated relatively quickly using a few easily obtainable figures. You’ll need:

  • Total revenue (or other metric that makes sense for your website) for a given period
  • The length of that period
  • The total number of customers during that period

This simple formula gives you the expected value of each customer.

Obviously, knowing your average CLV gives you a strategic advantage. For example, use your average CLV to estimate how much of your marketing budget you can spend to acquire a customer.

You can also use CLV to estimate the success of your deals and offers. And you can even use it as a metric for your A/B testing (by measuring and/or aiming to increase your average CLV).

Measure CLV with Google Analytics’ Enhanced Ecommerce features

Depending on your website’s function, you can measure different aspects of the lifetime value of your visitors. You might be interested in page views per visitor, or session duration, as success metrics.

However, most ecommerce websites are interested in growing revenue per user, since that’s their ultimate KPI.

Accurately measuring customer lifetime value in terms of revenue requires you to enable Google Analytics’ Enhanced Ecommerce features. Only then can your analytics track and record your site’s revenue and transactions made by individual customers.

Without Enhanced Ecommerce, you’ll only be able to track non-monetary values, like page views, goals, and sessions.

Google Analytics Customer Lifetime Value Reports Graphs
Graph of Customer Lifetime Value in Google Analytics

Google AnalyticsLifetime Value report consists of two parts: a chart and a sheet.

Both play their part in providing information about your visitors. We’ll go through these two components one by one.

How to read and modify the LTV chart using different metrics

This chart offers multiple ways to modify the information displayed, and analyze different aspects of your visitors’ behavior.

The “LTV metric” will be your primary way to modify the data you see on the chart. (Note that this feature is still in beta, so Google can make additions to all the metrics selectable in the drop-down menu here.)

Google Analytics Customer Lifetime Value Reports Dropdown Menu
Drop down menu selection

By clicking a plus sign on the right-hand side of the “LTV metric” drop-down menu, you can select another metric to compare it with.

LTV metric #1: Goal Completions Per User

Depending on what you want to track, a goal might be defined as viewing a particular page, or taking a certain action on the website. If you select the “goal completions per user” metric, the chart will display the number of completed goals per visitor.

Whatever you choose to track as a goal, make sure it’s a goal that works towards the objective you want, and has a positive overall impact on your website.

LTV metric #2: Pageviews Per User

The “pageviews per user” metric is another way of tracking and conceiving of lifetime value. Its usefulness depends on what your website is designed to achieve.

If you care how many pages of your website a visitor has seen, use this metric. For example, if you have a blog or similar resource, you may want to know how many pages visitors read.

LTV metric #3: Revenue Per User

All of the prior metrics mentioned will function perfectly and provide data without any additional Analytics configuration.

As mentioned before, though, the “revenue per user” metric will only provide data if you configure Ecommerce features in your Google Analytics account. These features enable Analytics to collect revenue and transaction data and report these directly to the Ecommerce Reports function.

When Ecommerce features are enabled, the Revenue Per User data will automatically be available in the Customer Value report.

The revenue metric’s main advantage is that it helps make a direct connection between other metrics and your website income. Because of this, it can serve multiple purposes — estimating the financial success and viability of your website, the effectiveness of individual traffic sources, and so on.

LTV metric #4: Session Duration Per User

Session duration per user allows you to estimate how long the average user engages with your website.

For example, much research has established that engaged visitors convert better — so if your session duration metric is rising, it means your visitors are more engaged and thus more likely to convert.

Additionally, if your website contains video or other content with a known duration, the session duration metric per visitor allows you to estimate the success of that content.

LTV metric #5: Sessions Per User

Another metric closely related to session duration is number of sessions per user.

If your website depends on multiple visits by the same user, this is the indicator you need to track. A growing number of visits per visitor indicates that your content is interesting in the long term, and that visitors frequently return to check for new content.

You can also use this indicator to estimate the purchase cycle of your customers by correlating it with other indicators available in Analytics, such as the Time to Transaction report that shows “Sessions to Transaction”.

It’s possible your customers need to do more research before purchasing from you, so they make multiple visits. Your aim is to reduce the number of sessions the average customer needs before they perform a transaction.

Google Analytics Customer Lifetime Value Reports - Sessions To Transaction Report
The “Sessions to Transaction” report

In the example above, most customers made their transaction in 1 or 2 sessions. Obviously, the ideal number of sessions per customer should be 1, or (at least) equal to the actual number of visitors. If this indicator is lower than one, so you have fewer sessions than visitors, it may mean that you have a problem with spam.

LTV metric #6: Transactions Per User

Another good indicator of the value of an individual customer is the number of transactions per visitor. Of course, this indicator will closely match the conversion rate of your website.

Use the Customer Value sheet to inform your marketing spend

At the bottom of the CV report in Google Analytics, you’ll find a sheet with multiple rows. Depending on the metric you select, these numbers will change, and you’ll see information that can lead to very interesting insights.

Google Analytics Customer Lifetime Value Reports Acquisition Channel
Sheet at the bottom of the report

This sheet contains the breakdown of all acquisition channels, and metric values for each. For example, in the above sheet, we used the “revenue per visitor” metric.

When broken down by channel, it appears that referrals brought the most revenue to the website. Visitors who reached the website by referrals spent $1.8 million on the site during the period observed. The revenue per visitor who arrived to the site via referrals was $35.

This means that referrals are the single most effective source of traffic for this website (the Google Merchandise Store, in case you’re wondering).

Of course, we could have gotten this data from other sources, such as the Multi-Channel Funnels report. The advantage of this report over that one is contained in the third column of this report.

The “Revenue Per User” column gives an indicator of the overall revenue each visitor contributes. Here, every visitor is worth $5.60.

The rows below show the revenue that visitors from each channel brought in. For referral, the most effective channel, this value is $35.01. That means you could invest as much as $35 to acquire additional visitors through referral and have a minimum positive ROI.

Using this sheet with the “Revenue Per User” metric makes it easy to adjust your marketing spend. For example, for this site, spending up to $2.50 on paid search yields a positive ROI.

But the usefulness of this report doesn’t stop there. The information in this sheet can be modified further for more valuable insights.

If you run multiple advertising campaigns using PPC, for example, you can use the dropdown menu at the top of the sheet to change the main criteria:

Google Analytics Customer Lifetime Value Reports Acquisition Campaign
Choose between acquisition channels, sources, mediums, and campaigns

Here, you can view not only acquisition channels, but individual sources and mediums, as well as your individual PPC campaigns.

Google Analytics Customer Lifetime Value Reports Acquisition Campaign View
Acquisition Campaign view

Using these criteria, you can determine which campaign is the most effective and increase your investment there, or adjust the spend to reflect the value of each new visitor acquired. (You can also see which campaigns are the least effective, and adjust your spend or kill the campaign accordingly.)

You can also check which source or medium results in the most revenue (or another indicator).

Which metric should you use?

As we’ve seen, there are many metrics in Google Analytics’ Customer Value reports, and all of them can be used to gain insight into your website’s performance.

As with every other report and metric in Google Analytics, the objective of your website will determine which metric or metrics are the most important to you. In general, though, ecommerce websites will mostly be interested in revenue and transaction metrics.

Eventually, no matter how long a visitor spends on your website, how many times they visit, or how many pages they see, what matters the most is how much money they spend on your website. Therefore, revenue per user and number of transactions per user should be the most important metrics to consider.

Just another reminder that in order to have revenue and transaction metrics at your disposal, you’ll need to make sure your Google Analytics account has Ecommerce or (better yet) Enhanced Ecommerce features enabled. Otherwise, this report will be empty, since Analytics won’t have data to calculate with.

All the other metrics, such as session duration, number of sessions per user, goals completed per user, and so on can be considered supplementary to the main objective: making more money.

Don’t forget that you should use this report to complement other available reports, like Acquisition reports, Ecommerce reports, and Attribution reports. Only by collating all of the data can you be sure that you’re not ascribing causation to correlation.

Google Analytics Customer Lifetime Value ReportsGoogle Analytics Customer Lifetime Value Reports

Customer lifetime value (CLV) is a metric that you can use to predict how much a given visitor contributes to the goal of your website. Put another way CLV helps you know how much revenue to expect from a visitor who converts. Google Analytics provides reports that can be of help and we go through them in this article.

Overcome Trust Issues In Order To Increase Conversions

Customer trust

Published by

Edin is a Senior CRO Consultant. Edin is into Google Analytics and testing (any A/B testing tool really) and likes to write about it. You can follow Edin on Twitter.

January 31, 2018

While we’ve touched on trust issues in some sections above, there is an important way to foster trust that needs more detailed explanation: leveraging your existing customers.

Customer Reviews

By getting your existing customers to provide reviews of your products and your customer service, you give prospects insight into how the product works for their peers. Reviews are the easiest way to achieve social proof — but successfully soliciting them can be tricky.

First, you need to motivate your customers to take the time to provide a review, which can be tough. You need to convince people to invest their own time in something that is beneficial mostly to you. If you want your customers to do something for you, you need to provide something in return.

You might offer customers who leave a review certain incentives (e.g. a lower price, free items, free shipping, or other direct benefits). These benefits can entice customers to provide feedback, but they produce additional costs for you.

You can also attempt to provide the perception of benefits — those that while not tangible, will still make your customers feel they’re achieving something. For example, you can give customers feedback on how helpful their review was to other shoppers, or create a ranking system that advances users as they provide more and more reviews.

One note about this: An achievement system should establish clear rules for how users can advance. A properly established achievement and review system not only provides you with proof that your products are useful to your customers, it also increases customer engagement with your brand and grows customer loyalty.

To get more reviews, don’t ask customers to go through a multi-stage process. If they click on “Yes, I want to review the product,” provide them with a single-screen form to grade your product and write as long a review as they want to. Follow our form guidelines to provide the best possible user experience here.

Increase conversions - customer reviews
An example of how Sony handles customer reviews

Ideally, each review should also contain information about the customer who wrote it. If you allow customers to provide some basic information, you can enable prospects to sort for reviewers who are similar to themselves, making the reviews they read more relevant. For example, if you sell cameras, and Joe Prospect wants to buy a camera to take family photographs on vacation, reviews by other customers with families will be more relevant to Joe.

You also need to make it easy for prospects to read the reviews, and make sure reviews are visible at the point in the purchase process where they can have the greatest impact.

Match your prospect’s frame of mind, and place product reviews at the correct position in the customer journey. Placed correctly, the right review has the power to allay doubt and tip prospects into a purchase decision.

Let’s back up for a second and review the customer journey:

  • New customers reach your website and begin searching for a reason to buy your product.
  • If they have not heard of your site or brand before, they’ll first look for information to confirm they can trust you (this is where the visual look and content of your site come in).
  • Once they ensure your business is legitimate, they’ll seek a product that can solve their problem.

In this next phase of information-gathering, product specifications are crucial — and so is including reviews of each product on its dedicated product page. One big mistake ecommerce stores often make is to lump together all of their product reviews and display them without any regard to what product the prospect is viewing.

Finally, you need to establish an internal system to react to customer feedback.

The capstone of a good review system is how you handle feedback from your customers. Customers will provide their honest views on your products, and they may have negative opinions. Handling these gracefully and respectfully goes a long way toward creating a positive brand reputation.

Social networking provides another venue for customer feedback. By establishing your business pages on the usual social networks, such as Facebook, Instagram, Snapchat, and others, you can take advantage of a ready-made, familiar mechanism for sharing feedback.

Plus, when customers share information about your products publicly on social networks, all their friends and connections on that network can see. You can benefit from this increased visibility and gain new prospects when customers’ friends see their reviews.

Create more repeat customers to grow revenue

Increasing conversion rate is only a narrow definition of the true aim of the optimization process. Through optimizing conversions, the real goal is to increase a site’s revenue.

One of the best ways to do this is to increase the amount of money the average customer spends on your website. This indicator is called “customer lifetime value,” and by growing it, you can virtually automatically increase revenue without changing anything else.

Research shows that return customers are likely to spend more and convert much more easily than new customers. Therefore, a simpler way to grow revenue is to try to get as many customers as possible to return and buy more products from your site.

Is creating repeat customers easy? Not always. Let’s look at a few ways to entice customers to return to your store and grow their customer lifetime value.

LCV generator #1: Loyalty programs

Large brick-and-mortar shopping chains have long used a method called “loyalty programs” to increase the likelihood of a repeat purchase. You’ve undoubtedly seen this in the form of various point accumulations, tokens, loyalty cards, punch cards, special rewards, and so on. The idea is to get the customer invested in the idea of receiving rewards for purchases.

Some rewards take the form of tokens or points that can be exchanged for a lower price on selected items. The more products the customer buys, the better off they are.

Increase conversions - referral programs
An example of a referral program from Who’s Hoo

Loyalty programs work, but there’s an even better way to secure customer loyalty and engagement.

Use the advantages of running an online store and social channels to create a group of like-minded people: a community.

Community creation is another step you can take toward engendering customer loyalty and increasing engagement. Communities work best for niche products, but you can create them for any type of store. The point is to create an environment where your target audience will gather, exchange information, and promote your products to new customers looking to solve their problems.

Your store community must be lively and engaging to succeed. It’s not something you can create overnight, and it requires a lot of effort to maintain. But the beneficial results of a successful community will far outweigh the effort you put into building it.

Not only does community increase engagement, it also provides you with a free source of qualitative data. Community discussions will help you anticipate the needs of your customers in advance and create more accurate buyer personas — not to mention it’s nice to have a marketing channel right there when you need it.

Increase conversions - community
Urban Beardsman created a community to engage its customers

The great advantage of a community lies in its ability to connect you to your target audience. And a vibrant community has a good chance of attracting new prospects, as well as growing traffic to your site.

Grow revenue through upsells and cross-sells

Let’s say a prospect comes to your store looking to buy a laptop.

She looks at various options, and finally finds a device she likes enough to buy.

She goes through the checkout process with no problem, and successfully buys her computer.

If the interaction goes this way, your store just missed a low-hanging opportunity to increase revenue.

Many purchases — especially more complex or higher-ticket items like electronics — offer clear opportunities to persuade customers to buy additional items or upgrade to different options.

For example, as this customer decides on the laptop she wants to buy, you can offer upgrades to it: e.g. more memory, a larger hard drive, or a better graphics card. This technique is generally known as upselling.

Increase conversions - upselling
HP offers custom laptop configuration

Upselling can take place during the process of purchase, since upgrading a product doesn’t lead to distractions for the prospect.

Here are the steps to optimize the upsell process:

  1. Provide an easy, clear, and distraction-free way to upgrade the product as the prospect starts their journey through the funnel.
  2. Clearly state the benefits the prospect will get from the improved product.
  3. Using personalization, you can offer prospects different types of upsells. For example, if we know that our laptop buyer is a graphic designer, we might focus on upselling her a faster CPU that will make her work easier.
  4. Check your analytics to find out which step in the process carries the best chance of an effective upsell.

Cross-selling is another technique to make customers spend more. Unlike upselling, which upgrades or improves the product in which the prospect is already interested, cross-selling attempts to sell products that complement that product.

For example, a cross-sell for our laptop example might be a backpack for carrying the laptop, or a mouse or printer. Another example of an upsell would be offering prospects gloves if they bought a winter cap or scarf.

Increase conversions - upsells/reduced price
Lenovo offers upsells at reduced costs

Because cross-selling adds a completely separate product into the process, it can cause distraction from the main goal if introduced too early in the funnel. Considering a new product can drive a prospect away from a purchase they were already almost done with.

That’s why cross-selling should generally take place at the end of a transaction. Once the customer completes the transaction and pays, then offer other products. The risk of distracting your customers is thus reduced.

It’s also worth noting that you can provide prospects with cross-selling opportunities in the form of “Related Products” before they even begin a transaction. Think of this like a navigational aid that offers your prospects an alternative to the product they’re viewing.

Here are a few steps you should follow to optimize your cross-sells:

  • Analyze the conversion process carefully to find out where cross-selling can be most effective, and make sure it doesn’t distract your customers.
  • Take advantage of personalization here, if possible, to make recommendations based on what you know about customers. That way you can make relevant cross-selling offers, increasing the chances prospects will go for them.
  • If you do offer a cross-selling option during the checkout process, provide as much information as possible along with an “Add to Cart” button, so the prospect doesn’t need to navigate away from the conversion process.

Improve your marketing efforts

Traditional methods of growing revenue and turnover also apply to ecommerce stores. Your acquisition strategy should focus on creating strong online positioning and differentiating your store. Your store should be immediately recognizable and distinct from your competitors.

Design can make a difference. Your website should look professional and be simple to use, while your copy should be engaging (even fun). A customer-centric orientation and excellent customer service will increase the probability of word-of-mouth marketing. You can also use PR strategies to generate interest and awareness of your store.

Put your customers to work for you

Don’t forget that happy and enthusiastic customers will do some of your marketing for you. In fact, they can become your greatest single marketing asset. For example, Apple has an enthusiastic following that’s willing to accept mistakes (and sometimes even goes to the extent of rationalizing them). This is a result of Apple’s great customer experience and commitment to customer satisfaction.

Create strong online positioning and branding to differentiate your store from competitors, generate PR, increase word-of-mouth, and grow market share.

As a second step and an upgrade to any existing loyalty programs, you can start a referral program. Reward your existing customers with additional benefits if they bring in a friend who purchases a product. By using your community or existing social networks, you can easily enable existing customers to share products with their friends, and you can track referrals using dedicated links, coupons, or similar tracking methods.

This is another way to obtain social proof, and it will carry more weight, since the recommendation will come from a friend or relative, and not an anonymous person.

Implement refer-a-friend programs that generate new customers — and customer testimonials — through positive word-of-mouth.

Nurture visitors with educational content

Not all of the visitors who reach your website will arrive with the intention to buy. Some of them may be merely researching products or seeking information before making a purchase in the future. Their interest at this stage can be nurtured, which is the actual name for this concept: “lead nurturing”.

For example, if you have a website selling TVs and a visitor reaches your website with the goal to research TVs, but without the intent to purchase one immediately… what should you do?

You could create an email series that provides relevant information about TVs over a certain period of time, building up the visitor’s motivation, gaining their trust, and providing them with firsthand information about types of TVs, ideal sizes, accessories, and other important information that can make customers more interested in your offer.

As the Harvard Business Review puts it:

“Remarkably few suppliers in business markets are able to answer those questions. And yet the ability to pinpoint the value of a product or service for one’s customer has never been more important. Customers — especially those whose costs are driven by what they purchase — increasingly look to purchasing as a way to increase profits and therefore pressure suppliers to reduce prices. To persuade customers to focus on total costs rather than simply on acquisition price, a supplier must have an accurate understanding of what its customers value, and would value.”

You need to provide your customers with enough information about your product(s), and this info needs to be relevant to both the product and to your customer’s stage in the purchase process.

Going back to the TV set example, your emails might explain what screen size best fits different rooms, what “screen resolution” means, what type of picture technology is better, etc. Providing this information will help a visitor both learn from you and push them closer to a purchase.

At every point in your customer’s experience and interaction with the website, you should be providing valuable information. Using email allows you to present this info directly to the consumer, without depending on the chance that they’ll find it on their own.

When you send educational content, make sure you’re targeting only customers who expressed actual interest in your offer by signing up for your email list. If you decide to email addresses of previous customers or registered users, make sure the information you provide is actually relevant to avoid annoying them.

Thanks to the advent of social networks, your website and email list aren’t the sole methods available for lead nurturing. You can further engage and nurture leads and prospects through social media. By tracking what your prospects and leads are doing on your website and providing relevant information through social networks, you can increase the likelihood that they will eventually purchase your product.

Ask yourself: what did they already download, what pages do they visit the most, what are they talking about on social media sites, and have you identified any of their pain points? Compare this information to past prospects to develop hypotheses on what they need to do or know next if they were moving towards a purchase.

You’ll also want to include offers in your lead nurturing emails. When you do, you provide prospects the opportunity to reconvert and tell you more about themselves.” – Impact

Profit from lead-generation and email-nurturing programs for visitors who are “still researching”.

Leverage the power of experimentation

Online stores have an advantage over brick-and-mortar stores in that it’s easy to conduct experiments to check the effectiveness of every aspect of your business process. A/B testing provides a way to either refute or prove the changes you make to your website in terms of marketing and sales.

How to start targeting and experimenting with your marketing

Your marketing campaigns should never be a “shot in the dark”. Put another way, you should always make sure that your marketing is directed towards your ideal customers.

By using analytics and qualitative research (including insights from any communities you build), you can get a pretty good idea of who your ideal customers are. Then, you can design a marketing campaign that will target exactly those customers.

And timing matters. When you catch your prospects at the proper moment in their quest to fix a problem, you increase the likelihood they will buy from you.

The same is true for serving clear and relevant content to prospects: the likelihood of getting conversions increases. In fact, studies have shown that relevant and clear ads that lead to a landing page aligned with the prospect’s expectations can increase conversions up to 50%.

By using A/B testing, you can significantly reduce the risk inherent to new marketing campaigns. When you know what marketing message will best resonate with different segments of your prospects, creating excellent marketing campaigns is easier, and the campaigns themselves are more successful.

Experimenting with content allows you to further iterate your marketing until you bring it to peak efficiency. This will also lower marketing costs, increase the ROI of every marketing activity, and allow you to outpace your competitors — all of which are good news for revenue.

Tie all your sales channels together for clear oversight

The great thing about online tracking and campaigns is that they can readily be adopted offline. When you discover what works well online, you can use the same concept for your offline marketing campaign. Using certain tools, you can now even track the efficiency of your offline campaigns.

Of course, if your business also operates brick-and-mortar stores, your offline stores can benefit greatly from any improvement in online stores. Knowing your customers online means you will know more about your offline customers too. Using the insights derived from detailed observation of online customers, you can improve the marketing and sales process offline and grow the brick-and-mortar aspect of your business.

  • Reduce advertising risk and marketing risk. Before a new campaign goes live, find out if it will increase — or reduce — sales.
  • Improve the return on investment (ROI) from existing advertising and marketing budgets.
  • Speed up time-to-market. Reduce the time it takes to plan, test, get stakeholder approval, and then launch new marketing initiatives and messaging.
  • Take advantage of multi-channel tracking, attribution and experimentation. Transfer winning online campaigns offline.

Take the optimizer’s approach to growing your product line

Increasing sales requires a systematic approach to every facet of your sales and marketing process. Conversion optimization, customer research, and testing are easily applicable to introduction of new products.

Using CRO methodology, you can improve products’ placement on the page, the copy and visual content you use to present them, and the marketing campaigns you use to acquire new customers. Together, these approaches can do a lot to reduce visitors’ perceived risk and increase your overall sales.

Another way to grow your business is by introducing new products to your store.

When you introduce a new product to your store, you first need to develop a unique selling proposition. It needs to be, as we have seen, clear, relevant, and unique to that product. Differentiate it from other products on your website, and use headlines and descriptive copy to point out its benefits. You’ll also need to add a new product page, whose design should be consistent with your existing product pages.

To promote a new product, use your existing resources. Try to identify previous customers who might be interested in this product, based on their previous purchases or activities on your site. This way, you increase the likelihood that a new product will find buyers from your existing customer base.

Testing and experimentation can also help you establish the best possible price for a given product. Test different price ranges and formats, and you’ll discover the price that drives the largest amount of conversions.

Keep in mind that the company that started the ecommerce revolution is still the leading online business: Amazon. It’s built on experimentation, and it remains one of the foremost ecommerce pioneers thanks to constant testing.

  • Test new products and services — and marketing campaigns — by leveraging analytics, research, and experimentation.
  • Test and optimize pricing models. Promote the most popular and profitable products across channels and devices with minimal drop-out.
  • Identify opportunities for growth that need no additional budget.
  • Create a culture of customer-centric decision-making. Know when and how to A/B-test marketing propositions.
Customer trust

While we’ve touched on trust issues in some sections above, there is an important way to foster trust that needs more detailed explanation: leveraging your existing customers. How to Increase trust in order to increase conversions?

Reduce shopping cart abandonment in order to gain more sales

Reduce shopping cart abandonment

Published by

Edin is a Senior CRO Consultant. Edin is into Google Analytics and testing (any A/B testing tool really) and likes to write about it. You can follow Edin on Twitter.

January 23, 2018

When we talk about addressing how to reduce shopping cart abandonment, we’re referring to measurably decreasing the number of visitors who abandon their carts during the checkout process.

As we’ve seen, cart abandonment is a common problem for ecommerce websites. According to the Baymard Institute, 68% of prospects add a product to the cart and leave the purchase process before it is complete. There are seven main reasons for this, according to the research:

  1. Distractions
  2. Poor usability
  3. Suspicion of fraud
  4. Insufficient information
  5. Don’t want to buy
  6. Can’t buy
  7. Additional cost

Let’s go through how to overcome these obstacles one by one.

1. Distractions

Often, customers do not complete their purchase due to distractions. Distractions are generally divided into two categories: internal and external. External distractions represent events that draw the prospect’s attention away from the purchase process — for example, they receive a phone call, or someone knocks at the door, or their kids call for them. Obviously, external distractions are by definition outside the influence of either the prospect or the website.

Prospects who abruptly stop their navigation and leave without taking any other action on the website were probably distracted by external events. You can mitigate the risk of external distractions by shortening the purchase process. The best way to deal with external distractions is to try to re-engage prospects who abandon the purchase process.

The second type of distractions are internal distractions. Internal distractions happen because something on your website has drawn the prospect’s attention away from the purchase process. They may have clicked on a cross-sell, or navigational options drove them away to another area of the website.

Reduce shopping cart abandonment - Crocs
Crocs has a link to its charity – Crocs cares leading prospects away from checkout process

You can lower the risk of internal distractions by analyzing your checkout process and eliminating all the navigational options that may distract a prospect from completing the purchase. For example, show cross-sells and/or upsells only when the prospect completes the purchase.

2. Poor usability

Obviously, poor usability is an issue that should be addressed early in the conversion optimization process. If you have remaining usability issues, these should be relatively easy to detect using analytics, like Google Analytics’ funnel visualization report.

Reduce cart abandonment - Google Analytics funnel
An example of the funnel visualization report in Google Analytics

You can detect usability issues in steps of the funnel where a significant number of prospects abruptly abandon the purchase process (unlike external distractions, which account for a small number of prospects).

But the best way to uncover usability issues is to conduct user testing and observe real people trying to complete the purchase process. The results of this testing might surprise you, as you realize that a step you consider obvious is actually confounding your prospects.

In addition to user testing, you can use session recording tools to uncover elements of the purchase process that prospects are struggling with.

3. Suspicion of fraud

If your prospects don’t trust your website or your offer, there is little you can achieve by improving usability or trying to engage prospects through clever content. Increasing the trust and credibility of your website is a job for every element of your website, from design to copy and social proof. Every little thing can help establish credibility.

Conversely, many little things can destroy your credibility. If your website has cheap-looking or stock photographs, or outdated or unprofessional design, your credibility will suffer. Make sure you follow best practices and established convention — research has shown that people tend to trust prototypical websites more.

For a growing ecommerce website, establishing credibility and dispelling fear of fraud is a critical requirement to continued growth. Some trust issues can be alleviated by using simple methods like guest account creation, social media login options, and third-party payment.

Third-party payment means using a payment gateway like PayPal to allow a customer to pay you without leaving any of their sensitive billing information with your store. If you suspect that there are trust issues involved with completing a purchase, especially at the ultimate step (billing/payment), you should try offering third-party payment options instead of requiring customers to provide you with their credit card numbers. You can also leverage trust indicators like security certificates to increase the perception of safety.

Reduce cart abandonment - Security indicators
An example how an ecommerce store leverages security indicators

To further increase trust, you should make sure your customers are happy with your products. Satisfied customers will be willing to provide ample social proof in the form of stars, reviews, and testimonials, so prospects can see that other people have actually bought the products.

4. Insufficient information

You must provide every possible piece of information about your product. Lack of product information can preclude a prospect from making an informed decision, consequently hindering conversion. Bear in mind that your prospects can’t touch or try on or feel the product you sell, so you need to give them the next best thing.

It’s unlikely that online information can alleviate every possible issue your prospects might have. If you sell clothes, for example, no amount of information can prove that the items you sell will fit your customer. For situations like these, providing a tryout period after which customers can return or replace the item they bought free of charge can make a big difference to sales. (Provided, of course, your bottom line can take the stress without turning red.)

5. Don’t want to buy

One of the most confounding issues crops up when prospects add products to the cart without any intention of buying. Those visitors may just want to compare your prices with other online retailers’ prices, or they’re just not ready to buy yet. It’s the online equivalent of window shopping.

While not much can be done to change unwillingness to buy, you can still try to engage those visitors later on with deals, free shipping, or other benefits that entice them to reconsider. Maybe they didn’t have enough cash at the time they added a product to the cart, but with a nudge, they might buy it at a later date. You can also check the prices of similar products in other ecommerce stores and try to match the lowest price.

Finally, you can use surveys to ask prospects why they abandoned their carts. Sometimes you’ll get an answer that will help you both solve the issue and make the prospect reconsider.

6. Can’t buy

Unwillingness to buy is a problem that’s potentially addressable by guessing. However, inability to buy is often related to real and solvable issues. One of the primary issues detected by the Baymard Institute study was technical issues, such as server errors. Errors resulted in abandoned carts in 90% of all surveyed subjects.

Sometimes, a prospect will not be able to buy because you don’t accept their preferred method of payment. Make sure you cover all major credit cards and offer third-party payment so your customers can actually pay. In addition, provide your shipping policies up front, and if shipping is not available to a customer’s location, notify them immediately.

Reduce cart abandonment - Amazon
Amazon reveals immediately that this product cannot be shipped to the customer’s location, helping avoid cart abandonment

7. Additional cost

People tend to have a low tolerance for unexpected and unannounced costs, so introducing these late in the conversion process will almost certainly result in losing the sale.

Luckily, this issue is very easy to solve by providing all of the price information up front. The best moment to give a total estimated cost of purchase is right before the customer clicks “proceed to checkout”.

Enable prospects to calculate the shipping cost to their location even prior to adding items to their cart. If you offer free shipping to some areas, but not others, don’t try to conceal this information from prospects since they might be led to believe they’re eligible for free shipping.

And if your product requires any additional items to function (like batteries, a cable, or an adapter), state this on the product page itself.

Why should you address these 7 issues?

You’ll reap numerous benefits from addressing the issues mentioned above. Here are three main ones:

  • By reducing the number of abandoned carts, you lower the load on your analytics. Your analytics software will be hamstrung when it comes to differentiating between different types of dropouts from your conversion funnel. Therefore, eliminating addressable issues can make it easier to analyze the funnel and figure out which issues need more effort to solve.
  • Solving technical issues with checkout can have a measurable influence on your costs. Any technical issue, especially if it persists, can trigger calls to your support center. Detecting technical issues as soon as they crop up and solving them promptly prevents the issue from developing into a serious drain on your customer support services. Serious technical issues that keep customers from buying thus represent not only opportunity cost and lost revenue, but also a lost chance to provide technical advice to prospects who need it.
  • You mitigate costs associated with engaging lost prospects. Lowering your cart abandonment ratio saves you the cost of trying to engage those “lost” prospects via email or retargeting ads. These types of enticements are necessary and potent tools to engage visitors, but their use should be limited to prospects who can potentially become loyal customers in the long run.

Speaking of which…

Save money by reducing unnecessary expenditures

Analyzing existing technical issues, missing information, and other potential issues that prospects (or customers) complain about allows you to identify and solve these issues before they become critical. This not only has a positive effect on conversions — it improves your brand image, plus helps you reduce the cost of customer support and recoup the opportunity cost of a missed sale.

In the last section, we covered technical problems that affect the conversion funnel. Now, we’ll analyze other potential technical issues that may have an adverse effect on customer satisfaction.

The most obvious? Issues like incorrectly displayed pages, bad coding, etc., that physically prevent the customer from seeing or purchasing products. You’ll discover these issues with a thoroughly conducted technical audit.

To quickly and efficiently solve customer calls and complaints, you can use an automated web-based platform (such as LiveAgent, or NABD). You’ll frequently see these platforms on websites that sell software or computer equipment. They start by eliminating the most obvious causes of the problem and driving the customer through the steps to fix it on their own. The advantage of this approach is that the support staff doesn’t have to worry about simple problems, so they’re free to solve bigger issues.

Reduce cart abandonment - HP
An example of how Hewlett-Packard handles customer support for simple issues

Automated support also offers benefits in terms of customer experience. First, with this type of support, customers can solve their problems without waiting for a live support representative. Second, customers experiencing more serious issues get served faster by technical staff. Both groups of customers enjoy a better user experience, which helps build a stronger brand image and increase loyalty.

For frequently encountered issues, you can set up a web-based application and walk users through the issue at hand.

Reduce shopping cart abandonment

Cart abandonment is a common problem for ecommerce websites. When we talk about addressing how to reduce shopping cart abandonment, we’re referring to measurably reducing the number of visitors who abandon their carts during the checkout process.

Your Ecommerce Store Copy Can Make or Break Your Sales (and Your SEO)

Ecommerce store copy

Published by

Edin is a Senior CRO Consultant. Edin is into Google Analytics and testing (any A/B testing tool really) and likes to write about it. You can follow Edin on Twitter.

January 16, 2018

To continue with the approach from our last article, you should be able to create a ecommerce user experience that delights your audience. But for all the importance of a good user interface, solid web design, and attention to customer relationships, none of it is complete without copy (AKA words) to back it up.

Using copy, you can point out the benefits of your products that cannot be discerned from product photos. Text is a natural and convenient way to transfer ideas, values, and facts — but to be effective, it needs to be clear and relevant.

Ecomerce store copy - Patagonia
Ecomerce store copy – Patagonia

Providing relevant copy and content, packed with information and an engaging story or narrative, is a must. Your copy should provide prospects with convincing reasons to spend their money. To be effectively persuasive (and thus convert more customers), you need to point out the ways your product benefits that prospect — by alleviating their pain, improving their life, or benefiting them in some other way.

Copy’s primary function is to establish trust and convince prospects to purchase products. Its secondary function is to provide content for search engines to hone in on. This is important to attract customers who research your products online and direct them right to your product pages.

Many ecommerce sites make the mistake of omitting product page copy entirely, or simply duplicating the product manufacturer’s copy. This is a mistake for two reasons:

  1. The product manufacturer’s copy is likely very different from the rest of your website copy, which creates consistency issues. Your prospects will be the first to pick up on the disconnect, and it will negatively affect their trust.

Think of it like this: If you go to a store to buy a household appliance, and the owner is only able to tell you information already written on the device’s box and nothing more, you’re likely to shop elsewhere for that appliance.

  1. Duplicating manufacturer copy on your product pages will also negatively affect your SEO. Why? Search engines notice duplicate content, and lower your page ranking.
Ecommerce store copy - Samys Camera
eCommerce store using product specifications from manufacturer page

To avoid both of these problems, use original, clear, and simple copy to explain the benefits of the product to your prospects.

Copy also needs to provide information that is relevant to the customer. To make your copy more relevant, you have to get to know your customers and target audience. The better you know them, the more relevant copy you can write.

Personalization is a key feature of most successful ecommerce sites that can help you create more relevant copy and offers. Take Amazon, for example. Amazon offers a deeply personalized experience by constantly collecting data about their customers (they facilitate the collection process by dangling attractive offers if you register for an account). Once you register an account with Amazon, every interaction with the site is noted and taken into account to provide more relevant offers.

Amazon tracks what customers buy, what products they view, what they write in their reviews, and what they put in their wishlists. All this data combined enables Amazon to surprise customers with personalized offers every time they log in.

Now, you’re probably not Amazon. But if your site offers a personalized experience, you must be able to distinguish between different groups of customers (or even individual customers) and offer pinpointed content.

Ecommerce store copy - Bombfell
Bombfell uses the data from each of their customers to provide a personalized customer experience

In a basic sense, this could mean distinguishing between male and female prospects or between different age groups. Or, if your site personalization is more advanced, then you can make your content relevant to narrower groups.

The other very important concept related to copy is clarity. Copy must be easy to read, understandable, written in plain and simple language. Complicated copy, or copy written with rare or unfamiliar words, generates friction.

If your prospect needs to spend time to understand your copy, they’ll seek clearer information on other sites. In effect, poor copy can actually drive prospects to your competition.

Finally, for product copy to be effective, it needs to be scannable. Research shows that 79% of people scan text, reading only the most important parts. Help readers by turning your most important points into bullets or headings, so they stand out from surrounding copy.

People tend to buy from the store that offers the best and most relevant information about the product. By taking care to write clear, original, relevant copy, your store can win this competition.

Message-match your advertisement copy

While product page copy is important for SEO and engaging prospects, your advertisement copy needs to attract attention to your product or landing pages in the first place.

Every advertisement should be part of a cohesive campaign that includes the ad (or ads) itself, and a landing or product page that shares specific benefits of buying the product in question.

Both the advertisement and landing page need to share key copy elements: namely, the headline and the actual substance of the offer. Those two elements must be in sync to avoid confusing or frustrating your prospect. This concept is known as message matching.

For example, if you offer a 10% discount in your advertisement, don’t change that to a “$5 off” message on the landing page, as that will confuse your prospect and likely result in a higher bounce rate.

Ecommerce store copy - An ad for a white cocktail dress
An ad for a white cocktail dress
Ecommerce store copy - resulting landing page
The resulting landing page

Advertisement copy needs to be relevant, too — but in a slightly different way. In this sense, relevance means that your advertisement and landing page must respond to the way your prospect navigates to the site. Prospects using mobile devices will benefit from simpler, faster-loading, and screen-fit content, while prospects on laptops or desktops can be shown a full-screen version of your store design.

Don’t overlook email copy — it’s your secret weapon

Ecommerce websites use email for a variety of purposes: sharing deals, notifying people of the availability of items they expressed interest in, inviting them to review products they purchased, thanking them for a purchase, and inviting them to shop again, to name a few of the most common ecommerce email functions.

As with every piece of copy and content on your website, you should leave nothing to chance, and deliberately frame these emails so they’re consistent with the overall tone and style of your entire website.

When you send an email to a lead who gave you their email address, strive for relevance. Use your analytical tools to find out what the user viewed on your website, and match their interests with specific offers.

Your thank-you emails are great opportunities to re-engage customers and trigger repeat purchases. Offer cross-sell or upsell items, or invite them to post a review of the product they bought. You can also survey customers to see what they did not like about your store or purchase process.

Finally, you can contact prospects with registered accounts who abandoned their shopping cart and invite them to continue their purchase, or ask what stopped them from completing it. These emails offer a perfect opportunity to reduce your shopping cart abandonment rate (read more on that in the next section) and engage potential customers who might just have forgotten about the items they put into the cart.

Often, prospects on mobile get interrupted during the purchase process by a notification. Alternatively, they sometimes just need a sufficient nudge to complete the purchase after encountering an issue (ex. the shipping cost was too high, the price of product was too high, or some other issue prevented them from buying).

Cart abandonment emails can help you identify the problem and try to address it either generally, if it’s something likely to affect a large number of prospects, or specifically, by offering a solution to a problem perceived by only a few prospects. Either way, you’ll be showing that your store cares — making customers feel appreciated, and evoking their loyalty and enthusiasm for your brand.

Of course, you need to be careful not to be intrusive or annoying. Avoid generic offers, hyperbolic language, and cliches in all of your emails. In fact, avoid these mistakes in all copy you write for your store!

For great cart abandonment email examples, we like the ones the ones Shopify picked here.

These are all methods you can use to improve your website and its content and grow your ecommerce store using common sense, best practices and customer research.

In our next article, we’ll look at how to use real measurements to impact the way your prospects behave. By using quantitative indicators, you can understand patterns and trends in your visitor behavior, and leverage this data to increase your revenues.

Ecommerce store copy

An amazing ecommerce user experience is critical but for all the importance of a good user interface, solid web design, and attention to customer relationships, none of it is complete without copy (AKA words) to back it up. Using copy, you can point out the benefits of your products that cannot be discerned from product photos. Text is a natural and convenient way to transfer ideas, values, and facts — but to be effective, it needs to be clear and relevant.

Great Ecommerce User Experience For Increased Conversions

Ecommerce User Experience

Published by

Edin is a Senior CRO Consultant. Edin is into Google Analytics and testing (any A/B testing tool really) and likes to write about it. You can follow Edin on Twitter.

January 9, 2018

Great ecommerce user experience will increase conversions! So what is UX? According to the Nielsen Norman Group:

”User experience’ encompasses all aspects of the end-user’s interaction with the company, its services, and its products.”

One of the first steps in conversion optimization is to identify any issues that force prospects to struggle to purchase something from your website or a mobile app.

So it stands to reason that providing a high-quality ecommerce user experience (UX) means employing different methods of engineering, design, advertising, and interface design. You need to ensure visitors face as few obstacles as possible on their journey toward their, and your, ultimate goal: purchase.

To ease friction and improve customer experience, your user interface must be simple and functional. It must allow intuitive navigation, cut down on visual distraction, and omit or hide unnecessary informational or visual content.

First, map your customer journey

Mapping the customer journey is a precondition for creating an engaging customer experience. A customer journey map is an analysis of the entire customer experience that aims to identify and eliminate bottlenecks.

What do we mean by bottlenecks? Every website has multiple customer touchpoints. Creating a map of the customer journey begins by identifying the most common funnel entry point. Then, you’ll create an actual road map of the way the average customer navigates your site.

The point of mapping the customer journey is not to create a “railroading” experience for your customers that limits their freedom and control. Rather, the point is to help them achieve their ideal outcome. To do this, you need to focus on helping your customer succeed in their goals (versus helping them achieve your store’s goals).

e-Commerce Customer Journey Map
Map the entire customer journey, identifying and prioritizing which touchpoints represent bottlenecks that need improvement.

Load speed: A little-considered factor in ecommerce user experience

One of the first things most users will notice is how fast your website loads. It’s a fact that faster websites convert better, so it is obvious that your website should load as fast as possible.

Site speed is influenced by many technical factors, but also by design choices.

For example: using Flash, large high-resolution images, or background video is a sure way to slow down your page load speed. While these design elements can make your website look pretty, they adversely affect load speed. (They have a few other negative consequences, too — such as taking control of the interaction away from the visitor, and adding unnecessary steps to interactions.)

Another frequent load speed issue is the tendency to use container files like .PDFs to show content. PDFs are great tools to present long-form content like books, guides, or instruction manuals. But other types of content are best presented using regular HTML code, so the visitors can view your information or products without interruptions, the need to download a file or plugin, or similar disruptive elements.

Simple ways to improve your visual design

The second element that influences user experience on any website or app is visual design. You can leverage your design to make your user experience simpler, to make information more accessible and easy to grasp, and to clearly, intuitively communicate the customer journey. Some important elements of visual design are:


Typography, or choice of typeface, can aid in conveying meaning, increasing the legibility of your copy, and establishing your brand tone and personality. Additionally, typography allow you to create a visual hierarchy by using simple variations of the same typeface (consider using different header sizes to communicate importance; bolding; enlarging; or otherwise varying the look of the copy.)


The use of color to elicit emotion, attract attention, convey meaning, and encourage exploration is a well-known concept in marketing and advertising. You should carefully consider and pick out a color scheme that conveys the desired mood for your brand. Depending on what you sell, you can use colors to project or provoke emotions that correspond to your offer.

While color is important, the interaction of colors and surrounding content can make or break user experience. Use contrasting elements on the page to make important information stand out, or to make your call-to-action more prominent. Used effectively, visual contrast makes it easier for your customers to accomplish their goals.

Page composition

Composition is all about the way design elements interact with each other. Effective composition eases the customer’s cognitive load (AKA the amount of mental processing they need to do to understand the page), controls the rhythm of interaction, and provides structure and flow to the purchasing experience.

The position of elements on the page should make it easy for your customer to understand the import and significance of different types of content. This is part of establishing a visual hierarchy.


Whitespace, or negative space, is a fundamental design concept. Allowing adequate space calls attention to crucial elements like calls to action, important information, or product images — so prospects will dedicate more of their attention to these elements. And, because it allows the brain to process information, whitespace can provides more clarity and order to your user experience.


In designing your website or app, you should follow a consistent visual scheme to provide the same experience all over. Different designs or experiences within the same site or app will confuse prospects and create an interruption in their experience. This may lead prospects to lose sight of their goals and ultimately abandon the purchase process altogether.

Offering a consistently designed experience also reinforces your brand and breeds familiarity, increasing the likelihood of creating loyal repeat customers.


One of the main issues of every ecommerce website lies in the fact that prospects are not able to touch and feel products prior to purchasing. Solving this issue depends on your ability to convey as accurate a representation of your products as possible. You can use photography, graphics, and other imagery to convey the story of your product, make it tangible, and captivate your prospects.


Movement (or more accurately, dynamics) is the principle used to convey subtle motion within the user experience — without using video elements. For example, you may use a sequence of images to enhance the story, or provide a hero image to help the prospect identify with a product or desired outcome. Moving elements can also be used to gather feedback — for example, you might allow a prospect to select different colors of a given product.

Here is an infographic with the main UX elements for better performing ecommerce websites:

Ecommerce User Experience Visual Design Infographic
Ecommerce User Experience Visual Design Infographic (easy embed link:

Don’t overlook the ease of navigation

Navigation is an important aspect of overall customer experience. If your navigation is hard and/or time-consuming, and your average customer needs to spend several minutes finding the appropriate category for a product and navigating to that product, by the end of the process, only the most motivated prospects will be left.

As Steve Krug put it in “Don’t make me think”:

Don’t force your prospects to puzzle out where you’re hiding what they’re looking for! Instead, make your site navigation intuitive and simple.

Let’s start by discussing the menu, which is the main navigational tool on most ecommerce websites. Your menu design should enable your prospects to quickly (and with minimal effort and time) locate a product they want to buy, and navigate to the page where they can purchase that product.

To make navigation effective, intuitive, and effortless, you should follow the Nielsen Norman Group’s 15 principles of usability:

a) Visibility

  • Use the largest menus possible, given space on your website
  • Put menus in familiar locations — either on the top left side (for hamburger menus) or at the top of the page (for navigation bars)
  • Make the menu look interactive, AKA visually distinctive and clickable
  • Give menus visual weight. The menu is an important item, and should be visually differentiated from all surrounding content
  • Contrast menu text links from the background color(s)

b) Awareness

  • Allow prospect to remain aware of where they are at all points of interaction. Techniques like breadcrumb navigation can help prospects answer the question “Where am I?” at any given moment.

c) Relevance and clarity

  • Label links clearly, so users understand what they’re clicking on
  • Make labels easy to scan
  • Enable users to preview lower-level items directly from the menu
  • Provide local level menus for related content
  • Make use of visual communication cues like icons, graphics, or colors

d) Ease of use

  • Make links large enough to be clicked easily
  • Hover-activated menus should stay open for long enough that the user can move the cursor. Don’t make these menus too wide or too long; otherwise, prospects may confuse them for a new page or have difficulty scrolling to the bottom part of the menu
  • Make menus sticky, so prospects who scroll further down the page can easily access them at any moment
  • Show the most frequently used items near the point where the menu is activated, so prospects have to expend minimal effort to reach them

Following these principles should work for most websites, but keep in mind that you should always test.

Interaction and feedback

The way your website interacts with prospects and customer is the final puzzle piece in customer experience. To operate effectively, an ecommerce website must have a way to collect customer data: billing and shipping information, registration data, and so on.

The best way to get this information is by using forms. Forms are a necessary way of collecting customer data, simply because they provide a structured way to both gather and process information.

User experience design - example form
User experience design – example form (Source: UX Matters)

But on the other hand, forms are a huge source of friction. They interrupt navigation, requiring prospect to stop and fill in fields with data. Since this data is invariably sensitive (ex. home address and credit card number), most prospects will be reluctant to provide it.

Even assuming you’ve done your homework and your prospects are ready to trust you with their personal data, your forms can still make this a moot point — costing you dearly in terms of conversions.

What makes forms frustrating?

Effective forms ask only for the information that is absolutely necessary to complete the purchase. Plus, by staying focused on just the critical information, you’ll shorten the form length and the time it takes to complete it.

Next, if your form is long, consider breaking it into several pages or screens so that prospects can fill in one segment at a time (which lowers their cognitive load). To facilitate progress, show prospects a progress bar that tells them how close they are to finishing.

One note on account creation/registration: Requiring prospects to register prior to purchase has obvious benefits for you as the owner of the business. But a truly customer-centric user experience reduces friction — so instead of requiring customers to register, allow them to proceed with their purchase using a guest account or linked social media account.

Even though a user who logs in using a social media account won’t provide you with as many data as one who registers on your site, they’ll still convert, which means they’re more likely to buy from you again in the future (and return customers also tend to spend more).

Grow your post-purchase relationships

This brings us to another important part of user experience: the post-purchase experience.

Till now, we’ve been focusing on the pre-purchase and purchase experiences — but the post-purchase experience is just as important because it has the power to create repeat customers. Even the best pre-purchase and purchase experiences can’t fix a lousy post-purchase experience.

Say you sell clothes. You build an excellent website with a beautiful interface, stunning photography, and a smooth, easy purchase process. You have a high visitor-to-customer conversion rate. These new customers then expect to receive the product they bought in good condition, free of any damage or flaws, delivered on time to the correct address.

Once a customer purchases a product, you need to do a few different things to engage them immediately:

  • Invite them to register (if they didn’t)
  • Invite them to follow or join your social media communities
  • Offer them information how best to use the product they purchased
  • Ask them to review the product
  • Cross-sell or upsell them on complementary products
Ecommerce user experience - secure checkout
Nordstrom guest checkout offers you to join their Borderfree community by registering for accountEcommerce user experience – secure checkout

One of the best ways to engage your customers is inviting them to take part in your social media community. We’ll return to this subject in more detail later on to examine the potential growth opportunities that stem from community. For right now, just know that community plays a role in user experience.

Your customers also expect you to provide post-purchase service. If there is anything wrong with the product they bought, customers want to be able to return it or exchange it without any hassle.

Naturally, your goal should be to make sure that your customers are satisfied and that all of their reasonable complaints are resolved in a timely manner. If your website allows customers to provide feedback after they buy, make sure you respond to negative feedback.

The way you communicate with dissatisfied customers shows how much you care for them. It also shows both prospects and other customers that they can count on your full dedication to solving any potential problems. If you show that you care, you’ll earn loyal and enthusiastic followers who become your best brand advocates.

View your website’s ecommerce user experience as a process that extends beyond the user interface of the site itself.

Ecommerce User Experience

To ease friction and improve ecommerce user experience, your website must be simple and functional. It must allow intuitive navigation, cut down on visual distraction, and omit or hide unnecessary content. You need to ensure visitors face as few obstacles as possible on their journey toward the ultimate goal: purchase.

Ecommerce Content Marketing: How to Create Influential Content for Your Store

Ecommerce content marketing

Published by

Edin is a Senior CRO Consultant. Edin is into Google Analytics and testing (any A/B testing tool really) and likes to write about it. You can follow Edin on Twitter.

January 3, 2018

Why is ecommerce content marketing so popular?

The rise of the Internet and increasing popularity of its new, lucrative ecommerce channel drove the competition for customers attention to ever higher levels. And traditional marketing approaches became less and less effective.

It was no longer enough to just flaunt great images of products on your web page, or write a few lines of cursory product descriptions. Customers, able to access hundreds of different sources of information in short time and with minimum effort, started to demand more from sellers.

This gave rise to a form of indirect marketing that tried to engage the customers in a learning process. Instead of providing only product promotion, marketing started to tackle education through helpful content.

The idea was that by providing useful information to prospects, you can create that coveted “know, like, trust” relationship between you and your prospects. This is high time to introduce a scientific approach to that concept.

Cialdini’s principles of persuasion and how they apply to content creation & marketing

Robert Cialdini, an eminent professor of psychology and marketing, formulated basic principles of influence in his seminal work Influence: The Psychology of Persuasion. Since its publication in 1984., it’s become a basic textbook for marketing and has found special use in online marketing.

Cialdini’s six basic principles are Reciprocity, Commitment and Consistency, Social Proof, Authority, Liking, and Scarcity.

We have a great article on our blog covering conversion psychology and elsewhere to help you get started with Cialdini, so be sure to read them. Understanding the basic principles of persuasion is necessary to create high-converting content.

So how can you apply Cialdini’s principles to enhance the effects of your content marketing for your ecommerce store? Let’s first briefly examine the types of content.

Common forms of content marketing


There’s no shortage of guides for how to write blogs on the Internet, so I’ll include just a few of the most important points for novice blog-writers:

  • Keep blog topics practical and useful
  • Identify your audience and address them in the tone and language they use and like
  • Format the article so it’s easy to read, and use imagery to break up the article flow

Blog posts are one of the primary tactics of inbound marketing, as a device to draw attention of people who never heard of your or your company. Practical and informative topics along with a well written blog can create reciprocity, liking and authority.


Making videos may be more technically demanding, but videos are becoming an obligatory element of content marketing. According to a recent study, over 70% of marketers say that video improved conversion rates better than any other content marketing effort.

Video content can be placed on your website, video sharing services like Wistia or Vimeo, and increasingly on social or specialized channels like Snapchat.

Videos can be educational, inspirational, entertaining, or even raise a question. If they are fun, the videos can generate liking.

Community participation (Social Publishing)

The existence of numerous social networking platforms has given rise to a huge number of communities related to all sorts of fields and interests.

By constructively participating in communities related to your niche, you can increase the recognition of your brand and drive users to visit your site. Do this by giving out some relatively valuable information for free, and answering other users’ questions. Valuable communities for this sort of activity can be focused question-and-answer forums like Quora, or social networks, like Facebook, Google+, and LinkedIn.

This is another way to draw attention of new prospects, as by providing useful information and help on topics close to your field of business can help create authority, as well as liking.


E-books complement blogging, and in fact, a successful series of blog articles can be turned into an e-book with comparatively low effort.

Like blogs, the contents of an e-book should share your knowledge of your field, help your prospects realize that there are solutions to their problems, and show them that you are qualified and able to solve those problems. E-Books can be seen as a way to delight your users and visitors with some smart content.

E-Mail content

Using email to get into contact with prospects is a good method of content marketing. By sending out newsletters, drip campaigns or surveys, you can establish a personal contact with prospect to increase liking and if you provide valuable content, you will have used reciprocity successfully as well. Before you do attempt it, make sure you personalize your email campaigns. Studies, such as this one, show that personalized email messages have 10% more conversions, 6X higher transaction values, and segmented and personalized email campaigns created 58% of revenue.

Emails are best used a mean to close the deals and turn your prospects into visitors.


As a special type of content, there are podcasts – podcasts represent audio content you broadcast on a regular basis. Audio content puts your prospects in direct contact with you and creates a link between you and them. Podcasts help build authority and reach, with the additional possibility of generating liking for you.

Other forms of content marketing

Other forms of content marketing are mostly tied into e-books or blog posts in the form of various content upgrades such as checklists, guides and infographics. These forms help further build up authority, and may even invite commitment if you require your prospect to turn up or a call or a webinar, for example.

Now that we summarized common types of content marketing, let’s look at a few examples, analyze which principle is at work, and dig into the tactics being used.

1. Reciprocity

Successful use of the reciprocity principle in content marketing depends on two things:

  • Your content must be genuinely useful to your audience
  • You must offer it for free

If your content provides something valuable, your audience will naturally want to return the favor. Virtually every company blog worth its salt uses this approach to inform its blogging strategy.

Some of the best examples come from SaaS companies, simply because it is very easy to link their products to content.

For example, Formisimo, a company that created a form analytics service, offers a free guide on how to make your forms more accessible, based on experiences of their subscribers. This content perfectly complements their own software solution for tracking forms. Thus by providing a free and valuable resource, they get prospects interested in how Formisimo solution can help them.

Of course, in order to get your whitepaper, you need to provide Formisimo with a lead in the form of e-mail address. Formisimo counts on the fact that prospects will be grateful for a great and actionable content provided for free and get them to subscribe to their service.

Here’s Formisimo landing page for their whitepaper.
Here’s Formisimo landing page for their whitepaper.

Crazy Egg maintains an extensive blog that not only promotes their service, but educates and informs readers on many topics related to online marketing.

As one of the leading CRO agencies puts it:

“Let’s say that you’re running a popular blog that offers its readers highly actionable and practical information that makes their lives better. Of course, all of this information is offered for free; they just have to visit your site and absorb all of the details. Based on the idea of reciprocity, your site visitors would be more likely to feel obligated to buy something from your website, providing you with an eventual conversion.” – Marc Schenker for ConversionXL

Symantec, a computer and network security company, also focuses on content marketing.

Their blog, Security Response, offers alerts and descriptions of the latest malware, spyware, and viruses, often including instructions on how to avoid being affected and even occasionally offering tools to remove serious threats.

Symantec, a computer and network security company, also focuses on content marketing.

Their blog, Security Response, offers alerts and descriptions of the latest malware, spyware, and viruses, often including instructions on how to avoid being affected and even occasionally offering tools to remove serious threats.

Symantec’s blog, where its targets find useful information.
Symantec’s blog, where its targets find useful information.

Symantec’s objective is to provide their prospects with valuable information or tools for free; their expectation is that a grateful reader will eventually return to buy one of their products.

By providing free tools, they foster trust in their audience — and make sure that they’re the first company that audience remembers when they need computer security.

2. Commitment and consistency

Commitment and consistency means to try to get prospects to commit to a course of action. People generally like to be consistent and if they make a commitment, they will likely to follow through. This tendency can be used by presenting aspirational image of your target audience predisposition so they will want to commit to it and follow through to conversion. To make this possible, you need to research your target audience and discover which “personas” identify with which content.

(We should note that, while “marketing persona” is a common buzzword, your personas should be based on real audience research, not imaginary.)

By being consistent with the prospect’s frame of mind, you will invite commitment. This is how Anthropologie, a clothing retailer, approaches content marketing. They created a blog that presents the base values of their brand and stays consistent with the aspirational self-image of their ideal customers.

Anthropologie’s style blog, which appeals to its target market.
Anthropologie’s style blog, which appeals to its target market.

Another good example is General Assembly, a website that provides online education in programming languages and digital marketing. Their blog and other content usually contains phrases like “become a better programmer” or “make yourself into a scarce resource,” and this theme persists throughout all of their content — course descriptions, blog posts, and email newsletters.

General Assembly’s self-reflective blog.
General Assembly’s self-reflective blog.

Like Anthropologie, General Assembly presents an aspirational image that its readers can identify with. They invite prospects to commit to this higher self-ideal, then invite them to subscribe to the GA email list, which fosters consistency with that self-image.

3. Social Proof

“Social proof” in marketing refers to showing the universal or near-universal popularity of your products or services, and the beneficial results previous customers have obtained from using those products or services.

It’s one of the strongest possible forms of persuasion, especially if it comes from a peer group of customers. To implement social proof effectively, it must be authentic and provide useful and detailed feedback to visitors.

Amazon offers an excellent example of social proof done right. They have an extensive review system, and even the reviews themselves are graded by other users for their value. That way, users don’t even need to sift through many reviews to find the ones that will provide them with the most information and help them make a purchase decision.

Amazon’s review system prioritizes the most helpful positive and negative reviews.
Amazon’s review system prioritizes the most helpful positive and negative reviews.

The reviews can be sorted by rating, or by whether they are positive or negative. This type of social proof is very effective and represents additional content and in of itself, volunteered by customers.

Another company that uses social proof to its benefit is Sony. Here is how they organize and sort reviews of their products:

Sony’s feature-comparison review system.
Sony’s feature-comparison review system.

Their system is very similar to Amazon’s, but they also support grades for specific features of each product, so you can compare which features are more important.

4. Authority

Content marketing also helps establish your website’s authority to your prospects. I’m not talking about “authority” in the SEO sense, but rather showing them that you’re a trustworthy expert in your niche.

If your website is a valuable source of new information that readers can trust and rely upon, they’ll visit more frequently. And if they become frequent visitors because your content provides something of value, they will be more likely to convert.

Consider Symantec’s content marketing again. By providing accurate and useful security information, Symantec establishes the impression that they know what they are doing in the field of digital security — so readers will turn to them when they encounter a security issue.

Another good example is Philip Patek, a Swiss watch company. Their website content is created around the importance of handcrafted items and a detail-oriented production process. By describing the process, they show their expertise and underscore the value of their product.

Ecommerce content marketing - Philip Patek
Ecommerce content marketing – Philip Patek

5. Liking

The principle of liking is simple: if your visitors like you, they will be more likely to convert.

So, if you provide interesting and entertaining content that people like, they’ll like you — and you increase the probability that many of them will eventually convert.

For example, Patagonia, an outdoor gear and sportswear seller, has a blog where they post many stories about the environment, preservation, and similar topics. They want their target audience, mostly people who enjoy nature, to like them for their environmentalist stance. That way, Patagonia will be the first brand those readers think of when deciding to purchase a new backpack for their next outdoor adventure.

Ecommerce content marketing - Patagonia
Ecommerce content marketing – Patagonia

6. Scarcity

The concept of scarcity is usually implemented in content marketing as imposing some sort of limitation on a product or service’s availability.

When you introduce scarcity, the prospect will perceive the offer as limited. You’re introducing the fear of missing out. If employed correctly, scarcity (and its sister concept, urgency) can push a prospect to “ACT NOW!” — namely convert.

Of course, scarcity must be authentic, otherwise it will backfire badly.

“Content supply is nearly ubiquitous, but time remains in constant scarcity for the consumer. Those who can attract engagement, particularly on a per-category and per-asset basis, are onto something powerful. – Josh Rucci, General Manager and Global Head of Bloomberg Content Service

An example of the scarcity principle in use is how Business Insider markets their exhaustive industry wide studies. By using an email campaign, they reach out to their leads and offer premium content for extremely low price for a limited time.

Ecommerce content marketing - Business Insider
Business insider intelligence uses scarcity to and FOMO to get leads to convert in their email content marketing campaign.

How to link content marketing to conversions

The hardest part of content marketing is actually getting readers to convert after reading your blogs or viewing your videos. The first thing you need to achieve through your content is to get people to like you. As I said in the beginning, people are more likely to do business with someone they like.

To that purpose, make your content as entertaining as possible! Insert witty jokes, use funny images to link the meaning of your article to something your audience can relate, like movies, books, or other popular culture.

Plus, by offering your prospects and readers useful information, you have created a bond between them and you. If they can immediately make use of this information to help them in some way, later on, they will want to return the favor in some way. This is why it’s a good practice to ask your audience to provide their contact information or register for your email list.

The most common way to get visitors to provide you with personal information is to offer additional content that’s related or relevant to the content they were just engaged with. You can offer a follow up article on a similar subject when they reach the end of a blog, for example.

This extra content can also take the form of an “offer” or “upgrade” (like an ebook, checklist, email series, etc.) that builds on or goes into deeper detail about the given topic.

Getting visitors’ contact information will enable you to remarket to them with additional content and calls to action to purchase your actual product or service.

Even if they only provide contact information and one small piece of personal information (such as country or gender), this info can be very useful for learning more about your audience, establishing personas, targeting more effectively, and coming up with new content ideas. By personalizing your content in this way, you will naturally encourage your audience to like you even more.

Getting to know your audience and target market is by far the greatest value of content marketing. By increasing your prospects’ awareness of your brand and your expertise, and fostering knowledge of your products and services, you ensure that people will remember you at the moment they need what you offer.

The best use of content marketing

The basic idea of content marketing is to help you identify your customers and initiate their buying cycle with your content. The typical buying cycle consists of the following steps:

Ecommerce content marketing - typical buying cycle
Image source: iFormat digital strategy agency

Buying Cycle Step 1: Awareness

As you have seen, a well-formed content marketing strategy can have the direct result of increasing your prospect’s awareness of your brand, your product, and problems your product solves. This provides the initial nudge to start moving your customer through the cycle.

Awareness-stage prospects are best addressed with content built around authority, commitment and consistency.

Buying Cycle Step 2: Research

Your content must provide the prospect with relevant, authentic, and clear information so they can confirm that what you’re offering actually solves their problem and that you know what you are talking about. When a prospect is in the research phase, serve them content that increases authority, creates liking, and builds reciprocity.

Buying Cycle Step 3: Comparison

While you cannot offer direct comparison between your products and your competition, using content marketing you can establish a baseline and ease your customers ability to compare different products. Social proof and liking is essential in this phase of customer journey. When prospects want to compare your offer to your competitors, social proof and liking of your company and brand will sway them to your camp.

Buying Cycle Step 4: Purchase

Proper implementation of the previous steps will result in your prospect purchasing your product, and becoming a customer. Once this happens, it is time for content marketing to take over again. The purchase phase is mostly influenced by scarcity and authority. Scarcity convinces prospects that they shouldn’t miss your offer, and authority helps convince them that they’ve made the right choice and will not suffer buyer’s remorse.

Buying Cycle Step 5: Retention

When a customer purchases your product, the relationship between you and that customer is far from over (unless you sell coffins).

In fact, the buyer relationship can last for a long time. Business Insider shows that returning customers are both more likely to convert again and more likely to spend even more money.

Retention is best achieved by continuing your efforts to build authority, encourage liking, and appealing to customers’ desire for commitment and consistency.

These three elements will make prospects come back again and again because they like you, trust you, and identify with your brand and products. (For example, think of all the die-hard Apple users you know.)

That means you need to do everything you can to retain your existing customers — and content marketing plays a big part. Provide customers with support in the form of content that explains how your product works, how to fix common errors, or how to customize it so it can do something desirable.

Avoid these common content marketing mistakes

There are a few mistakes you can make in content marketing that can severely hamper the chances of this considerable effort being beneficial to your company.

The first mistake is going into it without any strategy. Before you start creating content, create a detailed blueprint of what you want to achieve, who you want to address, what channels to use, and a brand story.

A second common mistake is focusing on yourself. Successful content marketing focuses on solving customers’ problems and providing customers with additional value. Failing to do that relegates content marketing to just another form of product or brand promotion.

As Phil Laboon, CEO of Eyeflow, eloquently puts it:

“Everyone likes to think they know what’s best for their audience, but too many companies write content that sounds good to them. They focus only on topics that cover their own interests. If you want your content to be successful, you have to write with your audience in mind. Craft your content so it speaks to them about the issues they are actually facing.”

Pitching your products all over your content marketing is a common symptom of the second error and only compounds the problem. Refrain from pitching the product until your prospect has reached the third stage in the buying cycle and is actually considering a purchase.

Finally, your content must be informative, useful, and substantial. Failing to create content that hits all three points is another fatal error. Empty content that conveys no useful or valuable information is just noise.

Consider Cialdini before you create content, and you’ll convert

Properly conducted, your content marketing efforts can create vocal advocates for your brand out of your customers.

By relying on Cialdini’s principles of persuasion while you create your content, you’ll avoid the pitfalls of many content publishers, and win the appreciation, trust, respect, and reciprocity of your target customers.

You can use many different types of media to create your content. Whether you choose to write blog posts, film and post video content on your site, build communities on social networks, or write e-books, remember to highlight the problems that your target audience faces and show how to solve them.

To avoid making mistakes, always keep in mind that your prospects come first. Try to offer a view from their perspective and help them fulfill their goals. And don’t forget that many typical content marketing implementation mistakes can be avoided by creating a strategy before you start.

Ecommerce content marketing

The rise of the Internet and increasing popularity of its new, lucrative ecommerce channel drove the competition for customers attention to ever higher levels. And traditional marketing approaches became less and less effective. Ecommerce content marketing is a way for all of us to differentiate.

Bandit Testing: What It Is, Plus When & How to Use It

Bandit Testing

Published by

Edin is a Senior CRO Consultant. Edin is into Google Analytics and testing (any A/B testing tool really) and likes to write about it. You can follow Edin on Twitter.

December 7, 2017

So, you’ve decided to start testing! Naturally, you’ve established a testing program, done your research — this is something we’ll always mention, since it’s so easy to get caught up in the test mentality without adequate research — and established sound hypotheses.

Only once you’ve defined the problems, devised possible solutions, and prepared hypotheses can you start testing with any hope of success.

You’ll often have multiple ideas for solving any given problem on your website. The simpler the problem, the fewer ideas that make sense to try. However, for complex issues, you will run into a situation where you need to devise a number of potential solutions, since you can’t really be sure what will work best.

As the number of experiments increases, so does the required sample size. This is important for a few reasons.

You can’t ignore the clock…

Bandit Testing Clock
Bandit Testing Clock

The larger the sample you need, the more time you will have to allocate for the test to run. For example, if you have a test that includes more than two variations, or a multivariate test, the sample size will likely go into the five figures or even higher, depending on the effect. Unless your site has hundreds of thousands of visitors per month, this will present a problem.

The time the test runs will be directly proportionate to the number of people who view it daily. If you require a sample size running in the tens of thousands and have daily visits of mere hundreds, your test will run for a prohibitively long period of time. And any test that runs for longer than a month risks suffering from sample pollution.

What is sample pollution?

Sample pollution is a statistical concept that signifies the amount of visitors who are subjected to your experiment multiple times. For example, you might run an experiment where you present two versions of your product page to visitors. After the test has run longer than one month, the visitor who viewed the page on the first day returns.

By this time, they’ve deleted or refreshed the cookie you left (a cookie is a piece of code left on a client browser by experiment software to track which variation was shown to that user). They will see a different version of your page than the one they saw the first time. Once this happens, that visitor’s actions can no longer be taken as conclusive. If this happens to a significant number of visitors, the test results will be unreliable.

Invalidating your test after such a long run would be a serious problem. That is why tests should be conceptualized in a way that ensures your sample size is manageable, especially for relatively low traffic site.

Iridion shares its testing benchmarks succinctly:

A test should run at least two to four weeks and include at least 1,000 to 2,000 conversions per variation. Activities (newsletters, TV spots, sales, etc.) should take place during this time. There should be as many different channels displayed as possible (total traffic mix, either through targeting in the preliminary stage or segmentation in follow-up). If the test has achieved a minimal statistical significance of 95% (two-tailed, that is positive and negative) and if it is stable, then stop the test.

Time also poses another problem. If you run a test with three, four, or even more variations, each variation that fails to deliver will mean lost revenue. The proper, rigorous statistical approach means you will have to maintain splitting your traffic in equal proportion the entire time, though you may realize that a certain variation is a clear loser. It means throwing 25% of your potential revenue down the drain.

Bandit testing allows you to steal back your time

Bandit Testing: Gaussian normal distribution
Bandit Testing: Gaussian normal distribution

The solution to this problem appears to be a simple one: observe the test as it runs and eliminate the variations that appear to be losing. Then allocate the remaining traffic to better-performing variations. The problem with this is that it breaks every statistical rule. You cannot possibly be sure that the losing variation is really losing without statistical significance, and that means having to wait until the test runs its course.

Multi-armed bandit
Multi-armed bandit

Scientists originally encountered this problem when they needed to consider allocation of efforts and resources on different experiments. The problem remained unsolved for decades, and it was only in 1952 that Herbert Robbins devised solutions for using a bandit testing approach on multiple concurrent experiments to select the best-performing one.

The solution quickly found its applications in both scientific research and the selection of the best-performing financial portfolios.

The name “bandit testing” or “multi-armed bandit testing” is derived from a simple analogy. The idea is that you are presented with a gambling machine that has multiple levers (arms). Pulling each of them results in a reward with a certain probability, which is different for every arm.

Your task? Find the arm that provides the most frequent reward.

There are multiple strategies for solving this problem, and each has its own advantages and disadvantages. They involve a fair number of complex statistical calculations — so for those of you who are really interested, check out the multi-armed bandit Wikipedia page.

For now, I’ll briefly explain one common solution: a two-phase approach combining exploration and exploitation.

Exploration is pulling levers to see what happens. Exploitation is pulling the levers that result in the most frequent rewards. The more time you spend on exploration, the more certain you will be as to which machine will provide the reward more frequently — but you’ll receive fewer rewards. The aim is to shorten the exploration phase to a minimum, and engage mainly in the exploitation phase.

Using bandit testing in CRO became popular relatively recently, and has gained traction by promising to deliver better results faster. But can it really be useful to all optimizers?

When you should use bandit testing

Using bandit testing in CRO is alluring, for obvious reasons. You can deliver a better conversion rate in less time and increase your revenue faster. But you have to be careful.

Why? First, due to the statistical methods used — especially frequentist statistics — interrupting an experiment before it reaches statistical significance carries an inherent risk that the result actually isn’t significant. This risk is hard (if impossible) to eliminate. However, given a large enough sample, we can with some amount of certainty and calculated risk, rely on bandit testing.

Here’s how Conductrics explains it:

While there are many different approaches to solving Bandits, one of the simplest is the value weighted selection approach, where the frequency that each optimization option is selected is based on how much it is currently estimated to be worth. So if we have two options, ‘A’ and ‘B’, and ‘A’ has been performing better, we weight the probability of selecting ‘A’ higher than ‘B’. We still randomly select between the two, but we give ‘A’ a better chance of being selected.

With this in mind, there are some guidelines you must follow to properly apply bandit testing. First off, the most appropriate use for bandit testing is multivariate testing. Here, you can use bandit testing to quickly eliminate the worst-performing variation, and reduce the amount of time required to run the test. Multivariate tests require a large sample size, and any reduction in that size is usually a welcome change.

Second, you can use bandit testing to test pages with a short shelf life: for example, landing pages for promotional campaigns. You want to run your campaign for two or three weeks, and you don’t want to spend a majority of that time experimenting on multiple versions of your landing page.

Essentially, if you stand to lose (or miss out on) a large amount of money during a promotional period, and you have the traffic, bandit testing could be the right approach for you. If not, then stick to A/B testing.

And if you have a website with sufficiently large traffic, you can use bandit testing methods for most of your tests, as you will hit statistical significance pretty soon. That way, you’ll have more faith in your estimate of losing variations.

Bandit test algorithms

Bandit test algorithms are automated pieces of software that solve the problem of selecting the optimal arm. There are several families of algorithms, with names such as:

  • έ greedy algorithm
  • Boltzmann exploration
  • Pursuit algorithms
  • Reinforcement comparison and
  • Upper Confidence Bounds (UCB)

Each of these families of algorithms performs better than the other on different types of tests. By far, the most often used are UCB and έ greedy algorithms. For an in-depth exploration of the merits of each with a sound statistical foundation, we recommend this article.

To see how a combination of a Bayesian test and a bandit algorithm performs, check out

AI is the future of bandit testing

Lately, there’s been a lot of buzz on the Internet over machine learning and artificial intelligence. One of the most talked-about potential use cases for AI and machine learning (at least in CRO circles) is applying it to A/B testing. The possibilities seem endless and the reward great.

The best-known, and one of the first AI software solutions for CRO, is Sentient Ascend. This piece of software relies on conducting large-scale multivariate tests and automatically discards lower-performing variants. Since it requires human intervention only to identify the variants to be tested once the research is completed and the hypotheses formed, the test program basically runs itself.

In the future, this type of testing will surely make testing easier and faster. It will never eliminate the human factor completely, however. Human input will still be necessary to point the algorithms to what to test and what combinations of variations to test.

Soon, AI will test for us

The point of the entire process of conversion optimization is to increase revenue — i.e. the amount of money ecommerce sites bring to their owners. The best way to increase the likelihood of visitors converting is by researching their behavior and devising changes on your website that you hope will deliver an increase in purchases.

Since you can’t know in advance what exact change(s) will improve performance, you need to test variations against each other and compare results.

The problem with this approach is that it requires following the rules of statistics to be sure that the variation you implement really is the winning one. Proper testing requires time, and when it comes to ecommerce, time is literal money. Every day you spend testing and sending traffic to each variation in equal proportion means revenue lost on the lower-converting page variation.

No wonder people were trying to find the solution to this problem! They found it in the form of bandit algorithms. The problem with bandit algorithms was that they require a complicated statistics to actually work. But once the computational problem was overcome, it quickly became apparent that bandit testing is in some situations superior to classic testing.

The combination of Bayesian statistics and bandit algorithm testing is a very happy marriage, since Bayesian statistics does not require the test to reach a significance threshold. Using Bayesian statistics, you can estimate the likelihood of the tests that will win, and eliminate those that have the least chance of success.

Bandit testing can also be combined with artificial intelligence (AI) to produce quick and successful test programs that use the AI to test many variations — much more than a human ever could. In the near future, a lot of the testing workload will be taken from humans and put on the shoulders of our electronic counterparts.

However, keep in mind that neither bandit testing, Bayesian statistics, nor machine intelligence are “Win automatically” buttons. They’ll never be able to function without optimizers’ thorough knowledge of what we are doing. Otherwise, we’d be entrusting our fates to a black box.

Bandit Testing

The name “bandit testing” or “multi-armed bandit testing” is derived from a simple analogy. The idea is that you are presented with a gambling machine that has multiple levers (arms). Pulling each of them results in a reward with a certain probability, which is different for every arm.

Fix Your Forms to Spur More Sales: Form Implementation & Analysis

Form implementation

Published by

Edin is a Senior CRO Consultant. Edin is into Google Analytics and testing (any A/B testing tool really) and likes to write about it. You can follow Edin on Twitter.

November 27, 2017

We constantly encounter forms while browsing the web. They require us to fill in data, provide our email address, our real address, or other critical data.

The role of forms is, at its base, to allow website owners to get to know their market — to contact customers, receive payments from them, and deliver the goods they buy.

And along with being a critical part of web infrastructure, forms are also one of the greatest friction points in the overall ecommerce experience. Frequently, they require giving away sensitive data, such as credit card numbers, home address, and so on. No wonder people feel apprehensive when filling out forms.

Overcoming this friction is an important task. Why?

Because if your customer fills out a form on your site, the likelihood of their eventual conversion is much higher. This is the consequence of a psychological pattern called sunken cost fallacy. Essentially, if someone devotes time and effort to filling out a form, they will be less likely to abandon that effort.

In a widely publicized and quoted study, How to Optimize Contact Forms for Conversions, the main recommendation was to reduce the number of the fields in a form. True enough, in the case the researchers analyzed, form-field reduction turned out to be the solution.

But extensive testing in multiple other cases revealed that the solution is not always that simple. The eventual answer that many practitioners (including ConversionXL, Unbounce, and others) discovered was that form friction largely depends on context.

Whether or not a visitor is willing to fill out a form directly depends on that visitor’s expectations, as well as the form’s length and complexity.

For example, when purchasing a custom, high-cost item or completing a job application, visitors may actually expect to have to fill out a long form with specific data. And if their expectations aren’t met, they may even react with suspicion!

As with everything, the answer is “Test, and don’t assume the best practice applies to your specific case.”

Amazon’s bare-bones, low-friction registration form
Amazon’s bare-bones, low-friction registration form

You can see here that Amazon only asks for a user’s name and email address for registration purposes. They know that all other necessary data, including payment info, will follow if visitors become buyers.

On the other hand, here’s a longer form that also serves its purpose. This one is from JCB, a company that sells construction machinery. JCB is interested into getting leads — and qualified leads, at that.

JCB’s lead-generation form
JCB’s lead-generation form

Since JCB is only interested in serious, qualified leads, they require much more information, including the name of the company asking for a quote. This makes sense, since they don’t want to waste their marketing department’s efforts on pursuing unqualified (AKA unlikely to purchase) leads.

Friction-reducing best practices

Having said all of that, there are still some general principles you can apply to reduce friction and increase your form conversion rates.

One best practice is to include only as many fields as are absolutely necessary. Including more fields increases the amount of friction that form-fillers will need to overcome, and will cause more of them to abandon the form.

Another important best practice is to reduce the amount of work the visitor needs to put into filling the form. There are many ways to do this: for example, you can add dropdown menus for country, state, year of birth, and other data fields that have a small or finite pool of possible responses. In some spots, radio buttons or checkboxes can be helpful.

Also, pre-fill or include a default answer for certain fields when appropriate. This further reduces your visitors’ burden, and thus eases friction. For instance, here’s what Google does with its registration form:

Google Registration
Google Registration Form

Google recognizes the country from which you’re browsing, and immediately inserts the correct values in the “Location” field. Knowing this, it also updates the “Mobile phone” field with the correct country code.

One note: Avoid putting the description of the data required inside the form itself. For example, if you want your visitor to give you their name, put the “Name” descriptor above or adjacent to the field. Don’t put it into the field to be replaced as the visitor starts to type — since people may forget what you were asking when they start typing!

Another useful method you can employ is to break the form into two or more pages. If you truly require a long form, this may be the best way to increase the likelihood of conversion. Be sure to provide your visitors with a visual indicator of their progress. Removing uncertainty in this way encourages them to complete the process and sets a clear expectation for when they’ll be done.’s referral program offers an option to create nice-looking forms with multiple pages and an indicator of progress.

A user-friendly form
A user-friendly form

Common problems with forms

Bryan Eisenberg, one of the original practitioners of conversion optimization, sorts form issues into three main categories: failure to address and reduce fear, failure to build trust and credibility, and failure to reinforce benefits.

Below are some questions to ask to ensure you don’t fall into these common conversion-killing traps.

1. Failure to address and reduce fear

Does your form ask for more information than necessary? The longer your form gets, the less likely it is that people will fill it in, unless they are really motivated. If it’s a simple contact form, you can almost always skip things like “home address”.

Do you ask sensitive questions? People will hesitate to give away private information for fear of abuse, fraud, and theft. You must establish credibility and ensure that collected data will not be exposed to unwanted third parties.

Does your form look intimidating? If your form is a wall of fields, it will visually intimidate your visitors. Try to reduce the number and density of fields, or break the form into multiple parts.

Does it create doubt and uncertainties? If you go out of your way to get information from visitors, or ask questions in your form that have no clear bearing on what visitors get in return (for example, you ask for a phone number and home address when giving away a free PDF file), don’t be surprised when people fail to fill it out.

Do you handle errors graciously? Point your visitors directly to any errors made in filling out the form. Don’t just generically, infuriatingly say, “There is an error.” Indicate where the mistake was made, and show your visitors how to correct it. Ideally, auto-validate fields as they are filled out and give a clear indication of the expected format — for example, show the “mm/dd/yyyy” format if you require a date.

2. Failure to build trust and credibility

Do you establish yourself as credible and trustworthy? If your website does not feel trustworthy and your offer lacks credibility, no matter how you design or present your form, it will not convert.

Is it obvious that your browsing environment is secure? Make sure your page is using the https:// protocol and that appropriate security badges are featured on form pages, especially if you require payment data like credit card numbers.

Do you leverage trust messages at the point of action? For example, provide relevant messages at each step of your conversion process and don’t cram in unnecessary messages that burden the process. For example, put payment security information on the step where you require payment info.

Do you assuage uncertainty as to why you’re asking for certain info, and explain what you’re going to do with it? Clearly explain your privacy policy, and assure visitors that you won’t share their data or use it for anything other than the intended purpose.

3. Failure to reinforce benefits

Remind your visitors what tangible benefit awaits them if they fill the form out. What will happen next? How will their life improve?

Is your message relevant to the ad or offer to which your visitors may have responded? Relevance can build trust, so make sure your copy and design are cohesive across the different elements of your marketing funnel.

Do you give your visitors sufficient payment options? Offer a third-party payment service, such as PayPal.

Are your forms user-friendly and simple to fill out? No one wants to squint at a form whose fields are packed too tightly together or displayed in 8-point font.

Solving for all three of these major issues is a must. Failure at any point will affect the form’s overall submission (AKA conversion) rate.

How to analyze your forms

Begin analyzing your form the moment you create it. Carefully ask yourself the questions above as you add or modify fields, so you can be sure you’ve created a functional form that’s as frictionless as possible.

Take, for example, a lead-generation form for a service or SaaS business. If you were making one of these, you’d need to consider what information you truly need and what questions will help unqualified leads self-select themselves out of the funnel.

In this case, you may want to include some fields on your form that will dissuade unqualified leads from completing the form — like asking for a budget range that begins well above $0.

But balancing friction and motivation can be really hard. That’s why it’s vital to carefully monitor form interaction on your website. Just knowing whether or not the form was submitted won’t cut it. In fact, it’s even beneficial to know the timing of each field being filled out, and when users quit.

This information can provide insight as to which fields in your form create most friction, and what questions should be rephrased or cut entirely. That’s why most form-tracking tools feature a timer that shows users’ hesitation to fill out a given field.

Conversely, a high completion rate for your forms may indicate that your form could be even be longer, and that you might be missing an opportunity to collect more data on your prospective customers. In this way, form analytics tools help you establish that delicate balance of friction and motivation.

Helpful tools for form analysis

So now that you know the methods used in form analysis, let’s check out some tools that can help.

The first tool to look at is Google Analytics — and often, you don’t need to look further. Establishing form-tracking in Google Analytics is relatively easy. If the forms are deployed in a standard HTML implementation, then adding a tag through Google Tag Manager is as simple as adding a new tag, selecting “Universal” tag, and tracking the type of event.

Using a trigger, you can easily set up form submission tracking on your website
Using a trigger, you can easily set up form submission tracking on your website

In the fields, simply input how you want your form to be reported and add a trigger. There is a predefined form-tracking trigger in Tag Manager that can be used with standard HTML form implementation.

However, since this setup involves making a tag for every form and carefully defining triggers to avoid triggering a tag on wrong forms and messing up the tracking (which invalidates any data collected), it can get fairly tedious.

Tracking gets a bit more complicated if your forms use Asynchronous JavaScript and XML, aka AJAX. These forms are not easy to track, since they are usually filled in and completed within the same page using a layer.

This is not easily interpreted by Google Analytics, and to add salt to the wound, the data layer — a crucial element in how Google Tag Manager tracks events — gets refreshed every time this happens, removing any variables stored within it.

Of course, there are solutions. They generally involve creating custom JavaScripts that populate the data layer with event variables after the form is completed. Google Tag Manager can then interpret these variables and actually track the form events.

And fortunately, most plugins for forms that use AJAX also feature easy-to-use tracking enabled through the server interface. For example, the popular Gravity Forms plugin for WordPress uses AJAX forms with this type of built-in tracking.

The results from this type of tracking are displayed in Google Analytics reports as “events”.

But as you might guess, there is very little in the way of visualization with this solution.

And often, having a visual representation may make all the difference in the speed of analysis. Humans are visual creatures. We’re faster at interpreting images than tables and numbers. That’s why there are quite a few tools made specifically to track forms and display the results in a more visual format.

The first of these are the mouse-tracking tools, such as MouseFlow or HotJar. All of these tools offer some kind of form-tracking. (They also track cursor movement, clicks, scrolling, and session recordings! Those features are all valuable, but form-tracking is the feature we are interested in right now.)

HotJar only offers tracking for the standard HTML-type forms. You just add the URL of the page containing the form, and the tool does all the heavy lifting for you. Pretty soon, you start receiving the tracking results, which look like this:

A sample HotJar report
A sample HotJar report

MouseFlow is even better in this regard, since it’s able to track AJAX forms virtually automatically. You just go to the Form menu and follow the simple installation instructions.

Once completed, MouseFlow will show all the fields of the form it’s tracking:

An example MouseFlow report
An example MouseFlow report

These tools are mainly used for mouse-tracking, and offer form-tracking features as an addition, While they’re often good enough, these features are, after all, afterthoughts in tools created for another purpose.

There are dedicated form-tracking tools, so let’s take a look at those now.

The first and most widely used is Formisimo. Formisimo focuses solely on form-tracking, using a script that is implemented on your website.

The setup procedure is simple: you enter the URL of the page where the form is located, and you can name the form you want to track if there are multiple forms on the page. Soon, you start receiving the results:

An example Formisimo report
An example Formisimo report

There are many alternatives, such as Inspectlet, Lucky Orange, JotForms, and others.

The main point of all these tools is the same. They track form conversions and offer powerful visualizations to help you get insight into your form performance.

Finally, there is another powerful tool at your disposal to help you with form analysis. You can employ remote user testing services to get a sample of real users to fill out your forms. This way, you can quickly see exactly what users are doing and how they interact with forms on your website.

A word of caution, though: since user-testing visitors are not your real customers, they may not act authentically. However, you’ll get a general idea of the hurdles that may be impeding visitors from filling out your forms.

Using any of these means, you’ll be able to spot difficulties with your form conversions. And knowing there’s a problem is halfway to solving it! The next step is figuring out what to do.

Surveys can help you troubleshoot problems with your forms

Once you set up form-tracking, you’ll soon have a clearer idea of which forms are underperforming, and which fields or questions are creating the most friction.

Sometimes you’ll be able to guess the reasons for the trouble immediately. Other times, well, you’ll have to ask your visitors using a survey.

The best way to increase form conversions is to match visitors’ expectations. This is obviously easier said than done. To determine what your visitors actually want, you should use surveys. Cleverly posed questions will reveal their expectations so you can meet them.

In order to analyze most effectively and efficiently, you should test different form implementations. Create several forms and put them to test using a A/B testing tool. Once you get a clear winner, implement it.

Use this checklist to create your next form

Forms are a vitally important (if not the most important) part of your conversion process. You need forms to find out who your visitors are, how to take their payment, and how to send them their purchases. For this reason, forms are at once a great source of friction and a great opportunity.

If a lot of people successfully complete your forms, you’re probably well on your way to improving your conversions.

Your 7-Step Form Implementation & Optimization Checklist

  1. Define what information your form needs to gather. Build questions around the end data you need.
  2. Design the form itself. If it’s long or complex, break it down over multiple screens or pages.
  3. Before the form is put on the site, user-test it. Fix any issues that this round of testing reveals.
  4. Implement the form on your site, connect it to your form analytics tool of choice, and analyze the results. You’re looking for spots where users seem to have trouble, or where they commonly abandon the page.
  5. Survey your visitors to get a better idea what might be wrong with your forms. Find out what information visitors are reluctant to provide, or if the forms are too long, by asking your visitors questions.
  6. Iterate and improve your forms, using your analysis and survey results. Test different versions of potential solutions to find a winning variation.
  7. Implement the winning variation!
Form implementation

We constantly encounter forms while browsing the web. They require us to fill in data, provide our email address, our real address, or other critical data. The role of forms is, at its base, to allow website owners to get to know their market — to contact customers, receive payments from them, and deliver the goods they buy.

Samples, Significance, & Statistics: Your Intro to Better A/B Testing

A/B Testing Statistics

Published by

Edin is a Senior CRO Consultant. Edin is into Google Analytics and testing (any A/B testing tool really) and likes to write about it. You can follow Edin on Twitter.

November 15, 2017

“Hey, why don’t we just test that?”

A/B testing is an indispensable element of conversion optimization. Backed by extensive and methodical research, testing promises to be the best way to improve website performance in a relatively short time. That’s why conducting A/B tests is considered the pinnacle of the CRO process.

And testing actually sounds like a pretty easy thing to do. You just create your page variations, put them into the experimentation tool of your choice, and that’s it. Sit back, wait for the results to roll in, and implement the winning variation — enjoy!

If it were actually that easy, this post would end right here.

I would have directed your attention to our blog on conversion optimization, and you would have clicked something like “How to properly conduct conversion research,” and that would be it. The reality, though, is a bit more complicated.

What the “black box” concept has to do with A/B testing

‘Wait, what? I thought we were talking about testing!”

We are. But first, let’s just briefly examine the concept of the black box.

What is a black box? Simply put, it’s any complex system (device or process) that uses inputs and process them to outputs (results).

The user does not understand the process used to turn inputs into outputs.

AB testing statistics black box2
AB testing statistics black box

If you use your testing tool in the way described above, you’re using it as a black box. In essence, you do not know why the outputs came out the way they did, or how the system processes them. The box, AKA the process, remains a mystery.

Using any system as a black box is inherently dangerous, since you can’t know if the results are valid. While detailed knowledge of the system is not necessary nor expected, at least basic knowledge is required.

Put another way, you need to know how a system works to use it effectively. Otherwise, you face a real risk of GIGO (garbage in, garbage out).

To make the experimental tools we use in A/B testing more transparent and reduce the risk of relying on false conclusions, we must learn their basic mechanics. Since A/B testing at its core is a statistical method, that means we need to learn some basic statistics and their role in the A/B testing process.

Hey, wait! Don’t close this tab just yet. I said basic statistics.

… Ah, well. For the two people who didn’t close the window, rest assured that I’ll put this in as clear and simple terms as possible.

I’ll explain some of the basic statistics terms involved in A/B testing. There will be no advanced math or formulas to remember. The tools will take care of that for you.

All you’ll learn here is what these numbers mean, and how they influence your test results, so you can test properly and confidently.

What is statistics?

Let’s start with a quick foundation. Statistics is a branch of mathematics that deals with the properties of large sets of elements.

Think back to your grade school, where you learned about sets and subsets. Those familiar concepts are used in statistics, too.

An element of a set in statistics is called an observation. This name fits nicely, because they are precisely the observations we make of the properties of an object, numerical figure, or person.

For example, imagine you write down the height of every person in a room. These data points together create a set of data that we can examine and use as a basis for larger, more meaningful observations.

Or, say you have 100 people in a room. If you wanted to measure everyone’s height, you’d have to measure each individually. That might be easy enough — but what if it were 1,000 people? Or 1,000,000?

At a certain point, you can’t measure everyone individually. So you have to take a sample.

Samples provide a representation of a larger data set

What you could do instead of measuring everyone is make a limited number of “observations” (measurements). This limited number of observations is known as a “sample” in statistics.

Once you know the properties of your sample, you can use it to deduce the properties of the entire set of elements (called a “population” in statistics).

And you won’t even have to measure 100,000 people!

Another quick example (this one from real life)

Let’s now look at a real example of how statistics are used. One of the most common uses of statistical methods is polling.

If, for example, you want to know the purchasing power of the population of New York City, you would need to gather data from every household in the city. But that is frequently not possible nor efficient to do.

Instead of relying on the time-consuming process of polling everyone, you could instead poll a number of random inhabitants, and calculate their purchasing power. This limits the number of measurements necessary, saving time, effort, and money that can instead be expended on analyzing the data.

This limited number is the sample, and all the inhabitants of New York City comprise the population.

So, now we poll a sample — say 1,000 people out of 500,000 living in NYC. What can we do with it? How do we derive the data we need and project it to the entire population? Let’s see.

First off, you need to be sure your sample is actually representative of the population. This means the ideal sample should resemble the entire population, but at a smaller scale.

To continue with our NYC example, if you took a sample of people living in relatively well-off parts of the city or in suburbs, but avoided polling people in poorer communities, you would not get an accurate sample of the population.

Therefore, sampling must be done with care — but not only that. You also need to take a sufficiently large sample to eliminate the possibility of error. There are methods for sampling without bias, and we will examine them later. For now, all we need to know is that a sample is a limited part of the total population that should ideally represent the entire population.

Averages indicate the “central tendency” of your data

Once you have a sample, you can use it to calculate many statistical properties that describe the sample and the total population.

One of the best-known properties is the “average” of the set. The average, or mean, is simply the value of the sum of all the elements divided by the total number of elements in the set.

Average = (Sum of all of the elements) / (Number of total elements in set)

Basically, if you have 5 different values in a sample, you can add them all together and divide that number by 5 to get the “average” value. This is also called the arithmetic mean or simple average.

An important thing to keep in mind when testing is a phenomenon called “regression to mean”. This essentially means that when you start testing, the results may vary significantly. Sometimes you’ll notice the initial values in the test that are far above the eventual mean of the sample.

Think of flipping a coin

Flipping or tossing a coin is a test with a binary result — the coin can either land on heads or tails, and there’s a probability of 50% either way. But if you start a coin-toss experiment, your first ten tosses might well look like this:

Heads, tails, heads, heads, heads, tails, heads, tails, heads, tails

Notice that heads came up 6 times in 10 tosses. If you stopped the experiment there, you might conclude that heads has a 60% chance of showing up in any random coin toss. But if you continued your experiment, chances are that regression to the mean would kick in, and after 100 tosses, you’d be tempted to conclude that it is 50%.

Coin toss
Coin toss

And after 1,000 tosses, you’d be absolutely sure that the chance of either side of the coin landing face-up was exactly 50%.

There are other types of averages, such as geometric mean, harmonic mean, and so on. All these are “central tendency estimates,” which serve to determine the basic characteristics of a sample or its distribution.

The distribution of a sample shows us how far each individual element of a sample is from the sample mean. Let’s dig into that a little more.


Depending on the sample size and the distribution type, the number and distance of your data’s outliers can vary.

This is represented by the curve of distribution, plotted on a graph. There are many different distributions; the most frequently used is called a bell curve (for its shape) or “normal” distribution.

The main characteristic of normal distribution is that its arithmetic mean and median (median being the element that lies in the exact middle of all of the data points) are the same. Normal distribution is most commonly used in testing, and all assumptions start with normal distribution.

Example of normal distribution
Example of normal distribution (Image source)

One of the most important properties of distribution is variance.

Variances can indicate the accuracy of your data, or provide a range

Variance is best illustrated and then explained. Look at the following picture:

Normal distribution and variance
Normal distribution and variance (Image source)

You see the tallest curve in the graph — the blue one, right? By its shape, you can instantly tell that all the observations in that sample are closely clustered around the mean. Can you guess what its variance is? Large or small?

If you answered small, you’re right.

Variance measures how much each individual data point differs from the sample average. It is used to determine how precise the average is. If we have many outliers, like in the orange curve on the graph, the mean will be a less accurate predictor of each individual element.

Let’s go back to our household purchasing power example.

We’ve collected 1,000 observations in our sample of the population, and discovered that the average purchasing power of a household is, say, $1,000.

—> If we calculate the variance at 9%, we can safely conclude that the purchasing power of the population is between $910 and $1,090.

—> However, if the variance is larger — say 50% — then the purchasing power of an individual household can be anywhere between $500 and $1,500.

If we detect very large variances in our data, it may mean the sample we used was deficient, and we should improve our sampling technique or include more observations.

Now back to A/B testing

So, now that we’ve got the basics out of the way, we can proceed with what we need to do to successfully conduct testing.

What is testing and why do we use it? In a statistical sense, testing serves to distinguish between two samples and determine the difference. In a practical sense, this means taking two variations of a web page and comparing the results they create in terms of engagement, views, conversions, etc.

The point of the test is to find out which of the variations performs better in some key measurement. Usually this is conversion rate. To produce meaningful results, this test must be conducted in a way that creates a meaningful comparison.

We will now see in practice all the theoretical concepts we just discussed, and… gulp… introduce a few new ones.

Significance gives you confidence in your results

In statistical terms, significance means ensuring that the effect we measured is not due to chance.

In terms of A/B tests, significance basically means ensuring that the A/B tests you conduct are meaningful — or in other words, that the difference between the two tested variations is real and reliable.

The usual level of significance we look for in testing is 95%. This means that the likelihood of the test results being due to chance is only 5%.

Note that the size of your sample will depend on the level of significance you’re looking for. The higher the significance you’re aiming for, the larger sample size you’ll require to confirm it.

Let’s do an example A/B test right here, right now

Say we’re conducting a test on a product page. We want to improve the page’s conversion and get more visitors to actually buy that product.

The existing page converts at a rate of 2.54%, and we want to increase that to 4% (which is the average conversion rate of other products on the website).

To decide how to test, we need to know the following elements:

1. Expected effect

In this case, the expected effect is equal to the difference between 4% (the expected/anticipated conversion rate) and the base 2.54% conversion rate. The difference between those two percentages is 1.46% — but since we are operating in relative terms, we want the percentage of lift achieved.

Expected effect = (Target conversion rate -Base conversion rate)/target conversion * 100

So we have to take one more step, and divide the difference by the base rate and multiply by 100 to get the percentage of relative lift.

Once we calculate that, we know that we’re expecting a lift of 57%. This is the effect we look for.

Generally, the higher the effect, the smaller sample size we need to detect it. This is the reason to go for bigger effects in testing, especially on sites with lower traffic.

2. Sample size

As we’ve seen, the second element that influences sample size is statistical significance, which is in direct proportion to sample size. We will always aim for tests with 95% significance, as anything lower increases the amount of error.

You will often read that tests need to be run until they reach 95% significance, and this is true. What you won’t know if you stop reading there is that this will happen multiple times in the process of testing. If you do lots of tests, you will notice that many of them reach significance at one point, only to drop below the 95% mark at other points.

Back to our example test.

When we plug these elements into Evan Miller’s excellent sample size calculator, we get “2,577”. This is how many visits each page variation needs to receive for us to deduce with 95% significance that one variation has won over the other.

And now, time enters the picture.

How long should you run a test?

Test duration is a function of two variables. The first is, of course, the required sample size. If your website has a large amount of traffic (in the hundreds of thousands of unique visitors per month), you can safely run any test.

Conversely, if the amount of traffic on the site is lower, testing will need to be oriented toward bigger gains or run for longer periods.

To explain the second variable, let’s return to our NYC household purchasing power example from above, where we explained sampling. The same principle applies here — AKA if you start testing with an inadequate or too-small sample, some tests will reach significance almost at once.

If you only test your variations on a sample of 50 visitors, you will not be able to tell with certainty which variation won. This is called a false positive. It simply happened that the first observations showed you a certain result — but the reason behind that result may be pure chance or some outside influence you didn’t foresee.

But even if we test the entire required sample size, or all 2,600 people on the first day or testing, we still can’t be sure we won’t end up with a false positive.

The reason may simply be something outside of our control. People may just come to the page on other days. Or we started testing at the end of the month, and the customers were out of money, or something else altogether.

In order to remedy this issue, we need to run our test for long enough to eliminate any outside influence. The changes we made should be the only factor that is different (i.e., the only independent variable), in order to make sure that any test success is due to OUR actual website changes or improvements. Therefore, it’s generally recommended to keep the tests running for at least two to three weeks, and/or through one to two complete sales cycles.

Due to the nature of how testing tools operate, running the test for longer than four weeks exposes us to the risk of sample pollution. Sample pollution is the result of repeating observations in a test, unbeknownst to us.

Why would that happen? Since testing tools rely on browser cookies to mark which visitors have seen which variation, after one month, those cookies may get deleted. If that happens, the same visitor would see a different variation of the page, spoiling the results. For this reason, most tests are called at the end of a four-week period.

Common problems with testing

Some of the most frequent problems related to testing are tied to the testers’ statistical errors. For example, selecting too small of a sample size. People sometimes assume that 100 visitors is enough, but it is not! You should always determine your sample size in advance to avoid this error.

Calling the test too early is the second, possibly even more common error. Regardless of the sample size you tested, you should run the test for a long enough period of time. As we explained earlier, that way you will eliminate outside influences.

Testing for small gains is an error that is easy to make, and it’s the most frequent reason for inconclusive tests. If you test for a small gain, you increase the chances that your testing tool will not be able to detect the change. This results in an inconclusive test (and you’re left wondering what to do next).

The final thing to keep in mind is the mantra of statistics, which we can’t say enough: “Correlation does not imply causation.”

This means that if two events happened at the same time or close to one another, it does not mean that the second one was caused by the first one. It could be a coincidence. Only sufficient testing can establish causation.

Don’t let this scare you away from testing

A/B testing is seemingly simple and (before you read this post) you might have been tempted to think there isn’t much to it.

The sooner you accept the fact that it is a bit more complicated, the sooner you will get better results. Creating your hypothesis and new page variations and setting up your test is just half of the job. Conducting the test properly is the only way to make sure the results you get will be true and sustainable.

As you’ve seen, most errors in testing come from lack of knowledge of statistics. While you don’t need to be an expert in statistics to conduct tests (though it couldn’t hurt), you do need to know the basics.

Here’s a checklist to help you run better, more reliable tests:

  1. Determine your necessary sample size.
  2. Test for at least two or three full weeks.
  3. Identify possible outside influences (weekends, holidays, shopping patterns, etc.)
  4. Resist the temptation to call a winner after you see a “trend” emerge. There are no trends in statistics.
  5. Correlation is not causation! Repeat this before, during, and after the testing is done.
A/B Testing Statistics

To make the experimental tools we use in A/B testing more transparent and reduce the risk of relying on false conclusions, we must learn their basic mechanics. Since A/B testing at its core is a statistical method, that means we need to learn some basic statistics and their role in the A/B testing process.

Google Analytics Audit Frequently Asked Questions

Google Analytics audit frequently asked questions

Published by

Edin is a Senior CRO Consultant. Edin is into Google Analytics and testing (any A/B testing tool really) and likes to write about it. You can follow Edin on Twitter.

November 8, 2017

I am the owner of my ecommerce store, can I do the audit myself

If you know how to use and setup Google Analytics, you can do most of the audit by yourself. Depending on your skills with JavaScript and knowledge of Google Tag Manager, you can potentially solve most of the issues. In fact, as an owner of a website, you would be uniquely equipped to know what you want to measure and what aspects of the website are most important to you. On the other hand, as an owner, you may have other issues that demand your attention too and setting up and fixing Google Analytics may not be the most efficient use of it.

When do I need to do an audit?

You should do an audit whenever you are not certain that your Google Analytics data corresponds to a real world data. You may spot discrepancies in reported and actual revenue, for example, or notice the acquisition channel data does not match your expectations. In short, whenever you think the data is not correct, run an audit of your account so you can ascertain your analytics is accurate. In addition, whenever you have different people making changes to your account, you will do well to run an audit.

My Google Analytics works so why should I do an audit?

The fact that you Analytics account appears to work and shows visits to your website does not signify it is without problems. Maybe you rely more on Social acquisition channel than you used to? Or maybe you started new paid search campaigns? Besides, Google Analytics is constantly evolving, so you may be missing some new features potentially adding a wealth of new data you can use to improve your website even more.

How often do I need to do Google Analytics audits?

There is no requirement to do audits, but you should audit your setup at least once a year to check if everything still works properly.

I want to audit my Analytics account – where do I start?

A good place to start an audit is always to make sure your Google Analytics snippet is properly implemented, filters and bot exclusion is turned on and demographics reporting features enabled. Secondly, make sure you have a test view. Read our audit guide for more.

I do not use any PPC campaigns, but my audit tool reports problems with PPC, what do I do?

If you used an automated audit tool, such as Online-Metrics, a tool we recommended in our article, it may happen that your AdWords integration and Branded and Generic paid traffic checks fail. This happens because a tool does not detect any of this data in your reports and assumes that they were not linked properly. In fact, since you do not use those means of acquisition, you can disregard those warnings for the time being. However, if you decide to start using PPC at a later date, make sure you follow the guidelines and ensure this traffic is properly reported and attributed.

Although the majority of my traffic does not come from social networks or referrals, I still receive fail mark in audit tool?

If referral and social traffic do not represent an important part of your acquisition strategy this issue will appear since in all probability a small amount of social channel will appear as a referral. This is not an issue as your traffic acquisition relies on entirely different channels and mediums. However, as in the case of PPC, you should keep a close eye on your acquisition channels and if your social traffic picks up, solve the issues that may appear.

I have set up goals in Google Analytics, but I still can not see funnel visualization?

Adding a goal by itself will not result in funnel visualization being displayed on your Analytics Conversion report. However, for funnel visualization to work, you need to set up a destination goal. For full instructions on how to set up goals, you can check out our blog article.

I have added a filter to my main view by mistake, what to do now?

If you happen to add a filter or change any other setting affecting your data in Google Analytics in the wrong view, the best (and probably the only) thing you can do is to remove the filter or alter the setting as soon as you realize the mistake. The longer it stays on, the more data you will lose.

Can I use the audit process for setting up a fresh Google Analytics account too?

Of course, you can follow the checklist for audit in setting up the new account. However, some aspects of an audit will not be possible to do unless you have actual data in your account. To properly estimate the setup of your PPC traffic, channel grouping, goals, and events, you need to have enough data to make the judgment.

What are some of the most common issues to look out for while doing an audit?

In the course of the audit, the most frequent issues involve bad event code, goals that do not reflect the purpose of tracking the performance of the website and non-existent measurement plan.

While automated audit tools will report no problems with event set up if they detect any event code on your site, they will not be able to determine whether this code is, in fact, correct or does it serve any useful purpose. This is an aspect of your account setup that you need to do manually.

Google Analytics audit frequently asked questions

This is the frequently asked questions (FAQ) section of the Google Analytics audit guide. We have listed the most common questions we receive related to GA audits and hope these will be of use to you.