Google Optimize, now in beta, is poised to rock the world of multivariate testing by integrating the capability to test with Google Analytics. Sounds great – right? Run your own tests with a tool you’re already familiar with?
So we took a very close look at Google Optimize to see exactly what it promises, and what it actually delivers.
What is Google Optimize
Google Optimize isn’t new, exactly – it’s Google’s replacement for an earlier (and severely lacking) experimentation tool within Google Analytics. With this latest incarnation, Google has decided to make an independent, full fledged experimentation tool. It relies in great part on Google Analytics data and can integrate results and inputs with Analytics (which is nice).
As with all Google tools, Optimize has a free version with basic features, and a paid version called Optimize 360 that is much more capable. However, as with other Google freemium tools, the free version of Optimize will suffice for many basic experiments.
Google Optimize has bundled Optimize as a part of the package that includes Analytics, Tag Manager, Surveys and Data Studio. All these tools are free and combine to make all of their data available in one place – which is a quite a differentiator. Google has been sharing invites for Google Optimize beta and if you have not yet received yours or haven’t applied for one yet, you should do it.
Here is a quick look at the most important stuff.
The free version of Google Optimize is as basic as tools get, but there’s enough there to conduct a viable testing program with the most common types of tests, including A/B, multivariate and redirect testing.
Setup is relatively simple. Google Optimize features an integrated WYSIWYG webpage editor and code editor in the form of a Chrome addon, making creating variations easy. As is the case with other Google Tools, if something goes awry, there is a diagnostics tool to help you identify what went wrong with installation.
Perhaps the standout feature of Optimize is its ability to integrate with other Google tools – a natural consequence of the entire spectrum of tools being made by one company.
Integration enables you to have your quantitative reports, experiments, results tracking and reporting in one tool. Addition of Data Studio, a separate product being rolled out by Google and allowing for visual enhancements to Analytics and Optimize reports by building custom dashboards and reports.
Now let’s look under the hood.
The method Optimize uses to conduct experiments is bayesian statistics. This makes it different from other major competitors who offer a choice between frequentist and bayesian methods.
The definitions of bayesian statistics and frequentist methods are mind-numbingly technical (feel free to Google), but if I had to explain the difference between bayesian and frequentist methods to my grandma, it would go like this:
Let’s say you lose your keys in the house, but you have a key locator on your keychain that will beep when you press the appropriate button on your phone. You press the button, and your keychain beeps.
The question is: What area of your house should you search?
You hear your keys beeping and you have a mental model of your house to help you identify from where the sound is coming. Therefore, you can infer which area of the house your keys are in when you hear the beep.
You can hear your keys beeping, and you have a mental model which helps you identify the area from which the sound is coming. But you also know where you’ve lost your keys before. Now you can combine both inferences – the current beep, the mental picture of your home, and where you’ve lost keys in the past – to identify where you should search for your keys first. (We can thank this guy and his grandma for this explanation)
According to most experts, Bayesian statistics offer more accurate results. That said, Optimize is the first tool to use Bayesian statistics exclusively and the interpretation of results will require getting used to.
The free version and the paid version differ in a few small ways, but here’s what’s really limiting:
- The free version allows only three concurrent experiments, but the paid version offers unlimited.
- You can’t use GA Audience data for experiment targeting in the free version, but you can in the paid version. Not having access to GA Audience data makes the free Optimize much less versatile and reduces targeting to what you can achieve by using experiment conditions within the Optimize itself. You will still be able to use Google demographics data and custom user attributes, though.
Another difference – but one that really doesn’t mean much – is that the free version limits variations on multivariate tests to 16. I say this doesn’t mean much because you wouldn’t want more than 8 variations anyway – any more than that and results become unreliable.
Some of these limitations are here to stay, but others might just be because Google Optimize is still in development. But Optimize as it currently stands is a great tool for beginners to start up their testing programs.
Our conclusion: Use it!
A brief summary of Google Optimize features
- Three types of tests
- Integrated editor for making variations
- Integration with Google Analytics and Data Studio
- Uses bayesian statistics
How to install it
As with all Google products, the installation procedure begins by creating an account for Optimize. The procedure is remarkably similar to both Google Tag Manager and Google Analytics. You select an email address tied to an existing Google Account (don’t have one yet? Congratulations for holding out this long, but go get one) and name your account. You can choose anything, as long as it makes sense to you. After checking what additional options you want to enable, which is easy to do with the help provided by Google (we, and Google, recommend you enable everything) and accepting the obligatory terms of service, you will get to the next screen.
Here you will be asked to name the container that will be placed on your website or other site you’d like to create experiments on. It is a generally adopted practice to take the site URL as container name. Once you do this you will get the container id number, similar to the id number of Google Analytics and Tag Manager.
Next you will need to enable Google Analytics tracking, without which Optimize will not function. Optimize uses Analytics to track how the experiments fare and create the reports, so the two accounts must be linked by inserting the Analytics id number in the field. Use the same property in Google Analytics as the website you want to experiment on.
Once you complete this step, you will receive an Optimize snippet, that must be implemented on the website. It is similar to the Google Analytics and Google Tag Manager scripts and there are detailed instructions on placement of the snippet within the code.
It can be implemented in one of two ways: by manually adding the code to the site or by using Google Tag Manager to add the code. If you choose manual installation, you must have the Google Analytics snippet implemented on your website already.
Because the Optimize snippet that is manually added consists of two parts. One line of code is added to your Google Analytics snippet like this:
and the other part of the snippet must be placed in your code, as high in the head part (and preferably above the Analytics snippet) as possible.
The last part is necessary to prevent a flicker effect in your testing. Flicker happens when the web page starts loading and the original or control shows up, immediately followed by the variation. This happens if the hiding script is placed in the body of the webpage.
Alternatively, Google Optimize can be deployed and implemented on the site entirely using Tag Manager. Google Tag Manager already has a predefined Optimize tag. All you need to do is to put the id codes of your Google Analytics property and the id number of your optimize account in the fields required and add the “all pages” trigger.
To make it easy and enable the use of the ids at any time, be sure to create constant variables with values being id codes. That way it is easier to insert these values whenever needed.
Google Optimize will verify every part of the process, except inserting the Optimize code itself.
One more step that is recommended by Google is to implement a hiding script. Google Tag Manager will not do it by itself, so you need to attend to it manually or by using Tag Manager.
Here is a play-by-play for how to do this.
To add a hiding script through the Tag Manager, just create a new tag and select ‘Custom HTML’.
Copy the script from the ‘View snippet’ in Google Optimize into the empty box and you are done.
To save it just select all pages trigger and the hiding script will be implemented on all pages.
Now for the Fun Stuff: How to create an experiment
If you followed all the steps, you should now have Google Optimize on your website. Now we can create some experiments and start optimizing.
“Really? It’s that easy?” you ask.
Of course not!
If you want your experiments to have any chance of success, you have to start with comprehensive research. Throwing spaghetti (in the form of experiments) onto a proverbial wall (your website) to see what sticks is the fast-track to saying “testing is lame. Never doing this again!” It’s not the test that’s lame… it’s the underprepared tester.
So let us return to setting up experiments in Google Optimize. This part is also very straightforward.
The first screen you see when you start Optimize and click on ‘experiments’ shows you a list of currently running experiments with information on active users and number of sessions recorded. Clicking on the running experiment brings up the full report.
The report contains the improvement delivered by variation (or lack of it thereof) and the number of days the test has been running, with number of total sessions that have been part of the experiment. This report is also available through the Google Analytics interface and you can find it under Behavior Report → Experiments.
To create a new experiment, simply click ‘Create Experiment’ button. You will get the dialog for experiment creation. You need to name the experiment and decide the type of experiment you want to run.
The A/B test is a straightforward split test. If you click on A/B test and give a name to your test, Optimize will open a draft interface. There you can create variations and assign the amount of traffic that will go to each one. Currently, Optimize allows only an equal amount of traffic for variation or simple split testing.
Multivariate testing looks at the matrix of variations on a single page in order to determine the best possible layout of a web page. The free version of Optimize allows for a maximum of 16 variations, which should suffice for most users.
The Redirect test allows you to test an entirely different page by redirecting 50% of traffic to a page that radically differs from the control one. Best used for split path testing.
Each of the types of tests allows you to create one or more variations. To create a variation, you can use either an internal WYSISWYG editor or an external one and copy the code to the editor. You can replace the existing code entirely, or just append the changes. The editor interface is simple and straightforward.
Once you have your variation in place, all you need to do is to decide on the objective you want to track. You do this through the objectives dialog.
You can add four main types of objectives to an experiment – Pageviews, Session Duration, Bounces and Events. You can have multiple objectives for one experiment. Always choose the objective that best reflects what you want to achieve with the change. For example, if you have a blog post and want people to hang out more on the page and read it all, the best choice of objective will probably be either session duration or scroll depth.
For an e-commerce site, the best metric would be the conversion rate, so you will need to have events and goals to measure the conversions.
Once you have made variation(s) and defined an objective, just click the “Start Experiment” button at the top right hand corner of the dialog box.
Once you start the experiment, you can verify if everything is displaying correctly. If everything checks out, you should now forget about it for about two weeks. Do not be tempted to sneak a peek into the results and draw any conclusions from them.
Tracking and reporting results
If you can’t resist but to check out on your experiment, you can get reports at any time. The starting screen of Optimize has a list of all the experiments currently running and by clicking it you access the reports on that experiment. There, you will see the following:
If you click on “Reporting” you will get more insights, such as the number of sessions that were part of experiments, the probability to beat the baseline and the probability to be the best. The last one only has meaning if you run the test with three or more variations.
Why would you want to use it?
Now that we have seen how Google Optimize works, why should we use it? After all, there are other tools available on the market and some of them have large market share and have been around for a long time.
First of all, Optimize is a very accessible tool. It is available to anyone who has a Google account. It is very easy to implement and start testing, with a very short learning curve. It is also conveniently integrated with the most popular analytics tool (Google Analytics), that incidentally comes from the same company.
Granted, this first edition is very limited. You cannot do bandit algorithm testing using Google Optimize, and you can only test three concurrent tests in the free version. Nonetheless, it has a significant advantage over other tools.
Best part: It’s free and has no hidden costs.
For now Optimize can be used as an excellent solution to jumpstart your testing program and get into optimizing your website quickly and relatively effortlessly. Once your needs outgrow the capabilities of Optimize, you can consider investing in more costly and more powerful solutions, such as Optimize 360, Optimizely or VWO.
And, perhaps, by that time, Google will add some more features and develop Optimizer into an even better starting tool. Either way, Optimize will probably bring about the same change in conducting experiment as the Google Analytics did in digital analytics. It is worthwhile to keep an eye on it. It could very well be the wave of the future.