Setting up a great AB test that gets you actionable results can be really quick and easy. It can also be a complete waste of time if you goof it up. So how do you not goof it up? You’ve got to channel some Bill Nye! Don’t know Bill Nye the Science Guy? Then it’s a little less fun, but you’ll manage. There’s a specific method to follow when you’re building any test that will get you actionable results every time. Blast back to high school science with me here- it’s the scientific method.
Seriously. Following almost the same steps you learned (and totally still remember, I’m sure) in high school will make you a top-notch designer of any AB test you might want to run. Stick around for a few minutes and we’ll walk it through and show you how each step is directly applicable to your testing needs. Let’s go!
Alright Wait, What is an AB Test?
Quick laying of groundwork here: if you’re not familiar with AB testing, the idea is simple. Let’s run with an easy example: I wonder if my newsletter form would convert better at the top of the page than the bottom?
An AB test is the perfect tool to answer this question. Using an AB testing tool, you can easily create a mirror of the page your newsletter form is on, with the only difference in the mirrored page being the form at the top of the page instead of the bottom. Every other part of the page is exactly the same. Your AB testing tool will split the traffic to your website, so that half of your users see the original page (Page A) while the other half sees the mirror (Page B).
The tool will also collect data for you and give you a report. You can look at that report to see which version of your form, A or B, converted better. And now you know!
Setting Up a Great AB Test
Step 1: Choose Your Question
- Would my form convert better at the top of the page than the bottom?
- Would my form convert better with 3 fields or 5?
- Would I get more click-throughs to my blog with a ‘Recent Posts’ widget on the home page?
- Would I sell more widgets if I changed my menu layout?
The most important part of this step, the absolutely critical thing, is to settle on 1 question. Just 1. Only 1. No more, no less. 1 is the number thou shalt count, and the number of the counting shall be 1. Seriously though, if you try to test more than 1 thing at a time the results you get are going to reflect both changes, and there’s no way to know which change influenced what, how.
Step 2: Identify Your Variable
Hey, this is easy. Whatever you want to test is your variable in an AB test. Let’s choose the question “Would my form convert better at the top of the page than the bottom?” for ours. The variable here is the location of the form. Top, or bottom? That’s what we’ll test.
It’s so important to just ask 1 question because you need to stick with testing one variable at a time in an AB test. There is a method for testing multiple variables. It’s called a multivariate test. Some of the tools I linked above handle multivariate testing, so if you just must test multiple variables, you need to look into a tool that handles those types of tests.
Step 3: Design Your Test
Ok, we have a question in mind and we know the variable we want to test. We want to know if our form does better at the top of the page than the bottom, and the location of the form therefore is our variable. Let’s say for sake of argument that the form is at the bottom of your page originally.
Using your AB test tool of choice, make a mirror of the page in question. Leave Page A completely alone, with the form on the bottom of the page. On Page B, move the form to the top of the page, and leave everything else alone! If you make any more changes, you done goofed. See above about asking 1 question and testing one variable only 🙂 …. Just for the record, Page A is now your control group, and Page B is your experimental group.
Step 4: Run Your Test
Two points to consider here:
- This could be considered a design thing, but you want to run your test on as many people as possible. Split your site traffic 50/50 between Page A and Page B.
- You want to run your test for as long as is reasonable.
If you cut your test too short, your data will be skewed (sort of like if you only poll a small number of people). Ideally you’d run your test for a very long time, but you have a business to run. How much time you can devote to it is up to you, just realize that the shorter the time the test runs, the less reliable the results will be. All sorts of factors can come into play with a short test: what day of the week you ran it on, what else is going on that week, etc. The shorter the test, the more short-term irregularities like that will throw off your data.
5) Analyze Your Data
Any of the AB test tools I’ve linked above have some form of reports and analytics. Put them to use. Consider factors outside the scope of your test that may have influenced your results before you come to a hard and fast decision as well.
For example, I may see that my form converted better at the bottom of the page in this test, but then again I realize that the top of my page is already cluttered with lots of widgets and menus. Maybe I should clean up the top of my page and then run the test again?
So there you go, the scientific method in your AB testing. Next week we’ll look at how algebra and physics can be used to… ok no, no we won’t. We’ll stop with this. But on a serious note, thinking through your AB test scientifically does not take a lot of extra mental bandwidth, and produces results that are actionable and reliable. It’s honestly the only way to do it and not get crap for results. If you have any questions, please air them out in the comments!