Split Testing – Knowing is Half the Battle, Part 1

I get energized when clients and my SEO colleagues catch the vision of split testing changes to a website. The potential for conversion rate increases is huge, and it’s also great fun!

It’s not all rainbows and fluffy bunnies, however. There are challenges that can sabotage otherwise promising split testing efforts. Awareness of these challenges will help you avoid some of the common pitfalls.

Here’s Part 1 in a four-part series:

Your website needs robust traffic to support a reasonable testing cycle.

If your website is brand new, it’s likely that you don’t have enough traffic to make split testing worthwhile. The more time it takes to complete a split test, the less valid your test result is likely to be because external factors can influence results.

These external factors are things like shifts in market demand (seasonality is a common example of this) and external media factors.

Want to know how long your test might take to reach a valid conclusion? Here’s how to figure it out:

1. Figure out how many test combinations you want to run. If you want to test one variation, that’s two combinations because you always test against the current version.
2. Calculate how much traffic per day passes through the test page. More traffic means the test will take less time, everything else being equal.
3. What percentage of traffic do want to include in the test? Exposing only a fraction of your traffic to the test will reduce the risk of your test negatively impacting revenue, but also increases the amount of time needed for your test.
4. What is the current conversion rate of this page? Whatever your success metric is, you need to know the performance benchmark you’re trying to beat.
5. What is your expected rate of improvement? The improvement needs to be high enough to have a measurable impact on your business. If a 10% improvement doesn’t offset the cost of executing the test, you need to aim higher.

Google provides a free test duration calculator that does the complex math for you. Just punch in the numbers to see how many days your test will likely have to run. If you need to decrease the duration, you can choose to run fewer test combinations or increase the percentage of traffic included in the test.