Simple Research Advice: A / B Testing
Posted on January 14th, 2006 in Usability | 3 Comments »
The hosting provider behind Pet Comfort Products offers up a really great discussion forum. Members can chat with each other about many different topics. Everything from Search Engine Optimization to writing great content to finding the right niche to setting up an ezine. It is a nice little community and I’m having fun interacting with these folks.
I just wrote a mini-article for the “Submit My Site for Review” discussion group. For folks interested in site reviews, marketing, and Adsense optimization, this is some simple advice. Nothing amazing and not written perfectly. However, it is reasonable advice. Here’s the posting…
–
As a usability specialist (webword.com) I’ve done a lot of research and testing over the years. Many people in this forum are asking for advice before they decide to take action.
My feeling is that action is better than planning if you have already done good work. If you are selling something, unleash your product. See how it goes. Don’t hold back: act now. Get real data.
Caveat: If you aren’t ready to let people see your pages or your product, that’s fine. But if you have something you think is good, let the market help you figure out what to do next. You’ll learn more from real customers than from reviewers.
If you want to figure out what works, consider setting up a very similar page with perhaps only 1-2 small changes. Drive an equal amount of traffic to both pages if possible (A versus B). Leave both pages up at the same time long enough to get a good sample size. One or two days of data won’t be enough, probably. I won’t go into the science behind the science…
In any event, keep the winner, drop the loser. As soon as you do this, tweak the page again and have another A versus B contest. Lather, rinse, repeat.
Advice is just that, advice. What really matters is how your advertisement performs in the real world. The A / B testing techinique is magic and I strongly encourage you to consider it.
For those folks paying attention, this is exactly what I recommend for advertising. Get some help proofreading and editing, but then fire away. Be sure you have multiple data sets so that you can make comparisions between your various efforts. In short, this basic technique generalizes to many situations.
I end by saying that site reviews are nice. There is value: fresh perspectives, proofreading, general satisfaction (preferences), lessons learned, and of course a little traffic bump. But, you probably care more about performance than preference. Folks here might like your site but that doesn’t mean they’ll buy. You won’t get the data you really want until you unleash your pages.
3 Responses
Thank you for the helpful article! Here are a few other suggestions that your readers might consider for basic A/B testing:
- Control your risk – depending on how “extreme” your new page/product/content is, try running the test on just a portion of overall traffic. For example, if you’ve got 3 page alternates that you’d like to test, split off 50% of overall traffic to go towards the the test, and then within that 50%, randomize the visitors into the three page alternates. The benefit here is that you’re better able control the risk/downside if one or more of your alternatives turn out to be stinkers (i.e. your risk exposure is only 1/3 * 50% = 16% for each of the alternatives.)
- Test running time – assuming you’re getting a decent amount of traffic to your site, consider running an A/B test for at least a week, ideally two weeks. This helps to minimize time bias that may occur if you’re only testing visitors during weekdays, for example. Of course if you’re getting alot of traffic, you may find that your test results start to look conclusive in just a few days. In this case, consider the suggestion above to reduce your overall test to 25% or 33% of total traffic. That way, you “spread out” your test over a longer period of time to balance out time bias.
It’s not uncommon to see seemingly conclusive results after a short period of time… but statistics can be misleading. To avoid the trap of making a hasty decision about what’s working (and what isn’t), try to wait until your best-performing page alternative has produced at least 100 “conversions”.
Happy testing :)
You may be interested in a new beta launch that we’ve put out there – a new A/B tool that will let you easily control the percentage of distribution manually (or you can let the system optimize on the fly).
Even if you do the auto-optimization option, you can still set a “ceiling” on the distribution so that you control the spiral effect we sometimes see in auto-optimized system (where it too quickly becomes a self-fulfilling prophecy and doesn’t give the other pages a chance to catch up).
If anyone is interested in trying it out, we’re in soft launch now and a free trial offer: http://pagelab.sitelab.com.
Would love to hear your feedback and ideas for next-gen!
To illustrate the concept of A/B testing, I have developped a simple script, that you can install easily on your site.
http://galide.jazar.co.uk/2006/02/ab-testing.html