Comments on: Simple Research Advice: A / B Testing http://www.webword.com/wp/2006/01/14/simple-research-advice-a-b-testing/ The usability blog of John S. Rhodes Fri, 21 Nov 2008 09:17:43 +0000 http://wordpress.org/?v=2.0.3 by: emmanuel http://www.webword.com/wp/2006/01/14/simple-research-advice-a-b-testing/#comment-11954 Mon, 27 Feb 2006 13:03:53 +0000 http://www.webword.com/wp/2006/01/14/simple-research-advice-a-b-testing/#comment-11954 To illustrate the concept of A/B testing, I have developped a simple script, that you can install easily on your site. http://galide.jazar.co.uk/2006/02/ab-testing.html To illustrate the concept of A/B testing, I have developped a simple script, that you can install easily on your site.

http://galide.jazar.co.uk/2006/02/ab-testing.html

]]> by: Dana Todd http://www.webword.com/wp/2006/01/14/simple-research-advice-a-b-testing/#comment-11837 Fri, 10 Feb 2006 00:00:17 +0000 http://www.webword.com/wp/2006/01/14/simple-research-advice-a-b-testing/#comment-11837 You may be interested in a new beta launch that we've put out there - a new A/B tool that will let you easily control the percentage of distribution manually (or you can let the system optimize on the fly). Even if you do the auto-optimization option, you can still set a "ceiling" on the distribution so that you control the spiral effect we sometimes see in auto-optimized system (where it too quickly becomes a self-fulfilling prophecy and doesn't give the other pages a chance to catch up). If anyone is interested in trying it out, we're in soft launch now and a free trial offer: http://pagelab.sitelab.com. Would love to hear your feedback and ideas for next-gen! You may be interested in a new beta launch that we’ve put out there - a new A/B tool that will let you easily control the percentage of distribution manually (or you can let the system optimize on the fly).

Even if you do the auto-optimization option, you can still set a “ceiling” on the distribution so that you control the spiral effect we sometimes see in auto-optimized system (where it too quickly becomes a self-fulfilling prophecy and doesn’t give the other pages a chance to catch up).

If anyone is interested in trying it out, we’re in soft launch now and a free trial offer: http://pagelab.sitelab.com.

Would love to hear your feedback and ideas for next-gen!

]]> by: Eric Hansen http://www.webword.com/wp/2006/01/14/simple-research-advice-a-b-testing/#comment-11627 Tue, 24 Jan 2006 18:12:23 +0000 http://www.webword.com/wp/2006/01/14/simple-research-advice-a-b-testing/#comment-11627 Thank you for the helpful article! Here are a few other suggestions that your readers might consider for basic A/B testing: - Control your risk - depending on how "extreme" your new page/product/content is, try running the test on just a portion of overall traffic. For example, if you've got 3 page alternates that you'd like to test, split off 50% of overall traffic to go towards the the test, and then within that 50%, randomize the visitors into the three page alternates. The benefit here is that you're better able control the risk/downside if one or more of your alternatives turn out to be stinkers (i.e. your risk exposure is only 1/3 * 50% = 16% for each of the alternatives.) - Test running time - assuming you're getting a decent amount of traffic to your site, consider running an A/B test for at least a week, ideally two weeks. This helps to minimize time bias that may occur if you're only testing visitors during weekdays, for example. Of course if you're getting alot of traffic, you may find that your test results start to look conclusive in just a few days. In this case, consider the suggestion above to reduce your overall test to 25% or 33% of total traffic. That way, you "spread out" your test over a longer period of time to balance out time bias. It's not uncommon to see seemingly conclusive results after a short period of time... but statistics can be misleading. To avoid the trap of making a hasty decision about what's working (and what isn't), try to wait until your best-performing page alternative has produced at least 100 "conversions". Happy testing :) Thank you for the helpful article! Here are a few other suggestions that your readers might consider for basic A/B testing:

- Control your risk - depending on how “extreme” your new page/product/content is, try running the test on just a portion of overall traffic. For example, if you’ve got 3 page alternates that you’d like to test, split off 50% of overall traffic to go towards the the test, and then within that 50%, randomize the visitors into the three page alternates. The benefit here is that you’re better able control the risk/downside if one or more of your alternatives turn out to be stinkers (i.e. your risk exposure is only 1/3 * 50% = 16% for each of the alternatives.)

- Test running time - assuming you’re getting a decent amount of traffic to your site, consider running an A/B test for at least a week, ideally two weeks. This helps to minimize time bias that may occur if you’re only testing visitors during weekdays, for example. Of course if you’re getting alot of traffic, you may find that your test results start to look conclusive in just a few days. In this case, consider the suggestion above to reduce your overall test to 25% or 33% of total traffic. That way, you “spread out” your test over a longer period of time to balance out time bias.

It’s not uncommon to see seemingly conclusive results after a short period of time… but statistics can be misleading. To avoid the trap of making a hasty decision about what’s working (and what isn’t), try to wait until your best-performing page alternative has produced at least 100 “conversions”.

Happy testing :)

]]>