90% of All Usability Testing is Useless
Lane Becker on Adaptive Path claims that 90% of All Usability Testing is Useless: “Ninety percent of all usability testing performed on Web sites is useless. This is not to say that it doesn’t have a significant role to play in user experience design. When done right, usability testing will improve your Web site and your development process, but the current culture surrounding Web site usability testing is such that it rarely benefits the design. Worse, this misapplication can undermine the acceptance of this important technique throughout an organization.”
What do you think?
June 16th, 2004 at 1:36 pm
I’ve been thinking about this a lot recently. Glad Lane wrote this.
I think he is mostly right. However, we must consider the source. Adaptive Path is much more of a design organziation than a usability organization. My impression is that they do a good job with information architecture too. Not to speak for them, but I think that in their mind that the “proof is in the pudding” and that design shows this. That is often true, but not always true.
More on this very soon!
June 16th, 2004 at 8:08 pm
Good article, but he’s attacking a bit of a straw man.
His angle is really that everybody (or almost everybody) in a design team should participate in usability testing so that the entire team internalises usability principles. This is instead of relying on outside expertise, which is expensive and tends to be tacked on at the end of the design process which makes it difficult to utilise.
The usability pros should heed this and start to think in terms of skilling up their clients. IOW, try and make themselves redundant!
Test early and test often!
June 17th, 2004 at 12:29 pm
Interesting article, but totally disagree. When you can come up with say 3 usability issues on an intranet site for example and the improvements from this can save employee time, thus increase productivity, you can…based upon a weighted hourly rate, determine cost savings to an organization.
The same goes for an ecommerce site, when you improve the purchase path and thus increase sales.
I think it is sometimes hard fro designers to have non-designers tell them that their work has human interaction flaws…. I would take it personally if someone’s job was to do nothing be essentially critique my work.
June 18th, 2004 at 12:55 pm
This article seems to me to be confusing and confused.
Confusing: At the start, the author says that the culture surrounding usability testing is the key, which is a different message to the sensational title of the article. A sort of “Okay, you got our attention but you didn’t provide compelling insight”. Next, the historic reference to usability testing (the “web is not software”)seems to imply that the technique is outdated or irrelevant (backing up the sensational title). But, further along usability testing gets praise. So which is it?
Confused: The real point of the article seems to emerge next which is that [organizational] insight is the key and context is more important than metrics, such as time on task. Note, the author assumes that time on task is standard with every usability test (not so if your metrics are set up to reflect context, ie. time on task is not a blanket usability testing metric). If you’re clear about what you are testing, this is not an issue. Are your metrics more qualitative or quantitative?
There is a call for usability testing to be used iteratively which is spot on.
The themes of the article include:
“Doing quantitative usability testing can result in useless results 90% of the time”.
“Integrating usability testing into your design process, including stakeholders and setting up a usability lab internally can result in 90% more success with usability testing”.
One of those would have been a better title, instead of throwing a bunch of broad issues under a provocative title. Help us be less confused.
June 21st, 2004 at 11:32 am
My thoughts on the subject.