If you want to know when new articles go
online,
subscribe to the WebWord.com
Usability Newsletter!
A
Proposal for Evaluating Usability Testing Methods: The Practical Review
System (PRS)
Article by John
S. Rhodes
Abstract
The purpose of this article
is to explain the Practical Review System (PRS). The PRS is an outline of
28 characteristics that can be used to understand any usability method,
thereby allowing any individual to decide between methods. This solves
many of the problems associated with understanding and explaining
usability methods.
How Can You Explain It?
I've been doing
usability work for at least five years but I barely understand it. Usability
is so screwed up in so many ways, I honestly don't think any person
understands it. There are no gurus, folks. There are some talented
people, that is certain, but I don't think any person truly understands
usability. It is the slippery eel of the production process. Face it,
understanding usability is not easy. And if it is hard to understand, of
course it is hard to explain to other people.
I'm less concerned
with understanding usability myself than with getting other people to
understand it. I want designers and developers to understand usability. I
want the engineers of the world to understand usability and I want them to
apply it where it makes sense, at the source. If we drive usability into the
hands of the people building products and services, everyone benefits. Also,
the usability specialists of the world will be able to focus on higher level
problems. Rather than focus on the little problems here and there, they
could attack bigger problems.
The Monkey
Test
If it is hard to
understand, and even harder to explain, where do we start? The answer is
pretty clear to me. There is need to give people outside the realm of
usability a tool to objectively evaluate the various usability testing
methods. The idea is that if designers and developers are given a tool to
easily decide between various usability methods, usability will get added to
the development process.
Why would
usability get into a product if usability was easier to understand? Simple. The
value of
usability is easy to understand, but usability itself is damn hard to
understand, explain and apply.
How do we make
usability easy enough for a monkey to understand and apply? We build a tool
to evaluate the various usability methods. We make it easy to compare and
contrast the methods with each other. We provide simple templates, simple
instructions, and simple decision tools.
The Practical Review System
(PRS)
In this section I
will describe the Practical Review System (PRS). At this point, I don't have
the system completely formalized, and it is definitely not populated. This
article is just the groundwork for the PRS and much more work needs to be
done to flesh out the details. Nevertheless, the outline of the tool should
provide us all with a common frame of reference.
Below I have
listed 28 characteristics that can be used define any single
usability method. The idea is to make the details of each usability method
available so that a person not familiar with usability can compare the
methods to each other, and then choose the best method based on the
constraints and needs of the project. At first glance, this will seem like
the foundation for an encyclopedia of usability. However, I will explain
that this is not fully accurate. Here are the characteristics.
-
Name --
Description of the method
-
Development
Lifecycle Stage -- When the test most relevant / effective
-
Number of Users Required
-- Number of users required to obtain sufficient / useful data
-
Type of Users
-- Novice, Expert, Mix; Applies to specific genders, races, ages, etc.
-
Evaluator Skills Required to Use the Method
-- Types of skills, knowledge, experience needed
-
Number of Evaluators Required
-- Number needed to obtain sufficient / useful data
-
Number of Evaluators Recommended
-- Number needed to obtain sufficient / useful data
-
Advantages of Method
-- Description of advantages in plain language; ROI
-
Disadvantages of Method
-- Description of disadvantages in plain language; specific costs
-
Level (or
Amount) of User and Evaluator Interaction -- High, Medium, Low;
passive / interactive
-
Data Recording
Method(s) -- Methods used to record useful data during testing /
evaluation
-
Total Testing Time Required
-- Years, Months, Weeks, Days, Hours, Minutes, Seconds required
-
Testing Time Per User
-- Years, Months, Weeks, Days, Hours, Minutes, Seconds required
-
Typical output from Test
-- Objective data, Subjective data; narrative, numbers, video, etc.
-
How to Run the Test
-- Basic instructions for running the tests
-
Related Tests
-- Pointers to similar usability methods
-
Example of Test
-- Template of a test; sample data; sample final output / report
-
Required Testing Materials
-- Video, laboratory, paper, 3x5 cards, microphone, etc.
-
Cost to Conduct Test
-- High, Medium, Low; actual estimated dollars to conduct the test(s)
-
References / Where to Learn More
-- Books, articles, authors, web sites, etc.
-
Type of System that Test Can Be Done On
-- Prototypes, actual systems, intranets, web only, etc.
-
Goals of Testing
-- Plain English description of what the test will produce; final
deliverables; ROI
-
Subjective or Objective Test
-- Estimation of experimenter bias; type of data generated by the test
-
Ease of Learning to Conduct the Test
-- Hard, Medium, Easy; e.g., Requires formal research experience?
-
Turnaround Time
-- Ability to produce results along the way? Typical schedule; payback
schedule
-
Focus of Evaluation
-- Narrow or Wide; Type of things that can be evaluated (e.g., button
vs. entire web site)
-
Focus of Task(s)
-- Real world or in the lab? Where can it be done?
-
Related Statistical Analysis
-- ANOVA, regression analysis, T-Test, Cochran's Q, etc.
The characteristics above overlap, and some might even seem to be
irrelevant. However, please be assured that I have chosen them carefully. I
expect I'll need to revise these characteristics once the methods are added
into the PRS template.
PRS Encourages Usability Standardization
I've started to
work on a few usability methods (e.g., heuristic analysis) and I've proven
to myself that the PRS works. I've been able to compare a few methods
because I have the PRS as a guide.
If you are a
usability specialist, or if you are interested in usability, you can
immediately apply the PRS to your own decisions, projects, and proposals.
You don't need me to do anything else. Simply look at the methods that are
available to you, identify the appropriate characteristics, and start
making decisions. I would also venture that the PRS characteristics are
sufficiently broad enough, yet powerful enough, to apply to other areas
where deciding between methods is difficult (e.g., information architecture,
psychology research, sociological research, and so forth).
The PRS is an
olive branch I'm extended to designers and developers of the world on behalf
of the usability folks. Of course it is subjective in many ways, but we can
al use it as a common framework for talking about usability. We can use it
to make usability more objective, and that is a good thing. We'll all be
able to better understand usability, and by its very nature, we can more
easily explain usability.
Phase Two and
Phase Three
As I stated above,
the first thing we need to do is create PRS profiles for all usability
methods. We need that first step, because it is like populating an
encyclopedia. But, as I hinted at, there is much more behind the PRS project
than what I have stated above.
After Phase One is
completed, where the methods are fully described in accordance with the PRS
characteristics, we need to move on to Phase Two. The idea of Phase Two is
to build a simple matrix that allows people to compare the usability
methods at a glance, in reference to the PRS characteristics. I can envision
a one or two page matrix whereby a designer or developer could quickly
determine which method(s) make the most sense. In some respects, this matrix
would act as a Table of Contents to the entire PRS list.
If that wasn't
ambitious enough, I've thought ahead to Phase Three. In that phase, we could
take the PRS data and the PRS matrix and turn it into an interactive
offline application, or a simple but powerful web application. Imagine
this. You visit a web site, answer a few questions related to your project,
and then you are given a list of best usability methods, along with
templates, analysis tools, and so much more. The Third Phase of the PRS
Project brings usability to the masses. It'll drive standardization and then
greater acceptance.
Epilogue
I'm interested in
working with serious individuals on the PRS Project. I've outlined the
vision, but now I need help. If you are truly interested, and you want to be
involved with this groundbreaking project, contact me.
Comments?
Please send them to me: john@webword.com
I want to know what you think about this article.
What next?
|