During a usability testing how do you know if users are telling you the truth?
I’ve been trying to figure out ways to tell if users are actually giving me good data or not. What I mean is that if you’re doing usability testing, don’t you want to know if users are actually telling you the truth? Don’t you worry about data integrity?
Look, the speak out loud method entirely depends on user self reporting. If they aren’t telling you the truth then how is that data any good? If they are telling outright lies, then you’re not getting their “real” perceptions and feelings. They are cloaking.
So maybe the data isn’t all bad. But hopefully you’re tracking with me here. I think “good” data is dependent on accurate and truthful self-reporting and truthful responses to questions from folks running a UX study.
Now that I’ve sounded the alarm, I need to provide a solution, right?
Well, I’ve started doing some research. If you want to catch malingering, you’ve got to understand it. Here’s one good resource I’ve found:
How to Detect Lies – “The following techniques to telling if someone is lying are often used by police, and security experts. This knowledge is also useful for managers, employers, and for anyone to use in everyday situations where telling the truth from a lie can help prevent you from being a victim of fraud/scams and other deceptions.”
How to Detect Lies (WikiHow) — “Watching facial expressions in order to determine whether a person is lying might just save you from being a victim of fraud, or it could help you figure out when somebody’s being genuine. Jury analysts do this when assisting in jury selection. The police do this during an interrogation. Of course a polygraph does this, but it is a little heavy to carry with you. Therefore, you have to learn the little facial and body expressions that can help you distinguish a lie from the truth.”
So, the next time you think you’re getting great data from your users, consider if they are really telling the truth. Learn to detect false statements and complete lies. This could be the “magic factor” which increases the validity of your testing results.
Think about it.
August 4th, 2007 at 4:13 pm
There’s a difference between not telling the truth and lying. IMO, lying is not an issue in usability testing. The issue is when users tell you things about themselves that they want to be true, but that aren’t. Like, “… now I would go carefully read the documentation…” or “… I carefully consider how usable a product is before I make a purchase decision…”
In other words, users think they are telling the truth when they say it.
August 4th, 2007 at 9:22 pm
http://www.uxmatters.com/MT/archives/000183.php
August 5th, 2007 at 5:22 am
John - maybe you have a natural ability to tell when people are lying or being insincere?
Check out this test, which shows you 20 mini-videos of people smiling and you have to determine whether or not the smile is real:
http://www.bbc.co.uk/science/humanbody/mind/surveys/smiles/
I found it fascinating.
August 6th, 2007 at 12:28 pm
Thanks for this post - its something I often think about when running labs (and focus groups etc)
Here is a good resource: Reading People by Demetrius and Mazzarella. She is a big shot jury selector, and in the book gives lots of tips about reading people. The main thing she says holds true here - look for things that are out of the usual, and intepret them in the light of the whole.
For example - someone may seem to be giving false smiles, but if its in the first few minutes of the test and she has just arrived at this new, strange place - then its quite understandable she is nervous yet wants to appear available. But, if the same person - one hour later - is still giving a false smile and saying platitudes, that would be unusual (most people have loosened up somewhat by then)….
August 6th, 2007 at 4:30 pm
There’s a ton of research in psychology around validity in testing, and there are about as many types of validity as there are people to write textbooks about them. The good news is that, if your testing procedure seems to have “face validity” (i.e. it seems reasonable on the surface), you’re usually ok. Testing experts have tried to devise all sorts of ways to subtly (and not so subtly) trick people into giving unbiased answers, but bias is incredibly difficult to unravel (and the trickery often creates entirely new kinds of bias).
Ultimately, I doubt most people are going to outright lie in a usability testing situation. People are probably going to tell you what you want to hear, to some extent, and the best you can do about that is try not to ask leading questions.
August 9th, 2007 at 8:14 pm
“Ultimately, I doubt most people are going to outright lie in a usability testing situation.”
No, but they might tell small lies which could be even worse.
In some cultures (help me out here, Dano), users work very hard to please superiors. That could drive false actions and activities.
August 10th, 2007 at 10:24 am
I see what you’re saying; one outright liar is just an anomaly, but an entire group of over-eager test subjects might skew your results enough to believe that you’re heading in the right direction despite some major issues.
In that sense, you’re better off with a more open-ended test approach. If you’re guiding your subjects and asking specific questions like “Do you like X?”, they may tell you what you want to hear. If you put the subjects on task (”Try to find a product you want and add it to your shopping cart”) and then just observe, the eager-to-please aspect won’t have nearly as much impact.
In other words, if you’re pretty sure that your test groups have a certain bias, there are probably procedural changes you can make to minimize that bias. That kind of procedural adjustment is going to be a lot easier and more reliable than trying to detect lies in individual test subjects. If you deal with lies on a subject by subject basis, you risk introducing even more bias (as you’re individualizing the situation too much).
January 29th, 2008 at 2:51 pm
Thanks Best Regards