Comments on: During a usability testing how do you know if users are telling you the truth?

by: mirc

Tue, 29 Jan 2008 18:51:24 +0000

Thanks Best Regards

by: Dr. Pete

Fri, 10 Aug 2007 14:24:19 +0000

I see what you’re saying; one outright liar is just an anomaly, but an entire group of over-eager test subjects might skew your results enough to believe that you’re heading in the right direction despite some major issues.

In that sense, you’re better off with a more open-ended test approach. If you’re guiding your subjects and asking specific questions like “Do you like X?”, they may tell you what you want to hear. If you put the subjects on task (”Try to find a product you want and add it to your shopping cart”) and then just observe, the eager-to-please aspect won’t have nearly as much impact.

In other words, if you’re pretty sure that your test groups have a certain bias, there are probably procedural changes you can make to minimize that bias. That kind of procedural adjustment is going to be a lot easier and more reliable than trying to detect lies in individual test subjects. If you deal with lies on a subject by subject basis, you risk introducing even more bias (as you’re individualizing the situation too much).

by: John Rhodes

Fri, 10 Aug 2007 00:14:38 +0000

“Ultimately, I doubt most people are going to outright lie in a usability testing situation.”

No, but they might tell small lies which could be even worse.

In some cultures (help me out here, Dano), users work very hard to please superiors. That could drive false actions and activities.

by: Dr. Pete

Mon, 06 Aug 2007 20:30:25 +0000

There’s a ton of research in psychology around validity in testing, and there are about as many types of validity as there are people to write textbooks about them. The good news is that, if your testing procedure seems to have “face validity” (i.e. it seems reasonable on the surface), you’re usually ok. Testing experts have tried to devise all sorts of ways to subtly (and not so subtly) trick people into giving unbiased answers, but bias is incredibly difficult to unravel (and the trickery often creates entirely new kinds of bias).

Ultimately, I doubt most people are going to outright lie in a usability testing situation. People are probably going to tell you what you want to hear, to some extent, and the best you can do about that is try not to ask leading questions.

by: Sandy

Mon, 06 Aug 2007 16:28:43 +0000

Thanks for this post - its something I often think about when running labs (and focus groups etc)

Here is a good resource: Reading People by Demetrius and Mazzarella. She is a big shot jury selector, and in the book gives lots of tips about reading people. The main thing she says holds true here - look for things that are out of the usual, and intepret them in the light of the whole.

For example - someone may seem to be giving false smiles, but if its in the first few minutes of the test and she has just arrived at this new, strange place - then its quite understandable she is nervous yet wants to appear available. But, if the same person - one hour later - is still giving a false smile and saying platitudes, that would be unusual (most people have loosened up somewhat by then)….

by: Yarone Goren

Sun, 05 Aug 2007 09:22:17 +0000

John - maybe you have a natural ability to tell when people are lying or being insincere?

Check out this test, which shows you 20 mini-videos of people smiling and you have to determine whether or not the smile is real:
http://www.bbc.co.uk/science/humanbody/mind/surveys/smiles/

I found it fascinating.

by: Daniel Szuc

Sun, 05 Aug 2007 01:22:34 +0000

http://www.uxmatters.com/MT/archives/000183.php

by: Terry Bleizeffer

Sat, 04 Aug 2007 20:13:53 +0000

There’s a difference between not telling the truth and lying. IMO, lying is not an issue in usability testing. The issue is when users tell you things about themselves that they want to be true, but that aren’t. Like, “… now I would go carefully read the documentation…” or “… I carefully consider how usable a product is before I make a purchase decision…”

In other words, users think they are telling the truth when they say it.