Research Tradeoffs
This weekend I took a survey. I’m kind of an odd duck in that I actually enjoy taking most surveys. The reason is that I like to analyze surveys as I’m taking them. What data do they want? What questions do they have? What’s the real point of the research?
The survey I took this weekend was long and seemed dreadfully redundant. I felt like I was answering the same questions over and over. It was dull. To make matters worse, there was no place to enter comments. There were no free form fields. I like to leave comments. It seems important to add some humanity to my surveys.
I wrote an email to the group that provided the survey and told them I thought the survey seemed to ask redundant questions, and I made it clear that I wanted to add comments. Within no time a response came back. The questions seemed redundant because they were using the conjoint analysis technique. I was told that this technique does a very good job of quantifying tradeoffs. I was told that survey designers excluded comment fields to maintain costs and confidentiality. Fair enough.
Questions:
Is there an optimal length for surveys? I’m guessing that it depends.
Are longer surveys intrinsically better than shorter ones in terms of data quality?
How can you balance survey length with data quality?
Do survey designers need to worry about user experience?
What are some examples of great surveys? How about survey best practices? (More…)
Possibly related: Can the effect of word-of-mouth be measured?
May 8th, 2006 at 4:27 pm
Hi John,
I have to confess that I also enjoy taking and reverse engineering surveys for the same reasons. And the funny thing is that the more I take the more I seem to attract.
Shorter is definitely better for questionnaires, especially on the web, where random sampling means you’ll usually be interrupting someone engaged in a particular task. I think the data quality issue is one part of it, but there’s also the amount of data you generate. Don’t forget that you’re going to have to *analyse* and cross-tab all the responses from that 50 question monster. So shorter means you can discover the findings and, more importantly, act on them more quickly.