4 Helpful Tips for Better Survey Responses

There was an instructive exchange in my recent interview with behavioral economist Dan Ariely. I asked him about how to apply lessons from his recent book, “The Honest Truth About Dishonesty,” to the world of commercial survey research. Could we apply some of those lessons to the way we structure the survey-taking task in order to maximize the accuracy of the answers that respondents provide?

In response to my question Professor Ariely gave a thoughtful response which included 4 distinct interesting, innovative and practical tips.

Here’s the relevant portion of the interview. I have divided his response into 4 sections containing each of the four tips.

Dana Stanley: There’s a lot of angst in the market research world about cheating among respondents taking surveys.  Many studies are done online these days, so there’s the anonymity of the online experience. And many studies are done using online access panels where people are doing surveys in exchange for incentives.  In many cases that’s a point system rather than cash. So what thoughts would you have that market researchers could think about to minimize cheating and maximize data quality when doing surveys?

Tip #1: Enable Greater Anonymity for Sensitive Questions

Dan Ariely: I think we need to separate here two types of cheating. There’s one type of cheating where I just don’t want to tell you the truth. You ask me a personal, embarrassing question. I don’t want to tell you the truth. And I basically look at this, and I say, I’m just going to lie here because I don’t want to admit to something.

So imagine, for example, that we ask people how often they cheat in golf. One of the chapters in the book is about cheating in golf. And it turns out people admit that they cheat in golf, but they say that other people are cheating much, much more than them.

Dan Ariely

Now personally, how do they know how much other people are cheating? What we think about is that this is actually a good way to get them to admit what they are doing. But you’re not admitting that you’re doing. You say other people are doing that.

So that’s one way to do it. I’m not asking you about yourself. I’m asking you about other people. And now, all of a sudden, you know what you know from your own experience. But all of the sudden, you don’t feel encumbered that you’re revealing something about yourself. That’s one question. And there’s other ways to do it.

By the way, one interesting approach, and I haven’t seen it online, but it works very well in physical cases. So imagine I want to ask you whether you had an affair with somebody else. And imagine you don’t want to admit it.

I say, please take this coin and flip it. And then look at the response, but don’t tell me. And if it’s tails, say yes, regardless of what the answer is. But if it’s heads, tell me the truth.

So what happens now is you flip the coin. You know it, and you tell me that you have an affair or not have an affair. Now if you it came on tails, you just say yes. If it came on heads, you tell me the truth. But me not knowing if it was heads or tails, I don’t know if you told me the truth, if you said you had an affair because it’s the truth, or you told you had an affair because it came on whatever side it was.

Now you can’t, in this method, you can’t know about the individual behavior. But you can know statistically about the behavior. So for example, if 50% of the people said they had an affair, and 50% of the people say they didn’t have an affair, you know that nobody had an affair. But if 75% of the people say they had an affair, you know what’s the percentage in the population.

Tip #2: Personalize the Researchers

But I think the thing that you are more concerned with is not this explicit lying. Because in marketing contexts, these are usually not the cases. It’s more about data quality. And it’s more about the fact that people are basically signed up to do a good job for you and to give you thoughtful responses and give you the information.

And it’s not that they really want to lie. They just don’t want to be bothered. They don’t want to read the instructions. They don’t care about it. And they’re giving you responses as fast as possible.

And sometimes it’s just about noise in the data. And you can say, OK, I can overcome it by running larger studies. But sometimes, if it’s a nuance that you really care about, and people don’t pay attention, it might be just a waste of time to do the study.

So now, let’s think about this point of being more steps removed from money or more steps removed from the consequences. How would you go about reducing those number of steps? I would argue that what you want to do is to first of all tell them who are they doing this research for.

And I would give them a picture. I would say, these are the three researchers, John, Linda, and Bob, and the other people who are interested in this research. And I would say to them, I would say something like, it’s really important for us that you take it seriously. You don’t have to do the study if you don’t want to. That’s OK. But if you do it, it’s really important to us that you’re doing it.

Tip #3: Respondent Pledge

And then on top of that, I want you to sign here to say that you understand that this is an important survey, and we are kind of entering into a social contract here. And we promise to ask you clear questions and to use it in a useful way. And you promise to answer the questions thoughtfully.

And it’s what we call cheap talk. What do you mean, I sign something here? There’s no ramifications. There’s no consequences for that. But we find that cheap talk is actually quite effective.

In fact, in our experiments, as you know, when we get people to recite the Ten Commandments, they stopped misbehaving, when we get people to sign an honor code. Even when you get a self-declared atheist to swear on the Bible, they start behaving better. So one thing you could say is you could say, let’s have moral and religious reminders on the survey, which would probably help to some degree. But it might interact with some things that we want to study, so I’m not always recommending it.

But there is kind of hope, and the hope is that people do want to be honest at some really important level. We don’t cheat enough. The question is, how can you get people. for these 15 minutes to understand what honesty is, to understand what their obligations are, and basically to follow through?

Tip #4: Expected Behavior Dashboard

And I’ll give you one other hint about this. You know the term social proof, right? Social proof is the idea that we don’t know what’s the right behavior, and we look at other people in terms of suggestions for what’s the right behavior.

And one of the things I really like in terms of mechanisms are dashboards, we did a dashboard study once with an online dating company. It turns out, people don’t know how to date online correctly. They don’t know how to write email and how many emails to write and so on. So they end up doing things like writing, “hey, babe.” And that doesn’t really work very well.

So we created in one study a dashboard for people. And we said, here is the behavior of good people. Here is what good people do who are successful in online dating.

Imagine you try to do the same thing in a panel, where you say, here is what people do. Here is how much time they take to read the instructions. Here is how much time they take to read the survey. And here’s how much time people take to read open-end responses. All of a sudden, you would create a social rule about what is acceptable and what is not acceptable.

And in the experiments on dishonesty we’ve done, we show that if we show people that other people are cheating in an egregious way, it helps everybody else cheat. Because all of the sudden, you say, hey, everybody’s cheating. It seems to be socially acceptable. People like me are cheating. And I can cheat as well.

And I think we basically need to do the opposite of that. We need to show people the good behavior, the desirable behavior, the behavior of good people around them. And with that create better norms for better behavior.

Click here to download the full interview from iTunes.

About Dana Stanley

Dana is the Editor-in-Chief of Research Access.

Speak Your Mind