Poor question design means questionable results: A tale of a confusing scale

I saw the oddest question in a survey the other day. The question itself wasn’t that odd, but the options for responses were very strange to me.

* 1 – Not at all Satisfied
* 2 – Not at all Satisfied
* 3 – Not at all Satisfied
* 4 – Not at all Satisfied
* 5 – Not at all Satisfied
* 6 – Not at all Satisfied
* 7 – Somewhat Satisfied
* 8 – Somewhat Satisfied
* 9 – Highly Satisfied
* 10 – Highly Satisfied

What’s this all about?  As a survey taker I’m confused.  The question has a 10 point scale, but why does every numeric point have text (anchors). What’s the difference between 1, 2, 3, 4, 5 and 6 that all have the same anchoring text?   Don’t they care about the difference between 3 and 5?  Oh, I get it, this is really a 3 point scale disguised as a 10 point scale.

With these and other variations on the theme of “what were the survey authors thinking?”  on my mind I talked to a representative from the sponsoring company, AOTMP.  I was told that the question design was well-thought out and appropriate, being modeled on the well-known Net Promoter Score.   Well of course it is  – like an apple is based on an orange (both grow on trees).  But not really:

1. The Net Promoter question is for Recommendation, not Satisfaction.  There were a couple of other similar questions in the short survey, but nothing about Recommendation. Frederick Reichheld’s contention is that recommendation is the important measure and also incorporates satisfaction; you won’t recommend unless you are satisfied.
2. The NPS question uses descriptive text only at the end points (Extremely Unlikely to Recommend and Extremely Likely to Recommend).  It is part of the methodology to avoid text anywhere in the middle in order to give the survey taker the maximum flexibility.  That’s consistent with survey best practices.
3. The original NPS scale is from 0 to 10, not 1 to 10.  Maybe that’s a small point, although the 0 to 10 scale does allow for a midpoint which was part of the the NPS philosophy.

Other than the fact that this survey question isn’t NPS, what’s the big deal?  Well, this pseudo 10 point scale really doesn’t work.  The survey taker is likely to be confused about whether there is any difference between “3, Not at all Satisfied” and “4, Not at all Satisfied”. Perhaps the intention was to make it easier for survey takers, but either they’ll take more time worrying about the meaning, or just give an unthinking answer, and the survey administrator has no way of knowing.  Why not just use the 3 point scale instead?  I suppose you could, but then it would be even less like NPS. Personally, I like the longer scale for NPS.  I don’t use NPS on its own very much, but the ability to combine with other satisfaction measures with longer scales (Overall Satisfaction and Likelihood to Reuse) means that I’ve got the option of doing more powerful analysis as well as the simple NPS.  More importantly, I don’t have to try to persuade a client to stop using NPS as long as I include other questions using the same scale.  Ideally, I’d prefer to use a 7 or 5 point scale instead, but 10 or 11 points works fine – as long as only the end-points are anchored. For more on combining Net Promoter with other questions for more powerful analysis, check out “Profiting from customer satisfaction and loyalty research”

There’s no justification for this type of scale in my opinion.  If you disagree, please make a comment or send me a note.   If you want to use a scale with every point textually anchored, use the Likert scale with every point identified (but no numbers). Including both numbers and too many anchors will make the survey takers scratch their heads – not the goal for a good survey.

Perhaps the people who created this survey had read economist J.K. Galbraith’s comment without realizing it was sarcastic.- “It is a far, far better thing to have a firm anchor in nonsense than to put out on the troubled seas of thought.”

Advertisement
About Mike Pritchard

Mike Pritchard is Principal at 5 Circles Research. Mike’s site, www.5circles.com, contains articles on research techniques including analysis and survey tips.

Comments

  1. Hi Mike – I don’t disagree at all. I’d love to know if there’s a significant impact on results or decisions if a company is doing the official NPS question – with the text anchored at both ends but uses a 1-10 scale.

    I’ve also seen (I think in Reicheld’s book) that companies use 5 point scales as well.

    What’s your take on that? Can NPS be useful using a scale that’s not 0 – 10?

  2. Ivana,
    as you pointed out in a different article, it depends on whether you want to follow the official NPS standard or not. If your surveys are loaded into an NPS database, then you’ll be using the standard question, and you’ll be paying big bucks so you probably aren’t reading this article;). But if you are doing it on your own you have more flexibility. Even so, you might want to use the official question scale for various reasons – it feels right, you can find some published information for a better comparison, etc. If you use a different scale you’ll have to make a decision on the cutoff points, since the 11 point scale is unbalanced and won’t directly map to a shorter scale. I’ve used 5 point scale questions for consistency with shorter scales I sometimes use for Secure Customer Index. In this case, I use 5 as Promoters and 0-3 as detractors. I’ve recently started seeing surveys where the Recommend question follows the standard NPS 11 point scale, and the other two Secure Customer Index questions are 5 points. It might look a little funky, but probably more to researchers than to survey takers.
    Mike

  3. Can’t say that I disagree either. It is far too easy to confuse respondents with too much detail. Parsimony is a better way to go.

  4. Thank you MIKE, as a survey taker I prefer to use 3 point scale that seems to me very easy to calculate.

  5. Jamal, thanks for your comment.

    I understand your point about shorter scales from the survey takers perspective. In fact I’ve just shortened the scales on a survey that’s just about to launch – because the questions weren’t as important as some of the others we prioritized ease for survey takers over the (theoretically) greater discriminatory power of longer scales.

    However, if surveys with an NPS question just asked the question in a way that matched the analysis (are you a detractor, neutral, or a promoter) that would undermine the fundamental approach of NPS as I understand it. The idea is that the underlying split is hidden from the user, with presumably the theory that survey takers will just respond to the entire scale. Some people do shorten the scales – particularly if they aren’t interested in comparisons outside their own surveys. Your approach is another interesting idea. I suspect the result would be more neutrals than the standard methodology.

    Mike

Trackbacks

  1. […] some wonderful comparisons between Net Promoter rating scales and satisfaction rating scales.   Read more about how to design a survey with the right rating scale. Possibly related posts: (automatically generated)How to Change From One Rating Scale to […]

  2. […] wonderful comparisons between Net Promoter rating scales and satisfaction rating scales.   Read more about how to design a survey with the right rating scale. 41.138388 […]

Speak Your Mind

*