NPS Requires Much Larger Sample Sizes Than Alternatives

test your strength game

At Market Strategies, we receive all sorts of questions about Net Promoter Score (NPS):

  • Where did it come from?
  • What are its major pros and cons?
  • Is it the Holy Grail of marketing research or not?

I answered many of these three years ago in an article for the AMA’s Marketing Research Magazine.

This post addresses the surprising volatility of NPS and how this continues to be a big challenge for users. We frequently hear statements like: “_____ seemed a bit surprised again this quarter by some of the [large] swings in NPS scores.”  Our clients see differences that seem like they should be significant, yet turn out not to be.  Just how volatile is this NPS measure, and how can we quantify this?

Let’s look at an NPS example for an average to above-average company: 40% Promoters, 40% Passives and 20% Detractors, with a sample size of n=500:

  • The sampling precision of this NPS of 20 is +/- 6.6, meaning that if we repeated this study the same way 100 times, we’d expect that about 95% of the time we’d end up with an NPS between 13 and 27.
  • What’s the comparable precision for the % Promoters—which is just a top two box score like researchers have used for decades? Its precision is 40% +/- 4.3%, and with identical repetitions we’d expect most results to come in at the 36% to 44% range.

The greater wobble we see here and the instability it causes is what’s behind the client frustration.  Having to explain to internal audiences or senior managers that a decrease or increase in NPS that looks substantial (but really is not) is an unwelcome task at best; makes for possibly stormy meetings; and at worst may undermine the credibility of the entire effort and the department.

The difference between these precisions may not seem like much.  But for this 500-interview example, how many interviews would we require for this NPS to have the same (I daresay, the anticipated) precision of +/- 4.3%?  The way the math works out, we’d actually need a base size of 500 * (6.6/4.3)2 = 500 * 2.36, or 1178 to accomplish that—over twice as many interviews!  This agrees with 1) another recent observation that when the American Customer Satisfaction Index (ACSI) showed a margin of error of +/- 3.3%, the NPS score based on the same results had a margin of error of +/- 10%, and 2) the statistician’s rule that if we want to cut precision or margin of error in half, we must quadruple our sample size!

Research suppliers may also fuel the NPS anxiety. Client conversations have shown that many suppliers use a too-simple NPS test that just treats it as a simple percentage. From the results above, we can see that this approach will yield considerably more significant differences than there really are (more false positives). Clients also tell us that suppliers say that NPS cannot or should not be stat tested at all because:

  • NPS’ variance is unknown (estimation via a method called bootstrapping has been proposed to address this, but a formula does exist, independently derived and verified by three marketing scientists at my company . . . rendering bootstrapping unnecessary);
  • NPS is a function of two related percentages, and no test exists (in reality, a little-known test is almost tailor made for this situation);
  • NPS is not normally distributed (this is another justification for mandatory bootstrapping, but given the often more-than-adequate base sizes, NPS is in fact approximately normal or bell-shaped).

As the article I mentioned above describes in some detail, NPS may be a great measure for your business needs.  But even when it is, keep in mind: To achieve the same precision and sensitivity typically associated with top box score findings, you may need to double or triple your usual sample sizes for NPS work. Taking this into account can move NPS’ high-volatility issues from being a frustrating, hidden cost to being a very clear and obvious financial cost.

Randy Hanson is a vice president of the Marketing Sciences Group at the market-research firm Market Strategies International. He has more than 25 years of research experience executing a wide array of research, consulting and statistical responsibilities.

Advertisement

Speak Your Mind

*