Is It Time to Redefine Quantitative Research?

.As simplistic as it is, market research is often defined as either “qualitative” or “quantitative.” In conversational use, most researchers use and define these categories fairly similarly. But this is changing. While “qualitative” use and meaning is fairly consistent, we are seeing a notable fragmentation in what the phrase “quantitative research” means to market research practitioners and related professionals.

How Do You Define “Quantitative Research”?

I don’t mean a Wikipedia definition; I mean how do you personally define it when asked by colleagues or clients? What do you mean when you use the phrase?

Common real-world answers we typically hear, whether we agree with them or not, include:

  • Quantitative research is a set of methods and tools for measuring attitudes, behaviors, emotions, and values. It includes data collection and analysis.
  • Quantitative research is survey research. It requires a sufficient sample size to generate results that are representative of an intended population and that would be replicable if tested.
  • Quantitative research is the collection and analysis of numeric data in order to test stated hypotheses.
  • Quantitative research is any research based on 50 or more participants. Data collection could be from a survey, interviews, or even focus groups[1].

Are these all different? Yes, however still somewhat related. But today, we often hear unexpected variations. Just last week I met a sharp market research professional, who defined quant as big data analysis—he grouped survey research with…wait for it…qual.

Shifting definitions can also be seen in recent job postings for quant researchers. Increasingly common requirements include data mining, text analytics and even HTML.

Quant under Fire

To some extent, fragmented meanings associated with quant research reflect the shift from data collection plus statistical analysis, to an increasing focus on analysis of existing or continuous data streams. But is there more to it?

Some of the fragmentation in what quant research means also seems to be coming from differently educated practitioners—market research newcomers who are from a data analytics or tech bent. We market researchers now commonly work with people who have different skills, training and jargon.

But some of the fragmentation is coming from within the industry.

Two powerful topics were recently written about—tackling some brutal quant research realities. Realities that do have implications for what the term “quantitative research” really means today.

  1. Articles on whether margin of error should be reported when based on non-probability samples (see posts by Annie Pettit, Chief Research Office at Peanut Labs, and related articles). This raises the issue: can replicable, valid data come from a survey that uses non-random sampling?
  2. Data quality is a hot issue for quant researchers. A May 2015 article from Quirk’s covers new online sample quality guidelines from ESOMAR and the Global Research Business Network (GRBN). While online sample quality is certainly not a new topic, it is becoming more urgent.

Quant is also picking up some heat from high-visibility polling errors. Most recently, the British Polling Council’s inquiry into the unsuccessful 2015 opinion polls that failed to predict David Cameron’s conquest. Even Nate Silver has said, “The world may have a polling problem,” citing other recent polling failures.

What Does “Quantitative Research” Promise?

Not only is the term “quantitative research” used differently by different people, and under fire, even within the market research profession there are questions as to exactly, “how quantitative is quantitative research?”

  • Just because data is collected using a questionnaire, does not mean the research is quantitative.
  • Just because a researcher delivered statistics, doesn’t mean they are reliable.
  • Just because a margin of error has been calculated, doesn’t mean it should have been.

When we researchers say “quantitative research”, what do people hear? A promise of representativeness? A measure of reliability? A promise of robust statistical analysis? The use of questionnaires? An assurance of large sample sizes?

The harder question is, what do we want it to mean? As a profession, what do market researchers want “quantitative research” to mean, and what are reasonable assumptions to make when we hear our colleagues use the term?

The Limits of Shorthand, the Promise of Clarity

Today the phrase “quantitative research” means different things to different people. That’s beyond our control. But as market research professionals we should have clarity—what do we mean? The next time a client asks us to define it, what will we say? And can we say it in a way that is meaningful and memorable so that we can help others, and our industry, communicate and manage expectations appropriately? I don’t have the answer; but I do want to make sure we start the discussion.

Kathryn Korostoff is president of Research Rockstar, a market research training firm. You can follow her on Twitter @ResearchRocks.

[1] Thus the answer to a classic market research question, “How many focus groups do I have to do to get quantitative results?”


Speak Your Mind