4 Kinds of Survey Error: Sampling, Measurement, Coverage and Non-Response

family feud error

There are 4 generally-accepted types of survey error.  By survey error, I mean factors which reduce the accuracy of a survey estimate.

It’s important to keep each type of survey error in mind when designing, executing and interpreting surveys.  However, I suspect some of them are more ingrained in our thinking about research, while others are more often neglected.

Imagine if we interviewed 100 researchers and asked each of them (“Family Feud”-style) to name a type of survey error.

Which type of survey error do you think would be mentioned most frequently?  Which type would be most overlooked?

Here is my predicted order of finish in our hypothetical example.

Note for the “Feud”-challenged:  Number 1 represents the most commonly named type of error in our hypothetical survey of researchers, while number 4 represents the least commonly named.

1. Sampling Error.

My guess is that sampling error would be the most commonly named type of survey error.

In a recent Research Access post, “How to Plus or Minus: Understand and Calculate the Margin of Error,” I explained the concept of sampling error and gave 3 ways of calculating it.

Sampling error is essentially the degree to which a survey statistic differs from its “true” value due to the fact that the survey was conducted among only one of many possible survey samples.  It is a degree of uncertainty that we are willing to live with.  Even most non-researchers have a basic understanding, or at least awareness, of sampling error due to the media’s reference to the “margin of error” when reporting public survey results.

2. Measurement Error.  

I believe measurement error would be the second most frequently named type of error.  Measurement error is the degree to which a survey statistic differs from its “true” value due to imperfections in the way the statistic is collected.  The most common type of measurement error is one researchers deal with on a daily basis:  poor question wording, with faulty assumptions and imperfect scales.

3. Coverage Error.

Coverage error is another important source of variability in survey statistics; it is the degree to which statistics are off due to the fact that the sample used does not properly represent the underlying population being measured.

There was generally more concern about coverage error in the past; these days, the combination of increasing internet penetration and fast/easy/cheap online survey panels has made it possible to accurately represent many target populations.  Concern about coverage error is still an important conversation; however, it is being discussed more in academic and thought-leadership circles than by the average day-t0-day research practitioner.

4. Non-Response Error.

My guess is that non-response error would be the least named type of error in our hypothetical survey.  Telephone survey houses historically have routinely made 20 or more call-backs to households that do not answer the telephone.  This practice has dwindled due to a combination of the expense of conducting so many call-backs, and the dramatic growth of online surveys, where it is just easier to replace non-responders with fresh sample.  It is also not considered acceptable in an online context to conduct scores of follow-up emails; that would get the sender sent to a blacklist post haste.

Meet the Data Triplets: Data, Metadata and Paradata

tripletsThere are three sorts of data, and very often you need all three to understand and use the data you collect from your survey.

Here are the three sorts:

1. The Data.

The data is the data, that is the actually numbers, codes or open ended text that the respondent enters into the survey. There should probably be a better way of describing this than “the data.”, maybe raw data is a better term to use ?

2. The Metadata

Metadata describes the raw data.

For instance the raw data for question 2 may be a “1” or a “2”. The metadata would say that the value “1” means male and the value “2” means female.  Or the metadata may say that the values for question 3 could be a range of 1-100, or 32-78.

Metadata gives meaning to the raw data, and so it is vital to the analysis process of the raw data that the metadata is present.  Otherwise the raw data is just a collection of numbers with no meaning.

One of the problems with metadata is keeping it connected to the right raw data. The wrong metadata with raw data can be a disaster.

Over the years data formats have got more complex, and one of the big reasons is to keep the metadata with the data. More recent data exchange formats/protocols such as JSON (Javascript Object Notation for the technically minded) have capabilities for attaching metadata to the data, which is a very good thing.

Raw data with no metadata is just a load of junk.

3. The Paradata

Paradata is the least well known of the data triplets. In the past decade or so it has become much more important for the survey research world.

Paradata is data which describes something about the way the raw data was collected.

It is data about data.

The most commonly used form of paradata used at the moment is data about questionnaire and question timings. That is, the time a respondent takes to complete a question or questionnaire.

This type of data is now one of the cornerstones of quality measurements for web surveys.

Obviously there can be many different sorts of paradata. For open ended text questions the length of text entered by the respondent can be measure, as well as the “level of vocabulary” contained in the text.

One metric used for web surveys is that of “speeders,” that is, the number of people who complete the survey extremely quickly. The paradata for time take to complete the questionnaire is used here.

Paradata can also be useful in revealing hidden biases; for instance, using paradata in the gamificaton of surveys is a rising trend. The time taken to do something in a gamified survey as well the action can have a great deal of meaning. Some researchers claim that hidden racism, some times unknown to the subject themselves, can be revealed by measuring someone’s reaction time to specific questions.

In a future post we will delve more into exactly how paradata can be used for quality control of web surveys.

Now We Have Smartphones; Shouldn’t We Try to Be Smarter about Surveys?

This is a presentation from Survey Analytics‘ President, Andrew Jeavons, from the Market Research in the Mobile World Conference in Atlanta in July 2011.

(By the way, it’s pronounced JEH-vons, not JEE-vons, as Andrew explains in the video…)

Andrew presented guidelines for conducting mobile surveys, and he made suggestions for adapting Net Promoter Scores (NPS) in a mobile survey environment.

Enjoy!

How to Use Facebook for Market Research Surveys

It’s an understatement to say that there’s tremendous interest in using Facebook for market research.  Indeed, among the most popular posts on Research Access is one written last year by Survey Analytics‘ CEO Vivek Bhaskaran, entitled “Social Media Research – Using Facebook for Survey Invitations and Market Research.”

What not everybody realizes is that companies are using the power of Facebook’s large audience to conduct research every day.

While Facebook-fueled surveys are not right for every situation, they can be extremely powerful in the right circumstance.  The biggest advantage is access to a massive audience of people who do not normally complete surveys.  However, even Facebook’s large audience will not necessarily yield a sample from the target audience you are trying to reach.  In addition, sampling through Facebook Ads can be expensive, depending on the particulars of your study.

Since Vivek wrote his Facebook sampling post last year, there have been many changes to Facebook, but the fundamental principle outlined in that post still holds true.   So it’s time for an update.

Also, I will explain how to use company or brand fan pages to get valuable feedback.

1) Use Facebook Pages to Reach Your Customers and Fans.

You can ask followers of your company or brand fan page (or your personal page, for that matter) to provide feedback in several ways.

  • Post an open-ended question asking for direct feedback.  For example, “We are looking for feedback on Research Access’ new look and feel.  What do you think?”   You can add language encouraging people to post their comments on Facebook, or you can give an email address for them to contact you directly.  The feedback you receive will be useful but will not be generalizable to all customers or fans.
  • Post a poll.  Facebook now has a “Question” option in the status update box allowing you to post a poll to your fans.  Please note: you can only do one question at a time, and the results will be visible to all fans.  Interestingly, there is an option to allow your fans to add responses which you didn’t necessarily consider when creating your question.

Ask a Question

  • Post a link to a survey.  Instead of using Facebook’s built-in question function, you can simply share a link to a survey.  You should also include explanatory text in the post.  Here’s a hypothetical example Research Access could use: “Please take 5 minutes to give us feedback on Research Access’ new look and feel. Everyone who completes the survey will receive a free eBook copy of QuestionPro for Dummies.”
Post a Link

2) Use Facebook Ads to Reach a Wider Audience.

Using Facebook Ads, you can open your survey up to a massive audience which can be targeted in very specific ways.  Here are the steps for directing Facebook users to your survey using Facebook.

  • Start creating a Facebook by clicking the “Create an Ad” link in the “Sponsored” section in the right-hand column of your page.
Create an Ad
  • Create an ad with an image and a message that will drive the right type of traffic and redirect those who click on the ad to an externally hosted survey.  Select “External URL” in the “Destination” drop-down list.  Put your custom survey URL in the “URL” field.  Use the “Title” and “Body” fields to create a compelling call-to-action for survey-takers.  Be sure to include an image that will garner attention.  In the “Targeting” section, you can target your survey by geography, age, specific interests and more.
  • Define your budget and schedule.  With Facebook Ads you have a great deal of control over your ad’s schedule.  Importantly, you can define a daily budget which will not be exceeded.
  • Finally, preview your ad, then start your campaign!  Good luck.

QR Code-Enabled Mobile Surveys: An Example

 

 

 

 

 

 

 

 

 

 

 

[Editor's Note:  this post was originally published on the Survey Analytics Blog.]

My friend Scott Liang from Parametric keeps telling me – the quicker you collect feedback from the Point-of-Transaction, the better the recall and quality is. While I have absolutely no way of verifying his assertion, in todays Blackberry and iPhone induced ADHD world, it seems logical – and anything that passes the sniff test, works for me!

If we accept that premise, then what the Washington State Ferries in conjunction with the Washington State Transportation Commission is doing is pretty innovative — collecting feedback directly from commuters while they are in the ferry. As passengers are commuting in the ferry, they have options for using their smartphones via QR Codes to give feedback on the ride.

Few innovative options:

Thumbs UP and Thumbs Down:

Instead of having just one QR code that takes them to a survey, the Thumbs Up/Down model has 2 QR Codes – each representing positive or negative emotion espoused by the passenger overall. This is similar to the the universal facebook “like” button that we are all accustomed to in the web world.

FROG on Board Poster

Integrated with MicroPanel:

Users who choose to give feedback are then asked to join a panel for future surveys and feedback. This allows the commission to build a long term relationship with the passengers — the commission can then use this panel for pricing, satisfaction and other kinds of research.

Once the feedback was collected, users are given the option to go to the mobile-optimized page for MicroPanel;

Completely Turnkey:

No custom development. This entire solution is off-the-shelf. This reduces cost and complexity. SurveyAnalytics has as question type that supports creating QR Codes. This enables you to create a survey with multiple QR Codes for each option:




click-here-to-download-the-case-study



How Not to Get Market Research Clients

I’ve worked in many roles in market research, but never as an in-house market researcher in a corporation.  I’ve always felt bad for them in that I know they get a constant stream of suitors among would-be market research suppliers.  In particular, at conferences as it appears they’re being accosted left and right by hungry supply-side sales people.

This Tuesday, November 8th, Tiffany McNeil, Strategy & Insights Manager, Innovation at Del Monte Foods was interviewed by Ray Poynter of Vision Critical on Radio NewMR.  She spoke about a number of topics, including how to – and how not to – get on her radar.

Note:  this interview has been lightly edited.  You can listen to the original interview on the Radio NewMR archive page.

Ray Poynter: What would you define as innovation in market research?

Tiffany McNeil: I’d say that, for me, good innovation is one of two things. Besides a general willingness to be sensible and try new things, which is actually one of the great things about BrainJuicer, I know Tom (Ewing) is the other guest today – we work with BrainJuicer a lot – they have a lot of proprietary tools that are trademarked, and those tools are fantastic, but if you have a question that doesn’t fit with one of the tools that they already have in their toolchest as it were, they are more than happy to try and find a new solution for you. And I think that a lot of other companies do have these trademarked tools, and kind of no matter what you ask them, try to shoehorn whatever your question is into one of those boxes. So I think that’s one thing – just a general willingness to be flexible and try things. And the other is sort of a more traditional answer, which is, kind of a better way to answer questions, either faster, more efficient, or maybe even just more engaging, so that when I present back to my internal partners, it’s something that’s more compelling for them to listen to or to pay attention to.

Ray Poynter: OK, we’ve mentioned one of your suppliers, but in general, when you’re looking for suppliers, what are you looking for?

Tiffany McNeil: It’s hard to say. I think that the suppliers who earn – I can’t speak for my company or my team, but for myself. The suppliers that I’m excited to go back to over and over again are the ones that are just great to work with – the ones who, when you call them, there’s always someone really smart and engaged on the other side of the phone, who are really responsive, who do lots of follow-up work without sort of claiming “scope creep” the second you ask a question you didn’t know the first time. And there aren’t actually that many people who fit that mold, unfortunately. There’s a lot of little mistakes, there’s a lot of sort of people who are really engaged when you’re trying to win the work and they disappear once you get it. So we have probably as a group a pretty small list of suppliers who we feel are always consistently delivering excellent work, and good partnerships.

Ray Poynter: So you have a good list, a small list of people – now, please do not everybody contact Tiffany offering your services…

Tiffany McNeil: [laughter]

Ray Poynter: …otherwise we’ll get nobody wanting to appear on the radio show, from the client side – but, what are the good ways of people getting onto your radar? If somebody is truly innovative, how would you like to find out about them?

Tiffany McNeil: It’s a really interesting question. I was thinking about this a lot yesterday. And I think that – how about this? The way not to do it is to call me and leave me a message, or send me kind of cold emails. I get – you know, everybody gets – but I get tons of them every day. And, not only do I not have time for them, but I also have a lot of guilt about them. It’s just not making me feel good about myself. But I think that when we’re looking for a new methodology or we’re looking for a new supplier, typically the first thing I’ll do is ask around on my team. There is a lot of word-of-mouth – people who have worked in other companies, and may have answered a similar question somewhere else and so might have an idea.

For example, I’m working on a question right now, which isn’t something that I’ve done before – the methodology’s going to require some pretty smart analytics, it’s going to be kind of analytically tough. So in my head I go through the roster of companies that fit that mold. So, you know, who do I know that’s really smart about analytics, and if I come up short, the next thing is I kind of walk around and ask everybody on my team. If I fall short there, I’d say I probably look to thought leadership next. So, Vision Critical is a good example. They are a company that I didn’t really know anything about, and we don’t use them – we do have a relationship with them now, but the way they got on my radar is two things: they showed up at the conference that Lenny (Murphy) chaired last summer – the NewMR new methods conference – but before that, I had gotten a piece of mail that was a list of the top – it actually wasn’t from them – I think it was either from BrainJuicer or from Synovate – one of those two – were also on the top of the list, but Vision Critical I think was top – and it just stuck out as a company that we don’t know, that apparently people like working with. So, that’s an example of how a company kind of got on our radar – now, frankly, we haven’t done any work with them, but I had a conversation with them, and I kind of feel like I know what they do and what they’re good at. And if the right question came up we would reach out to them proactively.

Ray Poynter: Super – and I think probably the list you’re talking about was a thing called GRIT, which is a study that GreenBook run in conjunction with lots of other people, and it has got some interesting questions in there, “Who do you think is innovative?” and yes, BrainJuicer do very well and a number of other companies do very well, including some of the large ones, which I think was an interesting part of that. I’m going to change tack now, and Tiffany, thanks for that, and I hope you don’t receive even more emails especially now that you said that’s not the way to reach you.

Here is Tiffany’s biography:

Tiffany McNeil is a client-side research manager whose circuitous career path brought her most recently to a Strategy and Insights role in the CPG world. Before that, she spent time in New York and London, where she worked primarily in the television industry, including content and editorial research roles at UKTV and Channel Five in London. She is passionate, smart, and opinionated – um . . . and modest charming, but she wrote this herself, so take that how you will. She lives in San Francisco(ish) with her family and spends most her time making lists.

Election Polls: 5 Tips For Navigating the Clutter

vote buttonTomorrow is Election Day here in the States.  The big vote (for President) isn’t until next year, but we have the usual spate of races for local offices as well as a wide range of citizen-initiated referenda on everything from the mundane (bond measures) to the highly divisive (social issues).

As the expected avalanche of survey results washes over me and everybody else (accompanied by a barrage of television advertising, internet electioneering, and attack mailers), I thought I’d share some tips for understanding which poll results to heed, and which to take with the proverbial grain of salt.

My experience in the world of election polling has given me a bit of insight into this topic; I hope these tips help you find your way on your journey through the sea of election polling data.

1. Understand the Methodology

The best way to judge an election poll – indeed, any survey – is to have a good understanding of the study’s methodology.  Unfortunately, reporting on survey methodology is often woefully inadequate, scarcely going beyond a reporting of the margin of error.  However, sometimes one can read between the lines of a story to gain a better understanding of the circumstances under which the poll was conducted, including: timing, data collection mode and survey length.

If the methodology is completely unclear from the article, be a good citizen and email the editor requesting clarification.  The rest of us will thank you!

2. Think Random

Generally speaking, in election surveys, relative to market research surveys, it is particularly important to have sampling methods that give all likely voters as equal as possible an opportunity to be surveyed.  Look for efforts to ensure a representative sample, including:

- surveying voters at different times of day and on different days of the week
- compensating for sampling limitations such as telephone coverage, cell phone coverage, and internet access
- using a “likely voter” screening question, ensuring only those both registered and likely to vote are considered.

Also, be wary of  polls conducted using automated telephone interviews rather than trained interviewers.

3. Evaluate the News Source

As good a data consumer as you are, you cannot realistically check the fine details of methodology on every study.  Therefore, pay attention to the reputation and track record of both the publishing entity.

More reputable organizations tend to higher publishing standards than less reputable ones; they have more to lose if they publish bad reporting.  Put more stock in a study reported by the Washington Post or the Pew Research Center than a study sponsored by a smaller or less reputable newspaper, website, non-profit organization.

In local elections, sometimes the most reputable (though hardly infallible) source is the state or locality’s largest newspaper.  However, look for critical analysis rather than simple reporting of results.

4. Pay Attention to the Sponsor.  Often a group with a vested interest in an election will privately commission a survey to be used for internal strategy; however, if some of the results support their public relations efforts, they will release an (often-misleading) subset of the data in order to influence the electorate.  Sometimes entire carefully-worded polls will be conducted which are meant for public release.  Be very skeptical of any data paid for by a group or persons with an interest in the election’s outcome.

5. Learn from the Experts.  There is a wealth of great content created by smart people who focus on analyzing election data.  Take advantage of their wisdom!  I’m partial to HuffPost Pollster (formerly Pollster.com) – check out my friends Mark Blumenthal (a.k.a. “@MysteryPollster“) and Margie Omero – and Nate Silver’s Five Thirty Eight blog at the New York Times.

These resources are primarily focused on U.S. Elections; there are undoubtedly many more good resources out there both for the U.S. and everywhere else; please suggest others in the comments section below.

I hope these tips are useful to you.  And if you’re in the U.S., don’t forget to vote tomorrow!

How to Plus or Minus: Understand and Calculate the Margin of Error

iPhone - Plus or MinusSometimes in the day-to-day work of conducting and interpreting market research, it’s easy to forget that many people who work with surveys on a daily basis have not had formal training in statistics. Even for those who have been trained, it can be useful to have a refresher from time to time.

UNDERSTANDING MARGIN OF ERROR

One of the most basic concepts in market research is the confidence interval, commonly referred to as the “margin of error.”  The confidence interval is a range of values within which a survey result can be assumed to accurately represent the underlying construct being measured.

Technically the margin of error is half the confidence interval; plus or minus 5 percentage points represents a confidence interval of 10 percentage points

The general public has a basic if vague understanding of this concept. Indeed, media reports of election surveys often report a result “plus or minus” a certain number of percentage points.

The confidence interval is important because it helps us as marketers and researchers understand the limitations of our survey results. The confidence interval estimates the inaccuracy of our results due to “sampling error,” that is, error stemming from the limitation of conducting our survey among a single sample of the population of interest (rather than the impractical or impossible alternative of conducting a census of the entire population).

Sampling error is distinct from other types of survey error – including measurement error, coverage error, and non-response error – but those are topics for another time.

Here are the factors that affect the margin of error:

  • confidence level
  • proportion in the sample
  • sample size

Confidence level.  You must choose how statistically certain you want to be.  The most common confidence level is 95%.  The conceptual meaning of a 95% confidence level is as follows. If you were to conduct your survey one hundred times with randomly drawn samples and everything else were equal, the result of your survey question would be expected to fall within the confidence interval ninety-five of those times and outside it five times.

Proportion in the sample.  Proportional estimates closer to 50% are subject to more variability than estimates near the ends of the spectrum, e.g. 10% or 90%.

Sample size.  The greater the sample size, the lower the margin of error because variability due to sampling anomaly is reduced.

CALCULATING MARGIN OF ERROR

There are three ways to calculate the margin of error:  use a formula, use a look-up table, or use an online calculator.

Use a formula.  There are a number of formulae you can use with slightly varying assumptions.  If you want to go through the calculations yourself using a formula, I refer you to this web page: “Guide to Computing Margins of Error for Percentages and Means” from Professor Ted Goertzel’s at Rutgers University, who explains the calculations better than I can hope to do.

Use a look-up table.  Here’s a table that will be appropriate in most circumstances.  This table is based on a 95% confidence level.  In order to find the confidence interval (the “plus or minus” amount) for a particular proportion, go the the row closest to the proportion of interest and the column closest to the sample size of interest.  For example, if an N=500 election poll showed a race tied at 50% to 50%, you would go to the 50% row and the N=500 column, yielding a margin of error of plus or minus five percentage points.

 N N N
Proportion 1,000 750 500 250 100
10% 2% 2% 3% 4% 6%
20% 3% 3% 4% 5% 9%
30% 3% 4% 4% 6% 10%
40% 3% 4% 5% 7% 10%
50% 3% 4% 5% 7% 11%
60% 3% 4% 5% 7% 10%
70% 3% 4% 4% 6% 10%
80% 3% 3% 4% 5% 9%
90% 2% 3% 3% 4% 6%

Use an online calculator.  The above exercises are great, but guess what, you’re in luck!  There are many online calculators out there.  Here are two examples:

American Research Group
Relevant Insights

I hope this post is useful as you navigate the world of survey research.  Good luck, and happy polling!

Note:  Table reproduced from The Roper Center at the University of Connecticut.

Determining Price: The van Westendorp Price Sensitivity Meter

Determining the best price for a product or service is a common marketing research question.  I usually start my conversation with a client asking whether their product has all of its features set or if they also need to test a range of features other than price.  If they are testing variable features in addition to price, we start to talk about conjoint (see here for a video on the current state of affairs in conjoint).  However, if they tell me that their product features are set and they just want to look at price, one of the things we’ll likely discuss is the van Westendorp Price Sensitivity Meter (let’s just call it VW).

I was recently corresponding with a colleague (Dave Lyon of Aurora Market Modeling) and the discussions led me to look back at the original VW paper (Peter H. van Westendorp (1976), “NSS – Price Sensitivity Meter (PSM) – A New Approach to Consumer Perception of Prices,” in Venice Congress Main Sessions. Amsterdam: European Marketing Research Society (ESOMAR), 139-167.)  In my conversations with Dave, one of the issues that arose was the way many modern researchers calculate the point of marginal cheapness.  Are most researchers incorrectly calculating VW’s outputs?  What might van Westendorp himself say about this?  How about a little background before going into this point?
[Read more...]

How to Control Your Customer Satisfaction Scores

We tend to measure customer satisfaction after the customer experience has already happened.  But that isn’t when the opinion about the experience is really created.

Customer experiences are actually created long before your customer ever reaches your doorstep.  They often start with impressions and perceptions created when your potential customer interacts with friends and colleagues who may tell them about your company, or they search online and find articles and reviews about your business.

This is the moment when expectations are created based on what messages are currently active about your business and the customer experience.  While you can’t control others — you can control and design a customer experience. Here’s how:
[Read more...]