Non-Traditional Paid Respondents More Tech-Savvy than Panelists

Tokyo crowds coming together in two crosswalks

On March 5, 2015, Survey Magazine presented the TrueSample semi-annual quality council meeting. The speakers were Chuck Miller and Mark Menig, reviewing findings from several research-on-research initiatives. The quality council is now in its seventh year.

Traditional sampling versus non-traditional sampling was the first topic of the day. The goal here was to assess differences in attitudinal and behavioral data when controlling for demographics. In short are there key differences we should consider in survey and sampling design if we plan to use non-traditional sourcing? Traditional sources included emails sent to online panels. Non-traditional sources included everything else, including social media sourcing. The study included questions on lifestyle and outlook, shopping attitudes and behaviors, technology and media usage, etc.

Eight sample providers contributed to the 1,800 respondents. Four providers were traditional panel companies, and four engaged their audience through multiple touchpoints.

Responses for each of the two groups looked similar overall. The notable exception was that respondents from non-traditional sample sources were more tech-savvy, especially with mobile phone usage. A key characteristic of the non-traditional sourced participants is that they engage in numerous activities to earn points including coupon usage, video watching, searching and surfing. In line with these behaviors non-traditional respondents spend more time online. The graph below highlights these differences.

TrueSample ROR

Respondents from non-traditional sample sources did four activities significantly more often: watching YouTube videos, posting photos to Facebook, browsing online ads, and downloading music. Traditionally sourced participants were significantly more likely to do online banking.

Technology ownership, while similar between the sample sources, still showed non-traditional respondents as slightly more tech savvy. Non-traditionally sourced respondents were more likely to not have a landline phone and less likely to use a cable or satellite TV service, indicating a higher percentage of cord-cutting. They are also more apt to rely on mobile phones for a greater share of their phone calls. They also leverage their phones more frequently for product research while shopping in-store (“showrooming”).

While online and technology usage varied significantly, the two groups were less likely to express differences related to brands and shopping behavior, with significant differences in only two of nine items: non-traditional respondents were more apt to report they enjoyed shopping, and traditional respondents were more apt to report price as the key influencer in shopping.

Key points to ponder and act on include recognizing that sample source, be it from traditional online panels or non-traditional sources, can be a significant source of variation. This is especially true for technology related issues. If you look to multiple sources for sample, then be sure to include measures to assess sample quality.

Greg Timpany directs the research efforts for Global Knowledge in Cary, North Carolina, and runs Anova Market Research. You can follow him on Twitter @DataDudeGreg.


Speak Your Mind