Reducing Cognitive Stress on Survey Respondents

PET scan of a normal brain

Data quality is directly proportional to comprehension and usability

Challenge

When it comes to survey and attitudinal behavior research, representative sampling and random probability sampling theory is often discussed; however, a simpler quality factor has long been overlooked:

The ‘cognitive stress’ the surveys place on respondents.

Cognitive stress measures the internal apprehension and anxiety humans face when they are unsure about a task relating to using a particular system. The confusion may arise due to various factors such as complicated statements respondents are asked to rate or simply ones that don’t apply to a particular individual.

Here’s a simple example:
Where do you live? TryMyUI - Stop Making your Respondent Work too Hard

The ambiguity illustrates the issue: does the researcher mean country, city or zip code? Such a question immediately increases respondent frustration (read: stress) as he or she is forced to guess what the researcher really wants.

While a simple QA/review process would likely catch this issue (or more likely the experienced researcher would never put it in a survey in the first place), this is just the tip of the ice berg. In addition, the real world “QA process” is often without sufficient time to ferret out every issue.

Let’s go a step further:
Rate this Question about Employees Feeling Free to Innovate - TryMyUI - Avoid Respondent Fatigue Due to Interpretations

The first option is loaded, posing a substantial burden on respondents to interpret the question in the same way. But is the question about management valuing employees or only those considered innovators?

The second question should be easier to understand with much less interpretation required.

As similar as they may seem, these two questions are measuring different artifacts – one measuring management’s ethos, and the other management’s practical implementations. To make an accurate recommendation to the CEO, the researcher must initially identify these as two separate items and phrase questions appropriately.

While researchers have long struggled with measuring cognitive stress, the questions themselves are often not tested for efficacy because of either cost or it’s assumed researchers are experts in crafting questions without loading or bias. While this is often true, they are also human and costly oversights can occur – especially when QA is performed as an afterthought.

Solution

User Experience and Interaction Designers have conducted usability testing very successfully in the web/app design space. In the last few years, as crowdsourcing has gone mainstream, remote testing has become increasingly popular.

TryMyUI has now pioneered a similar model for identifying and measuring cognitive stress in surveys.

With this new model, testers record their screens and verbally discuss their experience taking a survey. Strides in technology have made remote usability testing – where the testers are using their own devices and tools and verbally walk you through their experience – cost effective and easy to use.

TryMyUI is a remote usability testing solution – with a pre-recruited database of over 100,000 testers who have the TryMyUI screen recording software installed. And with their recently released Partner API, we have integrated survey usability testing directly into QuestionPro for One-Click access to the usability recording sessions.

The screenshots below show how to order Usability Tests from the TryMyUI tester panel.

TryMyUI Cognitive Stress Test

TryMyUI Cognitive Stress Test - Place Order

Conclusion

This qualitative and subjective model to measure and identify usability and cognitive stress, using remote testers and recording their video session, represent a step in the right direction to increase the reliability of survey data. Data collected via surveys fundamentally represent inputs into a larger decision making process – and as researchers we need to be cognizant about the quality of the data we collect. This process makes it that much more accurate.

Ready to learn more? Join us March 17th for an informational webinar on the benefits of testing the usability of your surveys.

References & Further Reading:
QuestionPro FAQ on Cognitive Stress & Usability Testing
http://www.questionpro.com/help/601.html

Bureau of Labor Statistics & TryMyUI – Case Study
http://trymyui.com/whitepapers/BLSCaseStudy_TryMYUI.pdf

Erik is the CEO of QuestionPro, where he leads the operations and strategic vision of the company.

Advertisement

Speak Your Mind

*