The Fall of Recall

tree shaped like head, losing leaves © fresh idea Fotolia

Recall: the retrieval of information stored in the memory.

We all rely on it every day – Where did I leave my phone charger? Which gas station has the lowest prices? What filename did I save that document under?

In usability research, we rely on recall to get feedback from everyday users, testers, and focus group subjects with surveys, direct questioning, and other methods – What did you think of the registration process? Did you have any trouble finding the contact information? How does the site make you feel?

So What’s Wrong with Recall?

A cursory reading of Wikipedia (Why you can trust Wikipedia – and crowdsourcing in general) reveals a number of factors that exert a major influence on recall accuracy, including:

On top of all this, your respondents’ answers can be subject to various other manipulations depending on the format, including social pressures coming from fellow testers leading to conformity, bandwagoning, or lying to hide what could be perceived as incompetence; inadvertent pressure from a test administrator to answer a certain way or confirm a given expectation; question-framing issues that influence responses (think ‘leading the witness’); and more.

The human mind is capable of endless shape-shifting, over-imagination, and self-deception – more than enough reasons to think twice about your focus group results.

So if waiting til the end to ask your questions is such a feedback faux pas, what’s the solution? Problems like division of attention or context dependency might be diminished by confining testers to a controlled environment, but at the cost of losing a genuine, true-to-life look at the user experience. Primacy, recency, and interference could be combatted by asking testers questions at the end of each individual task, but at the cost of obstructing the natural flow of their journey through your website.

There is simply no way to eliminate all the distortions at once from a usability study that follows the format:

  1. Have testers use the website

  2. Ask questions later

Replacing Recall with Occurrence

The only way to close the gap between what users do on your site and what they remember doing on your site, or what they say they’ve done on your site, is to look at their occurrent thoughts – that is, the thoughts that pass through their mind in the exact moment of those actions.

No recall is required – no flawed mental filters, no forgetting of middle elements or transitioning between mental states – just verbalization of the thought process as it happens. It’s something that comes naturally to us: most people talk to themselves, especially while alone. Remote usability testing allows you to tap into that natural instinct, listening to testers’ thoughts in real-time and getting the full, accurate, unadulterated picture.

Effectively, you’re looking into your user’s mind, understanding what they do and why at the most direct level. The pitfalls of relying on recall are avoided, and the problems that crop up when testers are influenced by judgmental peers, by hovering researchers, and by the wording of questions never arise. They have simply to open their mouths and let their minds flow out.

Occurrence is the key to getting the best feedback on your website. (See an example.) Are you relying on imperfect cognitive processes and delayed responses to understand your site, or are you peeking into your users’ heads?

Tim Rotolo is a UX Architect at TryMyUI, a remote usability testing service. You can follow TryMyUI on Twitter @trymyui.

Advertisement

Speak Your Mind

*