About Andrew Jeavons

With over 25 years in the market research industry, Andrew is a frequent writer and speaker for various publications and events around the world. He has a background in psychology, statistics and software development. Andrew is President of Survey Analytics.

Why Recall Must Die: Capturing the Point of Emotion

Emotions

Living in the Past

Market research relies heavily on human memory. Attempting to measure recall about what respondents thought or felt about a product or service is a standard approach for market researchers.

Surveys often consist of long lists of memory tests. So many surveys contain phrases like, “Thinking about the last time you used XXX”?  And of course, focus groups always rely on the subjective recall of emotional states.

The assumption underpinning the standard market research operating procedure of directed recall is that we can reach into our experiences and retrieve complex information.

But is that true? Can respondents accurately retrieve memories and emotional states in response to a survey questionnaire?

Most market researchers give little to no thought to their reliance on recall. They fail to challenge themselves to better understand respondents, and in so doing they fail their clients and themselves.

Market research lives in its respondents’ past. The problem is that the current market research modus operandi of asking respondents to recall memories and emotions may be faulty at its core.

Memory is increasingly being understood by academicians as fluid rather than a concrete object that can be picked up and read at will.

The dominant theory of memory for many years has been so-called “working memory”, with researchers such as Alan Baddeley, Graham Hitch and Nelson Cowan producing a robust literature. These researchers concentrated on the cognitive aspects of memory, acoustic and visual buffering systems, episodic memory formation processes, and, finally, longer term memory processes.

In parallel, neuropsychologists and neurophysiologists searched for the “holy grail” of memory research, identifying what was known as the engram – the physical imprint that a memory must somehow make on the brain. More recent work seems to be getting researchers closer to understanding the physiological nature of memory.

Almost all of this is resolutely ignored by market researchers.

Researchers tend to see memory as a concrete object, something that can be brought back and returned to memory. It can be lost like an object too. When we simply can’t recall things, we say we have lost the memory, as if we possessed a thing such as a key.

In Wired Magazine’s March 2012 issue, Jonah Lehrer provides an interesting summary of recent research on memory, focusing on the work of Karim Nader at McGill University in Canada.

Research on alleviating the terrible symptoms of Post Traumatic Stress Disorder (PTSD) has begun to challenge the idea that memory is like a set of photographs we can access, look at, and put away again in the same condition.

PTSD can be regarded as a super-strong memory. A memory has been imprinted so powerfully that it cannot seem to fade away, as many memories do over time. PTSD is a breakdown in forgetting.

Nader’s research overturns the idea that memory is static, that it is a concrete object that can be read repeatedly in the same way. He found that the process of recall can cause the memory to be rewritten, so that we constantly modify memories as we recall them. Hence there is hope for victims of PTSD.

There is no engram.

Memory is not static; it can be amended by the conditions under which we recall it. Recall is rewriting memory.

The ability for us to forget is vital for us to function in our lives. As the famous Russian psychologist Alexander Luria documented in his book “The Mind of a Mnemonist,” remembering everything can be crippling for someone. The man in Luria’s book, called “S”, was not able to forget. He lived a confusing, cluttered life.  Everything he did or heard brought back a flood of memories and feelings from the past.

Without forgetting we can’t have new experiences. We have to forget: we have to forget childhood, we have to forget most of what we do to remain able to function in the future.

The necessity of forgetting is itself forgotten by market research. There is a pervasive idea that respondents actually can remember all these subtle impressions and emotions and then record them on a 11 point scale days or weeks after an event.

The truth is, mostly, they can’t. We have to forget most of what we experience. Trips to the mall or the supermarket are low on the list of things we have to remember because, mostly, they don’t matter. What grades your child got last week are a much higher priority.

The obvious problem is that market research needs those impressions that are forgotten. While we may not be able articulate with any accuracy what we have felt, the emotional residue will influence behavior in the future.

The core dogma of recall has to be rejected.  The problem becomes: what will replace it ?

We can’t have interviewers follow all of our panelists or respondents around and constantly monitor what they do in the hope of catching those fleeting moments of emotion about products or services that they experience. Those memories of emotions are soon lost, washed away in the stream of consciousness that allows us to function from day to day.

The Point of Emotion

It’s not often that new technology is really a revolution. Too often, vendors hype technology way beyond its boundaries.

However, smartphones just may deserve the hype. The smartphones that 65% of the US population now carry around with them have astounding processing power and connectivity. This power is being harnessed to give us a view of the consumer which is radically different from anything we have seen before.

Consumers are also using smartphones in various settings that were heretofore unheard of: on the toilet, while waiting in line for coffee, in transit, and just about anywhere there is idle time for the consumer. Smartphones and their addictive connectivity have users carrying these devices every waking moment in their lives.

Our need to be connected drives this smartphone ubiquity. This also presents an opportunity for research and feedback to live “in the moment” – in real time, not in recall time.

We call this the Point of Emotion (POE).

The Point of Emotion is the point in time when a consumer is using a product – drinking coffee, using toothpaste to brush their teeth. Technology allows us to capture emotions as they happen.

There will be many technologies that will allow us to leverage the Point of Emotion, the current technology we see as the most significant is QR codes.

Smartphones with QR code embedded feedback systems allow us to capture four critical pieces of paradata:

  • Timestamp – When the emotion was experienced.
  • Location – Almost all smartphones have GPS or wifi-enabled location triangulation.
  • Context – Embedded QR codes give granular context about products.
  • Unique Device ID – Unique identifers enable linking of data from different temporal phases.

Researchers, rejoice! We no longer have to rely on recall to capture the customer’s viewpoint.

The Point of Emotion is closer than ever. It’s for this reason that mobile technology is truly a revolutionary force in market research. There are millions of people carrying around technology that gives us a window into their lives. All we have to do is shed our own biases and make of use of what’s right in front of our eyes.

Meet the Data Triplets: Data, Metadata and Paradata

tripletsThere are three sorts of data, and very often you need all three to understand and use the data you collect from your survey.

Here are the three sorts:

1. The Data.

The data is the data, that is the actually numbers, codes or open ended text that the respondent enters into the survey. There should probably be a better way of describing this than “the data.”, maybe raw data is a better term to use ?

2. The Metadata

Metadata describes the raw data.

For instance the raw data for question 2 may be a “1” or a “2”. The metadata would say that the value “1” means male and the value “2” means female.  Or the metadata may say that the values for question 3 could be a range of 1-100, or 32-78.

Metadata gives meaning to the raw data, and so it is vital to the analysis process of the raw data that the metadata is present.  Otherwise the raw data is just a collection of numbers with no meaning.

One of the problems with metadata is keeping it connected to the right raw data. The wrong metadata with raw data can be a disaster.

Over the years data formats have got more complex, and one of the big reasons is to keep the metadata with the data. More recent data exchange formats/protocols such as JSON (Javascript Object Notation for the technically minded) have capabilities for attaching metadata to the data, which is a very good thing.

Raw data with no metadata is just a load of junk.

3. The Paradata

Paradata is the least well known of the data triplets. In the past decade or so it has become much more important for the survey research world.

Paradata is data which describes something about the way the raw data was collected.

It is data about data.

The most commonly used form of paradata used at the moment is data about questionnaire and question timings. That is, the time a respondent takes to complete a question or questionnaire.

This type of data is now one of the cornerstones of quality measurements for web surveys.

Obviously there can be many different sorts of paradata. For open ended text questions the length of text entered by the respondent can be measure, as well as the “level of vocabulary” contained in the text.

One metric used for web surveys is that of “speeders,” that is, the number of people who complete the survey extremely quickly. The paradata for time take to complete the questionnaire is used here.

Paradata can also be useful in revealing hidden biases; for instance, using paradata in the gamificaton of surveys is a rising trend. The time taken to do something in a gamified survey as well the action can have a great deal of meaning. Some researchers claim that hidden racism, some times unknown to the subject themselves, can be revealed by measuring someone’s reaction time to specific questions.

In a future post we will delve more into exactly how paradata can be used for quality control of web surveys.

If you want to get ahead…

I was in a sales meeting a couple of days ago. We were telling a prospect about our new smartphone interviewing app, SurveySwipe. We were talking about apps and how they are used. There were several “old guys” i.e. over 40 and a couple of younger people (under 40) in the meeting.  For the record I am an old guy – way old. The discussion got on to who uses apps and what they are used for. My point was that apps are the new communication medium, because email is dying fast in the under 40 age group.

The older guys didn’t seem to agree, so I asked the younger guy how many apps he had on his smartphone – his answer was “50 – I think”. This clearly shocked the other older guys, they had smartphones but they used them like phones, not as computers to run apps. Smartphones are not phones. Smartphones are personal computing systems which are now the direct channel to respondents and consumers. And email isn’t going to cut it anymore.
[Read more...]

SurveySwipe Helps Kick Off MarketMix 2011

One of the most exciting marketing conference in the Pacific Northwest kicks off today: Market Mix 2011. This event brings together some of the most well-known and dynamic marketing practitioners in the country. The range of subjects covered is impressive, from mobile marketing strategies, partnership building and leveraging social media, to career development skills for marketing professionals and cost-effective lead generation approaches.

Along with this innovative programs comes another exciting announcement. SurveySwipe, the new smartphone data collection platform from SurveyAnalytics, is available to all of the attendees to MarketMix 2011. The SurveySwipe is available on the Android and iOS (including iPhone and iPad) platforms, with versions for Blackberry and Windows 7 coming soon. SurveySwipe provides a flexible smartphone survey system with a range of advanced features for realtime data collection. A related product, SurveyPocket allows data collection when users are out of range of an internet connection.

Using the SurveySwipe application, MarketMix 2011 attendees can provide realtime feedback and comments during their time at the conference. Instead of surveys carried out long after the conference has closed, the attendees can voice there their opinions during sessions, break; in fact anytime they like!

Vivek Bhaskaran, CEO of SurveyAnalytics, said “ We’re really pleased to be teaming up with MarketMix 2011. We know it is one of the most prestigious conferences in the Northwest for marketing professionals. SurveySwipe is the ideal tool for this sort of event; it means the organizers can get feedback on how the conference is going in realtime.”

 

Let’s Save Smartphone Surveys: 10-7-140

Surveys have a great new platform: smartphones! We have access (we can get to people pretty much anytime) and identification (we can be pretty sure who is taking the survey).  It’s all for us to screw up. Is the survey industry going to do what it always does? As Betty Adamou pointed out in her paper at the Newmr (www.newmr.org) conference last year, survey research takes a communication medium and beats it to death. Telephone? Web? All victims of over use and poorly designed, boring,  and long surveys.

There has been a “140 characters per question text limit” meme floating around recently started by Annie Pettit from Conversition Strategies. This made me think, “How can we apply this idea to smartphone surveys?” The 140 characters for the question text makes sense, but what about the rest? So here are my proposals for NOT destroying the smartphone platform:
[Read more...]

Will Smartphones Save the Survey ?

At Survey Analytics, we’ve been a bit smart phone focused recently, due primarily to our recent State of the Union dial test study. As I was testing the SurveySwipe software on which our study was based, it started me thinking about the whole idea of surveys on smartphones.

One thing that is clear is that the smartphone screen has some limitations. Obviously it is not a laptop or desktop monitor sized screen! You have a screen, but it is small. You can’t do anything really complex on the screen with a question in a survey without running out of room pretty quickly.

But this smartphone limitation is a tremendous opportunity for the market research industry. It may mean that the smartphone saves the survey as a way of collecting data in MR.
[Read more...]

Joining Survey Analytics

I just wanted to let you know that I have joined Survey Analytics (surveyanalytics.com) as Executive Vice President as of today.

I’m very excited to be joining such a dynamic company which is one of the market leaders in accessible in online survey and analytical tools. Survey Analytics clients include McGraw Hill, Experian, Roku, Career Builder, Texaco, Microsoft, Motorola, Qwest and many other leading companies.

Survey Analytics is fast becoming a leader in the consumer intelligence and “DIY” market research field, with innovative companies such as BadgeFarm.com and SurveySwipe.com, a new smart phone interviewing system.

Vivek Bhaskaran, CEO and founder of Survey Analytics commented:

“We are delighted that Andrew is joining us at this critical new phase of our growth. His experience and insights into the business domain are already proving of incredible value”.

You contact me on andrew.jeavons@surveyanalytics.com .

The Twitter Fire Hose

Have you ever heard of the Twitter “fire hose” or the “sprinkler” ? If you have, you have probably been doing software development work on connecting to Twitter. The “fire hose” and the “sprinkler” are just two of the terms Twitter use to describe the type of connections you can make to Twitter with an application. As the name implies the Firehose is everything anyone tweets anywhere, and very few companies need or can cope with the volume of data that is produced. Most applications filter or search the stream for hashtags or keywords or users. Absorbing the full force of the global twitter population is not for the meek. According to ComScore there were 25 billion tweets in 2010. That’s a lot of anything.
[Read more...]

Seems statistics in MR don’t matter…..

I was reading Ray Poynters excellent blog article here:
about the Likert scale. Ray stated:
“If a researcher makes the wild and unjustified decision (IMHO) to treat Likert numbers as an interval scale…”
I took this as him saying he agreed it was wrong to treat Likert as an interval scale, he did only say that it is “wild and unjustified” to treat them as interval. This is a decision that is made constantly in MR. I will go further and say it is plain wrong and bad statistical practice to treat Likert scales as interval data. It is an ordinal scale. It should be treated as such. You can’t take the mean and standard deviation of ordinal scales. They don’t have them. They have a mode (sometimes two modes), they have a median, they don’t have a mean.
I got these radical ideas some years ago. After my degree in Psychology and post graduate work I worked in a medical research facility called the “Institute of Neurology” (ION) in London UK. I worked in the “Computing and Medical Statistics Unit” in a basement in Guildford Street complete with IBM card punch machines, cockroaches and CDC UT2000 remote RJE terminal for the University of London Computer Center. I think the drum printer on the UT2000 gave me hearing damage, it sounded like a machine gun.
I worked for a medical statistician called Liz. I was what I called a “data monkey”, I ran programs, wrote them, cleaned data, killed cockroaches and generally helped out. Liz had worked with Sir Richard Peto, who is currently Professor of Medical Statistics and Epidemiology at the University of Oxford. Liz was passionate about her profession. I recall several studies we worked on together. One was a drug trial for a drug to cure Multiple Sclerosis, another was a long term epidemiological study of Multiple Sclerosis, another was a drug trial of a chemotherapy drug to be used against a particularly evil form of brain cancer called Gliomas. Then there were studies about stroke models in rats, muscular dystrophy and Tourettes syndrome.
We had a strict procedure for all data. First all interval data variables were plotted and looked at. Then these variables were tested for normality of distribution, if they were not normal appropriate transformations were applied to correct any anomalies and then they were re-tested for normality. If they still didn’t pass the normality test they were only analysed with non-parametric techniques. Other data variables were plotted too, and the histograms looked at carefully for anomalies. I can recall Liz spending quite some time researching if you could use a T-test on percentages. She concluded you could not. She decided percentages belonged to the Cauchy distribution, which has no mean or higher moments. Thus a T-test would be statistically invalid.
I asked Liz about this procedure of treating data and the rigour she applied. I came from psychology, we were a little more lax in our approach. She said we had to remember that the results we obtained mattered. They could be life and death decisions. A type I error on a drug trial could lead to more people dying because they were given a drug that didn’t really work. A type II error on the epidemiological work may miss an important antecedent to Multiple Sclerosis, a crippling disease.
We were working on data for the drug trial for treating Gliomas, a form of brain cancer. Gliomas remain a deadly form of cancer, with only a 50% survival rate within one year of diagnosis. We were using something called Survival Analysis to test the effect of a chemotherapy drug. Liz said we had to wait for more events to be sure of the results. An event was someone dying. I happened to look at the columns in the data which contained the ages of the subjects. 18, 19, 21 – these were people only a little younger than I was at the time. We had to wait for them to die to be sure the conclusions that were made about the drug (Vincristine) were correct. The results really mattered.
It seems to me the question about Likert scales is not so much about if you can treat them as an interval scale rather it is this: do the results matter ? Do you care that the results are correct ? Do the results matter enough to do the work properly ? If the results do matter, do it properly.
From what I can see very often it seems market researchers think the results don’t matter……

The Problem with Neuromarketing….

There are several problems really. The first is the name. It should really be called applied cognitive neuroscience (ACN), because that is what it is. Hopefully this would counter all the specious arguments about it being scientific. The New Scientist (http://www.prnewswire.com/news-releases/neurofocus-and-new-scientist-magazine-apply-neuromarketing-to-select-cover-design-100053049.html) test raised some comments about the science of ACN. I have to disclose in what seems like a previous life I studied cognitive neuroscience (we called it neuropsychology back then). Trust me, it is a science, it has been around a *long time*, many decades, and it also uses statistics correctly. This latter fact is a novelty for a lot of MR, I know. At least ACN tries.

There is also the privacy “discussion”. ACN is about as invasive as looking at someone who is blushing and deducing they may be embarrassed. ACN measures physiological correlates of mental states or processes. It happens to do them via electrical signals measures from the brain. We do this all the while with body language, speech tones and so on. Just because there is a lot of equipment in ACN and latin words doesn’t make it any different.

The biggest problem ACN has is sample size. N = 19, as in the New Scientist test, isn’t much. It is barely enough for a single quota cell. Making big decisions based on tiny samples mostly ends in tears. The sample size issue relates to the technology of ACN. The electrodes on the scalp can take time to set up and this limits sample sizes. However several companies have ways round this with either limited electrode placement (not so good – one electrode gets you nothing except muscle noise) and less “invasive” caps that hold the electrodes on the scalp without glue. The latter holds the most promise so far as I can see. Sample size is a solvable problem, scalability may take time, but compared to the rest of the technology used in ACN it is not the most complex problem. Several companies are building normative databases which will be hugely useful.

The problems with ACN are solvable, the potential is huge…