What Apple Got Wrong with Its Question about the 5c

iPhone 5c in jean pocket

The most popular research tweet on the Twitter #MRX hashtag in the past fortnight was Tom Ewing’s tweet of this screencap:

Cor, I can see why Apple don’t make a big thing about their market research – pages of grids, stone age stuff. #mrx pic.twitter.com/uV1rbPDTSq

— To Mewing! (@tomewing) May 1, 2014

iPhone 5c importance matrix question

Some folks on Twitter asked Tom what was wrong with this survey question. Here’s my take on some of the problems with this question, ranging from strategic mistakes to tactical missteps:

  1. Consumers often don’t know why they made a particular purchase – First, and most importantly, this question assumes respondents are conscious of the reasons they made the purchase. In reality, any purchase is as much an emotional decision as it is a rational one. The reasons we think we made a decision are often not the reasons we did (see “The influence of in-store music on wine selections“).
  2. The wording is not from the consumer point of view – The phrases “iOS 7″, “A6 chip”, “1080p HD video recording” and “LTE wireless” are not the sort of things you will hear in casual conversation. A good survey researcher speaks the language of the consumer, not the jargon of the engineering staff. Consumers are interested in benefits, not features: they want to be able to run the latest apps (the benefit of iOS 7), they want fast response (the benefit of the A6 chip), they want high resolution video (the 1080 pixel recording), and so on. I considered buying the iPhone 5c last month but didn’t because it didn’t feel “sturdy” enough – that’s the type of language consumers will use to think about their rationalizations. Apple should have run a pilot survey asking 100 consumers the open-ended question, “Why did you buy the iPhone 5c instead of a different phone?” and then developed a list of the most common reasons, in consumers’ actual words.
  3. The list of items seems incomplete – Perhaps the matrix was longer than the screen capture shows (heaven forbid!) but lots of common reasons why people might have chosen this particular device are missing: the Apple brand, price, being stuck in a contract with a carrier, wanting to continue to use existing Apple content (iTunes songs and shows, apps, etc.), and so on.
  4. Matrix questions produce less accurate results – A grid of questions shows less variance between items, as respondent think about each row less than if each item was an independent question. Plus, respondents hate them! – subjecting your own consumers to them is foolish.   (See Grid-Cheating Panelists on Trial.) Given the lack of variance between results for items, whenever one of my clients forces me to use a matrix like this, I try to follow it up with, “What feature or attribute was the single most important to you in making your decision to purchase?”
  5. The 5-point bipolar scale is suboptimal – The most reliable bipolar scale (a scale with opposites, in this case, importance and unimportance) has 7 fully labeled points, but bipolar scales are more cognitively difficult for respondents. A 5-point unipolar scale would be more reliable: “Not at all important, slightly important, moderately important, very important, extremely important.”

To the anonymous Apple staffer who wrote this, please take a course in questionnaire design from Research Rockstar or Burke Institute. And don’t feel bad: these are common mistakes. (And I’ve made them all.)

And, one last item: Tom Ewing is an incredibly talented blogger. It’s ironic that his tweet about an Apple survey was more popular than any of his many great posts. You can check them out at Blackbeard Blog.

Jeffrey Henning, PRC, is president of Researchscape International, which provides “Do It For You” custom surveys at Do It Yourself prices.  He is a Director at Large on the Marketing Research Association’s Board of Directors. You can follow him on Twitter @jhenning.

Advertisement

Comments

  1. With all due respect, Jeffrey-PRC, you do a nice job of illustrating why many are turned off by present day market research, and actively not using research or market researchers in business decisions, especially in the tech world.

    Calling out this “anonymous Apple staffer” feels a bit odd and a tad
    confrontational given that many make these same decisions. Perhaps there is some history or background. I don’t know.

    More importantly, I am pretty sure it accomplished little with respect to changing anyone’s point of view on the value of research, or the value of making the changes you suggested.

    As for tone, do you speak to your own clients this way? Hope not. I hope none of us are. Again, perhaps there is some history or background. I don’t know. Many of these issues (unfortunately) are commonplace, and I have seen many surveys with grids, etc from the likes of the Honomichl top 50.

    “Consumers often don’t know why they made a particular purchase” is your first point. Really? You are telling Apple this? c’mon.

    Many of your points are either a tad academic (kind of), trivial with respect to the overall business question (iPhone 5c sales), or not entirely correct.

    For example, the ‘incomplete’ list items that you mention are probably really obvious to Apple (or anyone) or better answered through analytics or other non-survey data. Wasting survey space is suboptimal.

    You mention the phrases “iOS 7″, “A6 chip”, “1080p HD video recording” and “LTE wireless” are not the sort of things you will hear in casual conversation. Perhaps. However, they are the sorts of things that Apple and iPhone customers do spend time talking and thinking about.

    And, if history repeats itself, the mainstream will likely be speaking this language very soon. I think you can count on it.

    Perhaps, mrx needs to take a step back and study what this company has accomplished the last 30 years with respect to revolutionizing customer experience, computers, phones, and communication. It wasn’t the optimal use of 7 bi-polar scales.

    Believe me, I wish that it was but really it wasn’t.

    I fear this blog and corresponding tweets probably solidify the notion that MRX is as irrelevant as ever. I am all for pointing out better ways of doing things, but tone, perspective, and balance are critical to the message.

    Apple has sold more than 500 million units of the iPhone
    worldwide. The 5c and 6 will probably bring many in the emerging markets to the web. In China, Brazil, Indonesia, Poland, and Turkey, iPhone sales
    grew by double-digits year-over-year, and in India and Vietnam, sales more than doubled.

    What does market research have wrong is the better question?

    “If you tell a joke in the forest, but nobody laughs, was it a joke?”

    Steven Wright

  2. Jeffrey Henning says:

    Thanks for the long and thoughtful response to my snarky blog post, Stephen. (It’s a small world, but I wouldn’t want to paint it.) My only agenda was that if you are going to do market research, you should follow best practices. Not enough of us do.

    There’s no backstory: I have nothing against Apple — I’ve been a happy user since the Apple ][ and a happy customer since the original Mac and I love my iPhone 5. Nor did I make any sweeping claims that one bad survey was going to have any effect on their business. Nor do I believe that criticizing bad practices imperils the future of the research industry. Quite the opposite — tolerating them does.

  3. Thanks for reply. I agree with some of your points, and I enjoy making snarky comments myself. So much so I find myself unemployed. Kidding..kind of.

    I may be a little sensitive as former Apple survey guy-a little bit of a nightmare for that poor soul I imagine.

    With Apple there always seems to be that noise about them not doing surveys and market research. And its all just very inaccurate. The original tweet seemed to insinuate the design was an artifact of the “stone age” mentality. Unfair a bit (and then the bit about training). KK is a rockstar though.

    Forget the emotional part for now. But do note that if I ever find someone giving me the business on twitter, oh man..

    Let’s think about it practically and with the goal of producing best results for business.

    I do think there is a difference in opinion with respect to what is prioritized and highlighted. And I do think that there is a wrong way to address some of these ‘bad practices’.

    My real concern is that as market researchers, who are expected to add significant, actionable, insight to the business at a reasonable cost, across different geographies, languages, platforms, carriers, models, devices, we are way off with respect to how prioritize vs. what the business may need. I would say in the technology/internet space it feels so.

    For some industries–CPG, I believe your points would certainly be well received and would “move the needle” so to speak with respect to the quality of the insights.

    Being on the client side, my number 1 priority is adding value to the business in a timely and accurate manner.

    There needs to be a more realistic and assessment of what the actual ($$) benefits are to employing the ‘best’ techniques are, and what is scalable in this new day and age.

    There is a very different cadence and speed in the tech world with respect to how biz decisions are made. Its not optimized for survey research but disruption, innovation, etc

    With global companies like the size of Apple, HP, etc. Its a very different research ecosystem. Not an excuse, just a reality. There still needs to be standards and quality but scale of things is different.

    Best practices and high standards are absolutely essential and when possible we should strive to follow them. I too love MaxDiff, and understand that it is a better technique for differentiating items than Likert scale questions.

    But as you know Apple (or HP, MSOFT, etc) are really huge. iPhones sold all over the world. Phones are sold via different carriers, retailers, and surveys done in many many languages (proof read by native speakers).

    As a researcher, you have $50K and 5 weeks to get information and presso ready. You need to compare China, France, Brazil, UK, Japan, and all those little EMEA countries that I forget the name of.

    You also need to look across models, and have the survey programed for both desktop and mobile. MaxDiff in Cantonese on an iPhone? Hmmm Maybe? That will be a zillion dollars for 3 countries says Vendor X.

    No China or Japan unless you pay $2 zillion more. But you could get all the countries done though if you just did scales. Now what?

    Keep in mind that you need to explain results to wide array of employees with different skills, strengths, and language skills. If the research is going to make an impact it needs to get out and be socialized, yada, yada, yada.

    Determined to do MaxDiff, you explain to Yiqing Yin, Wang Zhizhxi, Wang Shcvipeng that Jordan Louviere-a brilliant man-thinks MaxDiff is better. And they won’t get their results cause poor survey practices won’t be tolerated.

    Yiqing Yin and Wang Zhizhxi meet to discuss. They make a few phone calls and no one sees or hears from Jeff ever again. And they decide against the Maxdiff in the end.

    Point is, it feels like MRX is way way off sometimes, and that we are talking amongst ourselves. There needs to be a more consultative approach, and reorientation of priorities.

    And what I see/fear is slow but steady move away from using MRX as source of knowledge and consumer insight, and a preference for analytics and “big data” because there is a belief that its more accessible and aligned to business.

    Survey design is critical but mobile, globalization, “Big Data”, all will disrupt to the point where there needs to be major compromise. Not happy about. But it feels like its already happening.

    If the goal is to really maintain the standards, then there needs to be some flexibility with respect to how we do things.

    Keep up the standards but with the needs of the business as the most important goal.

    Good discussion and important points. I love all your other pressos and work too. More snark.

Speak Your Mind

*