I Hate Social Media Research Because…

We’re counting down our Top 10 blog posts of 2013. Coming in at #3 is this post originally published July 20.

Dislike emoticon

Oh yes, I’ve heard it all. Having been in the deep end of social media research for several years now, I’ve heard every possible complaint about social media listening research that exists.

I hate social media research because: there are no demographics

Social media research does not include demographic data. I’ll admit right off the bat that this is a commonly raised concern. Coming from traditional research where every single survey complete and focus group attendee is associated with demographics, it’s hard for experienced researchers to come to grips with datasets where they don’t know the stated demographics of even a few of the participants.

But let’s not look at the glass half empty. This is definitely a glass half full moment. I am still astounded when I see a social media dataset where 20% of the data has demographic information. Really, when was the last time you wrote a tweet like this?

“My name is Annie. I’m female, speak fluent English, and have a PhD. I like Kitchenaid products.”

So think about it this way. If you NEED detailed demographic data, then your main methodology should be a survey. Social media research should fill in the gaps, add flavor, show details that couldn’t be evaluated with the survey data. You must always choose the right method for your research objective and if that means doing a survey or a focus group so that you have demographic data, then so be it.

I hate social media research because: it’s not a representative sample

Are we really going to go there? I guess we ought to. In 99.9% of market research, we aren’t using a representative sample in the strict sense of the word. Survey panels aren’t probability samples. Focus groups aren’t probability samples. Market research generally uses convenience samples and social media research is no different.

But here is the difference. We’ve all heard the statistic that a tiny percentage of people answer the majority of all market research surveys. In other words, most people aren’t participating in the survey experience, and we never hear their opinion. Similarly, when we conduct social media research, we only listen to people who wish to share their opinions on Facebook, Twitter, YouTube, or any of the other millions of websites where they can write out their opinions. No matter what research method you choose, you only hear the people who wish to contribute their opinion in that mode.
Who is talking about the brand anyways? Alright, so we know SMR doesn’t use rep samples. Big deal. One of the reasons we use rep samples in traditional research is to ensure we are talking to the right people. We do a rep sample because a product is used by a rep sample. We do a male-only sample because a product is used by males only. In both cases, we choose a particular sample because it is most likely to reflect product triers and users. Guess what. The only people talking about your brand in social media are the people who care about your brand. Whether they hate your brand or love your brand, you have instantly reached people who are relevant to your brand. They have raised their hand to tell you, “Listen to me. I have an opinion about your brand.”

If you require a rep sample, use a survey because that is the closest approximation. Always use the right method for the job.

I hate social media research because: it cannot measure awareness nor incidence

What is awareness? It is a measurement of how many people or the percentage of people who have heard of the topic of concern.

Have you heard of Justin Bieber? Yuppers.
Have you heard of KitchenAid small appliances? Darn tootin!
Have you heard of Freelin-Wade? Uhhh…. should I?

Chances are that anyone reading this blog post has heard of Justin Bieber. It’s pretty hard to ignore this young man whose fine hair has graced many a magazine cover and news website. But here’s the important question: Have you mentioned his name in social media? Perhaps. Perhaps not. Even though you’re aware of him.

What about KitchenAid? If you’re a regular reader of this blog, you’ve probably heard me yammer on about KitchenAid at one point or another. So have YOU mentioned KitchenAid in social media? Probably not.

And what about case #3? The only reason I know the brand name Freelin-Wade is because I just thought of an obscure category and searched for any brand name associated with it. So plastic tubing it is. I’ve never heard of them before nor have I ever tweeted about them before. But even this doesn’t reflect a case where social media would accurately measure awareness. You see, there are probably plenty of people in the industrial business who know Freelin-Wade very well but they too have never tweeted about it.

So this brings me to the answer. As much as we’d like it to, social media can not measure awareness. You may be able use it as an under-reported approximation, but you won’t know by how much it under-reports. If you find that your research objective is specifically awareness, then you’ve come to the wrong place. Head on over to survey research.

I hate social media research because: it provides no insight

Let’s start with what I mean when I use the word insight. I don’t mean data. I don’t mean describing results. I don’t mean putting a series of results into a story that makes logical sense. When I say insight, I mean that magical moment when an idea you’ve never considered before suddenly pops into your brain and a light bulb explodes over your head.

First off, whether you’re conducting survey research, focus group research, or social media research, all of these research methods result in data and descriptive results at your fingertips. Tables, lots of tables, and charts, lots of charts, will be at your disposal. But no matter the method, just because you have data at your fingertips does not mean insights will spring forth like a popcorn maker.

Of course, in the case of social media research, if consumers don’t want to talk about
something, you won’t learn about that particular aspect. But in the case of surveys, if you don’t ask about that particular aspect, you still won’t get an answer. Either way, we aren’t talking about insight. Insight requires the application of brain to data. No brain = no insight. No brain applied to social media data = no insight from social media data.

I hate social media research because: it doesn’t do anything better than what I’m already doing

This is an easy complaint to address. How else are you measuring opinions in social media without using social media research? How else are you learning what people are saying about you behind your back without your prompting them on every item. How else are you gathering opinions from thousands of people all across the country on a daily basis? With that basic methodological issue put aside, let’s address the other issue.

There are so many other methods that people are already using that do just as good a job as social media research does.

Surveys address my very specific questions in very specific detail. Agreed, but social media research can go beyond the 30-minute barrier where the data quality of survey declines very rapidly. Use the survey to get the initial set of quantitative data as accurately as possible. Then, go to social media research to get the rest of the psychographic data.

Focus groups let you interact with and probe research participants as they use products. Agreed, but social media research gets you out of the lab and peeking into real life to learn how people talk to their peers in a natural environment. So do the focus group. Observe and learn. Then go to social media to find out so many more intricate details that didn’t come up during the focus groups.

I hate social media research because: Social media research is not accurate

Accuracy is something we address in every research methodology. We talk about probability samples, representative samples, appropriate research design, well-written questions, margin of error, and more.

We know we manipulate and weight research data so that it appears to be representative of the population we’re interested in. We do our best to choose and create a research methodology that is best suited to answer our research objective but have difficulty finding the one that does it all. We try to write quality questions but always find a leading question, a biased question, or a misleading question somewhere in our surveys. We know that margin of error shouldn’t be reported on most surveys and yet margin of error is everywhere.

So here’s the deal. As researchers, we understand that error is a part of every research project we conduct. We know when we start a survey or focus group project that multiple sources of error will be introduced at every step. We’ve learned how to work around these errors so that our research results are still meaningful and useful. That’s how we’ve been trained.

Why do we expect anything different from social media research? No, it’s not a probability sample nor a representative sample. Sentiment analysis is far from perfect, and content analysis isn’t perfect either. So in the end, if you decide that the error rate of your social media research is simply too great, then choose a better vendor. Quality does not depend on the method. Quality depends on the vendor.

I hate social media research because: it doesn’t have data on anything I’m interested in

This is a complaint I’ve heard a lot. There’s nothing new in the social media data. I’ve seen all of this in survey data before. Hmmm. That’s an interesting comment. You’ve seen all of this in survey data before. So tell me, why are you bothering to conduct this research now?

There are two solutions to this problem. First, people forget that when conducting research, it is to your great advantage to start with a research objective. Without an objective, you’re left to wander and ramble without direction, and it’s nearly impossible to find something when you don’t know what you’re looking for. With that in mind, don’t waste your time with social media research until you have a firm research objective in mind.

Second, there are tidbits of new information in every dataset. You just have to know where to look and how to look. Tidbits just raise their hand and say “Look at me!” For instance, one of my favourite things to do is look at the variables that generated very small sample sizes. For just a minute, ignore the one or two hundred variables that generated a few thousand records with quantitatively reliable results. Instead, focus on the variables that generated sample sizes of only 5 or ten people. These are the variables that either pissed off 5 or 10 people or absolutely delighted 5 or 10 people. These are the things that more often than not desperately need to be fixed or desperately need to be bragged about from the mountaintop.

I hate social media research because: it is too expensive

My sincerest apologies, but social media research companies aren’t charities. Here’s why.

When you engage a survey company, you expect them to write high quality survey questions that address your specific needs.
When you engage a social media research company, you expect them to build high quality variables that address your specific needs.

When you engage a survey company, you expect them to create appropriate Likert scales for the survey questions.
When you engage a social media research company, you expect them to score sentiment on Likert scales as accurately as possible.

When you engage a survey company, you expect them to clean out random responders, straightliners, and incentive seekers.
When you engage a social media research company, you expect them to clean out spam and irrelevant data.

When you engage a survey company, you expect them to ensure that the responders are an appropriate selection of people.
When you engage a social media research company, you expect them to ensure that the responders represent all of the relevant websites where people chatter about your brand.

When you engage a survey company, you expect them to behave ethically, to avoid interacting with children without permission, to avoid using personal information without permission.
When you engage a social media research company, you expect them to behave ethically, to avoid collecting information from children, to avoid using personal information without permission.

You see, when you engage the services of a social media research company, you have agreed to work with a company that holds the same high standards of quality and attention to detail that you expect from survey companies. Quality costs money. End of story.

I hate social media research because: it is too complicated

Well, I can’t argue with you too much there. Social media research isn’t easy at all. There are so many intricacies and nuances that it’s impossible to simply buy one, analyze one, and prepare a report on one.

Data quality is far more than just removing spam. It’s about including messages the spell the brand name wrong, use slang words, and don’t even mention the brand name at all.

Sentiment analysis is far more than love is positive and hate is negative. T of words and phrases and emoticons and slang carry emotion, and many of them depend on what category you’re talking abou.

Want to measure awareness? You had better know why that’s not possible. Want to know why the most expensive campaign you’ve ever launched is generating no data? You had better understand why. Want to know why I can’t give you a solid measure for a concept called “Makes customers feel at home?” You had better understand why.

Though the social media research I do has been designed to mirror survey research as closely as possible, there remain many differences, most of which require you to put your thinking cap on. So if you thought you could just pick up a single serving of social media research and be on your way without any help or training or guidance, then you are terribly misinformed.

But I’ll help. You just need to ask.

I hate social media research because: it doesn’t do laundry

I hate to disappoint you but not only does social media research not do laundry, it also doesn’t cook, clean the bathroom, dress the kids, make lunches or vacuum the house. Sigh.

But there are other things social media research doesn’t do. It can’t measure awareness. It can’t be used for In Home Usage Testing. It can’t measure brands that people know but never talk about. It can’t give you census rep results. It can’t ensure that all responses come from a specific country.

Essentially, social media research is not the ultimate, most fabulous, all encompassing research tool that ever existed on the face of the earth. It’s taking a while, but people are gradually realizing this. No, social media research can’t do everything. It isn’t designed to do everything. It was never intended to do everything. And yet, for some strange reason, some people are shocked when they find out it can’t do everything.

Fortunately, researchers have this extraordinary thing available to them. It’s called the market research toolbox, and it is overflowing with coolness! It contains surveys and focus groups and bulletin boards and facial tracking and mobile and neuroscience and, oh yes, it also contains a handy dandy little tool called social media research. None of the tools in that toolbox can do the laundry or the dishes or clean the toilet, but every tool has its own unfortunate disadvantages and its darn cool advantages.

When it comes right down to it, there is no one research method that can solve every problem nor solve the entire problem. Surveys need to be used to do the part that they do well – census rep quant. Focus groups need to be used to do the part they do well – in-depth qual. And similarly, social media research needs to be used to do the part it does well and that doesn’t mean mimicking other methods. It means large samples, unsolicited opinions, geographical diversity, and spontaneous unexpected topics of discussion.

So, no, SMR can’t do everything. Why did you think it did?

Annie Pettit is VP, Research Standards at Research Now and Chief Research Officer at Conversition, an e-Rewards company


Speak Your Mind