The Dilemma of Social Media Measurement

[Editor’s Note: The  following post by Michael Wolfe was originally published by and is syndicated with permission by The GreenBook Blog.]

Everyone with any sense of what is going on in the field of marketing right now knows that there is almost a mad frenzy about social media, the likes of which we have seldom seen. In fact, this frenzy has become so strong that it has spawned almost a gazillion self-proclaimed experts and gurus. It almost feels like the American Wild West, with the itinerant snake-oil salesman coming to town to sell us a bottle of his miracle cures.

As I have reviewed the literature on this subject, there appears to be three camps or approaches towards social media measurement. These are as follows:

  1. The word counters. These simply count the frequency of mentions of any given subject or brand name across social media such as Twitter, Facebook and various Blogs. The theory behind this approach is that word frequency is sort of like market penetration, the more the better. Some who follow this approach might go one step further and parse their word counts into positive and negative sentiment groups. This sometimes evolves into more sophisticated sentiment analysis via sophisticated text mining tools.
  2. The influence enumerators. This camp believes that it’s not the words that matter, but the influence of the person saying them. Via some algorithms for scoring individuals based on number of friends, followers, re-tweets or klout scores, it is possible to differentially score the word counts based on the level of influence of the person posting the conversation.
  3. The linguists or engagement scorers. This approach believes that language and semantics is everything. Through the science and rules of linguistics, it is possible to “quantitatively score” social media conversations and determine how “engaged” these conversations are.  nfluence and impact in social media is very much a function of language, word nuance and context.

Presently, there is a lot of confusion regarding which approach is most valid. The only research I can cite is from Bernardo A. Huberman of Hewlett Packard. Bernardo found that social media can predict such things as movie success. He also found that the quality of a message can pierce through the noise and influence an audience. He likewise found that the number of friends and followers on social media does not add much to the influence of the message.

The impetus for developing social media metrics is obviously targeted at developing a capability to measure the impact of social media conversations on people’s attitudes and behaviors. Significant here is the conversations on brands and how these conversations affect brand performance and ROI.

My own journey here tends to favor the linguistic approach. As a player in BBDO’s marketing-mix modeling consultancy, clients challenged me to come up with an approach to measure the impact of social media on their brands’ performance. My approach to this is fairly simple.   That was to find a metric which correlates best with brand sales of any of our 8 blue-chip clients. In this journey, I tested and evaluated many of the “word counting” and “sentiment” metrics from major social media data vendors. Sorry to say, none of them were found to have greater net correlations to brand sales of about 0.15. Some of the sentiment indicators, in addition, even had counter-intuitive relationships with client brand sales.

Our journey finally led us to a metric called the Semantic Engagement Index or SEI as developed by a company called Linguistics Insights. This metric was developed by Dr. Peyton Mason and Dr. Boyd Davis. Davis is Professor of Applied Linguistics at the University of North Carolina. Together, Mason and Davis developed an approach which first parses large quantities of social media conversations into positive and negative sentiment groups. From there, each group is quantitatively scored using Linguistics science and rules. The scoring algorithm used defines consumer engagement by scoring conversations on two dimensions. Those dimensions are the degree of personalization and emotional effect contained in the conversational language.’

With our first client, we generated a time-based SEI metric. We plotted this metric against brand sales for one of our clients.   As shown here, we plotted the ratio of positive to negative SEI. It’s correlation to brand sales was an amazingly high +87%. We have never found any single social media metric which tracks so closely and has such a strong connection to brand sales and performance!


We have since found similar and strong correlations of the SEI metric with brand sales of other clients. We obviously think we have made a very significant discovery here. Certainly, our efforts have enabled us to include the SEI metric in our marketing-mix models and directly measure the impact and ROI of social media on our respective client brands. Overall, the evidence so far seems to support the linguistics approach to social media measurement; but obviously we need more supportive evidence across many more brands in order to draw definitive conclusions.

In sum, our research has found at least three different approaches to developing social media metrics, all with the purpose of measuring its impact on brands and consumer attitudes. While I believe there is some evidence supporting the linguistics approach, I must say that the evidence to-date of which approach is superior is significantly lacking. The bottom-line here is that this really represents an opportunity for Marketing Research to step to the center and demonstrate leadership. Certainly, many researchers have access both to the metrics and brand sales data to do so. Whether they have sufficient moxie to do so remains to be seen.

About Michael Wolfe

Michael Wolfe has 25+ years of experience in marketing analytics that has taken him across firms such as Kellogg’s, Fisher-Price, Kraft Foods, Coca-Cola and his own firm, Bottom-Line Analytics. Michael has a masters in Economics from the University of Iowa. Presently, Michael helps lead the modeling practice called “The Worth”, a group specializing in marketing and media response modeling and part of the agency BBDO Inc. Some notable clients of “The Worth” include such firms as AT&T, L’Oreal, Hyatt Corporation, Starbucks, Federal Express and Daimler-Benz. Recently, Michael’s expertise in marketing –mix modeling has led him to develop innovative approaches for measuring the impact and ROI of social media on client brands.



  1. This is great great news!!! I am an advertising professional for the last 5 years in the Middle East and the last 3 years I have had all my clients asking the same question…’Can we measure Social Media impact and ROI?’ …I would love to promote this tool in the Middle East and India…..I see this as a great relief to all the Clients for the monies they have invested and they will be investing in the future. Wow this awesome news!!!!

Speak Your Mind