Social Media and the Sampling Problem

Midway through Elea Feit's presentation at the WAA Philadelphia Symposium on work that the Wharton Customer Analytics Initiative has done, I realized that one of her research examples had direct bearing on a recent series of posts that I've been doing on Social Media. I mentally kicked myself, because I'd seen the same basic presentation at X Change and had quite forgotten the relevant part.

Here's the setup. We've done quite a bit of Social Media measurement, scorecarding and analysis at Semphonic in the last year or two and so I've started a series on some of the learnings. In the course of that time, I've come to realize how difficult it is in Social Media to get the data you need and some of the perils of interpretation. I started my series with a description of six functions of social media (ranging from Customer Support to Customer Research to Social Campaigns) and the corresponding measurement of each. Each of these functions is valid and useful, and the measurement of each is quite distinct.

However, one function that I'm rather skeptical of is brand sentiment monitoring as a replacement to traditional brand tracking studies. I haven't laid out the full case for this in any post yet, but I did, in my next post, show some of the reason for my skepticism about this AND about the use of sampled posts and human readership to classify brand sentiment. The gist of my argument is that there are multiple levels of sampling involved in creating a social media measurement and that each of those samples had potential issues.

Listening tools sample the internet, they do not produce an exhaustive set of all verbatims (a particularly serious problem internationally). From their sample, the analyst must build a subset or sample using keyword classifications that are difficult to validate and can easily introduce significant bias. Most organizations give almost no thought to this step and don't realize how critical it is in achieving accurate representation of your topic-set. Finally, if you are further subsetting verbatims for human readership, you have a sampling problem that will either introduce significant bias (if you weight by source or influence or any other variable) or severely limit analysis of key sources like traditional media.

All this poorly controlled sampling makes me skeptical of the validity of most brand-tracking functions with Social Media as a true representation of consumer sentiment. I just don't think most companies have done the work to justify the data.

Well, it turns out that Wharton's research has, at least partially, addressed an even larger issue: how representative is social media chatter of word-of-mouth (WOM) conversation. In other words, if you look at social media mentions, are they the same kinds of subjects and attitudes expressed by people in real conversation.

If social media isn't representative of word of mouth, then the whole enterprise of brand tracking is doomed, sampling be damned.

The study Elea showed tackled this problem in classic academic fashion. They listened to a social community and they setup a Word-of-Mouth (WOM) experiment and they compared the results for two different industries.

As you might almost expect, the results were both positive and negative. In their study, automotive social chatter correlated reasonably well with WOM. Beauty-Supply chatter had zero correlation with WOM.

In other words, if you're Ford, then if you can listen very carefully, you have some chance of using Social Media to measure brand sentiment. If you're ProActiv, no matter how carefully you listen and sample, you're not tracking actual consumer sentiment when you listen with Social Media.

And the million dollar question, of course, is what if you're neither automotive or beauty supply? How do you know if social media measurement is correlating to actual customer attitudes?

You don't.

Of course, that won't prevent countless companies from selling you their tools and services on the assumption that it all works perfectly for you. It won't prevent countless organizations from presenting this data as if they were confident of its meaning.

But I actually took encouragement from the study - the correlations in automotive were better than I would have expected. Of course, these were academics designing a careful experiment (not your PR firm slapping together a Keyword Profile) and there are many other factors -such as social campaigns - that might visibly distort such findings over time. On the other hand, the social medium is becoming more reflective of the general population and perhaps conversations are becoming steadily more representative of WOM. It's certainly possible, and at the very least it suggests that Social Media Measurement for some industries might fruitfully include brand tracking.

The research findings do prove that assuming that such a relationship will hold is wrong. For at least some (most...a few...many?) industries, there is no valid use of Social Media Measurement as a reflection of brand WOM and sentiment no matter how careful your measurement.

That's important.

One of the great challenges of academic research is finding ways to tackle a problem that suggest the availability of a general solution or the lack thereof. Some findings have very broad applicability and can be used in a wide variety of situations. For others, the key variables are inherently local. The research here suggests that for understanding the relationship between Social Conversation and true WOM, the key variables are local - specific to every industry. That's never the most pleasing answer, but reality is not ours to choose.

It's not as if Social Media measurement is endangered by this finding. The functions of social media extend far beyond consumer brand-awareness and sentiment tracking. And even for that particular function, there is some merit in understanding "social consumer brand awareness". If social matters as a channel, then how you're talked about in that channel matters, even it is not representative of true consumer conversation.

It does mean, though, that if you want to use Social Media Measurement for broad brand awareness and sentiment tracking, then you have some work ahead of you. Work that involves first proving the validity of the relationship and then controlling your samples to the greatest extent possible.

When I first saw Elea's presentation, I found it tremendously exciting. To have academics tackling big real-world problems and producing definite answers using careful experiments is fantastic. it's so hard, within the enterprise or as consultants, to come to any similar place. Without such data, we might speculate endlessly about whether or not social media matches word-of-mouth and we would have, none of us, any real knowledge. With studies like these, we have a far better basis on which to make decisions, to evaluate the claims of vendors and agencies, and to decide how far it is prudent or advisable to push the uses of our data.

Comments

[Bryan]

Gary,

Great piece. It is important for any analyst to remember the age of social platforms, and the diverse landscape for user generated content. One thought I have shared with my team members has related to the idea that noise is often created around the volume of product sold, and the discounts that apply. Coupons, free offers, etc, plague the ProActiv market as you referenced, where as you are unlikely to type coupon for Ford 500 into Google and get many results. I read through the WCAI study and "Insights for Practitioners" but it seemed to lack this thought process.

As you mentioned though, I think its best that practitioners continue to apply tested measurement and optimization strategies to an emerging segment such as social. If we can optimize conversions or attract engaged visitors through social channels, we are doing something right.

Thanks.

[Kaushalam]

This is excellent thought. I believe social media will change the world of online marketing that we are looking today. What changes we can see in the near future is the topic for the internet marketer to discuss on. The things like Tracking, Monitoring social media campaign will have a more look to find out the what exactly people think via various social media networks facebook, twitter and google+ etc.