Media Measurement – The Assessment of Social Value

There were quite a few discussions at X Change that stuck in mind and that, in one way or another, I hope to blog about. Today’s topic started with a question in on my Huddle sessions – “How do I know the value of user posting a comment?”

The topic stuck with me largely because I had been talking about something slightly different and gave a glib answer in reply. I hate being glib. It’s worse than being wrong. There are any number of reasons for being wrong – many of which are intellectually respectable. But glibness is pretty much always a fault.

In the X Change discussion, I had been talking about how media sites were increasingly moving toward a world where they are able to measure the actual impression value of all the content they serve on a visitor by visitor basis. To obtain this level of measurement, a site has to be able to track the value of their ad impressions on every page and pass this (either real-time or on the back-end) to the web analytics solution. By marrying the ad value to the behavioral stream, a media site can track the actual revenue derived from each visitor and each visit.

If you can do this (and it takes a lot of work), the payoff is enormous. A whole range of very powerful analysis projects become available that simply can’t be done otherwise. Everything from PPC arbitrage, to remnant usage and optimization, to programmed search results, to lifetime value prediction, to home page engagement is either made possible or made much more accurate when you can measure the actual revenue produced by each visitor to the site. 

So when I heard the question about the value of a comment, I was thinking about measures of engagement. And I was thinking that measures of engagement become much less important when you know actual value. If you can see how much revenue a visitor produces and you can model how much they are likely to produce, you don’t need to measure how “engaged” they are. The engagement (doing key actions, viewing pages, etc.) metrics are just proxies for the revenue generated so if you have the revenue generated you don’t need the proxy. And so I spoke and so I have written.

But the question wasn’t how do I know the value of a page view or even a tool use. The question was specifically about a social action. And social actions like posting comments have community value – they drive usage by others and thus represent a fundamentally different type of value than is measured by ad impressions. How might we capture the size of that community value for a social action?

I can think of several ways to tackle this type of problem – and each may be appropriate to a slightly different set of circumstances. First, it’s important to realize that the value curve is not fixed. When you launch a forum, for example, getting enough user-generated content to make the space “come alive” is critical. The value of posts and replies and messages in the early stages of a community are absolutely disproportionate to the actual impression value they drive. Indeed, they are far more important in this early stage than they will be later on even though their impression value is likely significantly less than it would be once the community is well traveled.

It’s because of this that so many community efforts are at least partially pre-populated. And instead of trying to measure the impression value of user-content, it’s far more important to understand how much you can spend to generate it. The most relevant measure for early-stage community building is likely to be how much it would cost to actually build the content. You can pay for content generation around most things – and as long as it is cheaper to generate content than user-generate content, it often makes sense to self-generate.

But in a fully-functioning community or web-site, the problem shifts to more classic behavioral measurement. A comment added to an article has two community benefits: it will create additional page views/time-on-site and it will improve the perception of the community – leading, presumably, to additional comments, additional consumption and increased loyalty.

The first community effect – increased consumption – is fairly straightforward. Comment threads are easily measured as page-views and the consumption value of those pages (and net effect on total consumption) can easily be measured. It’s critical, of course, that you be able to measure the net effect on consumption since comment pages are often lower-value than content pages. If significant amounts of session time are being devoted to comment consumption or generation, you may actually be lowering your net consumption value.

You can also measure avg. page time and total click-outs based on the length of the comment thread. This is nearly impossible to do out of the box with a standard measurement setup, but as a research project it’s quite easy.

Once you’ve measured the value of net consumption change, you’ll also have a way to quantify the net value of content-producers (and topics) that generate lots of comments. 

Modeling the second type of community effect is trickier. What makes it tricky is the element of self-selection that dogs so much of web analytics. You can’t simply measure comment-writers or comment-readers as a group and show that they have higher consumption or are more likely to return. You can’t, because the question is more likely to return than what other group? It can’t be an average because those who write or respond to comments aren’t “average” and may not have ever been average.

As with all good market research, the biggest challenge you face is establishing a true control group. On the web, the challenge is tougher because cookie deletion can make it impossible to track visitors over extended periods of time.

Here’s the method I would suggest for this type of research. Start by selecting a group of visitors who have at least some behavior over an extended period of time – six months for example.

When I say behavior over time, I mean that everyone in the group should exhibit at least one visit in the first month and one visit in the last month. This is essential because otherwise you will be losing visitors to cookie deletion in ways that make the analysis unpredictable. By creating your study group in this fashion, you guarantee that you are tracking a consistent group of visitors.

Remember, no matter how bad cookie deletion is, if a visitor has the same cookie, it’s the same machine. The problem with cookies isn’t symmetrical: it’s only the absence of a cookie that’s problematic.

Next, you should identify the subset of these visitors who didn’t view or post comments (for example) in the beginning of the period (e.g. the first eight weeks). By extending the period well past the 1st month, you insure that you are removing effects relative to visitors just starting on the site vs. long-time visitors.

For this group (visitors with no comment behavior in the “pre-period”), isolate the smaller subset that in the next two months began reading or posting comments. Now compare the pre and post behavior (first two months vs. last two months) of the “comment-consumers” with the “non-comment consumers” in the subset.

If you can, isolate a further subset of visitors whose “pre-period” behavior is nearly identical in terms of your key aggregate measures (visit frequency, consumption, interest area). If you measure a significant net change in this “like” subset in the last period based on whether or not they are mid-period “comment-consumers,” then you really have something.

Using this method, you’ll have a significantly better picture of the “community” impact of a social behavior like comments.

You can use similar techniques to isolate the impact on attitudes using online surveys. Everything I’ve said here applies just as much to survey research as it does to behavioral analysis. People tend to forget that problems of self-selection are every bit as acute in voice-of-customer research as in behavioral analysis. A random sample of comment-readers vs. non-comment readers will tell you NOTHING about the causal impact of comment consumption on attitudes.

I going to repeat that last sentence. A random sample of comment-readers vs. non-comment readers will tell you NOTHING about the causal impact of comment consumption on attitudes. Many who struggle with behavioral analysis fall back on opinion research and mistakenly claim it’s easier to draw actionable conclusions. It might be truer to say that opinion research is even easier to misuse and misinterpret than behavioral data.

What this all adds up to is that measuring the value of community actions is greatly facilitated by an ability to measure the individual revenue gained from each visitor/impression. But it is not captured wholly or in a straightforward manner by that same revenue measurement. The community effects can be measured, but that measurement will often require a fairly significant research project that makes a genuine effort to establish a valid control group in both the behavioral and attitudinal realms.

Comments

[Jacques Warren]

Hi Gary,

I pretty much agree with you about the larger relative weight of determining value versus engagement. In my mind, it has obviously more predictive quality than engagement (for what we mean by "engagement" from the current variety of attempts for a definition).

Could you develop a little more on your assertion that random samples of comment-readers vs non comment-readers in an attitudinal analysis has no predictive value? I don't see why not.