I Comment – Therefore I AM…Satisfied/UnSatisified/Already Engaged (pick one)

Problems in Voice of Customer Analysis

I got a question/comment from Jacques Warren (who has always sent me the most interesting thoughts) that I wanted to expand on because it gets to the heart of critically important issue for understanding how voice of customer (VOC) data should and should not be used.

Jacques’ full comment is published, but I wanted to tackle just his question:

“Could you develop a little more on your assertion that random samples of comment-readers vs non comment-readers in an attitudinal analysis has no predictive value? I don't see why not.”

The comment Jacques is referring to comes in a section where I discuss the problems of self-selection on a behavioral analysis of a social tools’ (like commenting) impact. Here’s my quote:

“A random sample of comment-readers vs. non-comment readers will tell you NOTHING about the causal impact of comments on attitudes.

I going to repeat that last sentence. A random sample of comment-readers vs. non-comment readers will tell you NOTHING about the causal impact of comments on attitudes. Many who struggle with behavioral analysis fall back on opinion research and mistakenly claim it’s easier to draw actionable conclusions. It might be truer to say that opinion research is even easier to misuse and misinterpret than behavioral data.”

I see Jacques problem. That’s more of an assertion than an explanation!

So why I do think it’s true?

For the same reason that a behavioral study that shows that comment readers or posters consume more content than non-comment readers or posters do says absolutely nothing about the causal impact of the comment functionality.

There is no way to determine from the basic facts:

Comment users have a higher sat score than non-comment users (attitudinal)
Comment users consume more pages than non-comment users (behavioral)

if either relationship is causal. We don’t know if commenting self-selects visitors who happen to be more satisfied and consume more content or whether it actually contributes to that relationship.

We can probably guess that it may be some of both, but we have absolutely no way from the base numbers to infer the relative strength or direction of causality. Commenting may be totally non-causal to viewing content or have a negative impact on content views, even as it has a positive impact on satisfaction. Or vice versa. Or it may have no impact on either but simply be an artifact of the not unlikely proposition that those who are more satisfied and do more on the site are also more likely to comment.

Let me give a simple example.

Suppose that tendency to comment increases with usage (so that it is usage driving comments, not comments driving usage). Let’s say that if you view 5 pages, you have a 3% chance of commenting. If you view 10 pages, you have a 5% chance of commenting. If you view 20 pages, you have a 10% chance of commenting.

Such a relationship might yield numbers like these:

Commenters:    8.9 Avg Page Views
Non-Commenters: 2.7 Avg. Page Views

But let’s also assume that when you comment, you tend to view slightly fewer pages – perhaps because you focus on just one section of the site.

So you might actually measure numbers like these:

Commenters: 8.7 Avg. Page Views
Non-Commenters: 2.7 Avg. Page Views

In this scenario, comment behavior would still be highly CORRELATED to page views. It would look as if the more you commented, the more pages you consumed. But commenting would be causally negative to content consumption. A true model would show that, in this imaginary situation, commenting actually reduced page consumption. It is an important fact about the real world that not only does correlation not establish causality, it doesn’t even capture direction of relationship.

This argument applies to satisfaction analysis as well. Change the words “page consumption” to “site satisfaction” and the argument is identical.

It’s as simple as this. People who are highly-engaged with your site are likely to be more satisfied with it. They may also be more likely to view or post comments. This in no way proves that they are more satisfied because they view or post comments. They may be less satisfied as a result of commenting. They may be more satisfied. There may be zero impact. You just don’t know. Looking at the satisfaction scores for each tool on your site and inferring causality from them is simply a basic statistical fallacy.

My point is here was in no way to detract from voice of customer analysis – only to point out that it suffers from exactly the same problems that behavioral analysis does. In my previous post, I outlined the rather arduous path required to actually establish whether or not the relationship between social actions (like commenting) and overall site consumption and satisfaction is causal or merely correlated. This path is similar regardless of whether one uses behavioral or attitudinal research.

Ignoring the effects of self-selection is the single most common mistake in web analytics. It happens to be even more prevalent in attitudinal research.

Comments

[Jacques Warren]

Hello again Gary,

You make a very strong point here. I can see now, and, true, this is a question of wrongly taking correlation for causality, the ultimate crime for an analyst!

If I follow your argument correctly then, there would be a flaw with Eric Peterson and Joseph Carrabis Fi Index in their calculation of Engagement Index, since feedback would be predictive of nothing. I know Eric and Joseph suggest that the simple fact of giving feedback is positive ("...every session in which feedback is gathered is scored positively"), but if it can't predict whether the visitor is more or less satisfied, or even more or less engaged in terms of content comsumption (thus impacting their Ci and Di), how can it predict, or much less, determine *value*?

What I like about what you're doing here, is that it is counterintuitive. One would be easily drawn to conclude that comments *are* a positive sign of engagement, the more the better, thus the more valuable. Whereas they are not... necessarily.

I must say, however, that the measurement path you propose is quite complex (well, it is a complex problem),and I wonder if you see it as actionable only in a minority of organizations (using Visual Sciences?).

[Gary Angel]

Jacques,

This is a truly formidable question. I’m not sure I have ever fully understood Eric and Joseph’s work. But here’s my take: their measure of engagement is NOT a conversion proxy or a cause of success. In other words, it is not a stand-in for any other measure of success (at least as I understand it – and if it is intended to fulfill that role it is surely mis-designed). The engagement metric they are proposing isn’t put forward as a “cause” of success, it’s a definition of success. I take it to embody an “a priori” (prior to experience) claim about what constitutes success on a web site. In other words, nothing in the proposed framework is measured against anything to prove it actually has anything to do with any particular sites actual success.

This is a very different sort of beast than the type of analysis I was discussing which is designed to answer a real-world empirical question like: “Will I make more money if I add/remove comment functionality to my site?” I take it that not only can an Engagement metric NOT answer this question, it would also suffer from exactly the same issues of self-selection as any other metric if used as a measure of success. Such questions turn out be quite tricky to answer when the content has a significant social component.

Is there something wrong with making an “a priori” claim about engagement as a type of success? I’m not sure. As I’ve said before, where sites can measure their actual success per visitor (as is increasingly true for media properties), I am skeptical of the value of an engagement calculation. But where you can’t measure actual success, a well-thought out framework that makes some basic assumptions about engagement may be your best bet.

The type of analysis I originally proposed is indeed a complicated one. It can be done in most enterprise web analytics systems, but it often requires a certain amount of setup and pre-planning to accomplish well. With a little bit of extra tagging, it can certainly be done in Omniture (for example) though I won’t deny that it will take a good chunk of work.