Why KPI's are Context Specific

Eric Peterson's Engagment Metric

I’ve been meaning to comment on Eric Peterson’s series of posts on building an Engagement metric for some time. And while the posts are now dated (last year for god sakes!) in blogging terms, the subject – being real – is still interesting.

I’ll give a quick summary of Eric’s posts, but if you are interested, you should really check out the series here. Eric uses six different measures in his index:

1. Click-Depth Index: Percent of visitor sessions of "n" or more pages
2. Recency Index: Percent of visitor sessions occurring in the last "small n" weeks
3. Duration Index: Percent of visitor sessions of "n" or more minutes
4. Brand Index: Percent of visitor sessions originating directly or originating from search engine searches for terms like "eric t. peterson" and "web analytics demystified", etc.
5. Blog Index: Ratio of blog reading sessions to all sessions
6. Conversion Index: In this case, session- or order-based conversion

Since each measure is indexical, he can sum them and then divide them to produce a single index of percent engagement by visitor. This measure gives him the ability to rank visitors by their "engagement," derive a site-wide average of visitor engagement, and classify other variables (like source) by the average engagement or distribution of visitors by "engagement."

This sort of metric would be extremely difficult to do in most web analytic tools (he's lucky). But its power and utility seem pretty obvious. As a KPI, I’ll stand by my last post and say that – like any other single KPI – changes in it cannot be interpreted simply. I can think of a number of reasons, for example, why a decline in % Engagement might actually be a good thing.

That being said, I think this type of metric is extremely useful. I’ve written in other posts why Conversion proxies (which are really measures of engagement) are essential for optimizing sites without conversions or where conversion is only one of a set of goals. In addition, we’ve found that even where conversion measures exist, proxies often do a better job statistically. Conversion numbers are sometimes too small to be used well statistically in all the places you’d like to optimize. By employing a broader measure, like Eric’s engagement, you can optimize to a much finer-grained level.

What I really wanted to focus on wasn’t any of this, however. But another idea which is related to my immediate topic of reporting. Eric’s Engagement Measure is designed to be a fairly generic visitor profiling measure. And while I’d quibble with one or two of his choices, I think it’s one of the best metrics for this that I’ve ever seen in web analytics.

But as useful as a generic visitor profiling measure is, this doesn’t mean it’s appropriate to every analytic situation. One of the things I like about this metric is how broad-based it is – which makes it much less vulnerable to misuse. Suppose, however, that I want an engagement metric to be used specifically for evaluating the effectiveness of my Sources. Eric’s generic measure might be useful in most cases, but there are at least two significant biases built-in.

Numbers 4&5 will both tend to weight certain sources higher or lower – by dint of causes that might have nothing to do with "real" engagement. Indeed, all of the other factors may also be misleading since they may reflect landing pages (better or worse click-thru), recent linkages (higher recency) and even percent of users on dial-up (potentially longer sessions) rather than any real relationship of Source to engagement. I probably wouldn’t worry much about these secondary biases – and I could certainly get rid of most of them by setting up a simple control group with Landing Page and Connection held constant.

The biases introduced by 4&5 are real, however. And they might significantly color any analysis of Source effectiveness based on this metric. That’s why, if I was using an Engagement metric for evaluating source, I’d want to make sure that I chose a metric that had as few built-in sourcing biases as possible. This might not be the "best" KPI for measuring engagement in the sense of capturing every action that actually indicates increased engagement – but it would be the best for doing the particular analysis in question.

On the other hand, if I was trying to build a measure to use for producing email solicitations, I’m likely to be much less concerned with Source bias and more concerned with a measure that captures all the actions I think are relevant. That’s a situation where Eric’s measure seems perfect.

Similarly, if I’m using my metric to measure the "engagement" produced by visitors who used a specific part of a site (like the blog or the press releases), it’s vitally important that my metric not include a strong built in bias toward one of the areas (like blogging). Some analysts might argue that this represents a flaw in the metric Eric proposes. I don’t think so. Every metric carries with it some biases – and no metric is appropriate to every situation. This is part and parcel of my last post. Just as it is impossible to base an action on a single KPI, it is impossible for any single KPI to not carry at least some real-world bias. It’s the analyst’s job to decide whether the measurement in question is likely to carry a strong bias for the function desired.

In other words, a KPI is useful, correct and good only in relation to a specific set of problems. This isn't a criticism of any KPI, just a fact of life. Perhaps most analysts would accept that there is no single Engagement KPI appropriate for every site. But I would take that a step further and say that there is no single Engagement KPI appropriate for every analysis on any ONE site.

Just as no single KPI can ever drive directly to an action, no single KPI will ever be perfect for every analysis. Broad-based metrics like Eric’s Engagement Metric are likely to be useful in a wide variety of situations. But, that same broad-based quality can work against them since they are more likely to incorporate one or more biases (even while reducing them in importance). When you’re using a simple specific metric, you may be less likely to bias your analysis – but where your analysis is biased the effect can be disastrous. With a large compound metric, you’ll have a nearly impossible time avoiding some biases – but their impact is usually going to be much less.

In my next post, I going to discuss what I take to be the role of reporting (and, indeed, of much analysis); namely, to provide a meaningful context for decision-making.

To put information in a meaningful context, the analyst needs to place as much relevant information as possible in front of the decision-maker. And, of course, the analyst must pay equal attention to eliminating the measures that – in some specific situation – may be distorted by a hidden bias or relationship. Indeed, introducing a biased datum into a reporting context will usually have much worse consequences than leaving out a truly important piece of information. Decision-makers are much more likely to sense that something is missing than realize that a relationship is misreported!