Measuring Engagement without Excuses

As someone who has worked on various measures of Engagement quite a bit, I read Avinash’s  “Engagement” Is Not A Metric, It’s An Excuse” with considerable interest. As always, Avinash makes some excellent points and the post is well worth reading, but I don’t ultimately find in it much reason to change the direction of the work we’re doing (which often involves building Measures of Engagement).

There are five separate points in Avinash’s argument concerning Engagement – some powerful, some not. I’d like to consider each and see what they add up to.

Point 1: Each business is unique so it’s hard to generalize measures of Engagement.

I think this is true. I know of no satisfactorily generalizable Engagement Metric and I see only very modest hope of every attaining one. That being said, the desire for comparability is an illusion across much of web analytics. People always ask me if X% conversion or Y% lead generation is good. And I’m forced to answer that the question as stated is meaningless and always will be. Claiming that X% Conversion is good is akin to claiming that there is some "natural fair" price for a commodity. It's an easy mistake to make - but it's completely wrong.

Since I’m assuming we can all agree that this lack of comparability doesn’t imply that measuring Conversion Rate on your own site is meaningless, I don’t think the non-comparability of Engagement is fatal. Does it present difficulties in understanding? It may. I’ve found that more complex data definitions almost always create problems of explanation. When we use neural networks to generate complex visitor segments based on the information in hundreds of behavioral data points it’s nearly impossible to explain (without a great deal of gloss) how a segment is built. That’s a significant drawback to the technique – but one that is often outweighed by the ability to capture many complex behaviors in a single segment. The same may well be true for Engagement – when you capture multiple behaviors in a single measure you have both simplified your reporting and complicated your vocabulary. The benefits of the trade-off need to be judged by individual cases.

Point 2: It is nearly impossible to define engagement in a standard way that can be applied across the board.

This strikes me as Point 1 re-stated with slightly different words.

Point 3: Engagement tries to measure something deeply qualitative.

I think this point is misguided on several levels. I’m not sure it’s true and to whatever extent it is true, I’m not sure it’s meaningful. Let’s start with a simpler metric – impressions. Impressions has a specific meaning – an ad was displayed where a person could see it. It has an implied meaning, that of making an impression. But “impressions” doesn’t truly measure making an impression. It measures a behavioral fact about how often a visitor was in a particular situation where a psychological impression might have occurred. Most measures of Engagement are similar. They measure a specific set of Behaviors that in theory may represent a psychological state of engagement. If you think about it, you’ll see that even seemingly uncomplicated metrics like Page Views are this way. Did a visitor actually view (read or see) a page? That psychological question isn’t actually answered by the metric – but we tend to assume that the behavioral proxy is a reasonable measure. The same is true for measures of engagement. When you are measuring Engagement you’re typically measuring a set of behaviors that you believe are a legitimate proxy for establishing differing levels of psychological engagement. It’s an empirically verifiable fact that on many web sites these proxies are predictive of final outcomes and are, therefore, measuring something real. It is, of course, a much deeper question whether your measurements of engagement are right. But that isn’t what Avinash is getting at. So I’d tend to say that almost every web metric we use is a proxy for a psychological state and that this is untroubling. We don’t actually care about the psychological state – if the behavioral measure is good, it’s good.

I think there’s a second issue lurking here that’s deeper. Some types of questions are either extremely difficult or impossible to answer with behavioral analysis. We often talk about “why” questions as being like this. We see drop-off in a step in a conversion process. We can see the drop off, but the data will often tell us nothing about why visitors drop-off at a particular point or at all – their state-of-mind isn’t revealed by the behavior. I think Avinash is trying to suggest that Engagement is this sort of question; however, I think it’s clearly not true. Measures of Engagement are typically scores based on accomplishing some specific set of site actions – they are a pure representation of “What Happened.” Perhaps there is too much psychological superstructure in the word engagement for some people’s taste, but I don’t think it’s reasonable to suggest that a typical Engagement metric isn’t measuring a quantitatively different and potentially significant variance in site usage. It is.

Point #4: One of my personal golden rules is that a metric should be instantly useful.

This is just wrong and I’m surprised anyone still believes this. Old myths die hard. I thought I had conclusively shown that no single metric is ever instantly useful. For the complete and I think absolutely irrefutable argument, you can read this extended post on why a 100% Conversion Rate is a bad thing! (http://semphonic.blogs.com/semangel/2007/01/why_100_convers.html and the follow-up at http://www.typepad.com/t/app/weblog/post?__mode=edit_entry&id=15548545&blog_id=214250). If you don't want to dive that deep consider this simple case. I tell you your Conversion Rate went from 30% to 32%. Is that good? If you think so, what if I add that your cost of traffic doubled? What if I told you your Conversion Rate dropped from 10% to 7%. Bad? What if you sourced ten times the traffic for the same cost? Still think you can make instant decisions based on the change in a single metric? Almost anything we are ever going to care about is a system - and it is virtually impossible to represent a system in a single directional metric.

I’ll make a second point here and one that I think gets to the heart of a fundamental mis-understanding about reporting. The idea that you can with a single, simple uncomplicated metric tell a meaningful story is WRONG. Badly, completely and totally wrong. There is not a single idea that makes reports both worth less and misused more than this widely propagated idea. It is not so. And analysts that strive for this ideal are sorely misguided.

Conversion-Rate, as we have seen, is only significant in the context of a particular level and cost of traffic. If you ever build a report that leaves a decision-maker with the impression that they can act based on movement in a single number (like Conversion Rate) without providing the key contextual numbers (traffic and cost of traffic for example) then you’ve failed in your job. Giving a decision-maker too little information is much worse - and much more dangerous - than the sin of providing too much. Both are sins, but one is venal and the other mortal!

So typically, you have a choice between building complex metrics like Engagement if you want to represent real-world trends concisely (these aren’t instantly actionable either) or you can present a decision-maker with a matrix of 15-20 simpler metrics. Which is better? Not always a trivial call, but lots of times the former is better. It’s true, you’ll have to do some spadework within your organization so that everyone understands and buys into the Engagement metric. But once you’ve accomplished that, you’re reporting on a few numbers not 15 or even 50. That’s a big win when it comes to actually absorbing trends on an ongoing basis. I can tell you from personal experience that when you give a decision-maker a report and say: “Here’s our campaign data broken out across twenty different success metrics on our site,” the first question I’ll get back is “but which campaign is best overall?!”

Point #5:  Most of all engagement is a proxy for measuring an outcome from a website. Conversion is not enough, as mentioned above, so we try something else. The problem that we’ll define engagement as a measure of some kind of outcome but we won’t give it the sexy name of engagement.

I’ll go along with Avinash part-way here. If your measure of Engagement is just view / visitors then you might as well it call it that. But if your measure is based on views, interactions, events and weighted content areas, what exactly are you supposed to call it? And what, really, is wrong with calling it engagement? Some words carry too much baggage to be deployed effectively by the analyst. I just don’t see anything in Avinash’s post that makes me think Engagement is one of those words. In my experience, Engagement isn’t a proxy for measuring an outcome on a site at all. It’s more often a means of aggregating a set of outcomes into a single visitor score or segment. Since a set of outcomes can’t reasonably be described by just appending all of the events into one long name, the analyst is always going to have to pick a name that reasonably represents the overall concept. In many situations, and for many sites, “Engagement” does a pretty good job of carrying that baggage.

So I guess I score the argument this way: points 1-2 are compelling if you had unrealistic expectations of using an Engagement Metric to compare your site to others; point 3 is interesting but almost certainly misguided; point 4 is flat-out wrong; and point 5 misses the true object of most real attempts to use Engagement.

Add it all up, and I think the biggest idea I’d take away is that if you are going to call something Engagement it probably shouldn’t be reducible to a single simple metric (good point – I’ve made that mistake) and you should probably make sure you are comfortable with the trade-off between the long-term economy of collapsing multiple metrics with the price you pay both in explanation and loss of clarity (this isn't in the original post at all but I think it's what ultimately matters).

As is so often the case, the real-world complexity of being an analyst is rarely reducible to simple rules of “do this” or “do that.” Reporting is much more complicated than most people ever realize, and it involves an unending series of trade-offs to which there is no perfect answer. Can an Engagement metric be misused as an excuse? Of course. Can it be immensely valuable? Of course.

Comments

[Avinash Kaushik]

Great feedback Gary, especially from someone of your stature who has been engaging with engagement for such a long time.

I have learned a few things here, thank you.

See you in a couple weeks.

-Avinash.

[Eric T. Peterson]

Gary, great post!

It captured a lot of what I was thinking about Avinash's post. My own work on measuring engagement, some of which I know you've read, has shown that in the absence of being able to ask everyone "are you engaged?" (impractical, but not entirely so) that quantitative data can be used to calculate a measure of engagement that is useful as another input for the inevitable analysis required.

But I guess I question this statement: "if you are going to call something Engagement it probably shouldn’t be reducible to a single simple metric"

Do you say that because doing so is difficult using most currently available applications (and impossible in systems like Google Analytics and others that aggregate visitors into buckets), because you think it would be too confusing to the reader, or something else?

I guess I've spend enough time helping companies explain some of the basic stuff (page views per visit, etc.) it doesn't feel that much different to explain the nuance of a combined, weighted set of indices rolled up into something called "visitor engagement".

Anyway, I'd thank you for promoting a real discussion on this subject and not just fawning or navel-gazing but I'm becoming a broken record in that regard when it comes to your posts. You rule!

Eric T. Peterson
http://www.webanalyticsdemystified.com

[Michael Feiner]

Gary,

Very interesting analysis. I tend to agree with many of the points you raised.

I do feel you have interpreted Avinash's fourth point in a way not intended. I completely agree with your statement - no single metric is ever instantly useful.

My reading of that point is slightly different - a metric should be instantly useful (within the context of other relating metrics).

Your argument would hold if Avinash's statement read "One of my personal golden rules is that a metric should be instantly useful IN SILO".

Clearly engagement is a key issue in web analytics.
I often find it hard to define a set of engagement measurements (single or combo) across a single site, let alone compare it across different brands (e.g. engagement on a mortgages section would look completely different to engagement on the credit card section).

I'm pleased to see this issue being debated by the industry thought leaders such as you. I look forward to the continued discussion.

Thank you for your commentary.

Michael Feiner
AEP Convert

[Theo Papadakis]

Gary

I thought your post justifies the tradeoff between a simplification\economy of reporting and a complication of vocabulary very well.

You may want to check out my post on Ron Shevlin's blog http://marketingroi.wordpress.com/2007/10/11/more-thoughts-on-customer-engagement/ as i think it follows your train of thought.

[Robert Blakeley]

Let me see if I've gotten the implications: If engagement is a measure, what does it measure? Or to put it another way, why does the measure exist? I expect that its purpose is the same as the scoring used in Direct Marketing, it identifies your most profitable audiences. One would validate the accuracy of the scoring based on the resultant profitability. That is, can you show that a score of 2 represents a consistently more profitable customer than one with a score of 1. Correct?