Engagement as a Term-of-Art in Web Analytics

Part I

When I posted about eMetrics, I mentioned that of my three presentations the most challenging is probably a session with Eric Peterson and Joseph Carrabis on measuring engagement.

Part of my trepidation is that although I have written about engagement many times and it is within the class of problems/techniques that I actually use in practice, it’s an issue and concept that is broader than our core analytics practice and frequently crosses into terrain in which I not entirely comfortable. I’ll say too that I didn’t so much sign-up for this topic as I was drafted into it – I really didn’t go seeking this particular conversation.

But needs must fashion deeds, so I’m going to try and explore the topic in some detail in a three-part post on the topic.

When I first started writing about engagement, I approached it largely from the perspective of Semphonic’s practice – where something we call engagement is often used as a conversion proxy for the optimization of Search Engine Marketing campaigns. We didn’t pick the name Semphonic completely out of a hat – we do quite a lot of Search Analytics. And in Search Analytics one of the key problems our clients have faced is how to measure the success of a visitor when there is no straightforward conversion available.

We have a rather similar set of issues from lead-generation sites looking for ways to attach value encodings to individual leads. Pre-assessing lead-value allows organizations to tailor the extent and cost of their response to online leads appropriately. Using prior online behavior, you can do a surprisingly good job predicting lead-value.

In both these cases, our preferred technique was to create a measure (we called it engagement) that would serve as a proxy for either lead-value or success. This type of engagement measure was typically constructed by an analysis of how various site behaviors correlated to the desired outcomes. In some cases, we found that optimization could be done simply based on a single metric or behavior. Most of the time, we discovered that a fairly elaborate set of behaviors had to be combined and accounted for to build a good proxy of site success or lead-value.

This early experience in using “engagement” shaped most of my initial thinking on the topic.

First, it led me to believe that no standard cross-site measure of engagement of this type was possible. Even within a single site, apparently similar actions like generating a lead would – on analysis – result in dramatically different value. So to build a good success proxy, different lead types had to be handled differently. If different lead generation cues on a single site result in dramatically different values, then how is it possible that any standard measure of engagement could apply to multiple sites?

Well, I don't think it is; at least not when engagement is treated as a proxy for success or lead value.

Second, it led me to believe that the concept of engagement was a “term of art” with a meaning specific to web analytics practice and a clear and undeniable use within that practice.

But I soon realized as I followed the growing discussion about engagement that there were other ways people were using the word that transcend this function as a multi-faceted proxy for site success or lead value.

A second, very common use of engagement, is within the realm of media measurement – where it is used as a way to compare the relative performance and attractiveness of sites to a media buyer/planner. This usage is quite distinct from the idea of a “conversion proxy” and is expressly for site comparison.

Engagement has also become a core part of a third conversation – measuring the brand impact of site visits. This third usage may look very similar in treatment to engagement as a conversion or lead-value proxy but it is importantly different – both in terms of execution and intent. It is the only usage of the three which has a distinctly "psychological" aspect to it - where we are actually seeking some measure of the users state-of-mind. It is also a measure that may well be reasonably comparable across sites.

I’ll talk about each of these next two uses in the second and third installments of this post where I hope to show that each of these three concepts are quite distinct and should be given separate and clearly delineated names.

Every profession has its own “terms of art.” These highly specific uses of words can be frustrating to the outsider but are essential to the practitioner. I hope to describe a framework within which these three concepts will each have a specific “term of art” that will make their usage, meaning and intent clear. By doing so, I hope to remove a good deal of the confusion that surrounds the concept of engagement and, I hope, advance the conversation.

Comments

[Eric T. Peterson]

Gary,

Great post! I certainly look forward to our continuing discussion about engagement and our presentation together at Emetrics. Having spent a little time myself thinking about these same concepts I would propose that your three measures already have names.

The first measure of engagement, your proxy for conversion combined with other site measures, is more or less what I have referred to as "Visitor Engagement." Visitor engagement is measured directly from the web site using (typically) a census-based system and does attempt to fill the gap you describe --- a measure of desired behavior when more common metrics like conversion rate aren't as easily applied.

Your comment that "no standard cross-site measure of this type was possible" is, in my opinion, partially right. If you look back through my posts, and consider some of what Joseph and I have written lately at the Future of blog, there may not be a ** single or standard ** measure but I do believe there is a framework that can be universally applied.

Each site and sub-site will likely have different thresholds, which makes sense because Visitor Engagement is at its core just another type of key performance indicator. Just like you'd have different goals and thresholds at two different retailers, so will your Visitor Engagement calculation. This is key because it gives large sites the opportunity to have multiple levels of Visitor Engagement measures inside the Enterprise --- each of which have different thresholds but all working to describe the depth and degree of visitor interaction compared to a set of engagement goals.

The second measure of engagement, within the realm of media measurement, is something I have started referring to as "Audience Engagement." Audience Engagement has to be measured ** not ** using a census-based system as it requires cross-site visibility. This measure of engagement almost certainly includes some of the measures of engagement you'd include in visitor engagement (time, click-depth, recency) but would likely ** not ** include more site specific actions.

Audience Engagement gives media planners and buyers a different ruler against which to judge the audience visiting competing properties. It is still not completely clear to me how media planners will use a measure of Audience Engagement; it is only clear that they are actively looking for such a measure.

The third measure of engagement, focusing on the brand impact of site visits, is probably something called "Brand Engagement" and I agree, likely best left to those people having a slightly different background and point-of-view. Ironically this is why I invited Joseph to work with me on my measure of visitor and site engagement, and to present on our panel in San Francisco. I will let him comment but I suspect the NextStage technology would fit the bill to measure Brand Engagement.

Three different measures of engagement, each taken using a slightly different system, each informing a slightly different audience and having a slightly (or completely) different use. But all measures of engagement, at least in my opinion.

I look forward to your follow-up posts on the subject as I have little doubt your insights will advance all our thinking.

(From Helsinki!)

Eric T. Peterson
Web Analytics Demystififed, Inc.
http://www.webanalyticsdemystified.com