SemAngel

Two Cultures

About fifty years ago, C.P. Snow published a famous and by now almost trite conceptualization of the split between science and humanities as “two cultures” which could not communicate. It struck me at eMetrics that a similar problem exists today in web analytics – there is vast gulf between the culture of web analytics practictioners and the culture of web analytics theorists.

These two worlds sometimes presented themselves in such gross opposition as to make the individual hourly sessions that followed the expert/case-study format at eMetrics (giving one half-hour to each) either annoying, bewildering or amusing - depending, I suppose, on your cast of mind.

What is alarming isn’t that a gap exists – it does in every profession. What is troubling is the extent of the gap - the often apparent lack of ANY connection between the problems and methods of the practitioners and the words of the professional theorists and talkers. This is not a good thing.

And nowhere is this split more telling and more damaging than around the troubling issue of web analytics data quality.

Two presentations that I saw embodied this split for me and made it particularly vivid.

The first was a beautiful presentation given by Rufus Evison of dunnhumby. Though highly specific to grocery-store data (the sort of data we all wish we had but don't - and a bit frustrating on that account), Mr. Evison's presentation was more than just an excellent piece of real-world practice. It was a primer on the importance of finding and using the right data.

The dunnhumby presentation began with the "McNamara Fallacy." (The name is ironic since McNamara was a brilliant data analyst - possibly the best I have ever read or known of. But the fallacy was attributed to him by another excellent analyst - perhaps correctly - for McNamara's role in the Vietnam war. As with many a profession, analysts are better remembered for their mistakes than their successes!)

Here is the McNamara Fallacy around which Mr. Evison built his presentation:

• The first step is to measure whatever can be easily measured. This is okay as far as it goes.
• The second step is to disregard that which can't be measured or give it an arbitrary quantitative value. This is artificial and misleading.
• The third step is to presume that what can't be measured easily really isn't very important. This is blindness.
• The fourth step is to say that that which can't be easily measured really doesn't exist. This is suicide.

When an analyst begins with this quote, it’s fair warning that they take their data seriously. And I have come to believe that taking the data seriously is the hallmark of the culture of real practictioners.

Not too much later, I sat through a presentation by someone who apparently trains people in web analytics. It included a single slide with the words “data accuracy” and a big X written over them. I cannot quote the accompanying dialog exactly but it was essentially this:

“Web analytics data isn’t always accurate. So you have to trend it.”

In terms of McNamara's Fallacy, this is at least blind and probably suicidal.

I have written before on the utter fallacy of the “trending as a protection against data quality argument.” No worse idea has ever been foisted off on our community, and it is deeply disturbing to see this particular advice given to beginners. It is as if a teacher of mathematics calmly advised his students: “When you do an addition problem, don’t bother to double-check your work. If your answer is bigger than either number you added, it’s probably right.”

Trends, alas, are as likely to be the result of data quality problems as they are to protect against data quality problems.

Indeed, the whole idea of a trend as a protection against data quality problems is an intellectual muddle. It would be more correct to say that insight can be gleaned from imperfect data (luckily since we never get a perfect data set). It would probably be correct to say that, provided the data is not too imperfect and that the imperfections are (somewhat) random, changes in the data set may reflect changes in the real-world.

It would be about equally true to say the our inferences from a snap-shot (not trended) picture of the world are (if the data is not too imperfect and the imperfections somewhat random) possibly indicative of the real-world.

Trending, you see, has absolutely nothing to do with the underlying issue - "Is the data good enough to use and how much confidence should we have in what we find?" There is no difference between using the data to describe a snapshot of the current situation (how much content did prospects look at this month compared to how much content existing customers looked at this month) and a trend (how much content did prospects look at this month compared to how much content they looked at last month).

It is a good and useful warning to beginners to make it clear that web analytics data never achieves the near-perfection of some financial or manufacturing systems. It is an imperfect and damaging falsehood to suggest that trending is somehow the catch-all response to this fact.

Given this direction, beginners will take seriously and grossly misinterpret a range of web analytics issues. They may believe that a 13 month trend up in 1st time visitors is actually meaningful. They may believe that a trend in demographic data for their site as reported by a competitive research tool is meaningful. They may believe that an upward (or downward) trend in conversion rate after they have switched tools is real. Trends are meaningful only if the trend is not, itself, the product of flaws or shortcomings in the data. And trending NEVER, EVER provides protection against problems in interpretation resulting from a lack of statistical significance in the data.

Unless you understand where and how the problems in your data arise, trending is no protection against fundamentally misinterpreting what you are seeing. That’s just the way it is and every experienced, hands-on practitioner will know, understand and appreciate this simple fact.

Which brings me back to the two cultures problem.

There is nothing impossible or even unlikely about being both a practitioner and theorist. Indeed, it is hard to be good at the former without being at least adequate at the latter. And it is most certainly not the case that every theoretic presentation should be ignored or treated as the work of empty punditry.

But theory utterly divorced from practice is devoid of interest. And in no single attitude is this more apparent in web analytics than in a lack of respect for the data.

Web analytics data is messy. Difficult. Plagued by problems that are hard to understand and tricky to solve. It is so much easier if you can ignore these problems that for the unblooded theoretician, the temptation to over-simplify is almost irresistible.

This is an explanation of why the phenomenon is so widespread as to have created or at least personified two worlds. It is not an excuse.

A real data analyst will never lose sight of this giant problem. A real analyst may spend half their presentation telling you how hard they worked to get the right data and get it clean. It may be dull. It may not be the stuff of dreams. But it is, manifestly, true.

The gap is far too wide in our field between the words of our theorists and the work of our experienced practitioners. And in no place would it be more profitable to build a bridge than around this issue of web analytics data.

For it is my general observation that if you are getting a shallow, fake and unworried explanation of handling web analytics data quality, you should expect a similar lack of intellectual effort around everything else you are hearing.

Indeed, the particular “trend your data” presentation that so annoyed me (though it was, in fact, hardly different from and better presented than a half-dozen other ones I have seen) was not all misguided. It had a fair number of sound ideas jumbled up amid a nearly equal number of bad ones.

But if you cannot trust your theorist, consultant, strategic advisor or analytics educator on this fundamental point of intellectual honesty and effort, I do not think you will be well served to trust them at all. To paraphrase a famous dictum on the effectiveness of marketing spend, you may know that 50% of what you are hearing is true, but how can you decide which 50% it is?

[Personal Postscript:

Not all - or even most - was doom and gloom. I enjoyed eMetrics, the majority of the presentations I saw, and nearly all of the conversations I shared.  I hope my sessions on analytic planning, SEM Analytics and measuring engagement were worthwhile for people. Each had it's joys and disappointments from my perspective!

I missed most of the Keynote presentations since I had to work mornings. But in the afternoon sessions, in addition to the Rufus Evison presentation, I enjoyed among others: Diane Hoag’s presentation of a case study dealing with an organic search traffic decline resulting from a Google release (simple but real); the more practical elements of the panel on Multivariate Testing; Alex Langshur’s talk focused on public sector analytics – particularly the parts on categorizing SEO Terms and the compelling final report he showed; and Michael Stebbins’ highly-polished presentation on Universal Search (it wasn’t analytics focused, but if you want to do Search Analytics you have to understand Search and Michael provided a good, quick introduction to "Universal" in the session we shared). There were many other presentations I wish I could have seen - I tend to choose a bit randomly and go mostly to see people I don't know!

I especially enjoyed co-presenting with Nancy Abila – probably my single favorite experience at eMetrics. Thanks Nancy!]

Reporting on Forms and Conversion Processes

Part Nine in Series on Form Abandonment and Online Process Measurement
(and why I don’t give a Yahoo)

I seriously considered blogging about the collapse of the Microsoft-Yahoo deal but in the end, I decided it wasn’t really worthwhile. I don’t think I have much to add beyond what is, for the most part, conventional wisdom. On the whole I thought the deal probably made sense (at least for Yahoo). Management isn’t wrong to seek the best deal possible or even to desire to continue running the company themselves. But managers in a public company have a real responsibility to consider such deals fairly – perhaps even more than fairly to counter the natural bias of self-interest - and not simply do their best to sabotage them.

And while it is interesting, even occasionally gripping, watching these corporate battles unfold, I am usually relieved to return to the world of our craft. The concerns of craft are always meaningful; more meaningful to the craftsman than the ultimately rather uninteresting business of who owns what.

So I’m going to finish up my series on Forms measurement with a discussion of reporting KPIs that I hope will wrap up the series in a cohesive fashion.

In the first post in this series, I talked about the role for behavioral analysis in the development and improvement of Forms-based processes. This included a discussion of design and usability, and how CEM tools like Tealeaf and web analytics tools could extend the improvement of Forms on into their operational lifecycle. In the next post, I considered what kinds of process issues you actually study with web analytics and argued that the most prevalent web analytic measure (step drop-off) is not actually very useful. This post laid the ground-work for subsequent posts on measuring the qualification level of visitors to a process, measuring the need and success for selling INSIDE the process, and measuring and optimizing completer (thank you) pages.

My last three posts in the series have been on key extensions to the techniques of Form and process measurement including discussions of visitor segmentation, tracking process errorsand warnings and the implications of multi-session process usage.

At Semphonic, our focus in reporting has increasingly been on “systems” based reporting. The goal in a systems-based report is not simply to capture the state of a key variable (KPI) but to capture the relationship between variables so that the underlying business system is fully described.

For forms and processes, I think of the reporting system as involving several distinct systems that can be combined at a high-level into a global system.

The most important systems are typically these:

• The pre-qualification system: Measuring and summarizing the population that enters the processes.
• The marketing-system: Measuring how well the process re-commits, holds and upsells visitors.
• The operational-system: Measuring how well the process reduces friction and gets committed visitors through the process.

The pre-qualification system should measure the variables that tend to illuminate differences in process conversion that have nothing to do with changes to the actual process. Key measures in the pre-qualification system that should be modeled include:

• Completion and Step Fall-Out by Segment -Completion rates by key visitor segments
• Completion and Step Fall-Out by Previous Visits - Completion rates measured by # of visits before starting form
• Completion and Step Fall-Out by Previous Pages - Completion rates measured by pages viewed before starting form
• Completion and Step Fall-Out by Source Page - Completion rates measured by page name before starting form
• Completion and Step Fall-Out by Source - Completion rates by channel (PPC, SEO, Banner, Direct, Link, etc.)

The marketing system attempts to measure how well the process works from a sales perspective. The marketing tasks in most processes include providing re-assurance, reminding about sales benefits, surfacing necessary information, and re-engaging upon completion.

Key measures that should be modeled in the marketing system for a process include:

• Step Reassurance Rate - Percent of visitors who used a reassurance link (privacy, security, etc.) by step.
• Step Rejoin Rate - % of visitors who rejoined the process after abandoning on a specific step.
• Fallout Direction - The top next pages when abandoning the process and the top search terms after abandoning the process
• Completer Routing Performance - The % of controlled routes driven by the thank you page

In the operational system, you are measuring how well the form is accomplishing its basic task of collecting and processing information.

The key measures to model in the operational system are:

• Step to Step “Fallout” Rate - The classic funnel analysis
• Multi-Session Step to Step “Fallout” Rate - Funnel analysis extended to cover cross-session re-join behavior
• Step Mistake Rate - The number of user errors reported by step
• Step Mistake Type - The count of mistakes by type and by step
• Time on Step - Total time for completion, measured by start to complete
• Total Step Time - Measured by summing step times
• Step Time for Abandoners by step
• Step Time for non-completers
• Step Time for completers

Combining the high-level outputs of each reporting system into a global process report should capture the key changes in each system and function of the process.

Understanding how qualification levels varied can help protect site managers from misinterpreting changes in conversion rate. Understanding the difference between marketing abandonment and friction can help focus on the right type of change to actually help solve the problem.

By combining these factors into a high-level “analytic” report, you’ll have transitioned your thinking about conversion processes and web analytics from the sterile and often misleading focus on “step fall-out” to a deep understanding of the factors that actually matter in improving or draining your site’s process performance.

And not only will you have gained a much better understanding, that understanding will have deepened your thinking about the types of actions that will make a difference. You can spend – and waste – endless amounts of time streamlining a process whose real problems may lie within the marketing system. And, of course, you could add more and more sales material to a process and achieve nothing but increasing losses due to friction.

Only thoughtful, system-based measurement can insure that your process meets the true, real-world needs of your customers.

Forms Measurement and Multi-Session Behavior

In this, my penultimate post on the topic of analyzing Forms and Conversion Processes, I’m going to tackle the issues and measurement surrounding processes that are abandoned and then resumed in subsequent sessions. Multi-session form behavior greatly complicates the task of analysis – not least because most web analytic tools do a very poor job with any form of multi-session analysis. In the next installment I’m going to wrap up with a discussion of Form KPIs and reporting.

Not every form or process exhibits much multi-session behavior. It’s rare to see multi-session behavior on short, simple forms. These may be abandoned immediately and then tried again, but I generally think of multi-session behavior as belonging to the set of cases where a visitor fills out part of the form, abandons the session, and then resumes later on.

Understanding multi-session form/process behavior can help answer quite a few different questions including:

• What is the real conversion rate on the process?
• Does a Form require “Save and Return” functionality?
• Are there information requirements on the Form driving drop-off?
• Does a Form adequately inform users of key requirements?
• Are there information requests causing high-friction that can be eliminated?
• Can a Funnel be used to track the Form?

The right place to start is figuring out if – for a form/process – multi-session behavior actually matters. Intuitively, you can expect that any long form will exhibit a fair amount of multi-session behavior. But the extent is quite variable and often driven by the type of information required of the user. If a process requires a user to provide things like bank routing numbers, license numbers, tax-id numbers, etc. then a significant amount of multi-session behavior is almost inevitable.

I remarked that most tools make it quite difficult to understand multi-session Forms behavior. But the one thing you can usually get quite easily is the extent to which such behavior exists.

The easiest way is to simply generate a report for each page of the process that includes the views, visits and unique visitors. The visits to visitors ratio here will provide you with an immediate read on the extent to which multi-session behavior actually exists. Make sure (of course) that you aren't looking at a number like Daily Uniques or an addition of Daily Uniques.

If significant multi-session behavior does exist, you’ll want to focus on the VISITOR abandonment rate as the true measure of form/process conversion. For the entire Form, that rate is again, quite trivial. It is simply the Unique Visitor Count for Form Complete divided by the Unique Visitor Count for Form Start. This rate may be significantly better than a visit-based measure of form conversion whenever multi-session behavior exists.

This will also implicitly answer the question of whether funnels or fall-out tools can be used to track form processes. These tools are almost all visit-based. So if you have significant multi-session behavior, they will present a very misleading view of actual form performance. In addition, many such tools simply will not work correctly if your Form contains Save-and-Continue functionality.

Save-and-Continue functionality can be a big boon to usability. Chances are, good usability testing will have already shown whether or not it is important. But if you have significant multi-session behavior and can see that abandonment is occurring well into the Form (particularly by time spent), then a strong case for Save-and-Continue functionality can almost always be made.

At this point, however, we have left the realm of easily accessible KPIs. To understand when/where Form Abandoners who later-returned left the process is a non-trivial endeavor. You can’t simply look at Exit Rates by Page because you don’t know whether the exits are for returnees. Instead, you have to be able to segment your Form visitors based on the number of visits that contain the Form. Depending on your segmentation tools, you might also choose to look at a simple segment like "Form Completers."

If you have a segment of Form Completers, you can assume that any Form/Process Page Exit (other than complete) was followed by a return visit. Using this, you can pinpoint which pages in the process caused visitors to abandon and how long they spent on the process (on average) before they abandoned.

With these two pieces of information, you can probably make a better decision about the utility of a Save-and-Continue capability (or, possibly, a reorganization of the Form).

It’s also interesting to consider directional abandonment for multi-session Form Completers. Using that same Form Completer segment, you can look at Next Pages for each step of the Form. When those pages are outside the Form Process, you can see what types of information or re-assurance visitors who ended up completing might have been looking for.

This ability to pinpoint where multi-session completers abandoned the process is the best behavioral tool for deciding whether or not there are special informational requirements driving abandonment and whether or not the Form is adequately conveying those requirements. By testing up-front Form guidance, you can measure the impact on multi-session abandonment points and see if you’ve made a difference.

It’s a lot easier to see if you’ve made a difference with this functional approach than if you need to move the needle significantly on end-point conversion. Many changes that make a Form better, faster and easier for the user will have only a very small impact on final conversion. That doesn’t make them worthless.

One final piece of learning from multi-session behavior involves 2nd time abandonment points. If you find that Form users are abandoning in the same place on return visits (attempts to define user names on popular sites can have this issue), then you’ve certainly identified a high-friction area of a process. If you’re losing significant numbers of visitors committed enough to return for a second try, then you need to re-think your approach to the Form.

Multi-Session behavior – like nearly all of my other Form/Process topics – has a real impact on the reporting you do.  Ignore it, and you can significantly misread the real conversion rate of your Forms and the true potential for improvement. In my last post on Form/Process measurement, I’ll tackle reporting on processes with a systems approach (analytic reporting) and how the various points raised in the previous posts surface new metrics that are rarely used in reporting on Forms and make it clear how others (like Form Conversion Rate) can be easily misinterpreted.

Branding and SEM

There is an interesting thread on Jacques Warren’s blog that is quite distinct from my discussion of branding but has a definite relationship to it. Brand marketing is a surprisingly big part of even the most direct aspects of web marketing. Both the post itself and the ensuing discussion are well worth reading.

Jacques' post comments on the high-propensity to brand search on Google (and not just Google).

Indeed. The degree to which Search Programs are “branded” can be shocking given the industry-wide focus on the “long-tail,” on direct-response techniques, localization, landing-page optimization, etc.  Most of these techniques don’t make much sense or difference when 90% of your visits are people typing in your company name (though if you haven’t tried them maybe you can change that ratio).

There is nothing inherently wrong with “branded” search. But I can’t help but feel that it’s sometimes a bit of a con. Search Marketers often talk as if they are the world’s greatest channel for early-stage prospecting even when the programs they run are little more than glorified signposts to known web sites. This isn't always true, of course. Just true often enough to be scary.

I have yet to see a search program where the “branded” component didn’t have fundamentally different behavioral and performance characteristics from the non-branded part. This is common knowledge, of course, and it is pretty much accepted practice to treat and measure “brand” and “non-brand” components as distinct elements. So if your SEM program isn’t making the “brand” component of your search program visible and treating it distinctly, you need to think about changing your SEM program.

And, to me, it’s just another indication of how important brand often is. There are plenty of direct response people who sneer at branding at the same time that the success of their program is largely driven by the strength of a brand.

Engagement as a Term of Art in Web Analytics - Brand Engagement

Part IV in Series on Measuring Engagement

When working with some of our largest clients to quantify the value of their sites, I found myself repeatedly evaluating sites whose primary function business owners described as "branding." Since they'd spent money (often quite a bit) building and maintaining these sites, they not unreasonably wanted to know what their value was.

It's a tough question but not a foolish one.

Why should we care about brand value? I know that most direct marketers and many online marketers have a deep skepticism about brand marketing - viewing it as little more than an excuse to forgo any measurability.

It certainly can be that. But I am not in that philosophical camp. I don't question that "brand" marketing is both real and potentially very effective; nor do I doubt that "brand" is something real - albeit something difficult to measure.

Every customer interaction a company has is part of building a brand. Human interactions are uniquely powerful of course. One rude employee can ruin your brand for someone. A genuinely helpful one can cement a brand relationship for years. But advertising, ATMs, web sites, bills, statements - they all matter.

It's hard to say where the web site fits into this - and every reason to believe that it varies quite a bit from industry to industry and customer to customer. For Google, the web site just is the brand. For P&G it's quite a bit less. For a bank or an online brokerage it's a huge part of the everyday experience of the company. For an airline it's merely a significant piece.

But can any of this be measured?

In a way I think it can - but only indirectly. We all know that behavioral data has fundamental limitations in what it tells us about a visitor's motivations and attitudes. It's probably unreasonable to think that a purely behavioral analysis can, on its own, quantify brand value.

On the other hand, there is no simple way to quantify such things in any other channel or medium. What is the value of "friendliness" training for airline employees? Or wine instruction for waiters? These aren't easy questions to answer either.

I suspect that the only real method for measuring "brand" value will involve significant primary research using fairly traditional techniques. But part of that primary research needs to be tied to online behavior so that the true impact of differential levels of web site interaction can be assessed.

This is a topic I'm going to take up in more detail when I discuss visitor segmentation strategies (in the near future). But it seems likely to me that when you make that tie between changes in assessed "brand" awareness and satisfaction and differential uses of the online properties, you will be using a concept that looks remarkably similar to engagement. However, that concept will be neither a proxy for success (at least not in the sense in which I used the term originally) nor a measure of media-buying.

Presumably, differing levels of accomplishment will deliver a different brand value. For most sites, the longer you spend on the site, the higher the likely brand impression (this might even be true for Google). For most sites, the more pages you consume, the higher the likely brand impression. For most sites, the more interactive you are with the site, the higher the likely brand impression.

Why isn't this brand engagement concept the same as a proxy for success? It can be, but where the site has other functions besides branding, the relationship breaks down. In my lead gen example from the first post on this topic, if you are using engagement as a proxy for success, then you have to take the value of that success into account. Since one lead-type may be sourcing much more valuable customers, the proxy must weight that lead higher or it won't be an appropriate tool for optimization.

This simply isn't the same thing as the strength of the brand impression. It's perfectly possible that between two types of lead gen opportunities one is more valuable and the other creates a deeper brand impression. In branding, engagement is serving as a proxy - but the proxy is to a psychological state not a site success.

Although this measurement of brand impact is undoubtedly going to be different for every site (and certainly every industry), I see some promise and use for a standardized measure of engagement that is common across sites.

Many site owners would like very much to be able to account (and take credit for) the brand value they deliver. We know, intuitively, that they do in fact deliver this value. We simply lack an accepted framework for claiming it.

The difficulties involved in doing the primary research and building this analysis make it prohibitive for every company to study. So I think it likely that if a few companies where this problem is especially important lead the way, most other companies will simply accept their framework. This is never ideal, but it is a common practice in a great deal of market research where the barriers and costs to doing customized analysis are significant.

Summing Up

To discuss any issue usefully, you have to know what sort of thing you are really talking about. That just hasn't been the case in a great deal of web analytics thinking about engagement; the discussion has been much too liberal in the assumption that we "all know what we are talking about." By segregating out these three distinct uses (and I'm pretty satisfied with Eric's naming convention) and giving each a "Term-of-Art" we actually do achieve an improvement in clarity.

We often criticize disciplines for having specialized vocabularies. But while specialized vocabularies can be excessive and misused, they exist for a reason. Just as the Eskimo needs many words for snow, we - as web analytic practitioners - need more than one word for engagement.

This is an awfully long prelude to what will probably be 10 minutes of talking at eMetrics.  I do feel like these posts have helped me clarify my own thinking in preparation for that talk. And I'm starting to look forward to the eMetrics discussion and finding out how all of this fits into the work Eric and Joseph are doing. As I usually say at the end of my podcasts, talk to you then!

Measuring Engagement for Media Buys

Before I dive into the issue of measuring “brand” engagement, I decided to add a few thoughts and re-considerations from my last post on measuring engagement for media-buying.

In that post, I articulated a set of reasons why measuring engagement for media-buying might not be world’s smartest idea. The argument that underlies almost everything in that post is that virtually any measure of engagement as imagined from the site perspective can and will create situations where improvements in the measure don’t actually benefit the media buyer – and actions that do benefit the media buyer and make ads more effective don’t get credited in the measure of engagement.

That’s a bad thing, of course. But as I thought more about this problem, I realized that I hadn't captured everything involved in this issue.

Media-Buying seems to involve at least four separate dimensions:

• Effective reach: Trying to get the highest impressions and impression value
• Demographic Matching: Trying to find the most likely customers based on key demographic
• Brand Matching: Trying to find outlets that appeal to similar brand values
• Brand Commitment: Trying to find outlets appropriate for brand integration

The original measure used by companies to compare web sites (total page views) clearly gets to only the first of these functions. In my opinion, time on site is no different. And the behavioral alternative I proposed (based on Quality Click-Throughs) is designed to do the same.

None of these measures, in my opinion, address any of the other 3 dimensions.

Might they?

Demographic matching is sometimes more difficult on the web but there is little chance that any measure of engagement as commonly understood would address this very necessary function. It is perfectly to possible to re-imagine demographic targeting as interest profiling, but that still is nothing like engagement.

Brand Matching is similar in some respects to geo-demographic targeting but is based on perceived similarities in brand identity. Often, these similarities are heavily influenced by geo-demographics but they are not quite the same thing. As with customer matching, it is hard to see how any measure of engagement might facilitate brand-matching.

Brand commitment is a different story, though. Most media buying isn’t all that concerned with how attached people are to the outlet. We care that they are attached enough to show up, of course. But the buy is focused more on reach, demographic and brand matching than brand commitment to the carrier. Nobody has a brand commitment to a billboard, but that doesn't make the iPod ads less effective.

However, there are a set of situations where this isn’t true. Where the target buys are sponsored or will integrate with the outlet (e.g. having a radio personality like Rush Limbaugh or Jim Rome sell your product), then it’s important to understand how committed the audience is to the target brand.

Some programs/vehicles deliver plenty of audience but not the strong “brand” commitment that would make sponsorship or personality driven brand-integration meaningful and worth the extra cost/effort. The same is true on the web. And here, at last, is a case where a standardized measure of site engagement might actually matter in an intelligent way to the buyer.

If I’m contemplating integrated brand messaging with another web property, I would be very interested in evidence of strong brand commitment – probably via a measure like engagement.

It should be noted, in this regard, that use of averages to represent site engagement would not be the best technique. A site that bifurcated into very high and very low usage visitors might show similar engagement averages to one that had a much flatter engagement profile. But the former site would be much more likely to have passionate advocates. As with many analytic problems, this is a case where a distribution is considerably more meaningful than an average.

So I’m going to retract part of what I said in my last post: there seems to be at least one good use for a standardized measure of engagement in online media buying - just not as the primary means of comparing sites or pricing inventory.

All of which brings me to my last post – one I’ll tackle next time – thinking about “brand” engagement. Originally, I viewed the measurement of “brand” engagement as something relevant primarily to the owner site; a means of capturing another dimension of site value. I still believe that this is probably it's main function. But on this interpretation that same measure - understood and used correctly - might become a useful part of a media buyer’s toolkit.

Engagement as a Term of Art in Web Analytics

Part II - Engagement as a Media Metric

In the first post on this subject I proposed three distinct uses of the word engagement in web analytics and I covered the usage (as a success or lead-value proxy) that is most familiar to us at Semphonic. After that post, Eric Peterson sent me an excellent and quite detailed comment that I’m going to strongly suggest you read. Why? Well, a good chunk of it was stuff I was going to say but now feel stupid repeating. Thanks Eric….

Eric didn’t entirely replicate my thinking, however, and I’m going to take up my second usage of Engagement (providing a measure that can be used by media buyers to compare sites) and just tailor my thoughts a bit so as not to repeat his points. Here is what Eric wrote about this type of engagement:

“The second measure of engagement, within the realm of media measurement, is something I have started referring to as "Audience Engagement." Audience Engagement has to be measured ** not ** using a census-based system as it requires cross-site visibility. This measure of engagement almost certainly includes some of the measures of engagement you'd include in visitor engagement (time, click-depth, recency) but would likely ** not ** include more site specific actions. Audience Engagement gives media planners and buyers a different ruler against which to judge the audience visiting competing properties. It is still not completely clear to me how media planners will use a measure of Audience Engagement; it is only clear that they are actively looking for such a measure.”

I like the term “audience engagement” – and Eric’s comment that “It’s not completely clear to me how media planners will use a measure of Audience Engagement” echoes my sentiments exactly. In fact, I have deep doubts about this project.

My sense is that this usage of engagement has been shaped by historical and offline concepts in ways that make it quite suspect. At its core, I believe it is an attempt to approximate reach on the web - something like a GRP.

Media Measurement started out by focusing on Page Views as a comparative measure. This approach had a certain in-built logic since it matched the unit at which media was sold. Page-view based media comparison had its problems, however. It was increasingly broken for the very sites the traditional media measurement companies were most interested in – large media publishers. As video, flash and rich-media became common on those sites, the meaning of a page view was greatly diminished as a measure of reach.

To counter this problem, traditional media measurement has moved toward time on site as a better approximation for media buyers. It is, in fact, a better proxy for a certain type of site – most notably those same large media companies. But as has been noted in many places and by many people, it is hardly a universal measure. When I was speaking at SMX last month and this question came up, Brett Crosby from Google made the point that one of Google’s success factors is reducing time on site. The less time someone spends before clicking out the better.

What’s curious about this is that if you accept – as I think is correct – that this use of engagement is meant to facilitate the buying function, then we should reasonably expect it to show Google as a huge winner. After all, that’s where all the dollars are and I don’t think it’s because buyers are stupid.

The Google example illustrates an important point about media measurement in general; it only seems applicable within a certain class of sites (I’m indebted to a discussion with Joe Shantz of PHD for clarifying some of my thinking here).

We might think Time on Site is interesting as a comparison of two news sites, but find it meaningless as a comparison of a news site and a social networking site. Unfortunately, sites don’t always fit into neat classifications. Large properties tend to cross many boundaries; so any form of media measurement that demands site commensurability is problematic from the get go.

Nor is this the only issue at stake. At a deeper level, the issue is a fundamental difference between web display advertising and traditional mass media. In most traditional media, the experience of delivering eyeballs/ears is essentially synchronous. What I mean by that is that the programming doesn’t compete with the advertising – it funnels it. On the web, this is almost never the case. In nearly every experience on the web, advertising and content are delivered asynchronously and are constantly competing for attention.

It is the fundamentally asynchronous nature of the web that accounts, in my view, for the dramatic difference in metric performance by site type. The site experience of advertising on a social networking site is fundamentally different than the site experience of advertising on a media site.

However, the differences are hardly limited to site type. Even sites within the same paradigm are dramatically different in their “friendliness” to advertising. Known techniques like ad scheduling (showing different versions of ads to reduce tune-out), ad collapsing (removing ads or changing layouts also to reduce tune-out) and behavioral targeting can make a dramatic difference in the actual performance of advertising - not to mention simple things like ad placement. If these techniques aren’t measured by a media metric, then what good is the metric?

I’ve said before in other contexts that choosing the wrong metric for optimization is usually much worse than not having a metric at all. I think this applies in spades to the situation in media measurement.

Metrics like time-on-site and click-depth completely miss the asynchronous nature of the web and by focusing on the wrong metric they create a dissonance between the interests of an advertiser and the interest of a publisher. You can maximize time-on-site by reducing advertiser effectiveness. In effect, the metric becomes a self-defeating prophecy. Publishers can make more by making things worse for advertisers!

This problem simply does not exist in Radio and TV and it’s why any attempt to shift the mass media measurement paradigm to the web (as content is currently structured) is doomed to failure.

It’s also easy to see – on this paradigm – one of the reasons why Google works very well. It doesn’t compete for eyeballs at all. There is no there, there.

Is there a metric that would actually be useful to assist in media buying?

I believe that any worthwhile media buying metric should do all the following:
1. Show Google as a hugely dominant player on the internet
2. Reward sites for behaviors that improve advertising friendliness like:
    a. Behavioral Targeting
    b. Scheduling
    c. Collapsing
3. Not require an artificial categorization of large properties into some pre-defined group.
4. Create no dissonance between the optimization of the publisher and the optimization of the advertiser.

I can’t think of any traditional metric that will meet these criteria. However, there is a behavioral metric that might help (at least for traditional display – video is a different animal).

What I have in mind is a measurement of quality click-throughs. A click-through is just a click to an advertiser from a site. Quality might be defined as something like “a click-through to an advertising or sponsoring site that is judged non-fraudulent by the target site, consumes more than one content unit and does not return to the originating site directly.”

You’ll note that the measure of Quality Click Throughs (QCT) should meet the four criteria above. Google contributes a vast number of QCTs to the internet. It also contributes a fantastic QCT per impression. Techniques to improve advertising effectiveness will result in more QCTs and more QCTs per unit. QCTs are insensitive to site type because they reflect advertising friendliness. And finally, QCT’s foster a shared interest between publisher and advertiser. QCTs even provide a nice metric for pricing and evaluating placements inside a site.

It seems to me that using QCTs, you can get a good sense of site reach (total QCTs), site quality (QCTs / CTs) and efficiency or advertising friendliness (QCTs/impression or QCTs/minute and QCTs/dollar).

If I were a media buyer, these metrics would seem to me vastly more interesting as comparables than total page views, total time on site or any other similar metric. Of course, it’s not clear that these metrics are anything like a measure of engagement – except possibly with advertising.

One of the implications to all of this is that the measure of engagement that would be appropriate to media buying is not only different from but in many ways opposed to the measure of engagement appropriate to the site owner. Based on actual practice where issues about the costs and benefits of losing site visitors to networks like Google’s Ad Sense are routinely discussed, this seems reasonable to me.

There is nothing particularly revolutionary about the idea of measuring click-throughs. People do this all the time to evaluate advertisers and placements. However, adding the quality dimension sharpens the measure and makes it much more useful for advertisers as a comparable. That it doesn't necessarily measure how engaging a site is may be taken as a flaw - but given the asynchronous nature of the web I think any advertiser should be careful about using a measure that actually did capture that elusive quality.

In my next (and last) post on this topic, I’m going to cover a third important use of engagement – to measure the brand impact of a site. It's the one use of engagement where capturing that elusive state of "engagement" really is the goal.