The Great comScore Debate

By now, nearly everyone in the web analytics community is abuzz over the recent release by comScore of a study on cookie deletion rates. comScore tracked a specific web site (advertised as a portal) and one 3^rd Party Ad Serving network against a panel of 400,000 users. During the study, comScore measured how often the 1st Party cookies (issued by the portal) and the 3rd Party cookies (issued by an Ad Serving Network) were deleted and replaced. The results can fairly be described as startling in several respects. First, comScore pegged 1st Party Cookie deletion rates during an average month at 31%. Second, comScore measured the number of cookies and found that there were a significant percentage (about 7%) of serial deleters – who ended up having more than 12 1st Party cookies from the Portal site during a month. Taken together, these numbers imply a dramatic error rate in the total unique visitors a site measured even during a month timeframe. Perhaps as, or even more, surprising, comScore tracked relatively little difference between 1st & 3rd Party cookie deletion. 3rd Party cookie deletion was higher (as almost everyone would have expected) but by a margin best described as small.

Some of the reaction to this is predictable. First, there is considerable skepticism about the findings themselves. Perhaps that’s justified. Lord knows there are plenty of flawed studies done, not least when the studying party has a strong self-interested stake in the outcome. In addition, the frequency of deletion coupled with the lack of disparity between 1st & 3rd Party cookies (where automated Spyware tools might provide an explanation) makes it hard for many people to understand. Are internet users really this consistent about deleting cookies when they have to do it manually? It does seem hard to believe. Keep in mind, as well, that we are talking cookie deletion – not rejection. So though 3rd Party cookies may share a similar lifespan to 1st Part cookies, they are considerably less likely to find their way onto a computer in the first place. And while the comScore study wasn’t especially clear about this, it doesn’t seem to have focused on cookie rejection at all.

There is a second common reaction summed up in the idea that this level of errors in reporting doesn’t much matter because analysts are concerned with trends not absolute numbers. Anil Batra of Zaaz, for instance, argues (here) that it would make no (or little) difference whether your site got 5k visitor or 3K visitors – the important thing is how your site is trending in response to your actions.

I’m afraid I can’t agree with this thinking. At first glance it’s not necessarily wrong-headed though I’d argue that in fact it’s important to know your actual audience size on the web - that real numbers do matter and not just trends. Imagine a conference organizer who wouldn’t tell sponsors how many visitors the conference draws – only that it is trending upwards. I doubt I’d sponsor a booth!

But this is really just the tip of the iceberg when it comes to analysis. After all, total visitors IS pretty far down on the list of statistics of interest to the analyst. Unfortunately, almost every single statistic that matters is going to be effected – and sometimes devastatingly – by this level of error in visitor tracking. Perhaps Zaaz doesn’t track "New" visitors to the site vs. returning ones. But I do. And when half of my "new" visitors are really heavy repeat visitors I can hardly hope that my analysis will be crisp. Then too, I’ve always believed that tracking repeat customers on a site is central to most eCommerce analysis. But with this level of error I’m misclassifying a big chunk of that behavior.

Or again, perhaps people aren’t worried about what tools or pages drove repeat visits or cross-session sales behaviors. But it seems important to me. For a portal site like the one tested by comScore, the bellwether analysis is content impact on engagement. I’d like to say that this analysis may still be possible with this level of error, but I’m not sure that it would be.

Not concerned about these little things? Well how about the fact that every one of your campaigns might be significantly mis-counted in terms of conversion? That seems pretty consequential.

In short, just because comScore isn’t focused on analytics but only on traffic reporting doesn’t mean the impact of their findings is limited to traffic. And the impact for analysis of this level of error in 1st Party cookies would be very bad indeed. That’s not an admission you’d probably expect to hear from a web analytics consultancy and I’m sorry if this constitutes breaking ranks, but our first commitment ought to be to the truth of the numbers.

On the other hand, I too have some caveats about the general applicability of these results to every site. First, it’s important to understand the difference between each statistic. The 31% 1st Party cookie rate is, in my view, probably the most damning statistic in the study. Why? Because unless the behavior is somehow related to the specific site in question, you’d expect it to hold up for every site. Or then again, maybe you wouldn’t. If we accept that this statistic represents manual deletion of cookies, then there are two alternatives. The first is that people simply erase all cookies. That’s certainly possible – maybe even likely since it’s so easy to do. But a user may also scan the cookies on the target system and since cookies are often quite identifiable it’s possible that some cookies are much more likely to be deleted than others. It may be that an Amazon or Charles Schwab cookie is much less likely to be deleted than an AOL one. That’s one of the reasons it would be very interesting to understand the qualitative part of the comScore study – namely, what deleters (especially "serial" deleters) were actually doing. In addition, I’d like to understand if this problem is especially severe for certain browser populations (Firefox). And are users simply setting their browser to delete all cookies every time they close? This might explain the behavior of deleters – they aren’t doing any manual work at all. It would also imply that for this segment even same day uniques are overstated.

As comScore positions the study, however, the most damning finding is probably the one about the potential traffic impact of serial cookie "deleters." And for their particular chosen site, the impact would indeed be considerable. But for many of our sites, this group would probably have much less of an impact. Why? Portal sites are unusual in attracting a very high volume of repeat visits. comScore reported an average of nearly 13 1st Party cookies during a month for the 7% of visitors with 4 or more cookies detected. For most of our client sites (that aren’t portals), the overwhelming majority of visitors will come much less often than this. And if only a very small percentage of visitors are multiple repeaters and only a small percentage (7%) of those are multiple deleters, then the impact is likely to be considerably less (in terms of percentage of total visitors) than for a portal site. It’s likely true that no set of sites has a higher percentage of frequent return visitors than a true portal.

I’ll be honest – I find the study worrisome. It’s not that I haven’t realized that cookie deletion is a significant issue or that the potential impact of frequent return visitors on visitor traffic eluded me. But every analyst knows there is a line between data that is fuzzy but useful and data that is too messy to analyze. And I’ll be honest as well in saying that we see many sites where the number of visitors flagged as "New" would be better explained by the comScore view of the world than by the one prevalent (including by us) in the web analytics world. Depending on how the comScore study shakes out, web analysts may be facing a significant rethinking of where we are relative to that "fuzzy" line for almost every kind of interesting analysis. For some types of sites, at least, this might drive our short-term tool kit back to the pathetic same session analysis we were stuck with in tools a couple of years back.

comScore has their own agenda, here, obviously. But to me, the issue is ultimately about much more than unique visitor counts. Perhaps Anil is right to discount the importance of that statistic. But in doing so, he’s surely missing the larger point. In my view, the basic analytic toolkit of visitor segmentation and cross-session tracking is at risk in light of these findings. What’s the point of visitor segmentation when your visitors are lost every couple of days? This is an issue the measurement community just can’t ignore. I remember when 3rd Party cookie rejection rates began to climb and the ostrich like mentality that insisted this wasn’t an issue. Till suddenly the vendors made it possible to use tagged 1st Party cookies and everyone began implementing first party cookies. Sadly, we may not have solved our problem.

Are there short-term solutions? Possibly. And they begin with getting a handle on the scope of the problem for your site and understanding how to protect your analysis from these problems. You can begin to get a handle on the issue for your site by examining the trends in "New" visitors – especially if you can measure from significant events like a measurement start, a new cookie, or major traffic spikes. You might also want to look at the percentage of New/Returning visitors by Browser type. If you have strong behavioral cues on your site (like Login), you can measure the degree to which these behaviors are used by "NEW" visitors. This percentage compared to total visitor usage is an excellent way to get a reasonable read on how much or little of a problem you have. Handling bias is trickier, but if you want to analyze, for example, whether types of tools or content drive repeat visits, then you’ll be wise to start with a universe of visitors that you can prove has remained cookie constant. How might you do this? By insuring that you segment for behavior in both the 1st and last month of your study. This will narrow your analysis, but insure that you’ve ruled out cookie issues.

In the end, every problem in data quality simple complicates the life of the analyst more and makes doing good analysis just that much harder. I don’t much care who wins the great traffic reporting war. But I care very much that web analytics have a mechanism for tracking with some degree reliability the over time behavior of visitors. Where there is a will, I’m confident our truly fine tool vendors will find a way. It may be incumbent on analysts to provide that will.

Comments

[Jacques Warren]

I agree with you: this is no trivial matter. I think we all reacted like Anil, and still want to believe in our hearts that, well, trends, trends, trends. But this study, if really rigourous in its methodolody and findings, is really going to bring a lot of nerve racking dicussions. And we should have them: a visitor is going to have to be a person soon. What are we going to do when database and analytics integration is on every marketer's lips? How long can we live in a world where our visitors are likely to be suffering from multiple-personality disorder,and change identities when they commute?

I think you do a fine job in putting the percentage in the context of the web site type, but still, you're right to be very worried. I know I am. I am even wondering if getting back to same-session analysis won't offer at least more solid data... but we know that's not what we want to do.

Are we to say that maybe the problem, the anonymous nature of our data, has its solution in the "soft personalization" of the good'ol days of greeting visitors by their name? I know Amazon still does it, and I don't need to login. In fact, I hate deleting my cookies exactly because I end up not being recognized by the sites I use frequently. Which means that they can at least link me to some kind of profile.

I know, this sounds quite simplistic. But, again, we will need to get closer to the person, and people will only accept it with sites they have a vested interest in. I think one day we will determine acquiring an email, or anything else that permits starting a dialog, as valuable a conversion as a sale itself. And when we start knowing WHO's coming to dinner, we will care less about the anonymous part of the trafic, because of the tremendous value we will be able to extract from visitors who are willing to say hi.

Well, you will excuse my disorganized thoughts: hard to express all that in a comment.

[Clint]

Gary,
this is perhaps the most reasoned response I've seen to the issue.

My concern over the study revolves around the fact that it seems to be a single internet entity and a single ad network. Even if it is one of the top five, I wouldn't call it a representative sample in terms of the types of sites on the internet.

Furthermore, I don't really find the argument that 'comScore is up to something' argument. I might be inclined to believe it if the study had come from Nielsen - who has a competing site analytics product, but as far as I know comScore doesn't and isn't interested in competing in that (our) market.

[Anil Batra]

Gary,

I want to make it clear that views expressed on my blog are my own and do not represent ZAAZ or any other employee of ZAAZ.

I have posted a response to you post on my blog

http://webanalysis.blogspot.com/2007/04/great-comscore-debate.html

[Magid Abraham]

Gary:

I commend you on these two thoughtful postings.

I must admit that, when we conducted this study, we were primarily focused on the inflation in UV metrics and in campaign reach estimates provided by ad server logs. The issues you are bringing up are obviously very important and I am not surprised at the debate they are generating. We are in the process of writing up a more detailed paper on our analysis which I expect we will release next week. I hope it will answer a number of questions people have about the study specifics. In the meantime, let me clarify a couple of things relevant to this dialogue.

We chose the ad server and large portal cookies to get the broadest reach possible and provide the broadest view of cookie deletion behavior. In particular, the ad server we chose is so ubiquitous that you can hardly surf the web and not get their cookie. This is useful because it provides a good proxy for overall cookie ‘resets’ by the user, since a cookie reset will almost always translate into a new cookie for this server.

Obviously individual sites will be impacted differently. In general, the impact increases as a function of site’s frequency of visitations. At one extreme, a site that is only visited once during a one month period will not be impacted at all since it will have only one cookie per user regardless of how often the user deletes cookies. On the other hand, a site with a high degree of repeat visitors will tend to count a higher fraction of its visitors multiple times. The duplication is more acute if the site’s audience skews towards people who have a higher propensity to delete cookies.

As I think about the issues you raise, it occurs to me that it may be possible to come up with empirical adjustments to new and repeat visitors as well as campaign conversion metrics generated by Web Analytics software. We can specifically track the deletion rate of the site’s own cookie(s) and derive the appropriate correction for the various metrics.

I have always believed that audience measurement services like comScore are highly complementary to Web Analytics services. In that we tend to be focused on providing added value about competitive information and demographic profiling of the individual user, whereas Web Analytics go much deeper in analyzing the site’s own audience. This study may in fact imply that the two types of services are not just complementary but also interdependent. Each service enhances the value of the other, particularly if these correction factors are used. I would be curious whether you or your readers agree.