Defending the Indefensible

I’m not sure how hard I want to work to defend my last post (“Numbers it’s better not to know”). Jacques Warren called it “interesting but puzzling,” and I suppose that’s about how I meant it. In his comment on the post, Jacques points out that it’s risky and possibly inappropriate to give people an excuse not to care about or look at data. He’s right of course. There are many, many more situations where measurement is simply and wrongly being ignored than situations where it is being used when perhaps it shouldn’t be.

But the ideas underlying the post – that there are certain situations where measurement is particularly hard to use or particularly easy to abuse are, I think, valid and appropriate. It’s fairly important for an individual practitioner to understand the common causes of error, but it's critical for someone who is helping create or refine the web analytic processes that an organization will rely on.

In philosophy, it’s often essential as part of an argument to show how a theory both illuminates and explains why people go wrong. An error theory that shows how opposing views manage to capture only a piece of a truth or go wrong in an understandable way is very powerful.

When it comes to building good web analytics process, having a good theory of error is equally powerful. The goal of a good measurement process is to create an environment where the data we collect is analyzed and used effectively. As Jacques points out, in many environments, the primary focus for process development needs to be on encouraging the use of data.

But just as Jesse Jackson use to say “What a man can conceive, a man can achieve,” I’d say it’s equally true that “What people can use, people can abuse.”

So good process needs to be concerned with more than just encouraging the use of analytics. Good process needs to foster the effective use of analytics.  And figuring out what situations tend to foster data abuse – developing an error theory – is an obvious necessity for a web analytics process consultant.

In my original post, I enumerated several situations where the potential for abuse or unintentional misuse of data is particularly high:

1.    Situations where the decision-maker getting the data is also heavily invested personally in the underlying execution of the process (my blog was the example). This is extremely common in business and is one of the main issues that any good analytics process should explicitly deal with. Key process considerations include having a quasi-independent measurement organization and pre-committing to the measurement goals for every initiative.
2.    Situations where the action itself requires spontaneity or some level of freedom from concern about outcomes (again, I used my blog as an example). This may be rare in business but it’s not unheard of. And, by the way, I entirely agree with Jacques' take on one area where I don't think this applies.
3.    Situations where the measuring organization has a strong financial self-interest in the outcome. This is a classic problem for any truth-seeking activity and I used the example of having a PPC vendor provide you with the measurement of their own performance as a very common case.
4.    Situations where a political decision is being fought over and data is being grabbed willy-nilly to support one or both positions. I used the quite common case of open-ended anecdotal data from voice of customer research being used to support or attack particular policies.

There are many other common causes of error (inaccurate data and inappropriate understanding of techniques like trending as a possible corrective, poor training, inappropriate access to complex reporting solutions, complex report sets, incorrect use of KPIs, etc., etc.). If you don’t understand what these errors are, how likely they are to occur, and what types of organization or practice make them more or less likely, then your process is highly unlikely to protect you from them.

Nor is my argument that there are some numbers it is better not to know really all that unorthodox. After all, we make exactly that decision every time we eliminate some metric or view from a management report set. When I tell people they should never have a top 10 exit pages report, it isn’t because the information might not be valuable, it’s because in my experience the information is far more likely to be misunderstood than to be used judiciously – to be abused not used.

The role of a good process in web analytics is to create an environment where measurement is used and used appropriately. To get this right requires a deep understanding of the many, many ways in which we are all highly-likely to get things wrong!

Comments

[Jacques Warren]

Hi Gary,

I voluntarily left all nuance out of my comment for brevity purposes. Obviously, I am not promoting the use of false, or wrong information in the name of adoption. I am very concerned though with the capacity of a lot of organizations out there (or the sufficient merit of web analytics in the eyes of many companies) when it comes to invest time and energy in establishing the processes you talk about. I totally agree that, if one is at the stage of implementing those processes, not only use, but good use of data should be the clear focus/framework.

In my WA career, I have had to deal with several cases of the situations you describe (1 and 3 being the most common). In fact, I always have to deal with one, or more, of those situations with all new clients, and unfortunately the vast majority of them is still so new to WA that we are far from implementing processes.

I guess having them is a clear sign of analytical maturity, and maybe not all organisations are destined to get there. My question is: how can one bring a company that far, while fighting subjectivity, politics, and vested interests?