Resolving to have a Plan

Building a Web Analytics Roadmap for 2008

I’ve never been a big believer in trying to run your own life as if you were a department in a big company. You know the kind of thing I mean – building a five year plan for brand “Angel.”

But if planning can easily be over-talked, it is even easier to under do. It really is salutary to step outside the day to day hustle and bustle from time to time and try to figure what you really want and what you are trying to accomplish. New Year’s Resolutions are just one – rather silly – artifice for getting ourselves to do just that.

In the working world of web analytics, the story is pretty much the same. Even when analytics IS a department in a big company, the planning around analytics is often significantly under-done. And a lot of the planning that is done  tends to be focused on a relatively narrow set of process and resource issues. Questions like ‘how many staffers do I need’, ‘how should I set up my department’ and ‘what tool should I use’ tend to get all the attention. These are important, even vital questions. But they are only one part of a much larger story.

Meanwhile, the actual business of what you want to measure tends to get ignored. 

Why is that?

Part of the reason people don’t focus on the actual business of what to measure when building a plan is a basic misconception about analysis – the idea that data analysis is primarily exploratory. If that were true, then it would be meaningless to map out a plan of what you wanted to analyze at any great level of detail. Lewis and Clark didn’t have enough information to make a detailed plan of their trip across the United States – the essence of their voyage was discovery.

There is an element of discovery to any analysis – but the discovery is much more limited in scope than people who aren’t experienced and practicing analysts tend to believe. We aren’t often setting off across an uncharted continent. Instead, discovery takes place within a fairly narrow sphere defined by an investigation into a specific question or business problem.

Random exploration of the data is almost always a purposeless walkabout. There’s just too much data for this type of discovery to ever work. And, as a practical matter, web analytics tools tend to be very poor at the relatively few analytic techniques that are geared toward pure data discovery. To get an answer to a question – particularly a web analytics question, you almost always have to start with... the question!

This failure to recognize that analysis must be directed leads to all sorts of bad consequences in an organization. When there is no plan, there is no common set of expectations about what web measurement is supposed to accomplish. This leads to expectations that are equal parts grandiose and nebulous. That’s bad news, because when you look back on what you have accomplished it’s always more likely to look nebulous than grandiose! Worse, the absence of plan means that analyst time tends to get eaten up with ad hoc requests for information. If you’re in an environment where you never seem to do analysis because you’re always drowning in ad hoc requests, chances are it’s because you never protected yourself with a plan. If you can’t point to an analysis and say something like “We can tackle request X but it means we won’t finish analysis Y until April” then you are pretty much going to be stuck doing lots of X.

Unrealistic and nebulous expectations and a lack of defense against random ad hoc requests are both pretty bad consequences of not having a plan. But the most serious consequence of not having a plan is that your analysts typically don’t know what to do either. There are very few analysts who you could ever trust to conceive of an analytic project that is truly important and then execute on it. The inability to do this doesn’t mean an analyst isn’t competent. In fact, if an analyst can do this, they should probably be managing your business instead of analyzing it!

It’s the job of your management team to know the questions and issues that they really need to answer to make the business work. It’s the job of the analyst to take those questions and issues and provide back reliable and accurate information that can shape intelligent answers. To make that work, you need a process that joins your management team’s expertise and concerns with your analytic team’s knowledge and resources.

The process I have in mind for this is something we call an Analytic Roadmap. It’s nothing fancy: it’s a tactical plan that lays out in considerable detail exactly what an organization means to accomplish around measurement in the coming year. This should include the specific analytic problems the organization plans to study AND solve in the coming year.

Building the plan is about a four week excercise that involves stakeholders across the online channel. It's goal is to lay out a buffet of projects that meet a wide-range of business goals. From this wide selection of projects, a smaller set are incorporated into a formal measurement plan. The plan is constructed based on a combination of factors including likely business impact, interdependencie and ease of analysis. And each piece of the plan is a well documented and highly-specific analysis or measurement project.

I'll cover all this in more detail in my next post.

For now, I'll end by saying that the beauty of this sort of plan is the clarity and coherence it brings to a web measurement program. Most stakeholders in an organization simply don't know what to expect from web measurement. The plan is a great way to solve that problem. It can help build a broad consensus about exactly what measurement is for and what it can and should achieve.

It it also the only reasonable way to figure out what kind of resources you actually need to devote to web analytics. After all, if you don't know what you are going to study and you don't know what's involved, how can you say how many people and what kind of tools you need?

New Years is a traditional time to re-assess what we are doing and how we are doing it. Perhaps as part of that, it's time to re-think what you've been doing with web measurement and resolve that it is high-time to have a plan!

Comments

[geri]

It amazes me how many people plan to fail by not planning at all.