It's hard to get analysis right. Even when you do, it's hard to get it consistently right. Good process is very much about protecting ourselves from the things that cause mistakes so that we have a chance to be consistently correct. In two previous posts, I’ve listed eight different common sources of error in web analytics:
1. Self-Interested Measurement: Finding what you expect in the data
2. Lack of Statistical Significance: Believing small variations or tiny samples to have much more significance than they really do.
3. Unreliable data and what to do about it: Trending bad data or “getting over” critical data quality issues
4. Siloed Optimization: Improving one channel at the expense of others
5. Metric Monomania: Over-reacting to changes in individual KPIs
6. Tactical Focus: Concentrating on micro-analysis and missing the important information.
7. Self-Selection: Reversing cause-and-effect and thinking related items are necessarily causal.
8. Navigation Structural Influences: Reversing cause-and-effect and evaluating content performance without factoring in the influence of site structure.
In my last posts, I covered the first three topics and topics 4&5, today I’ll pick up from there.
Tactical Focus
Analytics is a very tactical discipline. Numbers most often suggest very discrete problems and they often get us thinking about small details. That’s actually a good thing in most cases. While you may not be able to see the forest for the trees in most web analytics, at least you can avoid running into tree after tree!
But if a strong tactical focus can be a strength of web analytics, an overweening concentration of small details can often contribute to huge missed opportunities. With web usage now so broad and representative, the web is a powerful vehicle for understanding your customers in every channel. And if a significant portion of your business is offline, failing to use web analytics to obtain a broader understanding of your business is a terrible waste.
Even within the context of the online business, too much tactical focus can often lead to siloed optimization and missed opportunities. Testing hundreds of small variations with a multivariate test tool is great. But if you’re not testing major creative alternatives and fundamentally different messaging strategies, you may only be finding the best variation within a basically unsound approach.
There is no guaranteed way to get the right balance between a healthy concern for details and a broader strategic perspective. There are, however, a couple of approaches that may help.
First, don’t completely silo your online effort – especially when it comes to consumer research. Traditional primary research materials should always be distributed within the online environment and online research should be regularly pushed back into the offline channel world. Making sure you have some cross-organizational meetings to focus on the combined research can greatly facilitate this. It’s also important to cross-pollinate people. Forcing some movement between the online/offline marketing and measurement organizations is extremely valuable. Creating organizational paths that facilitate this from a career perspective is absolutely necessary.
In planning your analytics, it’s also worth making sure that you have at least one effort specifically dedicated to offline learnings. So if you’ve mapped out an analysis plan and everything is about the web site, go back and try again!
If you’re creating an analysis template, always include a slide dedicated to offline/operational learnings. It may end up being discarded a fair amount of the time, but it will keep your analysts thinking about the possibilities.
Lastly, make sure you are disseminating your real learnings and not just bragging about your successes. I’ve written before that one of the biggest learnings for marketing people is what didn’t move the dial. If you are just reporting about how much of a lift your testing or analysis got – and not helping marketers understand what worked and what didn’t, you are missing out on the most important parts of the story.
Self-Selection
Web Analytics is greatly complicated by the fact that most correlations between pages and events are less a matter of content impact than visitor self-selection. Your “Sign-up Now” link is effective because the group of visitors who clicked on it were (mostly) ready to “sign-up now.”
The confusion of correlation with causation is an incredibly common problem with nearly all types of analytics – not just with web analytics. But we certainly manage to make the most of our opportunities for error.
Building good analysis processes is never going to fully prevent this type of error – nothing will – but it sure can reduce the number of your mistakes.
The most important counter-balance to self-selection in analysis is appropriate use of testing – especially controlled testing. There is no better way to eliminate self-selection from an analysis than to run a test that controls for the population in question. In some cases, this can be achieved with rigorous use of segmentation within the analytics tool. But nine times out of ten, the easiest way to ensure that Self-Selection isn’t driving your conclusions is to run a controlled test. You can spend many valuable analyst cycles trying to protect an analysis from self-selection or you can just build a quick test and measure the actual results.Once you’re set up for it, testing is nearly always a lot easier and more reliable.
Similarly, integrating your Voice of Customer(VOC) data into your behavioral stream can help protect against errors of self-selection in each. Many an analyst can avoid basic errors in self-selection by beginning with an understanding of the intent and characteristics of the visitors who are being studied.
Sadly, most organizations have chosen to isolate multivariate testing AND voice of customer research from web analytics. The measurement group should have a significant say in how each is used – and should be able to organize both tests and surveys in conjunction with marketing. What’s more, the capabilities of each of these tools (especially VOC tools which are often delivered in shockingly limited configurations that make their use in analysis almost impossible) should be evaluated from the perspective of the measurement /analytics team not just the marketing team.
Analytics professionals need a fundamentally different set of survey and test & target tool features than do designers. Failure to recognize this and integrate their requirements into the tool acquisition process will almost certainly lead to critical missing capabilities.
Navigational and Structural Influences
Your web site isn’t a pure research vehicle designed to neutrally solicit customer interests and attitudes. It’s designed to influence, motivate and drive the visitor to a specific set of actions. In practice, of course, that’s a good thing. But it means that the web site cannot be treated as a pure research vehicle – it has a powerful effect on what visitors can and choose to do. This simple fact is often missed in analysis.
One of the most important process steps in protecting against structural influences is to have a sound analytics methodology. We use a Functional approach for many basic site analysis projects – and one of the virtues of the Functional approach is that it often helps protect the analyst from errors caused by navigation and site structure.
By grouping pages into types that often capture the navigational distance between outcomes and by focusing the measurement on type-specific metrics, Functionalism greatly reduces two of the most common sources of web analytic error.
Almost as important is making sure that your analysts don’t just look at the numbers. Every analysis should begin with a careful examination of the pages / tool to be studies. Again, Functionalism encourages this step by forcing the analyst to classify pages by their type – a step that greatly improves the analyst’s understanding of each page.
And, as with self-selection, use of controlled testing and integration of VOC data can help the analyst resolve potential questions and illuminate potential problems.
Summing Up
Well, that’s about it. As promised, I’m going to distill my various process recommendations down into something much shorter in my next post. I’ll try to just outline the various process and organizational structures/processes I’ve suggested and group them into categories for easier consumption.
As I’ve said all along, building organizational processes from an error theory is only going to yield part of the necessary machinery for success. There is more to success than avoiding mistakes! But I do believe that taken together, the process suggestions I’ve outlined would greatly improve most organization’s ability to do quality analysis and measurement. Because when you get right down to it, no measurement is better than bad measurement and good measurement is much better than no measurement.
Comments