SemAngel

Getting it Right: Over and Over

It's hard to get analysis right. Even when you do, it's hard to get it consistently right. Good process is very much about protecting ourselves from the things that cause mistakes so that we have a chance to be consistently correct. In two previous posts, I’ve listed eight different common sources of error in web analytics:

1.    Self-Interested Measurement: Finding what you expect in the data
2.    Lack of Statistical Significance: Believing small variations or tiny samples to have much more significance than they really do.
3.    Unreliable data and what to do about it: Trending bad data or “getting over” critical data quality issues
4.    Siloed Optimization: Improving one channel at the expense of others
5.    Metric Monomania: Over-reacting to changes in individual KPIs
6.    Tactical Focus: Concentrating on micro-analysis and missing the important information.
7.    Self-Selection: Reversing cause-and-effect and thinking related items are necessarily causal.
8.    Navigation Structural Influences: Reversing cause-and-effect and evaluating content performance without factoring in the influence of site structure.

So now it’s finally time to tackle some possible organizational responses to each of these. There is no one single solution that encompasses them. In general, I’ll suggest a specific organizational structure, process or analytic tactic designed to combat the influence of each error factor. Add them up, and I’m hoping that they’ll form a significant part of an overall picture of a good web analytics process. Remember, I’m not suggesting that all process is driven by error theory – so I’m not going to trouble myself to elaborate a complete process design. But I am suggesting that ANY web analytics process design should include these components or risk significant and continued misuse of web measurement.

Self-Interested Measurement
This is such a large problem in all measurement that there are many different aspects of a cure. From an organizational perspective, I think the single most important point is that your web measurement should always be quasi independent from your tactical implementation teams. This goal is complicated by the fact that measurement needs to be close to those teams or it will not understand what to measure and it will not be able to communicate results effectively. So the really tricky goal is integration without co-option. I think this is best accomplished by having measurement report up at a fairly senior level and outside the daily operational responsibilities. In other words, if measurement is being directed and consumed at the VP Marketing level, it shouldn’t report there but up at least one-level. I’d also have measurement report up to two different organizations – perhaps the C-Level Strategy as well as within Marketing. In this way, measurement is never entirely co-opted to the interests of one department. No matter what strategy you adopt, you’ll never avoid the problem of self-interested measurement. On the other hand, if you build in too many checks-and-balances, you’ll never get anything done. It’s a question of balance, obviously. But for many organizations that simply grow measurement underneath the most important consumers, it’s a question of serious imbalance.

There are also some essential analytic processes that an organization can commit to that will help prevent too much self-interested measurement. One of the simplest and most important is to demand that all projects specify specific measurement criteria for success before they are deployed. This is a powerful tool for organizational discipline. And it is critical that it be done before deployment. As I’ve said often, it’s always possible to find some evidence of success when you are allowed to choose the measure of success after the fact. By committing to the criteria for success beforehand, the possibilities for egregious self-interested measurement are greatly reduced.

A third, extremely common manifestation of self-interested measurement is in the surprising practice of allowing vendors to measure their own performance. Whether it is SEO, PPC or web design, the idea that your vendor should be setting the success goals or providing the measurement for their own performance is dreadfully wrong. It isn’t a question of honesty. We are all simply too keen to read the data as we wish to see it. Independent measurement is essential just as is independent software testing. It’s not because developers are out to lie about bugs in their software – they just can’t recognize them. Believe me, this is true for all of us.

Lack of Statistical Significance
In talking about this before, I mentioned that most information users end up considering professional statisticians a royal pain in the ass. But if they are sometimes more like snotty gatekeepers than useful information processors, that’s because there is a genuine and necessary role in keeping bad analysis at bay. Web Analytics departments need not be stocked to the gills with professional statisticians. One is probably enough. But if you have a team generating reporting and analysis on a regular basis, you need at least one gate-keeper reviewing it and quashing the most abusive practices.

In addition, it’s good policy to build measures of significance and variation directly into your reporting. It’s not enough to say that your analytics-team's conclusions are free of gross statistical error if your reports make it all too easy for information consumers to commit those same breaches. One of the benefits to Analytic Reporting – the direction I’ve been preaching for the last year – is that you can build measures of variation and significance into the reports. That’s a huge win since the people most likely to make basic mistakes in statistical analysis (our report consumers) are the ones who have traditionally gotten the least help and training.

Unreliable Data and What to Do About It
Web Analytics data is notoriously unreliable. Unfortunately, this has spawned two diametrically opposed and equally pernicious views. The first and worst is to “get over it” and just use the data. The second is to ignore the data altogether. Neither attitude is reasonable. In a couple of decades doing data analysis, I’ve never worked with a data set that seemed really clean. Data is almost always dirty in some respects. So if we weren’t willing to work with dirty data, we’d never get any analysis done at all. On the other hand, “getting over it” is no solution. And neither is the disastrous advice to “trend the data.”

Combating data quality issues is a never ending process and it takes real work. First, an organization should make sure that the core infrastructure for data collection is reliable and complete. This means careful governance of tagging implementation, good standards for variable encodings, and regular checks of the data quality and collection. If you aren’t doing this, you aren’t doing what you should to protect your analysts. Second, good data quality often benefits from multiple, diverse collection points. Combining panel data and VOC data and analytics data can often highlight places where each method is struggling. Evaluating log data vs. tag data can often pinpoint significant issues with each. Deploying Google Analytics along-side Omniture can help too.

These combined data sources can help guide the analyst – making it easier to understand the types of analysis that can be done with each tool. It’s quite likely that you simply cannot do a new visitor analysis with a web analytics tool. And no amount of trending is going to help. But you can do a functional analysis just fine. And perhaps there are other ways to handle the new visitor analysis.

Just as with the law, ignorance of your data is no excuse for missing data quality problems. The only way to protect against data quality issues is to provide the analyst with enough understanding of how the data is collected and what changes take place so that informed decisions about the data can be made. The necessity for this should influence not just a host of decisions around tag deployment but also your thinking around change-control and reporting.

A great deal of bad or wasted analysis is generated in organizations because no one bothered to tell the analyst that something had changed. Change control is a critical element of good measurement – and your analytics department needs to be tuned into both the technical and marketing sides of the house. From an organizational perspective, this usually means regular meetings and a well-defined flow of information about new pages, new tools, changes in infrastructure, and new campaigns from the originating departments to the measurement team.

I can see this going to get quite long. Well, that’s no surprise. I’ll pick up next week with siloed optimization and go on from there. Half-time at the Super Bowl should be more than long enough to knock out a post!

Even More Ways Web Analytics Goes Wrong

Right before the end of last year, I wandered sideways into the huge and complex issue of web analytics process. I got there inadvertently, first posting some thoughts on why I’m reluctant to look at my blog readership numbers and then, in response to some questions I got, tackling the issue in a more logical fashion. In essence, I proposed that one of the most important factors in designing good web analytics processes is having a theory of error – why things in web analytics go wrong. And I laid out six of the most common causes of error in web analytics:

1.    Self-Interested Measurement
2.    Lack of Statistical Significance
3.    Unreliable data and what to do about it
4.    Siloed Optimization
5.    Metric Monomania
6.    Tactical Focus

I promised, this year, to lay out in more detail how appropriate processes can be designed to protect against these problems. And I fully intend to do that. Just not today.

As I thought about this list, I felt that I had left out at least two significant problems that really should be included.

The two additional causes of error that I had in mind are problems around misinterpreting self-selection and the influence of navigational structure. Both are problems that a good web measurement process really should take note of. Both are bound up with ‘Lack of Statistical Significance’ since they are technical problems in measurement. However, neither is related specifically to statistical significance and the issues around controlling for randomness and variation. I decided to call them out separately because, after thinking about them for a bit, I believe they drive to different aspects of good process.

Self-selection occurs all the time on the web. In fact, it isn’t a problem at all in the context of web design. It’s actually a desirable outcome. We nearly always want people to self-select into appropriate channels. But for measurement, the fact of self-selection makes understanding differences in page or tool performance incredibly challenging.

I talked about the issue of self-selection as it relates to online survey opinion research and satisfaction scores in a recent post. Here’s a short excerpt:

“There is no way to determine from the basic facts:

Comment users have a higher sat score than non-comment users (attitudinal)
Comment users consume more pages than non-comment users (behavioral)

if either relationship is causal. We don’t know if commenting self-selects visitors who happen to be more satisfied and consume more content or whether it actually contributes to that relationship.”

In other words, people who use comment functionality may already be more engaged and have higher satisfaction than those who do not bother. If so, the apparent (and statistically valid) relationship between using comment functionality and satisfaction is non-causal – at least in the direction we are hoping for. Comments are not driving satisfaction, they are being driven by it.

This is an incredibly common source of error. I think it’s fair to say that a simple majority of all uses of VOC (Voice of Customer) data that I see are nothing more than interpretive errors caused by self-selection. And, of course, the problem of self-selection isn’t limited to VOC data. It is rampant in behavioral analysis as well.

Self-selection errors may be rampant in web analytics, but they come from a class of errors that are extremely well understood in broader analytic circles because they happen to be common everywhere. This, at least, tends to make them easy to explain to both analysts and product marketers.

The other problem I wanted to add to my list – the influence of navigational structure – is not unheard of in other forms of analysis but it is less common in marketing analytics and is sometimes more insidious. Like self-selection, navigational structure is a good thing when it comes to web design but a problem when it comes to measurement. In many kinds of web analytics, we are trying to determine the impact of viewing one piece of content (page or tool) on some downstream decision (registration or purchase).

If we find that one piece of content is highly correlated with the downstream behavior, we tend to think it is working well.

Unfortunately for ease of measurement, web sites are not random presentations. They are more like a magician’s card trick where the design team is trying to “force” the user to pick the ace of spades. Because of this, a high-correlation between pieces of content is nearly always indicative of nothing more than navigational structure.

Thanks to the heavy influence of web site structure on behavior, a goodly percentage of web analytics research accomplishes nothing more than an abstract mapping of the navigational structure of the web site!

In a way, navigational structure and self-selection are two-sides of the same coin. In measurement, we are always trying to sort out the real impact of content on behavior. But we need to be constantly aware that our analysis is not based on a controlled experiment. On the one hand, we are looking at the behavior of a population that is far from random and that has strong pre-dispositions to certain behaviors. On the other side, the tool they are using – the web site – was designed to discourage or limit certain behaviors while making others nearly unavoidable.

Unless you are careful (and that’s part of what good process is all about), you can find yourself in a “Heads you lose, tails I win” situation when it comes to web measurement.

So now that I’ve added two more problems to my list, I’m going to tackle what I think are appropriate process responses in my next post(s). It's a daunting task.

So naturally, I'm also thinking ahead. I was thinking of doing a short series on the Omniture Reporting API after the process posts. API stands for application programming interface – and in this case, it’s a toolkit for accessing the SiteCatalyst reports from other software tools. There are a lot of cases where using an API is preferable to using a canned application like SiteCatalyst or even the Excel integration. I thought I’d talk a little bit about when to consider using an API and provide some guidance on using Omniture’s version.

Predictive Analytics – Getting a Leg-up on Where Analytics is Headed

It’s not as if I’m going out of my way to avoid writing about web analytics process, it’s just that things keep coming up. Or maybe it is that way. I’ll confess to a certain amount of…well perhaps fear is too strong a word…but maybe anxiety would fit the bill - about a post on web analytics process. I’m not really a process guy.

Thankfully, I am able to delay another week thanks to Predictive Analytics World (Feb. 18th and 19th in our lovely San Francisco) – a conference devoted, as you’d expect, to predictive analytics in all its many colors.

The conference is new and I think it sounds pretty interesting. From a web analytics community perspective, it has the misfortune to be scheduled at the same time as the Omniture Summit. And from the macroeconomic perspective it has the misfortune to be kicking off in 2009. But I think it may fill a void in a schedule over-crowded with me-too conferences.

A conference exists to help create a community of interest – be it around a vendor’s product (like the Omniture Summit), a shared expertise (X Change), or a shared field (SES or eMetrics). Most conferences tend to gravitate toward a very broad and low common denominator (since they are mostly done for profit and there are always fewer experts than those who are merely interested) – so they tend to cater less and less to the people who are most important to the community. That’s why I pitched X Change as distinctly focused on the experienced practitioner – I wanted to carve out a distinct niche.

It seems to me that Predictive Analytics World (PAW) has a chance to fill a similar niche for analysts and measurement leaders in a cross-industry space who are focused on using BI technology in the predictive space. There’s a lot of overlap (and learning) in understanding how different industries are using analytics – something I’d like to think about for X Change. And it seems to be me that the conference has a chance to bring a group of people together who can really benefit from each other and actually learn something new.

The point-man for the conference is Eric Siegel, and I did quick interview with him to help promote the conference. It’s pretty good stuff. There’s also a survey they are doing to help gather information for the Conference about the use of Predictive Analytics. You can take the survey here.

Here is the interview with Eric…

I’ll start with the obvious – Why a Conference about Predictive Analytics?

Great question.  You're asking about Predictive Analytics World, Feb 18-19 2009 in San Francisco (www.predictiveanalyticsworld.com), the only vendor-neutral conference that covers today's commercial, business-focused deployment of predictive analytics, across industries and across software vendors.

Why is this conference needed?  I'll start by defining predictive analytics and why it's needed. 

Predictive analytics is business intelligence technology that produces a predictive score for each customer or prospect.  Each customer's predictive score informs actions to be taken with that customer, thus automating operational decisions in marketing, web and other business functions.  For example, predictive analytics predicts response in order to target direct marketing, product recommendations and web/email content, predicts defection in order to target retention offers, and predicts debtor risk for credit scoring.  Business intelligence just doesn't get more actionable than this.  For more details on these and other business applications of predictive analytics, see my new article (recently published on BeyeNETWORK): http://www.predictiveanalyticsworld.com/businessapplications.php

Predictive analytics has penetrated across verticals, with wide adoption within some, such as certain financial sectors.  It's a proven method, growing in adoption across industries, so its leaders, professionals and newbees need a place to meet up.  Predictive analytics managers and practitioners will benefit greatly from this event, which brings them together to share lessons-learned and best practices across industries.

Predictive Analytics World's goal is to strengthen the business impact delivered by predictive analytics deployment, and establish new opportunities with predictive analytics.  The conference delivers case studies, expertise and resources to this end.  PAW-09 will have 25 sessions across two tracks, so you can witness how predictive analytics is applied at 3M, Acxiom, Affiliated Computer Services, Charles Schwab, Click Forensics, Google, Linden Lab (Second Life), The National Rifle Association, Netflix, Pinnacol Assurance, Reed Elsevier, San Diego Supercomputer Center, Sun, Telenor, Wells Fargo Credit Card Services, Wells Fargo Internet Services Group -- plus special examples from Anheuser-Busch, Disney, Hewlett-Packard, HSBC, IRS, Pfizer, Social Security Administration and WestWind Foundation.

See the full program at http://www.predictiveanalyticsworld.com/agenda_overview.php

It seems to me there is something of gray area between what I’d call classic analytics and reporting (which is mostly what we end up doing) and predictive analytics. When do you really feel like you’ve crossed over from one to the other?

Above I defined predictive analytics above as BI technology that produces a predictive score for each customer or prospect.  But it's a bit more than that - these predictive scores are assigning by a predictive model which has, in turn, been trained over your data, learning from the experience of your organization.  On the other hand, the model can be quite simple and still perhaps qualify as predictive analytics, such as a "univariate" model that simply takes one customer variable, e.g., number-of-items-purchased, and converts it to the customer's predicted propensity to respond to a certain offer.  Having said this, once you pull your historical data together to discover such a thing, it can only help to take the next step and bring together multiple customer variables to improve predictions.

But you can't stop there, because the whole point is to make use of the predictive scores to drive action.  Once the scores are deployed, integrated into existing systems or processes to automate operational decisions, then by gosh we're definitely doing predictive analytics.

When I first got involved in analytics and database marketing people were all excited about modeling and segmentation and targeted personal marketing. And while it did work exceptionally well for companies in certain industries, it never seemed to penetrate to many others. Today, I still hear about a lot of the same stuff as cutting edge that we were working on fifteen years ago. Do you have similar experiences – and why do you think analytics – especially advanced analytics has proven challenging for many industries to really embed?

Predictive analytics can only succeed with the right ingredients in place:
•    One or more experts in-house or deeply engaged
•    A business case for predictive analytics deployment, such as one of the business applications I listed above (i.e., a way a predictive model can and will be used, rather than just being a nifty model that may not provide business value); management buy-in for the integration and deployment of predictive scores
•    Sufficient data to train a predictive model for the prediction goal at hand
•    General understanding and buy-in of a predictive analytics initiative by stakeholders across business functions
•    Implementation of organizational process best practices.  For analytics, this means CRISP-DM (Cross-Industry Standard Process for Data Mining -- www.crisp-dm.org) or equivalent.  An iterative process that ensures comprehension, feedback and buy-in is attained across a group of relevant managers at key phases of a predictive analytics project
•    When initial deployment success is achieved, sufficient executive buy-in to facilitate long-term maintenance that keeps the deployment alive and effective
Some of these are elusive; if one goes astray, adoption or longevity is not attained.

The good news is that in fact these ingredients usually do exist for mid-tier to large companies – and often for smaller companies, if they have data pertaining to enough customers or prospects.  And, with these ingredients place, predictive analytics delivers high returns – significantly higher than analytics that are not predictive in nature.  An IDC study showed that predictive analytics initiatives show an average ROI of 145%, in comparison to 89% for non-predictive analytics ("Predictive Analytics and ROI: Lessons from IDC's Financial Impact Study," September, 2003).

What makes the difference is a certain level of global understanding,  a great deal of widespread buy-in, key executive buy-in (and perhaps a bit of executive understanding :), and adoption of best practice business processes (on top of killer core analytical methods).  This is where Predictive Analytics World comes in.  This stuff, the basic analytics concepts and business benefits of predictive analytics, needs to be made more widely accessible.  There’s no better way for non-experts to learn what predictive analytics does and how it works - and to become convinced of its effectiveness - than named case studies, which is why PAW’s program is built primary of such success stories, across verticals.  See the full program, at http://www.predictiveanalyticsworld.com/agenda_overview.php

I guess this is a related question, but do you sometimes find yourself surprised at the low-level of analytic sophistication in even very big organizations with very large marketing budgets?

First the subjective answer.  Sure, when I transitioned 9 years ago from an "academic" to the commercial world (read: "recovering academic"), I was in fact surprised.  I had been living in a more abstract world focused on analytically abstract challenges.  From that vantage, the business world's pragmatic challenges at first seemed mundane and you just wanted to kind of dismiss them.  But then it turns out that the pragmatic challenges are just as wild and interesting, just as gratifying to overcome.

More objectively, we can look to survey results to tell us concretely how far along and sophisticated organizations are in their adoption of analytics, such as:
•    The Predictive Analytics World survey on applications of predictive analytics.  This survey is in process, and we'd like the reader's help - take a few minutes, answer a handful of questions, and help us keep you informed: https://www.surveymonkey.com/s.aspx?sm=8dHx_2bFz7yxw3FPKlbi3OVg_3d_3d .  To make sure you don't miss the survey results, sign up for PAW event updates, at http://www.predictiveanalyticsworld.com/notifications.php
•    The Rexer Analytics 2008 Data Miner Survey report, covering the most popular software tools, which verticals have embraced modeling and more.  Available for immediate download at http://www.predictiveanalyticsworld.com/survey-signup.php
•    Davenport and Harris' results in their book, "Competing on Analytics"
•    The Web Analytics Benchmarking Study: http://www.benchmark-analytics.com/d/?q=Web+Analytics+Benchmarking+Study+-+Aug+2008

I noticed that your top keynoters were all Ph.D.’s so it’s no wonder I’m not speaking! I couldn’t handle that much schooling. But I wondered what is the target audience for the conference? How knowledgeable and experienced do you expect/hope/plan for attendees to be?

PAW is meant for a business audience, not only a technical or analytical audience.  Anyone who wants to understand the value of predictive analytics for their organization, learn how to run predictive analytics projects, or see how to advance and expand existing initiatives, will find great value in PAW's program.  If you're new to predictive analytics, we do suggest you read three short articles before attending the conference so that you can be sure to make the best of it.  The articles, as well as training options, can be found here: http://www.predictiveanalyticsworld.com/faq.php#q4
The conference program is designed to speak the language of marketing and business professionals using or planning to use predictive analytics to solve business challenges. Since the best way to catalyze commercial deployment is to show the people it really works outside "the lab", PAW's program is packed primarily with named case studies of commercial deployment. And for the hands-on practitioner or analytical expert focused on commercial deployment who wishes to speak this same language, it's an equally valuable event.
So regarding the higher academic degrees, I would venture to claim that myself and the other two keynoters are delivering great, actionable talks despite our Ph.D.'s, and based mostly on our experiences outside the ivory towers  :) .   PAW’s program of speakers has been optimized for commercial viability, not academic credentials.
If you'd like our informative event updates, sign up at http://www.predictiveanalyticsworld.com/notifications.php
For an overview of predictive analytics, and additional reading and resources, see the Predictive Analytics Guide http://www.predictiveanalyticsworld.com/predictive_analytics.php

Hey I might tease Eric about it, but if I had a Ph.D., I'd never say "in spite of my Ph.D". I don't buy into the idea that you can know too much about your subject and it seems to me that people often discount academic learnings only because they aren't doing anything as remotely interesting or rigorous. This seems like a great opportunity for people to actually get ahead of the curve a little bit and hear about some of the most interesting applications of analytics out there.

If you’re interested in attending the Conference, check out their web site. I believe I can also get you a 15% discount – so if you’d like to get a promotion code for that just drop me a line.

And damn it - there's even some process stuff in Eric's answers. I swear that by next week I will at least begin my process post!

Talk to you then.