Operationalizing Analytics

After a brief detour into Marketing Frameworks (and movies), I wanted to return to the series of posts focused on the principles behind creating a successful Analytics Center of Excellence. In the initial blog in this series, I laid out a set of principles for standing up a truly differentiating analytics capability:

1. Business driven not exploratory analytics
2. Investing in analytics as a portfolio
3. Agile methods for analytics
4. Creating operational strategies for every analysis
5. Coupling analytics and reporting to drive data democratization
6. Making collaboration a core capability not an add-on
7. Creating virtuous cycles inside analytics

The first three principles are largely concerned with identifying and creating analytics. That’s obviously essential. But creating analysis is far from the only thing you need to worry about when building a Center of Excellence. It’s true that most organizations don’t create much real analysis. But even when analytics is getting done, it often has far less impact than it ought to. So while the absence of analytics is the most common enterprise measurement problem, the inability to effectively operationalize analysis is by far the most common problem once you start doing real analytics.

There are quite a few reasons why organizations struggle to operationalize analytics:

• Exploratory analytics
• Technology Gaps
• Structural Issues
• Collaboration Issues
• Political Problems

I’ve already laid out my case against exploratory analytics. I’m highly skeptical that turning analysts loose on a data set without specific business problems to tackle is useful – mostly because it rarely generates insight. But even where exploration does generate insight, the chances are high that the analysis will never be operationalized. Given an exploratory approach, you have no opportunity to lay the groundwork for operationalizing the results or for choosing analysis projects that have an easier path to fruition. More often than not, this inability to select for and support operationalization will make even a successful analysis sterile.

I hate to blame problems on technology; it often seems like a cop-out. But there are times when you simply can't execute against problems because of the technology stack in place. For example, I’m a huge advocate of the potential for personalization in the digital space. Digital is fundamentally a direct response medium. The more personalized you can make the digital channel, the more likely you are to succeed. Unfortunately, and this is the biggest reason why so little personalization happens on the Web, there are serious technology channels to driving personalization. If you don’t have a testing tool, if your CMS cant’ support targeting, if you don’t have a digital data feed, a place to store customer profiles, and the ability to generate real-time decisions, you lack the fundamental infrastructure to drive personalization. Lacking this infrastructure, it does you no good to build a segmentation analysis for targeting.  It doesn’t matter that a segmentation analysis for targeting is a huge potential win. It doesn’t matter if it’s a great analysis. Without the technology investment, it’s useless. One of the benefits to the portfolio approach to analytics that I've suggested is that it helps you identify places where fruitful analytics opportunities exist but are operationally constrained by technology gaps.

If technology gaps are real but less common than we like to think, structural problems are rather more common than we'd generally like to admit. I was in a conversation on Thursday (in the frigid tundra of Arkansas) where one of the main topics was organizational structure. In the course of the conversation, one of the EY partners made the point that there is NO ONE RIGHT STRUCTURE. Every organizational structure has advantages and disadvantages. That’s probably why large enterprises seem constantly to be in an endless cycle of centralization and de-centralization. But while there may not be a perfect structure, there are problems and challenges unique to any given structure and these can often present daunting or fatal impediments to operationalizing analytics. Common structural barriers include lack of ownership, lack of control, resource allocation and budgeting.

Ownership issues show up in myriad of places. A common structural issue is the tendency for enterprise’s to split their digital marketing team and their digital Web team. The marketing team generates traffic, the Web team generates engagement and conversion. Alas, it’s nearly impossible to split these two in a sensible fashion since the effectiveness of each is deeply interconnected. Optimizing to quality of traffic is very different than optimizing to volume of traffic. But the only really good measure of quality of traffic is conversion. By splitting these two functions, organizations often make it impossible for analysis generated by either group to be operationalized.

Even more frustrating from and analyst's perspective is a situation where there’s a clear analytic opportunity but the organization has ceded control over the function to a vendor/agency. We’ve sometimes discovered startlingly ineffective and ill-conceived Display and PPC programs being run by agencies where the relationship is too high-up to do anything about it. Sometimes, we’ve even been unable to run that type of analysis because the agency has no incentive to hand over any data to the enterprise that owns it. As maddening as that is (and it's crazy maddening), it's probably better not to the get the data because it spares you from spending any cycles on what will surely be a useless analysis.

At a deeper level, there are times when the way an organization has structured its business units and budget process can make operationalizing certain types of processes nearly impossible. I worked with a company once that had built a great customer database capability around their rewards program but had  no customer data at all for those not in the rewards program. There were huge opportunities around targeting and messaging to non-rewards customers, but there was no owner and no way for anyone to get budget to tackle the problem.

While structural challenges are ubiquitous, they are also fairly well understood. Most people in an enterprise quickly come to understand the limits, barriers, and problems inherent in the organization. In fact, so powerful are these barriers that we tend to accept them even when we shouldn’t. On the other hand, challenges around collaboration are less recognized even though they are often blindingly obvious. In many of our CPG and Life-Science clients, analytics is heavily brand oriented. That makes sense, since each brand is often quite distinct and analytics and optimization require very specific knowledge of the brand. On other hand, most of these companies have done very little to facilitate any kind of sharing between brands. An analyst in Brand X will often have zero visibility into the learnings of an analyst in Brand Y. An analysis of re-marketing technique in Display might have a significant impact inside a Brand, but be completely unknown to the 40 or 100 other brands currently running Display campaigns. This inability to collaborate and share analytics results has an interesting side-effect – it limits the investment in analytics. If you have 40 different units running 100K Display campaigns, you’re spending 4 million on Display. But for a $100K program, spending $40K to analyze the techniques being used isn’t worthwhile. If that $40K yielded a 10% incremental improvement, it would only be worth $10K to the Brand. That's not a very good return. Overlooked in this is that it would yield $400K for the whole organization if analytics were consistently shared.

I have less to say about political problems. On the one hand, politics is everywhere in the large enterprise. It just is, and there is no avoiding it. Like technology gaps, though, I sometimes think blaming politics is a convenient excuse for doing nothing. Truly, when you choose a project, there’s no point picking something that will either support the current orthodoxy or be squashed by it - but there should be no shortage of reasonable projects in most enterprise environments.

This probably seems like a fairly intimidating laundry list of what can go wrong when it comes to operationalizing analytics, but I’m sure I’ve neglected or missed numerous other possibilities. When it comes to operationalizing analytics, there’s lots of dumb ways to die!

Fortunately, there are lots of ways to improve your odds as well. It's a good idea to start by evaluating the operational challenges when you first prioritize your analytics projects. Building the portfolio of analytics projects provides a natural step where you can decide whether you’re actually well-positioned to use the fruits of an analysis. Not only will this greatly improve your hit rate, but by forcing you to explicitly think through operational issues, it will lay the ground work for successful deployment of your project.

Whenever you start on an analytics project, you should have both a clear path and an explicit plan to operationalization. Not only will this help prevent the single most common cause of failed analytics projects, it will help drive your analytics decision-making. There's nothing like an operational imperative to crisp up your thinking about what an analytics project should deliver.

What’s more, each of the three topics I’m about to discuss – coupling analytics to reporting, focusing on collaboration, and creating virtuous cycles are all deeply concerned with supporting operationalization of analytics.

Related articles

Creating a Roadmap for Data Science

Comments

[Judah]

Gary,

Thought provoking blog as always.... :-)

I wanted to note that "exploratory analysis," which you define here as "without specific business requirements," is NOT the same as "Exploratory Data Analysis" or EDA. EDA was coined by Tukey in his book of the same name. Essentially, EDA always has context (a question), and the idea of "exploring the data" is basically to visualize to understand the relationships exposed in order to frame a model. In other words, fit the model to the data, not the data to the model. As you know I wrote about EDA fairly elaborately and for the first time in the context of digital analytics in both of my books. ;-)

To your point, I've always called analysis "without specific business requirements" to be "observational analysis" so as not to confuse it with EDA. On the idea of "observational analysis," I tend to agree with your statements. It is unlikely that any stakeholder is going to care about some observed relationship in the data unless tied back to what I always say: 1) generate profitable revenue (and thus growth) or 2) reduce cost (which can boost profitability).

If an observational analysis is done to that/those goal(s), then it inherently will always have context within a business requirement (i.e. to create value). But the risk can be that the analysis does not meet expectations or is deemed to have resulted in conclusions that are a lower priority (forest vs trees). Thus, I think "observational analysis" is inappropriate at worst and unnecessary at best unless a stakeholder has asked specifically for an opinion and some ideas based on observations you think are interesting (which happens...).

Judah

[Gerhard Klassen]

Hello Gary,
thank you very much for this post.

I agree that its important to prioritize the analytics project. We prioritize by Revenue impact, complexity of implementation and estimated time.

But sadly I see a lot of analytics projects with high potential for example in increasing revenue which end up in the drawer. As you said these were structural problems and political problems.

In this case its a huge problem for the analyst. He can not improve the website. So what is left is more implementation work and less improvement work. Which lowers the value of the analyst in my eyes.

As you said its important to discuss with the stakeholders in a project the following point:
you should have both a clear path and an explicit plan to operationalization.

Thanks!
Gerhard Klassen

[Peter14]

I got your point that we should really focus on a result-oriented or business driven analytic where we just don't explore things but execute decisions in order to really attain the best possible result. It's one good way in order to perfectly do the things needed to be done.