Military writers love to emphasize that it’s not the glamorous stuff that wins wars. It’s
the dull game of getting the most men and firepower to the battle. Many a
brilliant strategy has miscarried due to poor planning.
We’re not quite down to logistics yet but we aren't far away either. We are at
the point where the resources necessary to execute our digital measurement strategy need to be
fleshed out. With the model of the business and the Data Science roadmap, we’ve
identified what we need to accomplish. It's time to figure out what's needed
to get the job done.
We do this in two steps. The first step is to map out the
data and technology necessary to tackle each project in the Data Science
roadmap.
When doing this, we group these projects back into systems (and their stages and components) to tie
them directly to the models we’ve built. For each system, we map out the data
science projects and the resulting data/technology requirements:
The Campaign Targeting Analysis highlighted above is designed to separate out
campaign performance into its constituent components: targeting effectiveness,
creative effectiveness and offer effectiveness. To accomplish this, we need campaign buy data, intercept survey data on campaign responders and converters, plus Web analytics
data on site behavior by campaign. With this combined data, we can measure targeting precision
and targeting effectiveness (both of which are critical and nearly always entirely
unmeasured in digital) and we can separate out the impact of targeting from
creative and offer. Separating out creative and offer efficiency is more
challenging, but we have some behavioral analytics techniques for proxying each
fairly effectively.
The beauty of this type of analysis is that it creates
optimization strategies that are better than “drop this campaign” or “spend
more on this campaign.” The analysis provides deeper insight into why a
campaign is working or failing and, with that insight, often comes direction
for better optimization regardless of how good the initial performance is.
For a strategic perspective, what’s compelling about the diagram
above is that we create a clear tie between the business model, the data
science projects, and the technology and integrations that are needed. If you want to optimize the targeting of your digital campaigns, this is what you MUST do. If this what you MUST do, these are the technologies you MUST have.
By aggregating up the requirements across every model, we
can create a full technology ecosystem. The diagram below is for the Web system only (there would be separate diagrams for customer analytics, call-center, etc.):
We provide high-level detail on each element of the
ecosystem and we use the Data Science Roadmap to create an explicit
prioritization of the full technology stack.
Obviously, these last two slides are the sort that might be
created in any strategy. Anyone can lay out a Web analytics ecosystem, assess
which tools are currently installed, and recommend some prioritization of the
rest.
It’s the first slide tying the data science project to the
required technology that makes this approach powerful. Remove the Assessment,
the Model and the Data Science Roadmap, and you may have exactly the same set of
technology stack recommendations. What you don’t have is any real
justification/explanation for what you’re asking for. Even if (and I think it’s
unlikely) nobody in your organization cares why you’re spending all this money,
the rest of the plan isn’t just window-dressing. It’s true that one of the main
functions of a Strategic Plan is to justify a budget request (indeed, building
a high-level budget plan is the next step in our process). But just as the
Grinch discovers that Christmas is more than just presents and toys, a
Strategic Plan really should be more than just a way of putting presents under
the tree.
The Data Science Roadmap that lives at the hub of the plan
(the Assessment and the Model drive TO the Roadmap; the Data, Technology and
Budget steps drive FROM the Roadmap) is the guide-map for the entire digital measurement
effort. You can have all the technology and people in the world and still get
nothing useful done. Without leadership, all the firepower in the world will
generally accomplish little. And leadership, more than anything else, is about
knowing which direction to go!
In my last two posts, I’ve described two foundational steps
in building a comprehensive digital measurement strategy. These two steps: an
objective assessment of the state of your digital measurement system and the
creation of high-level model of the digital channel comprise the “here” and the
“there” of a good strategy. The model describes the digital channel and, at a
high-level, what’s required to measure and optimize it. The objective
assessment describes the current state of the system and the tools and
resources available to tackle the missing pieces.
So what’s missing? We still need a plan for getting from
here to there. In the next three steps, I’ll show how the process we’ve created
fills in the route map for getting from here to there – creating a truly
comprehensive strategic plan.
Building a Measurement Framework
The first of these three steps is a combination of our
measurement framework (creating a Two-Tiered Segmentation and corresponding
success framework) along with a fairly detailed data science roadmap. Both are
vitally important.
I’ve written so extensively about the Two-Tiered
Segmentation that I’m reluctant to reprise those discussions here. However, I
do want to touch lightly on why a measurement framework is strategic. In the
model above, some activities (like Prospect Acquisition) are easily described
from a measurement perspective. We may not know some of the details (such as
how to calculate a Predicted Lifetime Value), but we know that we need measures
of cost, reach, conversion, and value. Other activities aren't so
simple.
One aspect of retention, for example, may be Customer
Support. When a visitor visits a digital channel for Customer Support, we need
to know what the apporpriate measure of success is. At a high-level, the measure might be
retention and share-of-wallet growth, but iIt’s unrealistic to expect page or visit
measurement to be able to show statistically significant differences over these
types of variables. You can’t optimize your existing customer support site with
a retention metric.
How you choose to
measure success is going to have a significant impact on your digital
measurement strategy. If the best measurement of success requires a page-level
rating system, then you’ll need to have the technology, people and processes
for using that data. If the best measurement of success is pre-post experience
sampled research, you’ll need a different set of people, process and technology
to handle that. In laying out a strategy, we want to account for these
decisions and provide a framework for prioritizing them appropriately. Our goal
is to tie EVERY single technology and resource decision to a specific set of
digital measurement problems we intend to solve.
In previous posts, I’ve described how we create a two-tiered
segmentation and then elaborate a full measurement system by tying business
goals to customer intent and then creating a strategy for measuring the success
of those business goals for each type of visit.
This is our measurement foundation and it’s a fundamental
piece of a really good measurement strategy.
Creating a Data Science Roadmap
It isn’t, however, a complete map of the route from getting
from “here” to “there” in digital measurement. For that, you also need a Data
Science plan. The Data Science Roadmap is an analytics plan. Its purpose is
to describe the analysis projects that need to be accomplished before the
business can truly understand the digital system. In our Acquisition model
there was at least one obvious example of this: calculating a Predicted LTV:
In many acquisition systems, you simply can’t do an
effective job of optimization unless you have a good model for predicting
Lifetime Value. If you’ve invested hundreds-of-thousands of dollars in creating
an attribution system but haven’t bothered to find out if the campaigns you’re so
carefully crediting source fundamentally different types of customers, you’ve missed
the boat.
So it’s a good bet for this particular system that creating a Lifetime
Value Model would be a core part of the Data Science plan. And just as
decisions about success measurement in the foundation drive decisions about
people, process and technology, so too do decisions about Data Science
projects. To create a predicted LTV model for digital customers requires the
combination of digital campaign data, digital behavioral data, customer cost
and value data, and, often, customer demographics or 3rd Party data.
If you have to do this analysis, you need a platform (not Web analytics), you
need the core data sources integrated, you need the tools necessary to solve
the problem, and you may need 3rd party data enrichment.
Knowing that you have to do the analysis just IS the justification
for all of those things. The model explains why the analysis is necessary. The
Data Science plan describes the data, techniques and resources you’ll need to
accomplish the analysis.
A Predicted LTV Model isn’t the only type of analytics
project that might flow out of an analytics system like this.
We often, for example, recommend a technique we call
variation analysis. It’s designed to study and isolate sources of variability
in a Pay-Per-Click (PPC) Program. By studying the relationship between spend,
traffic, audience, and conversion over time, this analysis seeks to identify
the causes of variation in a PPC system. Why is that important?
If you know what drives variation in a system, you almost
always have significant opportunities for program optimization. In one of my
favorite examples of this, we found that for an online Traffic Site, weather
systems were a source of dramatic variation. Big surprise, right? Super-storms,
blizzards, and general bad weather drive enormously greater interest and use of
online traffic. So if you're an online traffic site, tying your buying strategies to regional weather events can significantly improve the use of your budgets both intra and inter-day. It’s
obvious that weather impacts usage of online traffic – but it also impacted
quality of visitor and potential acquisition opportunities. Optimizing buying
for variations takes more work (and it’s something the average agency might
not bother with), but it’s a critical factor to account for in creating your
digital measurement strategy.
We’re not done yet. The pass-off from Website
to Call-Center introduces an obvious and all too common gap in the measurement
system. You can’t measure true conversion of campaigns unless you can track
from Web to Call. And if you can’t measure, you can’t optimize.
Adding this level of measurement doesn’t just fill a basic
gap in the measurement system, it also enables several potentially valuable
Data Science projects.
By tuning the placement and highlighting of phone numbers,
you have a significant amount of influence on how many visitors will pick-up
the phone and call. Finding the optimal balance between web and call-center is
tricky. It often requires a combination of controlled testing and regression
analysis to identify the best campaign and page-level strategy for providing
phone drives. It’s not dissimilar, in many respects, to the merchandising
analytics methods I described in a previous post and whitepaper.
For this data science project, you’ll need online campaign
information, the intercept survey (audience) data, Web analytics data, and
call-center data. Ideally, you’ll need visitor-level integration of this data
to be able track patterns of conversion.
This, in many ways, is a classic site-side analysis and
it’s the type of thing that’s bread-and-butter for Semphonic. It’s a mistake,
however, to only think about site-side analytics.
In any lead-generation system, one of the most important
analytic projects you can do is to create a model for matching leads to
operators.
In some systems, this is largely about classifying the leads
by quality and optimizing for performance. However, that’s not always the best
way to think about the problem. Most operators are better with certain types of
callers. I’ll always remember a time many years back when I was working for a
company writing real-time trading software. We’d hired a German intern and when
a company from Germany called and he answered the phone, we got one of our
earliest sales. That’s pure chance, of course, but most operators will do
better with certain demographics and certain regions. By matching lead types to
operator skills, you can significantly improve call-center conversion. This
call-matching is a fourth Data Science project I’d recommend for this
Acquisition system.
Bringing it all Together
These four projects might constitute the Data Science Roadmap for the Acquisition system. Keep in mind that there are five different
systems in this business (which is about the minimum). So it’s not unusual for
the Data Science Roadmap to contain 20-30 different analysis projects.
These projects are a REAL plan. They constitute the work of
many months and, usually, multiple years. They are also the driving force
behind the selection and prioritization of technology, the resourcing requests,
and the data integration plan that flesh out a good digital strategy.
What’s particularly compelling about this approach is that
the Data Science Roadmap provides a clear connection between what you are asking for and what you
are going to deliver. If you get the people and the technology you’ve asked for
in an area, these are the business questions you are stepping up to answer.
In my next post, I’ll show how we take the Data Science Roadmap and translate it into a Technology Stack and then a Resourcing Plan. With those
steps we’ll almost (but not quite) have a complete digital measurement
strategy.
End Note: The Road (much) Less Traveled
Webinarmaggedon is over but I haven’t gone completely
quiet. I’m doing a more personal presentation; it’s most definitely not a Web analytics presentation – this is a “softer” lecture
series internal to single large enterprise, in which I’m just one of many participants. I ended up creating an offbeat
presentation that covers a blend of modern analytic philosophy, my own take on how
ethics and modern neuroscience might more fruitfully meet, and some reflection
on how this might actually apply to life-thinking. I’m not sure if the deck on
its own is “scrutable”, but if you’re interested drop me a line. I’m happy to
pass it on.
The Assessment Framework I outlined in my last post is only
the first step in creating a really comprehensive digital measurement strategy.
If a strategy is a plan for getting from here to there, an objective assessment
of your current state is all about figuring out where “here” is right now.
That’s a vitally important step in the process. But the next step is perhaps
even more critical: creating a high-level model of how the digital business
works. This model is, in fact, the central step in creating a real strategy and
not simply delivering a pile of industry standard best-practices.
At the tactical level, you’ll hear digital measurement folks
moan constantly about how business goals drive measurement and how essential it
is that the business know what they want to accomplish before they can hope for
effective measurement. I moan about it myself. There is probably nothing more
frustrating that being asked to measure something for which no purpose can be
ascribed!
The exact same truth holds at the strategic level. Knowing
where you are is great - that's the "here" in going from "here to there". But what's the “there”? It IS NOT the people,
process and technology required to do effective digital measurement. That’s
resources and logistics, not strategy. “There” is what you need to know to run
every aspect of your digital channel effectively and optimize it on an ongoing
basis.
How do you describe what you need to know to run
your digital channel effectively? Build a model. This isn't a predictive model or a statistical model - it's just a simple, high-level conceptual model of how the digital channel works.
The modeling starts at a very high-level with the core
functions of the digital channel:
Within each core function, you need to map out the
high-level process that takes place. I'm going to show how we might expand one piece of this by walking through an expansion of the Digital Advertising function:
In this particular case, Digital Advertising is focused on Google
Adwords, which drives visitors to the site. Some visitors convert leads off the
site. Others move from the site to the call-center and convert there.
This is an
example of a very simple but not uncommon digital advertising ecosystem.
With the basic flows in hand, the next step in building the
model is to figure out what you MUST
know at each step to understand how the business is functioning.
For PPC advertising, you start with what you are buying and
how much you are spending on each keyword:
The next step is to understand how much traffic that keyword
spend level generated and who that traffic consisted of:
Knowing how much traffic you generated seems (and is)
trivial. Audience mix, on the other hand, is frequently ignored. To
successfully measure and optimize campaigns you have to understand whether the
targeting was successful and whether the offer/creative treatments were working.
When a campaign succeeds or fails it can be hard to separate out the causal
impact of these factors. To isolate targeting effectiveness, we think it’s
critically important to understand the Audience Mix by source.
The next step in the Digital Advertising Process is the
Website:
To measure campaign effectiveness you obviously need to
measure Website Conversion. But if, like many businesses, you have significant
Call-Center conversion then it’s absolutely imperative that you also measure
the % of traffic sourced by campaigns that go from the site to the Call-Center and the
subsequent Call-Center conversion on those campaigns. If you don’t do this, you
simply cannot optimize your system.
For this particular business, not every application was
accepted. So matching acceptance rates to campaigns is essential. While most campaigns
tend to have similar acceptance rates, it’s perfectly possible – even likely –
that a few campaigns will vary significantly from average and be either more or
less effective than conversion rates would suggest.
Finally, there was a significant retention component to this business. From a digital
advertising perspective, what’s important isn’t measuring retention (that's a separate function that needs it's own model) but
predicting quality of customer. As with acceptance rates, different campaigns
are highly likely to source different types (and hence quality) of customer. If
your business has any ongoing customer component, and most do, optimizing for
Predicted LTV is essential to effective digital advertising.
This is the high-level model for one aspect of the Digital
Channel. And using it, it’s easy to highlight exactly where the gaps and needs
in a measurement program are:
It should be obvious how a model like this (expanded across all the core functions of your digital channel) is essential to
creating a good strategy. By describing the system and the ideal measurement
state, the model provides a complete map of where you want to get to. And by
highlighting gaps in the measurement system, it provides a map
of the ground that needs to be covered.
I have no doubt that most analysts inside a company carry a
complete (and probably much more detailed) model like this around in their
heads. But by making the model explicit and embedding it in the strategy, you
have a tool for making measurement requirements explicit and understandable.That's critically important. Because just as I have no doubt that most analysts carry a model like this around in their heads, I also have not doubt that the majority of enterprises we engage with have critical missing pieces like the ones I've described above that no one is addressing!
Why do you need online survey tools? To measure Audience Mix
so you can understand and optimize targeting effectiveness of campaigns. Why
do you need dynamic 800% numbers? So you can understand Web to Call-Center routing and
optimize campaigns by conversion rate. Why do you need data enrichment on
applicant emails? So you can build a better LTV model to optimize digital
campaigns by quality of customer generated.
The model not only provides transparency about why specific
technologies and analytics projects are necessary. It also provides direction (Strategy!!) about how to deploy your technologies and people within
the measurement system.
I hope from this it’s clear why there is no more
important or central step in creating a digital measurement strategy than this
modeling exercise. It is not quite the “whole of the moon”, but it is the
single element without which no part of the strategy can really be deeply
understood.
Recent Comments