The Two-Tiered Segmentation has become a fundamental technique for us at Semphonic. It forms the basis of our approach to everything from KPIs to Report Set Design to Use-Case Analysis to Data Warehouse Modeling. It’s probably the single most important technique in our toolkit.
What’s surprising is that our approach to the Two-Tiered Segmentation didn’t come directly out of Database Marketing efforts though it’s now a fundamental part of those efforts. It was actually part of an effort to significantly improve our Management Reporting Products and our frustration with the current industry “best-practice” around KPIs (for more about how our industry has misinterpreted KPIs check out my colleague Phil Kemelor’s recent post).
I’ve been writing for years that our industry’s approach to reporting was deeply flawed. The idea of reducing Management Reporting to a small set of actionable KPIs is both misguided and harmful. I’ve shown before – in terms that I think are all but indisputable – that the idea of an actionable KPI is wrong. In practice, however, it isn’t just the concept of “actionability” that’s in question; it’s the whole idea of a site-wide KPI.
Think I’m wrong?
Consider the following statements that you might commonly hear in a discussion of Web analytics:
- Our Traffic (Visits) is up
- Our Conversion Rate is down!
- Our Online Revenue is up!
- Our NetPromoter Score is unchanged.
Each of these statements includes a typical and obviously interesting KPI: visits, conversion rate, revenue and customer willingness to recommend. If metrics like these aren’t site-wide and actionable, then what would be?
Each statement has an obvious interpretation: Traffic is up seems good; conversion rate down is certainly bad; online revenue being up is clearly good, and NetPromoter Score unchanged is obviously neutral.
Interpretation is a snap…
Or is it? What if all four happen to be true?
It’s certainly possible that for a single site, all four of these statements could be true at the same time; in which case we’d have two positive indicators, one negative indicator and one neutral indicator. A muddle. This underscores what should be an obvious point: it’s almost impossible for a single KPI to capture enough information to correctly understand a system. The goal of reporting is understanding, not action, and there is no single guideline for how complex the metrics needed to understand a system actually are.
I picked these specific examples because in each case they’ve presented me with real-world problems in interpretation where different people actually put forward radically different interpretations of the data. Consider the following explanations of each statement:
Statement: Our traffic is up!
Explanation #1: We have more visitors with customer support issues because our last release sucked.
Explanation #2: Our SEO has improved.
Statement: Our Conversion Rate is down!
Explanation #1: Our new site design is broken.
Explanation #2: Our SEO is improved so we’re getting more visitors who are less qualified.
Statement: Our online revenue is up!
Explanation #1: We’ve improved our order process.
Explanation #2: Offline customers are shifting online but they spend less with us than they used to.
Statement: Our NetPromoter score is unchanged!
Explanation #1: We aren’t moving the needle on customer satisfaction and have to add deeper experience to our site.
Explanation #2: Our satisfaction with existing customers is up but our marketing has added more prospects that tend to have a lower score – the two trends pretty much balance out.
In each case, both explanations are perfectly plausible. It’s possible that one of the explanations is right and the other is wrong. It’s also possible that both explanations are correct and account for some part of the observed measurement. Or, of course, both explanations may be completely false and another, altogether different, explanation is correct.
I can, off the top of my head, list 15 or 20 different reasons why Conversion Rate might fall for a typical site or why Traffic might go up. Only a few have anything to do with real site performance. If I worked at it, I’m confident that an endless series of explanations could be manufactured. A KPI by itself does not and cannot select between those explanations. That’s why a KPI, taken in isolation, is neither actionable nor explanatory.
I’ve discussed this problem before and while I think the critique is damning, I used to end this discussion by saying that an analyst’s job is to create – in as few metrics as possible – an accurate description of the real-world. There is no one right answer for how many metrics or how complicated that description needs to be because the right answer is different for every system you’re trying to model.
I still believe that’s true, but it’s not exactly action guiding. Yes, it frees an analyst from the harmful tyranny of “actionable KPIs” but it hardly helps the analyst in deciding how to actually build a report.
Our Two-Tiered Segmentation scheme is a much more explicit solution to the problem.
In every case, as you think about the KPI statements presented above, a segmentation is an excellent first step for choosing the right explanation. If traffic for a site is rising, the first question I’d ask is “With whom?” Is traffic rising for existing customers, for online only customers, or just prospects?
If revenue is going up, I’d want to know exactly the same thing. Is revenue going up with all customers, or is it going up with customers who purchase high-end merchandise or is it up because we’re getting more customers?
Ditto with NetPromoter scores. An average hides all the interesting stuff and often moves very slowly because of the amount of noise in a large population. Is my Customer Sat. really unchanged or are there shifts in segments or micro-segments?
In the real world, when I present a metric to a client, the first question I want them to ask - the first question I’d ask - is “Who does this number apply to?”
The “Who” is the first tier in a two-tiered segmentation. It’s a classic database marketing style segmentation and it’s essential context to understand almost any metric. It doesn’t matter whether the metric you’re reporting on is Page Views, Visits, Conversion Rate, or Revenue – if “Who” is the first question you should ask when you see a change in the metric, shouldn’t the answer be baked into the reporting?
We discourage our clients from even looking at site-wide metrics for things like Traffic and Conversion Rate. They just don’t mean anything until they’re placed into a more specific context.
This first segmentation is powerful and it’s necessary. In the Digital world, however, it’s not enough on its own to create meaningful KPIs or serve as a framework for a data model in the warehouse.
For that, you need the second tier of the segmentation.
Let’s go back to our examples to see why. Suppose traffic is going up and I find that the increase is due to more customer traffic. That’s interesting, but it’s still not enough information to create real understanding. The question I want a decision-maker to ask when I present a fact like “Customer Traffic is up” is “Why? What are they trying to do?”
Are customers coming to the site to buy more, or because our latest product release is generating a dramatic spike in customer support visits?
It makes a huge difference, right?
This 2nd level of segmentation – Visit Intent – is actually the most important segmentation in Web analytics. It forms the 2nd tier of our segmentation scheme and it is essential.
It is in the intersection of the Who and the Why (Visitor Type and Visit Type) that metrics and KPIs should be placed. Every combination of Visitor type and Visit Type should have its own distinct KPIs and descriptive metrics.
If you tell me that Customer Visits for Installation Support are up, that’s starting to be meaningful. If you tell me that Revenue and Conversion Rate are up with Prospects coming to the Website with an “Intent to Buy”, that’s starting to be meaningful. If you tell me that time on site is down for Customers coming to find out a telephone number to call for information, that’s starting to be meaningful. Metrics have meaning (and the appropriate metrics can only be chosen) in the context of a segmentation – and that segmentation should have at least two tiers.
Putting metrics and KPIs in the context of a two-tiered segment doesn’t suddenly make them actionable and it doesn’t guarantee that they tell the full story. It does, however, advance the story dramatically. Metrics placed in this context will always be vastly superior to those provided on a site-wide basis. They will be cleaner, more accurate, and far, far more meaningful.
As I've said, this Two-Tiered Segmentation turns out to be essential to nearly every significant Web analytics task: from KPI development to Report Set Design to Deep-Dive Analysis to Data Warehouse Model.
We find that when most clients start with us, they are working with KPI designs that are all SITE-WIDE – often created at considerable expense – and that lack ANY concept of segmentation much less a Two-Tiered Segmentation. It’s unbelievable.
For reporting, we often have to fight our clients to convince them that the site-wide metrics for Traffic and Conversion Rate that they’ve stuck front-and-center on their Executive Scorecard aren’t useful – are, in fact, worse than useless since they encourage poor interpretations of the data. We have to struggle against a set of “best-practices” that have been deeply entrenched in our industry despite their evident flaws.
Management Reporting based on a Two-Tiered Segmentation is completely different than what people are used to. The most important KPIs emerge out of the segmentation – the simple counts of how many of each Visitor/Visit Type the site actually receives. These simple counts are often the most important KPI for any site and I venture to suggest they are not even in the conversation for most expert report designers and KPI authorities.
Metrics like Time-on-site, Visits, Revenue and Conversion Rate shuold be used selectively because they fit the Segment and Visit-Type not because they have some inherent value. Success metrics become Visitor and Visit specific as, surely, they should be.
Put in the framework of a Two-Tiered Segmentation, reporting suddenly makes sense – it becomes elegant even beautiful. It isn’t a matter of sparklines or striking colors or fancy ratios; you can pimp a bad report, but it's always going to be the equivalent of putting lipstick on a pig.
As to the Data Warehouse, the creation of Visit-Type as a form of meta-data turns out to be the single most important “fact” you can add to the data in the warehouse to simultaneously aggregate it and preserve its meaning.
In my last post in this series, I explained why a good measurement implementation demands Analytics and DB Marketing knowledge. In a way, this post explains the same about a good KPI and Management Report Set. The idea that Segmentation is fundamental to KPIs or Management Reporting is often given lip-service (if that) by the punditry who don't really practice, and the contents of that segmentation are left largely mysterious or defined in the broadest terms. It turns out that the necessary segmentation is too fundamental and distinctly digital for such vague generalities to suffice.
This has been a long, long post, but I’ve barely scratched the surface of the power and importance of this technique. In my next posts, I’m going to show real world examples of Two-Tiered segmentations for Financial Services, Travel, Media, and Public Sector sites. I’ll show how the segmentation scheme fits together and the ways it can be adapted to cover almost any industry. In subsequent posts, I’ll show some of the techniques for building Visit-Type Segmentations (which are extremely challenging). Then I'll tackle how it can be used to transform KPI development, Management Reporting, and, finally, how it forms the basis of a really good data model for the warehouse!
[If you're interested in hearing more, I have a webinar this Wednesday with IBM in which I'll be discussing some of these concepts including the Two-Tiered Segmentation. You can register here.]