SemAngel

Ernst & Young Acquires Semphonic – Details, Details, Details

It’s been so nice hearing from people throughout the community with kind words and congratulations regarding the acquisition. Makes for a very nice weekend (if only Duke – my alma mater – had won in the 2As)!

In my last post, I laid out some of the background to the acquisition, why we went in this direction, and why I think it’s important not just for us but for the entire industry. In this post, I just wanted to lay out some of the immediate implications – especially for folks I think of as “Friends of Semphonic”.

So here, in no particular order, are some of the details:

 

X Change

For those who have been around or followed Semphonic for a while, and particularly for the body of X Change loyalists, one of the first questions I get about the acquisition is “What about X Change?”

Don’t worry. X Change Berlin and X Change Laguna Niguel are on, intact, and unchanged. Let me underline the unchanged part. X Change has obviously been a big part of the Semphonic brand and it’s very unusual format and feel have always been representative of how we like to approach things. Fortunately (and I think this is one of the things that makes me believe we’ve found a great home), the E&Y folks feel very much the same way. I will admit that the craziness attendant on an acquisition have put me a bit behind on X Change here in the U.S. – but I plan to rectify that in the next couple of weeks. Berlin, meanwhile, is humming along. Matthias will continue to be the Conference point person. We’ve got our Huddle Leaders, our Keynote (which should be terrific) and we’re already more than half full with months to go. So if you’re thinking about registering, do it!

As for X Change here in the U.S., not only is it going forward, but we're even considering taking the concept and building it out in some other analytics practice areas. Maybe someday we’ll have multiple analytics X Changes and even an opportunity to cross-pollinate a bit and share some time with colleagues in other disciplines.

Longer term, the story around Europe might be a little different. Our unit is going to be North American focused. I’m hoping to have the European analytics practices of E&Y come out and check-out the Berlin Conference this year – we’ll see how it goes.

 

Blogging

Lots of people have also asked about the blog. I understand why. I’ve seen other acquisitions and it often seems like people go dark – especially when they become part of a big organization. I really don’t want that to happen. This blog is one of the many enjoyable aspects of my work – but that probably understates it a bit. Like X Change, it’s one of my favorite things to do. Here too, the news is good. I fully expect to be blogging going forward. So if my productivity flags, it will be on my head alone. I can’t promise to always blog – sometimes the press of work is just too much - but I’m going to do my best to continue the once-a-week pattern with very much the same type of output. I may be moving the blog over to an E&Y location (something I had long planned to do with Semphonic) but that isn’t all worked out yet.

 

Our People

As I mentioned in my first post, virtually the entire Semphonic consulting team is coming over. So for our clients the disruption should be almost nil.

Because we’re U.S. focused, Matthias Bettag will be spinning out but will continue to work with us on X Change. That hurts – I know we’re both disappointed. He has been a pleasure to work with in every respect. I count Matthias not just as a great colleague and analyst but a friend.

For those who know Semphonic as something more than a brand and from our public persona, it’s also the case that there’s a much bigger change that I can't help but touch on. Joel Hadary, co-founder and partner for the last fifteen years is going to retire. Joel has run every aspect of Semphonic’s business for the last umpteenth years – from HR to taxes to IT to contracts to billing to finance. At every step along the way, he’s been the guy keeping Semphonic whole, running smoothly and working well. He’s freed me up to work almost exclusively on the things I really enjoy. I suspect that all my readers (who are all people who do or manage analytics after all) know how much that means.

He’s also a friend who I’ve shared decisions with for so long I can hardly remember a time when it wasn’t that way. Over all those years, we had plenty of disagreements but we never had the least doubt that we could talk them through and jointly decide what to do. It’s been a remarkable partnership and one of the best business experiences of my life. We’ve trusted each other absolutely and never had the slightest reason to regret it. It’s fair to say that we went down this path of seeking an acquisition at least partly to make it possible for Joel to retire (in style). Joel’s kids are all grown up, he’s got grand-kids, he’s got a lovely, lovely place in Sonoma with olive and apple trees, and he wants to be able to relax a little. I get it. Give me ten years, and maybe I’ll be there too. For now, though, you’d have to drag the laptop from my cold, dead hands to get me out of the business!

I’ll miss him.

 

The Brand

Hey, I love Semphonic, but the brand is going away and while I will admit to feeling more than a bit sad about that, I don’t think any other decision would  be even plausible. Let’s face it, there are better names out there. Even most of our competitors had cooler names, which always bugged me a bit. The story behind the name is a bit odd; and, annoyingly, I have to spell it constantly. Even our client’s struggle with it. And, of course, outside of the digital analytics community Semphonic might as well be "Ziblikgrhven" for all anyone’s ever heard of us. I do hope to continue to build the digital analytics brand for E&Y. While I know that leaving the name behind will be hard for all of us old-line Semphonic folks, a name is what you make of it. The people and work are what drive brand, we plan to continue to excel at both.

 

The Personal Stuff

Readers of my blog know that I rarely dip into the personal side of things. This space is meant and will likely always be intended as place for professional reflections. But I write so much that at least a little bit of the personal will always leak through and with such a profound change, I think it’s fair to dip briefly into the personal side of things.

As I more than hinted at above, I don’t plan to go anywhere else. Every acquisition carries with it risks. So does stasis. Semphonic as a thing I owned has been a huge part of my life – and the very best, richest part of my working life. I’ve come to love my work rather more than I ever expected to - starting out as someone who never had much career direction and once swore I’d never have a 9-5 job (at least that part’s “literally” true). It’s gratifying to have brought it to this level and to have a clear milemarker of success. But to put this in basketball terms, I’m not one of those teams ready to measure success by making the Sweet 16.

There will be changes of course. Big and small. Everything from having a boss, to giving up my old jeans in the office routine, to commuting into downtown San Francisco instead of Novato, to having people know the name of the company I work for. Does it scare me? Sure it does. It’s the biggest jolt in my professional life since the dot.com bust, maybe even since before that.

There were probably safer choices in the acquisition space; choices that would have involved less change and less risk for me and for all of us. But this felt like the right choice given where I'd like the business to go. Six years ago (or thereabouts), Joel and I made a conscious decision to change our practice from a personal consultancy to a consulting business. To start talking and writing about our work. To engage with the broader community and to try and grow a Web analytics practice that mattered. It worked pretty darn well.

I believe this is the next step in that journey.

Semphonic Acquired by Ernst & Young

Normally I work my way around any topic to deliver (after at least 3 pages) the punchline. But not this time. The news is just too big to put anywhere in the headline. I am so excited about this – it’s been quite a while in planning and quite a while in executing. But now that it’s here it feels as sudden as an earthquake.

Here’s the Press Release announcing the acquisition.

And here’s a bit of background. It’s certainly no secret that my goal for Semphonic has been to build the leading digital analytics consultancy in the U.S.  It’s also clear that we’re at a critical inflection point in our industry. The consulting environment has been almost ridiculously fractured and dominated, to some extent, by small boutique companies like Semphonic. Yet our clients are demanding ever larger scale, more projects, more comprehensive digital measurement coverage, and more advanced technologies. In a big-data digital analytics world, the days of the Mom-and-Pop are clearly numbered. Digital has become an integral part of the enterprise marketing world; for many enterprises it’s the single most common customer touchpoint and often the most important.

Digital has grown up, and it’s time digital measurement did too.

2012 was a spectacularly successful year at Semphonic. By far our best year ever. We registered something like 70% growth in every important metric to a consulting business. That’s exhilarating, but it’s scary too. Managing growth can be even trickier (though not nearly as depressing) as managing contraction. I’ve done both and I know. But it also made clear to me how great the opportunity is around digital analytics and how little of that opportunity Semphonic could hope to capture.

So the decision to seek an acquisition wasn’t one we made lightly and it wasn’t driven by need. It was a conscious decision made with deliberation and carefully executed. We hired an M&A firm (ClearSight Advisors) and began a process of talking to potential partners. We talked to every sort of company – from interactive agencies to database marketing companies to technology companies to…E&Y. We ended up with almost two handfuls of offers. But in the end, it seemed to me that E&Y was the perfect fit.

E&Y provides massive scale, an aggressively growing advisory practice, the opportunity for Semphonic to come over intact and be THE Digital Analytics Center of Excellence, a company that’s not seeking to own creative or platform in digital – just measurement and performance – and a true people-driven business. We get world-class HR, recruiting, training and IT capabilities - all fundamental to fueling growth in our business. We get the ability to integrate with a much larger team of enterprise analytics specialties across everything from Supply Chain to Health Care analytics. We get the opportunity to work from the top of the enterprise down to drive measurement across the entire spectrum of digital. We stay in the same tool-neutral, customer-centric solution world-view that has always driven Semphonic and that I believe is fundamental to effective measurement.

As I look at the landscape for digital analytics consulting, I see technology vendors who provide, at best, capable support around tool-specific implementations and tactical usage. I see interactive agencies providing fairly sophisticated media analytics (though only in support of their own buying) but little in the way of deep site or customer analytics and who simply can’t engage around broader customer problems or across the spectrum of customer experience. I see boutiques that spun up in the Web analytics world and are struggling to even stay relevant in the more sophisticated world of big data and real analysis. And I see large traditional practices that can provide scale and professional consulting but lack any coherent digital methodology or depth.

At Semphonic, we’ve fueled our growth largely by junior hires and training. It’s a slower path than body-shopping, but it creates a practice with a stronger culture and a more coherent practice. I think we’ve gotten pretty good at this – taking smart people who haven’t done digital analytics and giving them the chops, the methods, and the ways of thinking that drive world-class digital analytics. From VoC to Big Data to Enterprise Reporting to Site Analytics to Customer Data Models, there’s a clear, strong, deeply considered point-of-view here that’s backed by a wealth of hands-on practice experience.

In short, I think our practice within E&Y can provide something completely unique in the analytics landscape. Something deeply interesting and valuable. Fold a practice with a deep understanding of digital analytics and the best methods in the industry into a company with the ability to scale across entire enterprise digital programs and deliver all the management, resources, and professionalism necessary to drive very-large engagements, and I think you have something absolutely transformative. Something that will shake up and change the entire digital measurement industry.

All this year I’ve been focused in these blogs on doing something that I think we all fail at far too often – driving measurement that matters. With E&Y, I believe we can deliver that.

VoC Research : Site Surveys aren't Always the Answer

In my last couple of posts, I’ve hammered away on what I think are some pretty big issues with the way most companies are using online surveys. I don’t want to leave the impression that online surveys are all there is to building a Customer Intelligence System (CIS). I love online surveys because they are so flexible. You can create an online survey for many, many different types of research and almost any type of question or situation. But "many, many" different types isn’t quite the same thing as all types; and there are some important problems for which online surveys are ill-suited.

After all, every research method has limitations and in this case, many of the most fundamental limitations are inherent to the basic mechanisms of online surveys. These mechanisms limit your ability to target certain kinds of samples, to isolate your research from your marketing, to understand certain classes of problems which exist outside the Web, to tackle operational problems, and to find ideas outside your known research program.

I want to tackle each of these in turn to show why integrating multiple sources of VoC data into a CIS is essential and to make it clear that even the most comprehensive online survey program will necessarily miss a great deal of importance.

 

Targeting Certain Kinds of Samples

When you use online surveys, you’re working with a specific population – your Website visitors. Ten years ago, that population was not particularly representative of your entire customer base, limiting the uses you could make of online surveys. Today, it might be a very good representation. But even if your Web population is a near-perfect match for your overall population, it is almost certainly NOT a perfect match for your entire Prospect Audience. If you want to explore Prospect attitudes and opinions, online surveys will limit your sample to the group of people giving you enough consideration to visit your Website. That’s always an important limitation and it can be a crippling one depending on your research question. If you’re Samsung looking to compare the attitudes of Prospects about the iPhone and Galaxy, intercept surveys on your properties aren’t worthless but they are missing a big chunk of the story. Even if you explore segmentation based on respondents who say they’ve visited the Apple site as well, you’re missing respondents who never considered Galaxy or didn’t consider Galaxy seriously. Remember the old adage in real-estate – location, location, location. In ALL Voice of Customer Research, it’s sample, sample, sample. Your ability to control and define your sample determines what type of research you can do. For early stage prospect research, traditional offline surveys, off-site online surveys, and social media research are the right pieces to use in a CIS.

 

Isolating Research from Marketing

The dirty little secret of online surveys is also bound up with sampling. Unlike pretty much ALL offline research, the sample for online intercept surveys is directly tied to your digital marketing and Web functionality. When you change your digital marketing, you change the shape of the visitor population and you change the distribution of visit types on the site. Since visit type and population are two huge drivers of high-level scores like NPS or Satisfaction, these numbers are heavily influenced by changes in the underlying sample. Because of this dependency, it’s much, much harder to insulate your online survey results from trends and changes in marketing than it is in the offline world. This isn’t such a big issue if you are stratifying your sample and are focused on site usage questions. In such cases, as long as you look below the site-wide numbers you’re probably okay. But if you’re running intercepts inside public content and you have active marketing campaigns, you’ll find it very difficult indeed to trend that research over time. To isolate your research from your marketing efforts, Social doesn’t work well either. You’ll have to rely on off-site intercept surveys and traditional offline VoC studies.

 

Understanding Some Non-Web Problems

I’m a firm believer that you can use your online survey program to understand and research issues that go far beyond the Website. In my last post, I argued for a fairly deep re-write of the content of surveys to do exactly that. I suggested, for example, that online survey research was an appropriate place to delve into customer drivers of choice around product and competitive set. Not every type of problem is equally accessible, however. Survey’s focused on drivers of choice are highly relevant to the consumer in the Web channel. But asking questions about your stores, or their call-center experiences, or your mobile app is going to be much less effective. I don’t think this is controversial and it’s not something I see anyone really getting wrong. On the other hand, what I do see people missing is the implication that VoC research needs to be tightly embedded in-channel to be maximally effective. Trying to do mobile app research on the Web (or, far worse, offline) just doesn’t make sense. This doesn’t mean you can’t email a survey to app users. It does mean that trying to randomly find users of Channel X by surveying in Channel Y is unlikely to work and unlikely to be interesting. Embedded research and email surveys are the right VoC techniques for most channels.

 

Tackling Operational Problems

The big drawback to online survey research when it comes to operational problems is sampling. But it isn’t the nature of the sample – it’s the very existence of a sample. Many operational problems on today’s Websites exist for very small sub-populations and in relative “edge” cases. An operational cart problem may only exist for 1 in very 10,000 site visitors – but still be an important defect. From the survey perspective, however, it’s difficult to catch such situations and difficult to track them down when they occur. You’ll have to collect a lot of surveys before any pattern emerges – so many, in fact, that from a statistical perspective this will always just be noise. Of course, it’s quite possible to study the specific behaviors of negative responders. But this can be a thankless task. You’ll have negative responders for many reasons, most of which aren’t operational problems. Making matters worse, most survey instruments aren’t setup (rightly) to research operational issues. So they often leave the user with little room or incentive to detail the real problem. So I’m not exactly against using online survey research to find operational problems. I am against using it as your only mechanism. Tools like OpinionLab are really better suited to this role. Though anecdotal, most operational research is, effectively, anecdotal. You’re not looking for general problems, you’re looking for specific ones. By providing an immediate and direct venting mechanism focused on customer-care, OpinionLab provides a more reliable source of operational defect discovery than almost any intercept-based survey program. Most of our clients get this right, but some still treat Opinionlab and Online Survey programs as if they were competitors and make either/or decisions about which to use. 99 times out of a hundred, the right answer is both.

 

Discovery Outside Your Research Program

Survey research is great for finding out what you don’t know. Just as long as you know you don’t know it! That isn’t a paradox. Most survey instruments are much more effective with close-ended questions than open-ended ones. I take quite a few surveys and I probably qualify as both a critic and a know-it-all, but I rarely end up filling out open-ended questions. As a respondent, open-ended questions are just too much work! So while it’s possible to build a survey instrument to study almost anything, it’s not so easy to build something that helps guide you to new research programs or illuminates issues you haven’t thought about.  For discovering new research questions, there are behavioral techniques and there are other VoC methods; in particular, Focus Groups and Social Media Research. While Focus Groups can suffer from some of the same problems as surveys, they provide a more open-ended forum where a skilled facilitator will often find un-explored and novel customer attitudes and issues. Even richer is Social Media. Because Social Media captures un-guided and un-fettered conversation, it can be a great vehicle for exploring outside your research paradigm. This takes real work though. New research questions don’t just fall out of social media reporting. To effectively mine this data, you’ll need real analysts specifically targeting this type of research.

 

Most of what I’ve called out in this post isn’t startlingly new. Much of it is obvious and a good portion of that is already standard practice. The really important point is that if you’re trying to construct a comprehensive warehouse and reporting system around customer attitudes you are necessarily in a multi-source world. Site surveys are central to that world, but they have limitations that simply can’t be got around. Fortunately, we don’t lack for sources. From online feedback systems to social media to Focus groups, there are ways to explore VoC data that can dovetail perfectly with an aggressive Site Survey program.

 

Speaking notes: I have some speaking gigs coming up in April – two of which are around Tag Management. The first is an AMA Workshop in Atlanta on Tag Management. If you’re going or in the area, check it out. I’ve created the content and it’s both a pretty wide-ranging survey of Tag Management and a pretty decent deep-dive too. Not long after I’m providing a short overview of Tag Management Systems for a free webinar with Tatvic.

Re-Thinking Your Survey Research – Asking the Right Questions

In this first part of a series on doing measurement that matters, I’ve been focusing on one of my main themes this year to Semphonic’s clients – that it’s time to fundamentally re-think Voice of Customer and customer research programs. I started the series with a look at how the 2012 Obama campaign broke from traditional survey research to create a massive customer feedback cycle that not only allowed them to better predict battleground states but let them rapidly test and experiment with campaign messaging targeted toward key states. The use of massive survey collection to supplement behavioral modeling and the integration of VoC with field experiments were radical alternatives to traditional small sample opinion research. In the next post, I showed how much more sophisticated enterprises are with behavioral and transactional data warehousing than with customer research data and I argued for similar attention to data quality, centralization, standardization, and dashboarding. In last week’s post, I showed how survey research could fill a critical (and largely un-noticed) gap in digital marketing programs - allowing the marketer to understand whether targeting or creative was responsible for performance. Today, I want to dive right into the heart of online customer research and spend some time on the type of questions that should make up an online survey.

With all due respect to site usability, I think it’s fair to suggest that in 99 cases out of 100, site usability is the LEAST important factor in a customer’s decision-making process. I can count on my hand the number of times in the past year that a site’s usability actually discouraged me from transacting with them. It does happen, and yes, we as analysts do have more control over site usability than product quality or product pricing or branding. But if you’re limiting yourself to a small set of incremental cases, you’re simply not doing measurement that really matters. Good research should include the Website as a tool, but there’s no reason to limit yourself to a narrow range of possible improvement.

For several years now, I’ve been telling our clients to stop asking dozens of questions in their online surveys rating site navigation, site images, and site functionality and start asking customers about drivers of choice – the things that make customers decide to purchase or not. Often enough, they give me a blank stare and ask “Like what?”

The answer, of course, is highly specific to each business. But there’s some general themes that emerge as Semphonic increasingly tackles this type of research. Let’s start at the top. One of the most basic site survey questions is “visit intent” – the “Why did you visit the site today?” question.

Nearly all of our client’s ask this question in some form or another, and the available answers are usually something like this:

A)     To  Buy or Research a Product

B)      To Check the Status of my Order

C)      To Get Customer Support

D)     To find out more about the Company

E)      Other

If the client has a few broad business lines, they’ll often follow up with a question like:

Which type of product were you interested in?

A)     Product Line AB

B)     Product Line B

C)     Product Line C

D)     Product Line D

E)     Other

Then it’s off to the races with a bunch of questions about their site experience.

There’s nothing wrong with these two questions except that, in practice, they are nearly useless. Useless because in most cases there are strong, clear behavioral proxies for the intent and because they live at a level of generality that makes it impossible to use them to understand any deeper behaviors.

In every site I’ve ever looked at, the overwhelming majority of visitors who say they came to the Website because they were interested in buying Product Line B went and looked at…drumroll…Product Line B pages! What exactly is this telling me that I didn’t already know from the behavior? You may claim that there’s something interesting in looking at the behavior of people who said that and then didn’t look at those pages, but most of the time there just isn’t. Sure, in some less common use-cases, there is real interest in what we call behavioral mis-matches – cases where stated intent and demonstrated behavior don’t match. But on really broad levels like this, it’s beyond rare for such mismatches to be common enough to be either studied or interesting.

So what should you be asking?

When it comes to understanding customer behavior on site, you have to go at least one level deeper. Let’s take Product Research and Purchase as an example. One of the most important things I’d like to know is where in the sales-cycle the visitor is when they arrive on the site. This is something very few survey instruments tackle well. But it’s vitally important when assessing site behavior to know whether the visitor arrived on the Website “ready to buy”, “shopping for price”, or “researching the brand.”

It’s powerful to structure questions around what the customer knows when they arrive at the site. If they are interested in Product Line B, did they arrive having a specific product(s) in mind? If so, we’re they trying to decide between 2-3 products? Or were they shopping for the best price on a single product? Had they decided on a product but were unsure whether or not now was the time to buy? These are questions that, once understood, can help illuminate the site (and site content’s) role in actual persuasion.  Not only are they powerful in and of themselves in understanding your customer base, they are far more useful than the big broad questions in deciding whether or not your site is doing the necessary work in any given customer use-case.

Once you understand where the customer is in the sales-cycle, you can start to tune your research to drill down even further.

For customer’s without specific products in mind, you want to drill down into what features, price points, and brands are potentially interesting. For  customer’s trying to choose between products, you can explore what the key decision points are. Then you can match those decision points to product lines and products actually viewed and/or purchased. If a customer is doing price comparison, you can explore what other sites they are going to or have already checked out. You can cross-tabulate this with the products viewed and conversion rates to decide how your prices need to stack up to the competition to get sales. Do you need to be cheaper? By how much? Can you be at the same price as Amazon? Can you charge more? This is vital research for any eCommerce shop and online opinion research is the lynchpin of a research program to decide where and when you need to adjust your pricing relative to the competition.

Is the customer trying to decide between your product and a competitors? You can explore the competitive set in the customer’s mind. You can ask about the key decision points. Then you can match to your visitor’s actual behavior to see which of your product’s they considered and differential rates of conversion based on their competitive set. This type of research can help you build a profile of when you have a competitive advantage or disadvantage relative to any given competitive set and feature ordering. That’s vital in knowing whether or not you’re likely to win with a given type of customer – and whether or not you need to offer a discount, can re-market effectively, or will likely win without any loss of margin.

Or you can ignore all this and continue to ask customers how they like the left-navigation on your Website.

It’s not that I mean to diss any questions about site functionality. But even when it comes to Website functionality, most survey instruments are deeply flawed. Let’s say you want to understand whether or not the search functionality is working well on your site. The wrong way to do this is to just include a question on the survey to rate search functionality from 1 to 10 (hint - this is what everybody does).

Why is this a bad approach? First, a significant number of folks who rate search didn’t use search. We’ve done this behavioral match many times and it’s not unusual for 20-30% of the folks who give an opinion NOT to have used the actual functionality they are rating.

Even worse, this simple rating doesn’t explore the customer experience enough to understand the nature of the feedback. Here’s a real-world example we found a couple of years back and have since seen duplicated many times. Search was one of the lowest rated functional components of a client Website. When we drilled down into the behavior, we found that search scored very differently when it was used early in the session vs. later in the session. Early session searchers had a much more favorable opinion of search functionality. That’s a clue, and a very common one. The tool doesn’t get worse the longer someone sits in a session! Further research showed that there was a fairly significant use-case where the site simply didn’t have the desired functionality. Users would thrash around looking, try a search, then give up. They were very dissatisfied with search in the resulting opinion research (it was the last thing they tried and it failed) and that experience generated a strong correlation between search-usage and site dissatisfaction. A correlation that had, obviously, almost nothing to do with search functionality.

So my advice for site functional research is quite similar to my advice around customer drivers of choice. If you want to understand search performance, only ask people who used search. Focus your research and use branching and pre-sets to make sure you only ask useful questions. Don’t just ask for a rating. Explore the drivers of the attitude – not just the attitude itself. For search, you can save yourself a lot of time by finding out why they used search. Did the user start with search? Did they use Search because they couldn’t find something in navigation? Or did they use search because where to start in the navigation wasn’t clear? It makes a huge difference in understanding their subsequent behavior and opinions.

If a visitor was using Search to find a product, how much did they know about the product? Did they have a model number? Did they know if it was no longer sold? Did they have the product to hand?

Every use case should have its own set of questions, its own unique path. A survey without branches is about as useful as a tree without branches: a barren stump not much good to the world and very hard to use productively.

I want to hammer home this demand that the survey instrument be responsive to your individual business. This is a bad place to copy.

Here’s a common example of indiscriminate copying that also illuminates a failing of survey research for certain types of problem. For ecommerce sites, it’s certainly important to understand whether or not product images provide enough detail to help users pull the trigger.

Because it’s important for some sites, it’s become a kind of standard question that all sorts of sites use – the "Please rate the images/pictures on the Website" question. Even Financial Services sites where the products don’t have ANY physical appearance at all will often ask this question. Of course, those sites do have graphics, so they probably think the question still applies.

It doesn’t.

It’s one thing to explore whether or not the product images on a Website provide enough visual information to support a purchase decision. It’s true that you can measure this behaviorally very well, but you can supplement that knowledge with targeted questions to help explain the underlying attitudes. Customers will tell you whether or not they thought the product images gave them enough information.

This is much less practical with general site graphics.

Like any research tool, survey research has its strengths and limitations. There are things you just can’t explore directly in opinion research. It would be stupid, for example, to ask a question like:

“Were the calls to action on the Website aggressive enough?”

It’s an important question. But it’s a case where your goals and the customer’s perspective don’t necessarily align even with perfect information. In such cases, opinion research methods break down.

It’s similarly ineffective to ask the user a question about your general site graphics and images. The user doesn’t really care, isn’t responsible for, and isn’t particularly cognizant of the psychological impact of your site’s general images. Knowing the user thinks your images are nice, ok, or even a little ugly will tell you next to nothing about the impact of those images on their behavior.

If you really want to use opinion research for this type of question, you need to approach the topic elliptically. You might, for instance, ask a series of brand perception questions to test groups to see if more aggressive calls to action (or different pictures) influence the respondents.This is much more meaningful than a direct ask. A user may love the graphics on your site but be left with a brand impression that is quite different than you'd really like. A user may really appreciate the genuine restraint of your "calls to action" but fail to convert precisely for that reason.

There are a whole range of cases where the direct approach just doesn’t work well and if you don’t recognize those cases, you can waste a lot of time and energy in useless research.

I started this discussion by admitting that finding the right questions for online research is necessarily specific not general. There is no one right set of survey questions for an enterprise. Good opinion research should be more finely tailored than the gowns worn on the red-carpet at the Academy Awards. The questions you ask need to fit your business and your research questions – not some arbitrary standard drawn from other sites. What’s more, the work in building and using survey instruments is never done. A static survey (any static survey) will outlast its original function. As your knowledge of customers grows and deepens, your survey instruments need to adapt to explore ever deeper areas of interest.

At a high level, however, the message should be clear. Online survey research is an amazingly flexible, inexpensive tool for conducting all kinds of customer research. And it’s easy to decide if you have a good program or a bad program. Just look at the questions you’re asking.

If that’s not enough, here's a set of basic principles to use when evaluating or designing a Voice of Customer program:

1. A good online survey instrument uses branching to deeply explore drivers of choice while keeping total questions to a minimum
2. A good survey instrument won’t ask questions to which the answers can be reliably inferred from behavior
3. A good online survey instrument doesn’t ask about things the user hasn’t used
4. A good online survey instrument isn’t static
5. A good online survey instrument  isn’t just about the Website
6. A good online survey instrument isn’t one instrument – you can’t investigate search functionality with the same survey you use to understand competitive choice.

Precision, brevity and depth are the hallmarks of good opinion research as they are of good writing. And though - when it comes to writing - I regularly fail in at least one of these, at least it’s not for want of trying.

Targeting Precision – Using Opinion Research for Campaign Measurement

In this recent series of posts, I’ve been mapping out a complete re-formulation of the use of Voice of Customer (VoC) and traditional customer opinion research in the organization. It’s a part of the analytics puzzle that is shamefully under-utilized in most enterprises – especially given how inexpensive it is to put the people and technology in place to do a much better job. There are parts of this that I’ve been thinking and writing about for quite some time. Other parts are inspired by recent real-world examples – especially the use the Obama campaign made of customer attitudinal research in 2012. The way the data science team used survey research in that campaign was fundamentally different than (and significantly more effective) the traditional research paradigm and is highly relevant to digital marketing.

The first post in the series described why I think that campaign’s use of survey research was importantly different. In last week’s post, I described the creation of a full Customer Intelligence System. Essentially, that system is designed to apply the principles of transactional data warehousing and reporting to the customer attitudes space. The goal is to achieve standardization of the data across multiple sources (from survey to social to call-center), deeper segmentation, and full enterprise visibility into customer attitudes.

Today, I wanted to tackle one of the unexpected ways you can use a full Customer Intelligence System.

 

The Targeting Problem

Pre-Semphonic, I spent many years doing database marketing in the Credit-Card industry. I learned that three pillars drive database marketing success: targeting, creative and offer. Probably none is more important than offer. If you’re putting lipstick on a pig, you have to be one fine artist or you have to be able to locate the very small segment of pig admirers to be successful. But a good offer/product will often succeed even with poor creative and haphazard targeting. Offer matters.

But offer is sometimes (not always) outside our domain as analysts, and even a good offer can be dramatically improved with the right targeting and good creative.

As a digital marketer/analyst, it’s your responsibility to optimize each of these pillars. We all know the old adage: “You can’t improve what you can’t measure.” So here’s the question to digital marketers. Can you measure for your digital campaigns whether the offer, the creative or the targeting was responsible for their success or failure? If not, then how can you hope to optimize each or even know which component needs optimization?

Digital marketing is, of course, highly measurable. We generally know how many visitors we sourced, how expensive it was to reach them, how many engaged with the content, and how many converted.

Suppose you have Campaign X and Campaign Y. Campaign X is for beauty products and Campaign Y is for sporting goods. You have the following information:

Campaign            Reach    Clicks     Engaged               Converted

X                             100K      3K           1K                           200

Y                              100K      3K           500                         50

It doesn’t take a genius to see that X is out-performing Y. The real question, though, is what do about it. Is Y worse than X because the offer isn’t as good, because the targeting isn’t as good, or because the creative isn’t as good?

It’s pretty much impossible to answer that question with this data. Detailed examination of the Engagement behaviors MIGHT shed some light on whether or not the Offer or Creative for Y was a problem. But nothing in the behavioral stream will help you decide whether or not the targeting was good or bad.

That’s a problem.

I think it’s fair to say that in the vast majority of enterprises I work with – even those spending millions of dollars on digital marketing and using highly sophisticated optimization techniques – no one has any idea whether their digital campaigns are well targeted.

That’s a problem.

And you can probably see where I’m headed for a solution.

Suppose you targeted a very small tight survey to digital responders to recover 1-2 core demographics and perhaps even some exploration of the offer/creative…you might end up with data like this:

X converts slightly better for its target audience (Female) than Y does for its target audience (male). But the difference is slight (8% to 7%). The big difference between the two campaigns is the poor quality of the targeting for campaign Y. In campaign Y, more than half the target responders are outside the target.

By adding demographics to the analysis, we're able to pinpoint what's going wrong with Campaign X. When the campaign reaches its audience it works just fine. But it's not reaching its audience most of the time.

This is, obviously, an artificial example. But when we’ve been able to measure targeting precision for campaigns it has been consistently illuminating – often revealing mismatches between expectation and reality.

Digital marketers make lots of bad assumptions – and one of the them is that click-through is a sufficient indication of targeting precision to live with. All of our experience suggests otherwise. We’ve frequently seen campaigns with 95% bounce rates. If click-through was proof of targeting, that would never happen.

Using survey data, you can measure targeting precision – the degree to which the respondent audience matches the target audience for a campaign. With behavioral integration, you can also separate out the impact of targeting from creative and offer. You know not only how accurate your targeting was, but whether or not the campaign worked when the right audience DID respond. This is a huge benefit to optimization.

 

Brand Marketing

There’s another use-case where targeting precision isn’t just an add-on to measuring campaign effectiveness but can be the primary technique. If you’re spending heavily on digital brand marketing – campaigns that have no immediate conversion outcome – you’re single best optimization point may be targeting precision. Measures of site engagement may  be interesting, but for Brand Marketing it is often more important that the RIGHT people see the message than that the WRONG people engage with the message.

Many of our clients have logged-in sections of their site. I regularly see digital campaigns targeted toward prospects where half or more than half of the click-throughs logged-in. That’s a good check, but what if you don’t have a log-in behavior to look at?

Much of the same thinking applies to social campaigns.  If you are running marketing campaigns to acquire Facebook Fans, then you need to make sure they really are useful relationships to have. Yes, that does mean more than demographics. But at least demographics are a start.

If you’re primary focus is digital brand marketing or social acquisition, then you simply cannot do your job well unless you measure targeting precision.

It’s one of the many ways a re-focused survey research program can add tremendous value to your digital marketing.

 

 

Building a Customer Intelligence System - a Data Warehouse for Customer Attitudinal Data

In my last post, I described how in the 2012 presidential campaign the data science team for President Obama fundamentally re-engineered their use of survey and customer attitudes research to gain a significant competitive advantage.

Two pieces of what they did really stand out as significant.

First, they broke decisively from the traditional opinion research paradigm focused on broad-brush random samples to predict the election outcome. By collecting massive amounts of research and tying it to individual voter behavioral scores, they were able to use survey research to fine-tune predictive models of turnout and choice. This system created a remarkably sensitive research instrument that allowed them to predict close precincts much more accurately than the Romney campaign or media pollsters. In a tight election, this strategic information advantage ended up playing a significant role in everything from debate strategies to media targeting.

Even more important, in my opinion, however, was that way in which the Obama campaign USED this massive, behaviorally integrated opinion research database to test campaign messaging fine-tuned to very small segments of the voting population. They created a tightly coupled system where collection and analysis of voter attitudinal data were used to develop, refine and test themes used at EVERY level of the campaign from individual voter out-reach to mass media.

Think there’s nothing new about this?

Sure, every campaign uses opinion research data to tune and analyze the effectiveness of campaign themes. But those surveys are high-level, collected from small samples, and non-integrated with knowledge of the underlying voter behavior. As research tools, this makes them a very blunt instrument. By vastly extending the reach of online research and integrating it tightly to behavioral models, the Obama campaign built a system where they could test messaging to micro-segments on a continuous, real-time basis at very little cost (here’s a very fine article describing in more detail what they accomplished).

The implications for enterprise marketing are plain. Senior decision makers and “marketers” in the Obama campaign had daily, near real-time access to detailed information on how their message was playing in every key region of the country and to every potential swing voter. Even better, they had the ability to rapidly test messaging strategies to evaluate their real-world effectiveness by segment.

Do you have anything remotely similar?

Do decision-makers in your organization use opinion research data to make decisions on a daily basis?

Have you ever even looked at one of your enterprise online surveys?

Have you seen how fantastically uninteresting the questions are?

Have you ever tested actual communication strategies with your online research?

Do the folks running your site tests and email campaigns even look at opinion research data?

Enterprise customer research is deficient in almost every respect by comparison. Few people are surveyed. The resulting data is rarely integrated into behavioral data and even when it is the sample size is too small to be useful behaviorally. There are no models of decision-making and choice exploiting the combination of behavioral and attitudinal data. The information is rarely disseminated throughout the enterprise, and decision-makers almost never use survey data to test marketing strategies at any but the broadest consumer level.

Why is that? How is possible that even a well-run and well-financed but ephemeral operation like a presidential campaign should have  better customer attitudes research than the marketing departments of Fortune 500 companies?

They shouldn’t and the remedy is simple. You already have a great model for what you should be doing with customer attitudinal data in your Customer Data Warehouse. Think about how the mature enterprise routinely treats customer transactional data:

1. The data is centralized in a warehouse
2. It’s collection is standardized and  there are rigorous quality control procedures to ensure it’s integrity
3. Data from multiple sources is normalized and placed in a standard model making it easy to join the data and extract it in interesting business combinations
4. Powerful reporting tools are used to provide direct access to the data
5. Regular reports are distributed to decision-makers trending and contextualizing the data to increase its value.

This is a terrific blue-print for using customer attitudinal data and paying attention to this blueprint can cure a multitude of problems in your current research program.

Right now, the vast majority of enterprises don’t consolidate information from ANY of their customer attitudinal research programs. OpinionLab data is one silo. Online Survey research is another. Offline surveys are in another. Social Media is somewhere over there. Call center data is somewhere else. In very few enterprises is there a single centralized place where customer attitudes research comes together.

Which also explains why there seems to be virtually no standardization in that research. We routinely see frustrating inconsistencies in the various components of the customer research program. Online and offline surveys may categorize visitors differently (using different age or income categories), have different wordings on basic questions (like role or relationship), and use different qualification criteria. Many efforts are sporadic and the results never tracked over time. These differences make comparison of data across sources difficult for impossible. It’s silly and easily fixed, but until you actually put the data together, no one need to solve the problem because no one is using the combined data.

In a customer data warehouse, architects think hard about how to combine and model the data to make it useful and predictive. There is no similar function when it comes to customer attitudes research. The result is that this data is almost NEVER used in combination. Have you ever even seen (much less used) a report that shows how customer attitudes about key decision factors are changing in comparison with evolving call-center call-types? Enterprises act as if there’s no relationship between customer attitudes monitored in one space and customer attitudes (or behavior) monitored in another. Clue phone. It’s the same customers.

In the Customer Data Warehouse, your line managers usually have direct access to the data with modern BI tools. They use this transactional data on a daily basis to monitor the health of the business and make decisions. No equivalent system exists in most enterprises for customer attitudes research. Not only isn’t the data available in any appropriate tools, but the siloed tools around customer attitudes are often extremely poor at data democratization. There’s a good chance that most of your marketing managers have never accessed customer attitudes data in anything other than a flat report. You just can’t USE data this way.

Finally, most enterprises have worked hard in the past two years to create scorecards that track key performance measures across the business. If you’ve incorporated any aspect of customer attitude research into those scorecards it’s probably an NPS or site-wide satisfaction score. This is useless. I’ve written elsewhere how misguided this approach is and there is a better alternative. You customer’s deserve more than a single score supposedly representing your success with them. Do you think the Obama campaign had a single election-wide score? Do you think that’s even meaningful? What the Obama campaign realized and acted upon is the understanding that customer attitudes matter at a very deep-level. To build good messaging strategies, they needed to understand key segments of swing voters (whether swing in attitude or turnout). Your marketing problems are almost certainly identical. This makes customer attitudes worthy of an enterprise scorecard unto themselves. One that unifies all the sources of customer attitudinal data in a single, powerful view. A view that segments customers in a consistent, thoughtful fashion; that tracks key attitudinal measures for each of those segments over time; and a view that constantly pushes the envelope of learning around those customer segments by continually exploring their evolving drivers of choice.

At Semphonic, we call this attitudinal warehouse combining these five directions a Customer Intelligence System. Building such a system may be the single most important, valuable, and cost-effective means in your arsenal of making digital measurement matter. 

[If you’d like to meet at the Omniture Summit drop me a line – would love to chat!]

Lessons from the Obama Campaign – Re-thinking Opinion Research

I dislike politics; probably not least because I worked in political direct mail for a half-dozen years after graduating from college. Some people take to it and some don’t. I was a don’t. But whatever your political opinions or however sanguinary or pessimistic your view of American politics, there’s no doubt that Presidential elections represent fascinating case-studies in persuasion, media-buying, PR, and, these days, analysis.

The recent election was particularly interesting because the two camps (and numerous media outlets) were unusually varied in their predictions of the outcome leading up to the election. What’s more, those predictions appeared to have a significant impact on the candidates strategy in the closing weeks of the election. That, at least, indicates that the opposing camps actually believed their own projections – which isn’t always the case. And given that a preponderance of media attention in an election is focused  on the “horse-race” not the issues, it’s probably no surprise that the accuracy of various forecasts was hotly debated right up to election night.

That night turned out to be an impressive vindication of the analysts behind Obama’s team and in the time since the election there’s been quite a bit of attention given to the methods they used. I particularly want to reference a very fine piece in the MIT Technology Review that provides a fascinating look at the two campaigns and their approach to analytics.

There’s a great deal of interest in the MIT article, but the part I particularly want to focus on is the role of survey research and the tight integration of survey research and behavioral methods to create a continuous process of persuasive experimentation. In my kick-off post for 2013, I suggested that one of the most important ways that enterprises could “make measurement matter” was to significantly re-tool their online opinion research program.

Virtually all Semphonic clients do at least some level of online opinion research.  Unfortunately, that level is by no means high. By far and away the most common situation for our clients is to have a fixed online survey instrument reaching several thousand respondents each month. That survey instrument is usually long, heavily focused on site experience and site elements, and designed to provide enterprise-wide reporting on site-wide customer satisfaction. Each one of these is problematic.

Let me contrast this small sample size, focus on static questions, and horse-race mentality with how the Obama campaign used opinion research. The Obama campaign’s use of Opinion Research was innovative in two respects.

First, they conducted massive amounts of opinion research – far more than is commonly necessary to achieve statistical significance. They did this because they were integrating the opinion research with behavioral data to create micro-targeting and very fine-grained turn-out predictions.

Here’s the description from the MIT article:

“As the 2010 midterms approached, Wagner built statistical models for selected Senate races and 74 congressional districts. Starting in June, he began predicting the elections’ outcomes, forecasting the margins of victory with what turned out to be improbable accuracy. But he hadn’t gotten there with traditional polls. He had counted votes one by one….His congressional predictions were off by an average of only 2.5 percent. “That was a proof point for a lot of people who don’t understand the math behind it but understand the value of what that math produces,” says Mitch Stewart, Organizing for America’s director. “Once that first special [election] happened, his word was the gold standard at the DNC.”

The significance of Wagner’s achievement went far beyond his ability to declare winners months before Election Day. His approach amounted to a decisive break with 20th-century tools for tracking public opinion, which revolved around quarantining small samples that could be treated as representative of the whole. Wagner had emerged from a cadre of analysts who thought of voters as individuals and worked to aggregate projections about their opinions and behavior until they revealed a composite picture of everyone. His techniques marked the fulfillment of a new way of thinking, a decade in the making, in which voters were no longer trapped in old political geographies or tethered to traditional demographic categories, such as age or gender, depending on which attributes pollsters asked about or how consumer marketers classified them for commercial purposes. Instead, the electorate could be seen as a collection of individual citizens who could each be measured and assessed on their own terms…

The scope of the analytic research enabled it to pick up movements too small for traditional polls to perceive. As Simas reviewed Wagner’s analytic tables in mid-October, he was alarmed to see that what had been a Romney lead of one to two points in Green Bay, Wisconsin, had grown into an advantage of between six and nine. Green Bay was the only media market in the state to experience such a shift, and there was no obvious explanation. But it was hard to discount. Whereas a standard 800-person statewide poll might have reached 100 respondents in the Green Bay area, analytics was placing 5,000 calls in Wisconsin in each five-day cycle—and benefiting from tens of thousands of other field contacts—to produce microtargeting scores. Analytics was talking to as many people in the Green Bay media market as traditional pollsters were talking to across Wisconsin every week. “We could have the confidence level to say, ‘This isn’t noise,’” says Simas. So the campaign’s media buyers aired an ad attacking Romney on outsourcing and beseeched Messina to send former president Bill Clinton and Obama himself to rallies there. (In the end, Romney took the county 50.3 to 48.5 percent.)”

 

This should be sobering news if you’ve listened to your online survey vendor about how many completed respondents you need to have a viable opinion research program. I’ve argued for years that traditional views about sample size are based on out-moded techniques where survey data ISN’T married to behavioral data. Marrying online survey data to behavioral data fundamentally changes and opens up the research program; largely invalidating traditional thinking about sample size.

Even more important, however, is the USE that the Obama campaign found for this remarkably detailed, real-time survey information. Instead of just focusing on the “horse-race” (who’s ahead), they focused on how individuals were responding to key messages. They used this information to craft and field-test persuasive approaches:

“One way the campaign sought to identify the ripest targets was through a series of what the Analyst Institute called “experiment-informed programs,” or EIPs, designed to measure how effective different types of messages were at moving public opinion.

The traditional way of doing this had been to audition themes and language in focus groups and then test the winning material in polls to see which categories of voters responded positively to each approach. Any insights were distorted by the artificial settings and by the tiny samples of demographic subgroups in traditional polls. “You’re making significant resource decisions based on 160 people?” asks Mitch Stewart, director of the Democratic campaign group Organizing for America. “Isn’t that nuts? And people have been doing that for decades!”

An experimental program would use those steps to develop a range of prospective messages that could be subjected to empirical testing in the real world. Experimenters would randomly assign voters to receive varied sequences of direct mail—four pieces on the same policy theme, each making a slightly different case for Obama—and then use ongoing survey calls to isolate the attributes of those whose opinions changed as a result.

In March, the campaign used this technique to test various ways of promoting the administration’s health-care policies. One series of mailers described Obama’s regulatory reforms; another advised voters that they were now entitled to free regular check-ups and ought to schedule one. The experiment revealed how much voter response differed by age, especially among women. Older women thought more highly of the policies when they received reminders about preventive care; younger women liked them more when they were told about contraceptive coverage and new rules that prohibited insurance companies from charging women more.”

Not one in a hundred of today’s enterprises are doing anything remotely as useful with their opinion research. Using these techniques, the Obama campaign was able to adapt individual voter communication strategies in near real-time. It should be hard not to see the applicability of this to business. Online survey techniques provide an extraordinarily low-cost technique for recreating this type of micro opinion-research in the commercial sector.

It’s hard to believe that an ephemeral activity like a Presidential campaign should be able to do analytics better than an established large enterprise. But for all their challenges, campaigns are a unique crucible – the intense win or lose environment rewards (and allows for) aggressive innovation. That analytics has become fundamental in political campaigns and that the techniques used have often advanced beyond those common in the private sector isn’t, perhaps, quite so surprising.

There are important lessons to be learned here and I’m confident that the learning won’t be limited to Presidential campaigns and political parties. Enterprises that fail to adapt these techniques will find themselves out-guessed, out-communicated and out-strategized by their competitors.  

In my next few posts, I’m going to lay out a vision for creating a Customer Intelligence System in the enterprise based not just on these learnings but on a broad vision for how customer attitudes can be collected, standardized, integrated and used. The fundamental tenets of this vision are:

1. Online Survey Research is the most powerful, flexible and cost-effective means of understanding  customer attitudes, decision-making and drivers of choice. To be effective, this research must be flexible, constantly configured and business-specific, and focused on specific customer issues and decisions.
2. Customer attitudes can be tracked with a variety of mechanisms, from online survey research to social media measurement to call-center to opinion-cards to offline research studies.
3. The standardization and integration of all these sources is every bit as necessary and valuable as the standardization and integration of transactional information in a warehouse.
4. The integration of behavioral data and attitudes data is fundamental to the appropriate use of this data and changes the shape of the research program including necessary sample sizes and types of questions that should be asked.
5. Dashboarding and reporting around customer attitudes is the next frontier for Enterprise Intelligence and presents unique opportunities to dramatically improve the enterprise-wide understanding of customer’s and drivers of choice.

Meeting at the Adobe Summit

I’ve got several weeks of travel coming up – on a plane as I write this - including a somewhat abbreviated stay at the Adobe Summit. If you’re going to be at the Summit and would like to setup a meeting, just drop me a line.

On a totally non-business note, I took a rare three day weekend last week over President’s Day. We took the girls down to Monterey, which they love, and had a chance to re-visit the Aquarium. It reminded me of my all-time favorite X Change event –held there several years back. I still remember that one fondly.

What I really enjoyed this trip, however, was day-hiking at Pinnacles National Monument. It’s been quite a few years since I’ve been to Pinnacles and when I was last there my girls were too young to attempt the High Peak trail. No longer. This time they bounded up the very steep Juniper Canyon Trail – my youngest daughter was practicing dance steps near the summit while I huffed up the final steps laboring a bit and regretting my high-cookie diet. From Juniper Canyon you navigate the Steep-and-Narrow section of the High-Peaks trail. It lives up to its name. This is about as good as day-hiking gets.

You are, quite literally, in the Pinnacles with Condor’s nesting scant feet away and vertiginous drop-offs on either side. I know Pinnacles is in the middle of nowhere, but if you ever have the chance, the High Peaks trail is worth the drive, worth the time, and worth the climb. It ought be to famous. I’ve included two shots from Hikespeak.com that give a small sense of the experience:

 

 

Real post to follow!

Why Measurement Doesn't Matter : A Guide for Confused, Annoyed, or Disappointed Decision-Makers

I started off this year with Luther-like epistle on “Measurement that Matters.” It was a small (much smaller than Luther’s) set of theses about how enterprise measurement can be made to matter. The projects I described were chosen because they force or are strongly tied to driving actual use of data in the business. Why? Because far too often measurement is something that the enterprise DOES but doesn’t USE.

In upcoming posts, I’m going to tackle those projects in more detail and show why I think they are powerful, how they can be done, and why, when put together, they create a transformative measurement program. But before all that, I wanted to take a quick look at the opposite end of the equation – why enterprise measurement is so often divorced from change or action.

 

Setting Yourself up for Failure

In a large enterprise, no single factor may be more important in your long-term success or failure than how you’re organized. Madison’s famous dictum: “If men were angels, there would be no need for government” is apropos in the private sector as well. Good people will often transcend organization. If you happen to have a great measurement lead, you may have success no matter how poorly organized you are. But individual successes are rarely sustained in the large enterprise, if only because success breeds promotion (or, more likely these days, poaching). So over time, it’s hard to sustain a level of human excellence in a large organization that is much above average. Running a large enterprise is fundamentally an exercise in generating good outcomes with average people – a task utterly different from what’s demanded in small, entrepreneurial settings.

Unfortunately, digital measurement has not been well structured in most large enterprises today. Here are the most common and most serious mistakes in organizing enterprise measurement teams:

Failure to Integrate Measurement into Teams: Good measurement can’t be distant from teams if it’s actually going to drive change. Transformation has to come from within since only hands-on workers have the expertise to take advantage of data. This is particularly true in the digital world where information changes rapidly.

Failure to Isolate Measurement from Teams: This isn’t a misprint and it isn’t a contradiction. Measurement has both an internal and an external function. The internal function has to support day-to-day operational change and customer conversations. The external function has to provide senior managers the ability to channel resources appropriately and make broader organizational investment decisions. These are two completely different functions and they MUST be treated separately if your organization is to work. Centralize both functions and no one will use information. Distribute both functions, and you might get measurement use, but you’ll NEVER know what’s effective or where to invest.

Think about it this way, decentralize and embed analytics to drive customer conversations or process; centralize and make analytics independent to drive external reporting, audit and investment functions. Nearly every enterprise I know blends these two functions.

It’s a big mistake.

Here’s a related organizational issue that can just about cripple the entire digital measurement effort: separating testing from analytics. About half the enterprises I see have completely isolated site testing from site measurement. Most (but not all) of the time, these enterprises have a vendor managing their testing. This makes my head spin. What exactly is your analytics team for if not to create test plans for the Website? Talk about measurement without use! If your site tests aren’t being driven by your measurement team, something has to change. If your internal measurement team can’t think up good, data-driven tests, find a new measurement team. If they can, and you won’t let them, re-organize your testing.

I can’t pass onto the next topic without mentioning the single worst mistake you can make as an enterprise – letting your creative agencies self-measure.

Not only are most digital agencies pitifully bad at measurement and analytics, they have a strong self-interested bias in producing “win” oriented measurement. If you think you can optimize your business spend by asking people to self-assess and then paying them based on their own assessment, by all means have your agency measure the channel they own. Paying your digital agency for measurement is like funding their marketing department. Stop the madness.

 

Hiring all the Wrong People

If your digital measurement department is filled with…well…digital measurement people, then you’ve probably missed the boat. There’s tremendous demand for analysts skilled in tools like SiteCatalyst. There’s nothing wrong with that. It’s a good and useful skill. On the other hand, these folks are too expensive right now AND too limited to be a complete solution. Useful digital measurement requires more. In fact, it requires a blend of at least four different functions: analysts skilled at data manipulation, traditional customer analytics and targeting (usually in SAS), analysts focused on information formatting and delivery (Excel, Tableau, etc.), analysts focused on Voice of Customer and traditional customer research, and analysts focused on digital/site analysis. If you’re missing any of these skills or you've segregated these teams, then you’re team or team structure is probably a limiting factor on success. The very best departments I see are thriving because of the creative blend of traditional customer analytics folks with digital marketers. If enterprises were just a little more aggressive in blending in opinion research folks, they might really have something special.

 

Skipping the Foundation

Measurement is all about use. So the drive to create product is natural and I’m sympathetic to it. But there’s democracy and then there’s anarchy. The difference between the American Revolution and the Arab Spring is all about having the right foundation for change. Before you start building reports and democratizing data, there’s a foundational exercise you simply MUST do.

The single biggest challenge in digital reporting is that your marketing stakeholders almost certainly don’t have a strong understanding of how to measure success in the digital channel. In some cases (like purchasing products) it’s fairly straightforward. But comprehensive digital measurement for the large enterprise will demand success measurement across a wide-variety of digital touch-points and functions: from ecommerce to branding to customer support. You can’t leave decisions about how do this measurement up to individual managers or business units. If you do, you’ll get metrics that simply don’t capture the necessary business success or capture only some piece of it.

From page views, to time-on-site, to site-wide satisfaction or NPS, the history of digital measurement is littered with failed paradigms. Don’t be a measurement Marxist! Failure here has deep ramifications for the whole of your measurement and marketing effort. Bad success metrics mean bad optimization. It’s that simple. Getting this right is a demanding exercise that requires business acumen, deep understanding of the digital channel, good methodologies, and quite a bit of tool knowledge. It also takes a significant up-front effort before you start churning out reports. Do it.

 

Accepting the Status Quo

If you’re a senior manager in digital or a C-level wondering why your company doesn’t seem to be winning the data revolution, here’s one very good place to start: a mirror. I’m not sympathetic to the tendency among analysts to disparage real decision-makers. I loathe the term HIPPO (derogatory slang in our industry for Highest Paid Person’s Opinion). Making decisions with data is far harder than people think.

But here’s my advice to senior folks frustrated with the (f)utility of their enterprise measurement. Don’t accept the status quo. Don’t accept measurement that siloes everything. Don’t accept reports that tell you what’s happened but provide no insight into why or what to do. Don’t accept analytics that are divorced from every possible channel of action. Don’t be intimidated by the buzzwords, the data-blizzard, the fancy reports, or the digital slang. If it’s not helping to make decisions, it isn’t any good.

Far too often, decision makers fail analytics by failing to demand enough.

 

So there it is - a whirlwind tour of the biggest failure points in digital measurement with a few side-notes from Luther, Madison and Marx to keep you on your toes. I’m sure that’s enough philosophy to last a good long while – next up a far more practical and prosaic deep-dive into the six projects that make up my measurement manifesto.

What is ‘Big Data’ – Shortening the Path to Enlightenment

I was down at Splunk this Friday and they reminded me that a few weeks back Svetlana Sicular of Gartner wrote an excellent piece about the state of ‘big data’ (with, as it happens, a glowing mention of Splunk). At its center was Gartner’s truly wonderful concept of the hype cycle. Instantly recognizable to all of us, the hype cycle looks like this:

 

Svetlana argues that big data may be passing the “Peak of Inflated Expectations” and entering the “Trough of Disillusionment.” If this sounds like a digital analtyics version of the tots game, Candyland, that just makes it all the more priceless.

“A six. Yay! I’m at the Peak of Inflated Expectations.”

“A three. Oh no! I’m in the trough of disillusionment and have to go back to where I started.”

I’m not completely sure that Svetlana is right about where we stand. Within the sophisticated measurement and enterprise intelligence community, I think she’s dead-on. A lot of analytics pundits now are feeling that same chilling breeze and beginning to take the “Big data? it’s all really the same old stuff” tack as a way to keep themselves ahead of the curve.

But what could change that dynamic is that analytics and big data – for the first time ever – show signs of punching out into the broader business and cultural realm. Within our community, the hype around big data has gone about as far as it can go. But measured across the broader business culture, the hype is still nascent.  It could grow much, much bigger if it truly metastasizes from us to the broader community. Extend the hype-cycle concept to say that it’s community specific and you see that hype that leaps boundaries will, gangnam style, re-draw the curve.

However, if we are indeed about to enter the “Trough of Disillusionment”, I’m going to remain contrarian (as is Ms. Sicular if I read her correctly). I’m not here to defend the hype around big data, the sloppy thinking, or the vendor overload. But I believe that the purveyors of “it’s all really the same old stuff” are selling a different form of snake oil. In 2002, after the collapse of a truly epic hype cycle, there were plenty of direct mail guys running around  saying ‘I told you so.' At least now they’ll have their Saturdays off.

No doubt, one of the reasons ‘big data’ is vulnerable to hype is that it’s so poorly defined and fuzzy a concept. The general consensus in the community is that "big data" involves the four V’s: Volume, Velocity, Variety, and Veracity. The IBM Website has a pretty good summary of what I take to be the conventional wisdom.

I don’t much care for this definition. It’s fuzzy, incomplete, and misses the truly key points.

To see why, let’s work backward through that list of V’s.

Is the need for veracity (read data quality) really different in the big data world? I don’t think so. Most big data systems lack anything like the quality and governance of traditional transaction systems and, you know what, they don’t even need that level of quality. So not only doesn’t big data push the envelope in terms of data quality, it often relaxes it a little bit.

Variety comes closer to the truth but it’s also a bit of canard. At least in digital, the range of big data sources is varied and challenging. On the other hand, big data folks often talk about mixing structured and unstructured data and then misdescribe most digital data as unstructured. In the digital space, only social media listening data is truly unstructured. Nearly every other source, including Web analytics data, is structured. What’s more, the vast majority of big data systems and technologies have NOTHING to do with mixing unstructured and structured data. There aren’t really any analytic methods or tools for dealing with unstructured and structured data at the same time and there’s usually not much reason to do it either.

I’m not sold on Velocity either. Are we really getting pushed into big data systems to do real-time analytics? I’m not seeing that. In fact, I’d argue that real-time is a completely different technology stack with its own unique set of problems and solutions.

I’m not going to argue against the last 'V' – volume. Now that would be contrarian! But we all know that just saying big data is new and different because it involves lots of data isn’t a good or complete answer. There was lots of data in those credit card transaction systems I worked with back in the ‘90s, but I don’t consider them big data solutions.

So if we have one ‘V’ not four and that ‘V’ isn’t very illuminating, you might be left with the increasingly popular pundit's view that big data really is just the ‘same old stuff’ with higher data volumes.

Except it isn’t.

I’ve expounded Semphonic’s definition of big data before. In its most interesting form, big data is about the move from aggregate level data analysis to detail level data analysis. It’s about the difference between analyzing data in streams and analyzing data as entities.

The vast majority of traditional data analysis techniques work on blocks of data that are discrete entities and have direct meaning. 'Customer' is a good example. Traditional marketing segmentations, for example, worked on data at the customer level. The customer’s demographics. The customer’s relationships. The customer’s purchases. The customer’s attitudes. We built segmentations where each record represented a single customer and each field was a discrete representation of that customer.

The vast majority of analytic techniques (not just segmentation) are geared toward this type block, entity-level analysis.

And it isn’t just analysis, it’s reporting too. Almost the entirety of the BI world in both the analytics and reporting senses, are driven by this concept of analysis at the entity level. Cubes work because you can meaningfully aggregate the underlying entities.

So what’s different about big data? In the digital world (and in a number of other big data scenario’s), the data is no longer collected and analyzed at the entity level. Web analytics data (which I consider to be a paradigm case), consists of lots and lots of server calls (page views). You can treat these server calls AS IF they were the entity of meaning. That’s exactly what systems like SiteCatalyst do. They aggregate server calls as if they were meaningful. Unfortunately, they aren't very meaningful, and this aggregation doesn't answer most of the important questions marketers have.

When we use Web analytics data, we want to understand what the stream of page views says about the Customer - so just counting page views doesn't work.  But here’s the tricky part – we can’t just add up page views at the Customer level instead of the site level to get the answer. The page view in and of itself doesn't have any real meaning and summing them doesn't add meaning. Instead, we need to infer meaning from the pattern of the stream. How broad the stream is, how fast it goes, which pieces flow together, and what comes first and what comes later. These “stream” concepts are what drive things like Semphonic’s Two-Tiered Segmentation (they form the visit intent part) and, believe me, they are different from traditional segmentation.

Let me give you an analogy. You might decide that to better understand how best to swing a bat to hit curveball, you're going to collect thousands of sensor data points about the location of the hitter's legs, torso and arm as they swing. This type of analysis makes perfect sense and it's truly a big data application. Looking at any single sensor data point is completely uninteresting. Adding up data points doesn't make sense either. You can't just sum arm movements - that a hitter has 700 different arm movements doesn't tell us anything about their actual swing. To analyze this data, you need to infer meaning from the stream. That's the hard part. That's the part that traditional data analytics techniques DON'T handle.Traditional analytics studied hitters by things like batting average, on base percentages and slugging percentages. That stuff is cake in traditional data and analytic solutions, and these methods are great for deciding which hitter is best. But those methods are completely useless for making any one hitter better. Analyzing swing data? They won't do that.

Web behavioral data is just the same.

This task of inferring meaning from streams is the real difference between big data analytics and traditional database reporting, BI, and analytics. Yes, along with the fact of streams comes, in many cases, much higher volumes. When you go down to the detail level you almost always create much higher volume. And certainly volume complicates the problem. But it isn’t the core problem. You can have just 10,000 records of stream data and it’s STILL unanalyzable and unreportable with traditional statistical and BI methods.

That's why a view like this that casts the big data world as the same old problems but with more data and different sources is so wrong. If you've never actually done any analysis, that story might seem right. But if you can see why analyzing batting-swing sensor data is fundamentally different than analyzing batting averages, you'll realize why thinking big data is just more of the same is going to land you deep in that trough of disillusion.

So that’s the Semphonic story. Big data isn’t about the four V’s. But it isn’t just hype and it most definitely isn’t the same old thing. It’s a fundamental shift in the techniques necessary to understand data when you move analysis to a level of detail where individual entities only have meaning in the context of their broader stream.

I hope you feel as I do that this is a beautiful, elegant definition that captures far more of the truth about what big data means (or should mean). After all, we make the definitions, and it's still early enough in process to decide exactly what we mean by big data.

This shift to detailed stream data really does change everything. It changes what you need from your ETL tools and it significantly increases the demand for complex ETL. It changes the demands you place on your data model and even what a data model is for. If re-shapes what you might expect from your data collection infrastructure. It changes and radically increases the demand for algorithmic analytic solutions. It changes the analytic methods appropriate to the data. It’s why solutions like Hadoop really do matter (and it’s also why languages like Pig are still missing important capabilities). It’s why constructs like Aster’s nPath are really important. It’s why solutions like Splunk don’t look anything like traditional data warehouses. 

Big data isn’t just hype (though it’s much hyped and most of the hype is nonsense) and it isn’t the same old stuff. I’m pretty sure I’m not going to spare us all a journey into the trough of disillusionment. But finding a way to intelligently define big data goes hand-in-hand with starting to think productively about big data problems. Knowing why big data is different, and why that matters is the first step forward.

“A five? Hooray! I’ve reached the Slope of Enlightenment.”