Analytics Club - Talking Optimization Post #7 by Gary Angel

Kelly,

Terrific post. I think your five steps capture a large and important part of how enterprises ought to think about the integration of testing and analytics. Everything, from the deep integration of segmentation to the use of customer satisfaction impact is, I think, dead-on.

In fact, I think we’ve done such a good job covering this that I want to delve into a slightly different topic; namely, the way that a testing program can be used by an analytics team (and an organization) to answer fundamental questions that are impossible to cull out of any analytics method, no matter how sophisticated.

Here’s an example of a case that came up recently in conversation with a client. This client has a fairly large number of Websites that each support a separate brand. The sites are structured in a fairly similar fashion but are independent, have separate content, and are built by quite a variety of agencies and creative teams. In the last few years, one of the questions the client has been asking is around the value of short-form video on these sites and the best strategy for integrating that video. Good question, right?

But if you let each brand and creative team decide how they’re going to integrate video, you’re quite likely to end up with a situation where many strategies are untested and a few are over-tested. With no coherent plant to test video integration, it is HIGHLY likely that you’ll lack data to actually answer the best-practice question.

Here’s the dirty little secret about analytics – it requires variation. It doesn’t matter how powerful your statistical techniques are, you can’t analyze what isn’t there. Big data, machine-learning, advanced algorithms – it doesn’t matter a whit unless the data has enough variation to answer the questions.

So let’s think about video. Right off the bat, it seems to me that I’d like to be able to tell my creative teams:

1. What types of video are most interesting and impact by customer segment
2. Whether video should be implemented on it’s own page or integrated
3. Whether video should be on the home page and, if so, in what area
4. Whether video should be auto-play or not
5. Whether video should be sub-titled
6. What’s the optimum length of a video by type and segment
7. What’s the best strategy for integrating calls-to-action into a video

I’m sure there are other important questions. None of these questions can be answered by a single set of videos and a single navigations strategy. With a single strategy, the best – the absolute best – that analytics can do is tell you which audiences that single strategy is more or less effective for. Useful, but simply not enough to create best practices. But suppose you have five sites with different implementations – can you answer the best practice questions? Well, it seems to me you can take a stab at it, but there are some severe limitations. First, you can only analyze the actual variations that have been implemented. If none of those sites used auto-play, you can’t analyze whether auto-play is effective. Second, you have to hope that you can achieve a high-degree of comparability between the sites. After all, you can’t test the impact of different video integration strategies unless you can compare placements, engagement, and impact on an apples-to-apples basis. That’s hard. Really hard. Sometimes flat-out impossible. The whole point of a careful test is that you’ve created a true control group. That control group makes it possible –even easy - to measure the impact of variation. Without a true control, you’re only taking your best guess.

If you really want to develop a good set of best practices around something like video, you need to develop a comprehensive test plan: a test plan that includes different types of content for different audiences, different lengths, different integration techniques, different locations. Then you have to test. And analyze. And test.

So in my video example, a good test will include developing different types of video for each customer segment and use-case, trying two or three creative approaches to each video (remember the creative brief!), modifying each type into two different lengths, testing video in various site placements including early to entry, late to entry and key pages, testing auto-play and various other video configurations, and testing different call to action strategies across different permutations of length and type.

Whew!

This is certainly daunting – especially when it comes to expensive form content like video (these same tests are ridiculously easy for text and are almost all applicable to article form). Which, to me, highlights an important organizational truth. Some enterprises choose to treat testing as a pay-to-play service. For some things, that works fine. But guess what, it also means that this kind of comprehensive programmatic testing to establish a cross-brand or cross-business-unit best practice will NEVER GET DONE. No single brand is ever going to invest in the necessary content or the necessary experiments to create the variation necessary to answer the analysis questions. I’m not against making testing and analytics departments get some of their funding from their stakeholders on a pay-to-play basis. That’s a good discipline. But if they don’t have independent budget to attack this class of problem, it doesn’t seem to me that anyone ever will address these bigger questions.

What do you think? What’s your view on the best way to budget a testing organization?

This whole question, it seems to me, is the flip side of the close relationship between analytics and testing. Testing plans should be developed at least in part to answer analytics questions that require more variation. When you create these kinds of tests, you’re learning with every variation. Wins and losses aren’t what’s important because every test yields analytic knowledge that then feeds back into broader strategies.

To me, this kind of testing ought to be a significant part of any good enterprise testing program – and it’s driven by the analytics team because they are the ones who have the best sense of the questions they can’t answer without more variation. What do think? Are testing departments and testing consultancies out there doing a good job of this kind of testing? I sure don't seem to see many cases where testing programs (even where they are led by expensive professional testing consultants) have definitively answered this type of open-ended best-practice question.

In one sense, all testing is really just a way to create variation for analytics. But a testing program that’s designed to establish best-practices around some kind of content or tool fills a unique niche that is separate from testing that’s designed to optimize a single path or problem. How much of this kind of testing should organizations be doing? How can they decide what variables to test to create a comprehensive plan? What’s the right way to balance this kind of focused testing-program with analytics testing driven by specific problems and issues? And finally, how, organizationally, can you work with agencies or creative teams when this kind of focused testing program is what you’re trying to build?

That’s a heap of questions, so I look forward eagerly to the next installment of Kellyvision!

Gary

Comments

[Allison Hartsoe]

Hi Gary,

So now we are really talking about analytics leadership which is notoriously thin. In order to get the variation you need and the cross-business unit testing the senior team must see analytics as a priority. I think that is occurring more but still lagging.

Meanwhile, until we get that overarching data science budget and quantified customer focus through testing, we are left with the small wins strategy. Small wins is are created when one business unit finds testing synergies with another to lift both thereby creating a live example which proves the need for "landscape" analytics leadership.

Here's an analogy. Testing today within business silos is like asking all the people in the NorthEast what they think of your brand. The assumption is that no one from Portland (Oregon) ever crosses over to Pittsburgh. And while you can gain insights this way, the lack of variation from the rest of the country will clearly limit the macro learning and more powerful insights that could result.

All the best,
Allison