In my last post, I argued that even fairly sophisticated testing programs at many companies have lacked a strong testing plan, that such plans are essential to driving deeper business goals instead of small creative optimizations, that the basic methodology commonly recommended for testing on a full run-of-site visitor basis was deeply flawed, and, finally, that Web analytics Use-Case analysis was the appropriate method to provide a better testing framework: a framework focused on solving real-world business problems not on manufacturing alternative creatives.
I ended that post with the comment that if you accept this critique of testing, then multivariate testing (as opposed to A/B tests) probably won’t seem like a very good idea.
I think the connection is fairly straightforward – so straightforward in fact that both Peter Ahl and Jim Novo in commenting on that post basically jumped my point! Both their comments are well worth taking in.
The central claim in my last post is that intelligent development of creative alternatives can only take place in the context of a real-world business problem and that these problems nearly always are best understood in terms of a specific visitor segment and visit intent. Creative doesn’t just happen – good creative is deeply responsive to a specific audience and to what that audience is trying to accomplish.
Because of this, it’s almost impossible to imagine how good creative can emerge from a strategy based on run-of-site visitor tests.
Instead, I argued that good creative is far more likely to emerge when you begin with an audience segmentation, add visit intent, and then identify potential use-case improvements – the business problems you want to solve. When you do this, your creative developers are working in a much stronger context for the test – and will produce much better material.
Multivariate testing almost inevitably breaks that paradigm. Once you’ve abandoned the run-of-site visitor method and are building tests that are targeted to specific audience segments within specific use-cases, how attractive is it to build multiple creative alternatives? I admit, it’s not totally unreasonable to do so, it just doesn’t seem very compelling.
Multivariate testing also suffers from the same problems around creative development as run-of-site visitor testing. It’s much, much harder for a creative developer to build multivariate creative because it’s much harder to do so within a business hypothesis.
I won’t argue that it’s impossible. It isn’t. A designer may have a very strong hypothesis about how to reach an audience segment but still be unsure which of two images or placements within a larger creative effort is more compelling to that audience. Multivariate testing would provide an easy way to get an answer within a perfectly logical testing program.
On the other hand, everyone realizes that one of the primary drawbacks to multivariate testing is how much work it is to build multiple creatives. That work is easier to justify when you’re doing run-of-site testing. It’s hard to imagine lots of effort being invested in multivariate testing options when all of your tests start out focused on particular segments and visit types.
Although I can't argue that multivariate testing is always wrong, I do believe that it routinely encourages exactly the wrong sort of testing by making it extremely difficult to build a creative hypothesis and discouraging tests focused on a single population segment. It’s the sort of testing that blindly throws creative’s into a “reality-mixer” hoping that a superlative cocktail will emerge. It won’t. As Peter Ahl of Serenata puts it, the temptation with MVT is very much to “test because we can.”
I think there are strong theoretic grounds for my argument that multivariate testing isn’t a very good direction. But there are strong observational grounds for believing this as well. When I listen to the organization’s who are doing testing, the one’s that seem to me to be doing the most interesting tests have hardly touched multivariate and certainly haven’t made it a focus.
Not everyone in the U.S. may be familiar with the U.K.’s Serenata Flowers, but they have consistently been in the forefront of sophisticated Web analytics usage. And if you paid close attention to Dylan Lewis’ presentation at the last San Jose eMetrics (I didn’t make his X Change Huddles – wish I had!), I think you’ll see that my proposed methodology and focus on A/B testing owes much to listening to his experiences. Dylan runs what I think is (rightly) one of the most admired Web analytics testing efforts around. How much multivariate testing did he do to get there? Pretty much none.
So if you’re thinking about testing tools, here’s my number one recommendation. Don’t worry about multivariate methodologies. Don’t worry about multivariate. Don’t even think about doing multivariate testing till you’ve created a real testing plan, built up a strong framework for segmented tests, and have created a testing culture that is focused on solving real-world problems not manufacturing creative alternatives.
Some technologies are risky but potentially very rewarding. Some technologies are useless. Some are worse than useless because they are dangerous and not all that rewarding even when done well. The more I think about the problems organizations face when doing online testing and the more I hear how people are approaching the problem, the more convinced I become that multivariate testing isn't useless, it’s worse than useless.
Comments