My Photo


  • Clicky Web Analytics

Your email address:

Powered by FeedBlitz

« Another Sunset, Another X Change | Main | Multivariate Testing – Thoughts from X Change »


Gary, agree completely.

"Brute force" testing, where one keeps throwing stuff at the wall to see what sticks, seems like an incredible waste of precious resources. And too often no actual learning takes place, people don't stop to ask "why" certain things work better than others. I hope your Use Case approach teaches people the value of taking the time to understand "why".

Test outcomes are not as random as people think; there are patterns that repeat over and over, and segmentation is the key first step to being able to recognize these patterns.

"Optimizing for the average" is truly suboptimal!

Great post. We have been continously testing for the past 3 years and one of the first european clients of first Optimost and later Offermatica (now Adobe Test & Target), users of Google Optimizer and finally we decided to build our own testing tool integrated with our CMS.

Whilst often guilty of the stepped methodology critisised here, we wholeheartedly disagree with the vendors and agree with you, on the value of MVT over A/B testing. A/B tests are easy and flexible and can be launched in no time. Analysing the results are straightforward. Often when running MVT test one is tempted to test "because we can" and consequently add ever more superficial and marginally diferent creative elements into the mix.

The fundamental problem with most testing as a service offerings as you rightly point out is the lack of capability of testing deep business strategies that goes beyone simple changes in creatives.

Excellent blog, cannot agree more than from experience very few companies are linking MV testing with analytics.
Taking this a step further, its very rare for usability testing to also be supported by analytics.
In both cases this is normally a case of ownership, where we have all these things controlled by a user experience director (rare but starting to happen) we see the best results

The comments to this entry are closed.