Hey Gary –
You’re absolutely right that companies continue to tell us that the greatest bottleneck to their testing process is creative. Should it be? NO!! Bottlenecks should be due to your site traffic – never your analytics, creative or development or you’re not getting maximum benefit from your program. So why is creative most frequently the bottleneck? If we’re talking copy only – there’s really no excuse as you point out. BUT – when we’re talking UX and design – well – that takes more time and those people tend to be very busy folks being pulled every which way supporting multiple programs. As far as how I’ve seen companies scale creative effectively – I haven’t seen it yet to be honest. I think one company that is closest decided to actually designate an agile-run creative team of 6 people to the testing team. And even with that level of 100% support from 6 designers and copywriters, they still could only crank out creative for about 10-15 tests every 2 weeks. I’ve not personally seen but I’ve heard of other teams who have lower standards for their test creative than they would if it were going permanently on the site – and those folks tend to move much faster – often with only 1 designer (or someone who is both developer and designer). However – those programs were all newer programs that had yet to gain quite as much company-wide attention (and oversight) so they were able to sort of fly under the radar and move much faster by removing some of the common “review rounds”.
You asked about creative briefs. Analytics can’t tell us everything for a creative brief – but it can turn the lights on so you aren’t taking a shot in the dark (though of course you can still miss)!
And yes, it’s true that even great creative briefs will only inspire more possible variations of creative approach – but why is that a bad thing? If you have the creative bandwidth to support it – I think it’s an awesome thing! The testing tools we have today are so powerful (when used properly) – we can handle the extra variations and not just win – but win big!
So let’s talk about how analytics can be used most effectively for creative briefs. A truly helpful creative brief is more than an “I want to fix page X or experience Y.” It requires providing some analysis to the creative team. This can be a point of frustration and even contention in many organizations if analysts are not taught how to tell stories with the data. Simply slapping some tables or charts on a slide and sending that over a cube wall ain’t gonna cut it if your goal is for the creative team to really understand what the data is telling you. Instead – YOU have to tell the story – and tell it in a compelling way. Let the data talk to you – then translate into a language most likely to inspire the creative team. One of the most effective tactics I’ve ever seen with creative briefs is for the analyst to essentially provide use case examples based on the analysis they’d conducted with full stories showing persona and intent for each. For example, in the Obama campaign example you provide, an analyst would have a rich volume of great data to create some pretty compelling segmentation and, ideally, would also have a pretty good idea of some potential reasons visitors would chose to open and read the email. So an analyst might present story topics something like this:
- Passionate Supporter – Looking for inspiration
- Frequent Donor – Looking for fundraiser goal status
- Hillary Supporter – Just checking out the competition
And as you might imagine, the top two segments could easily swap intents giving you a passionate supporter looking for fundraiser goal status and a frequent donor looking for inspiration. With that information (and a nice story backed up by the analysis to give more color to the two-tiered segments), creative teams are ready to hit the ground running and create at least 3 if not 5 unique and compelling email campaigns for the testing team to consider. They might decide to send a Michelle-featured email to the Hillary supporters and create two options for the passionate supporter – one that features an inspiring video showing Obama speaking to a highly-supportive crowd at the top and one that features the current fundraiser goal status.
I’ve also seen some very good “auto-optimization” efforts where we use analytics to make our best guess about what type of creative will work best for each segment and launch the “test” that way but leave some chunk of the audience to randomly receive random creative regardless of which creative was “intended” for them. Then, if a different creative than analytics predicted would “win” begins to “pull ahead” for that segment, the tool can begin siphoning traffic for that segment into the “surprise” creative.
Lastly – analytics CAN get in the way of good creative – if analytics become to prescriptive instead of providing a general framework, it can serve to hobble the creative team and leave them nowhere to go. At the end of the day, analysts are the numbers experts and designers and copywriters are the creative experts. They must work together and respect each other’s efforts to have a successful creative brief and outcome. In an ideal situation like what occurred with the Obama campaign, I would imagine the analytics team would have sat down with the creative folks and explained the different “audience segments” they were hoping to target and give them some ideas on what might resonate with them (based on past data maybe or any other information that was used in the segment creation process). Then the creative team goes back and attempts to use that information to guide their creative process. Ultimately – only clean A/B testing provides confirmation – but if set up properly, you can learn very quickly with a sample of the population and then roll out your learnings to the full population AND use those learnings to feed future creative briefs.
So yes, it’s absolutely true that you can just go to creative team and tell them that you want to test 3 versions of landing page X or email campaign Y – provide the goals of what you hope the page or email will accomplish and then let them do their work. They will nearly always create copy and creative that is fun/interesting/entertaining/pretty – and if it’s different enough from status quo without removing any important functionality – it will probably out-perform default. However, I have seen again and again that an analytics-driven design recipe out-performs the pure-creative design recipe when run head-to-head. So can you do pure-creative without analytics? Sure. Will it win? Probably. Could it do even better if analytics drove the design? Highly likely.