Evaluating Internal Search Performance - Thinking about Optimization

Most problems in web analytics are much more involved than people are led to believe. This is the seventh article in a series on analyzing just one part of a web site – Internal Search. And my posts are not famous for being short! Take brevity issues out of this; it’s still true that to do a creditable job of analyzing a tool or part of a web site takes quite a bit of work. In this series I started out with some basic tips for insuring that Search is measurable and for evaluating overall search usage. This basic knowledge helps the analyst understand the role of search on the site. Next I tackled the analysis of Search sourcing. Finding pages that source either too little or too much to Search. Functional analysis of Search was third – treating Search as a class of Router Page and outlining the special nature of Search next steps. After that, I extended the Functional analysis to cover specific categories of search terms – and the necessary steps to evaluate the performance of a topic-specific subset of Search. Failed Search Terms and how to use them absorbed the next article. And my last post covered Search cues – ways you can use Internal Search to answer questions in other kinds of analysis.

In this last post, I thought I’d address a broader issue about Internal Search optimization – namely, where to focus your optimization effort.

Most of what I’ve talked about in terms of measuring Internal Search treats it as something of a black box. But any state-of-the-art Search tool in today’s world provides a platform from which Search Results can be customized. Each tool has its own vocabulary for this, but the end result is the same. When you customize Search, you provide either rules for Search to re-rank content or you simply provide specific content suggestions to be returned in special positions on the page. Of these two approaches, the second is the most common.

Many sites have discovered that their single most effective optimization point for Search is returning customized Search Results for key Search Terms. By providing Custom Results, you can provide a greatly improved path to the most important places a user can navigate to when searching on a specific concept. You can group key links, maps, photos and information in a way that will never be equaled by a simple listing of Search Results ordered by a mathematical index of relevance. You can emphasize the best path to the right content. You can add context that helps visitors decide where to go.

Done well, this ability to customize Search Results can dramatically flavor the actual paths users take in response to key queries. In our experience, customized Internal Search will almost always yield at least a 10% improvement in routing efficiency for a key concept. And it’s quite common to get improvements on the order of 30-40%. That’s a dramatic improvement – especially for sites where Search is the primary navigation mechanism.

In an environment where Search is the first recourse, you should probably never stop customizing search results. This optimization is MUCH more important in most cases than Landing Page optimization – and it should be treated with similar respect and sophistication. That means you should always be working in two directions – adding customization to more search terms to expand the breadth of your customization; and testing variations on the customization of your most important terms to improve the quality of your customization.

For many sites, you’ll never, ever be finished optimizing Search; because you’ll never be finished customizing it.

What’s the measurement implication to this? The most important is that you’ll need to establish measurability around customization. There are two aspects to this that MUST be in place. You need to insure that you’re tracking customization impressions – the number of times a specific customized result is shown to a user. You also need to insure that you can track customization clicks - the number of times a visitor chose your customization.

Achieving this is somewhat tool dependent, but the basic technique is easily grasped. To capture customization impressions, you need your Search Results page to populate a web analytics variable with the name of the customization. This should be done when the Search Results page loads. This will provide the impression count.

Getting customization click-through is trickier. Remember, you can’t use Active Viewing and you can’t necessarily track links or next pages either since these are often duplicated in the "organic" results. What we recommend is coding all Search customizations with an OnClick handler that will record a Custom Variable or a Link Click in the analytics solution. Doing this will provide the click count. If your customized suggestions include a variety of links, it's also a good idea to make sure you can track which types of customized links are most often used.

When you have the customization impressions and clicks by customization type in your Web Analytics tool, then you can answer the key customization issues: how often was a specific customization selected, how often were other results selected, and how did the behavior of each alternative actually compare.

I’m confident, based on our experience, that most sites will find a lot of value in the customization (and measurement of customization) of Internal Search. This is one of the most common "slam dunks" in web analytics – at least for sites where Internal Search is important.

That brings this series on measuring Internal Search to a close. Taken together, these posts outline a method by which ANY analyst should be able to do a pretty good job measuring and helping optimize Internal Search.

I’m going to take up SEM analytics next – a topic area that I’ve always loved and which can be even richer than Internal Search. In addition, look for a special post this Tuesday or Wednesday. We’re going to be announcing something I’m very, very excited about!

[Stop here unless, like me, you are Soprano’s fan]

And speaking of teasers, I can’t wait for tonight’s final installment of the Sopranos. I’m hoping Tony makes it through alive! It’s probably not kosher to admit it, but I still like Tony at least a little. If this last season had been all about Tony’s success and power, I wouldn’t have minded seeing him fall in the last episode. That would have been poetic justice and more. But can anyone who has watched the show for years be happy seeing Phil Leotardo triumphant? This isn’t life, it’s art. And in no art that I can get behind, can Phil Leotardo end triumphant.