Evaluating Internal Search Performance By Topic

(Part IV of a Series on Methods in Web Analysis)

In the last post, I covered the evaluation of Search Performance using Functionalism. The basic idea was simple – Internal Search is a navigational tool that should be judged in comparison with other routing devices in terms of how effective it is in moving visitors to the intended content. The complexity for Search is that unlike most web pages, it isn’t clear what the intended content really is. In the last post, I chose to ignore that and just consider every non-navigation page on the site as an intended route for Search. There’s good logic behind this, but it can mask significant problems and opportunities in search optimization. It is in the very nature of Search to perform differently depending on the Search Term entered. So if you are doing a real deep-dive analysis of Search, it’s essential to evaluate Search performance for the key types of queries on your site.

To do this, first isolate the Search traffic generated by a specific query or set of queries. In most cases, this will involve using your tool’s segmentation capabilities. But first, you have to find the appropriate term(s). Start with a report on Internal Search Terms.

Most web analytic tools will provide this report (if your tool has this report but shows no data, then the tag for the Search page isn’t properly configured). Typically, it will show a list of Search Terms and a count of the times they were used. To find queries of interest, subset this report using your tool’s "Search" functionality or by just scanning through the list. Chances are that for any single category of interest, at least several terms will apply. You may want to choose a single key term or take as many of the highly used terms as your tool will allow in Segment Definition.

If Search is a heavily used component on your site, this can be a problem. If a concept has hundreds of terms with significant volume, it’s probably not possible to use them all when creating a Segment. Content or large retail sites that have significant Search volume may well want to consider a different approach – one that relies on a change to the tag.

Usually, measurement solutions require that at least one special variable be passed from the Search Results Page – the Search Term used. Quite often, you’ll also pass a value for the number of results returned (this is used to determine whether a Search Failed or not).

For most sites, that’s the extent of customization around Search Results. But if you need to do a lot of analysis around search, then consider adding (if possible) a category code to the results. There are two ways to approach this – the first is to categorize the "Searched Terms" – the phase the visitor entered. Some Internal Search engines will let you do this – but many won’t.

If you can’t easily translate the Internal Search Term into a Category variable, here’s another technique. Most content-based sites will tag their pages with Category and Sub-Category designations. This is usually necessary to drive advertising optimization and, of course, good content measurement.

You can use the Category Code of the #1 result returned (or use the Category of the Clicked Result) to code the Search Page by Category. The advantage to using the Category from the #1 Result is that you get a Search Category even when the visitor doesn’t click on any of the results. This technique relies on your Search tool’s ability to identify a reasonably good match for the Search Term with its #1 result.

If you want to record the Category of the clicked result, you’ll add an OnClick handler to each Search Result. This handler should make a dynamic call to your measurement solution (or http server if you’re log-based) – and register the Search Category Event. One benefit to adding OnClick handlers to Search is that you can also record the position of the clicked Search item. Tracking the position of the selected Search Item can be used to measure the quality of your returned Search Results.

If you’ve tagged Search Category, then it’s much easier to create appropriate segments in the measurement tool. If you can’t do this, don’t despair. Even if you can’t bucket terms together to analyze a Category, an analysis conducted using the central term (usually the highest single volume term) is still very worthwhile.

I’ve spent this much time on getting ready to build your segment because once you’ve built it, the subsequent analysis is straightforward. Just apply the segment and track your routing performance (as per the previous post). It’s usually interesting to compare the routing performance of categories to each other and to the Search page as a whole. When you’ve isolated a single concept (like a product) in Search, and you can compare the performance of Internal Search to the corresponding Router Page for that area of your site, then you’re in a much better position to really evaluate how well Search is working.

In addition, this concept-based analysis of Search will often turn up very considerable tuning opportunities. In our experience, the most effective tuning of Search is in customizing the output in response to specific categories or terms. This takes real work, of course, but the impact of customizing your top 20 search results is generally much higher than what you’ll get my replacing your Search Engine with a significantly better one.

By measuring concept-specific routing performance, you’re likely well along on the path to truly optimizing your Internal Search.