Part IX in an (epic) Series on Functional Web Analytics
(A quick note – I’m in NY next week teaching a class on Functionalism at the WebSideStory ActiveInsights Conference. If you’re an HBX customer - I’m afraid you have to be - and you’re interested then go to: http://www.websidestory.com/company/news-events/active-insights/overview.html)
"They Seek Him Here, They Seek him there – those Frenchies seek him everywhere…."
Like the Scarlet Pimpernel, good analysis of Search Functionality can be hard to find. I just finished an excellent post by Matt Lillig on his new analytics blog (http://mattlillig.blogspot.com/) about looking at Internal Search, and it inspired me to bump Search higher up on the ladder of Functional components and talk about some of the peculiar issues associated with its measurement.
Matt’s analysis focuses on the analysis of failed keywords – and pretty much everything he has to say is dead-on. It sounds like the site he was working on had extraordinarily bad performance in this regard. I don’t think I’ve ever seen numbers quite so high for failed searches – and for a retail site bad search functionality is truly disastrous. This analysis of failed keywords is one of those very simple "checks" you have to do - it won't always or even often yield dramatic results but when it does the payoff is immediate and enormous!
I’ve found that there are three types of Search Analysis that are fairly common – and each is useful – though also limited in its own way. One is Matt’s analysis of failed searches. This analysis is typically a slam-dunk in most web analytics solutions. It does, however, require that you pass the number of results returned on the search results page.
A second type of analysis focuses on successful searches – and is germane to a certain kind of site – one for whom search is a fallback not a first option. For sites that aren’t search focused, it’s often interesting to view what visitors are searching on to spot holes in your navigation. For sites like this, what you want to see is lots of search terms with very little percentage going to each. When you see a search term getting a high percentage of total clicks, it’s often a cue that you should change something about your pages to cue visitors where to find that information. This technique, by the way, is increasingly problematic. We see many cases where the visitors’ first action is to Search – this being the default navigational style for an increasing number of users. So for a growing segment of the population, it doesn’t matter what else you put on the site Search will be their likely navigational choice.
Finally, Search is often analyzed in terms of endpoint conversion. In this regard, Search is primarily viewed in comparison to non-search sessions – with designers trying to analyze whether search sessions are more or less productive than browsing sessions. This analysis is complicated for ad-based sites by the fact that search sessions are nearly always shorter than browsing sessions. Conversion analysis can also be quite interesting and useful – but it’s necessary to put it into perspective. In my last post, I described epiphenomenal effects – and Search can actually be a prime example of this. On both retail and ad-based sites, the visitor who searches often has a fundamentally different mind-set than one who browses.
On retail sites, search behavior is often indicative that the visitor is further along in the sales-cycle and has focused in on a specific product. On publishing sites, a search is generally indicative that the visitor has a particular information need. In either case, the main point is that a simple apples-to-apples comparison of search vs. non-search sessions may be misleading.
None of these, however useful, is the primary focus of Functional analysis. We tend to understand Search as a type of Router Page – essentially competitive with Router pages as a class. Router Pages, after all, are typically the alternative navigation path for visitors. So to analyze Search, we like to look at the classic measures of Routing – how effective Search is in moving visitors to deeper content in the site.
As part of this, one of the key statistics to focus on is how often visitors refuse any of the suggested search results. Refusal can take the form of any of the basic router exits: site exit, back-to-home, top navigation, etc.
Secondary Router measures like re-surface behaviors have their own interpretations for Search. For a typical Router, re-surface is often a function of navigational structure. This is sometimes true of Search – but not always. One of the interesting types of re-surface analysis for Search is to analyze subsequent searches for a specific term.
In addition, you can use web measurement to track how visitor’s usage of links tracks to their position. Significant disconnects likely indicate poor search results. This is a actually a specialized case of real-estate analysis – something we do commonly on Router pages to match the usage links to their prominence on the page. To do this in many solutions, you need to add special onClick handlers to your Search Results to track clicks by position.
One of the distinctive attributes of Search is that it’s site wide. Since Router pages across the site will often perform quite differently, it’s important to be able to compare Search Routes by Site Area. This is tricky to do, since you can’t just use outcomes (next routes will only include successful routes – not failures) and specific Search Terms may (or may not) map cleanly to topics. Even where Search Terms do map reasonably, it’s obviously going to be quite a bit of work to map Search Terms to site areas so that Search to Router performance can be compared.
Some sites solve this problem for us by categorizing Search Terms. As with tracking link position, tracking category may also require onClick handlers in the Search Results page. This greatly facilitates analysis of Search in many respects. Where this isn’t being done, we typically try to focus on a small set of "central" search terms for each site category. Our hope is that by comparing Search Performance for these terms vs. comparable Router Performance we’re getting a reasonable picture of how well Search is doing.
I mentioned earlier how often Search is a first recourse for site visitors – even on sites where Search is not meant to be a primary router. This underscores how important it is to insure that search pages are highly tailored in effectiveness for top searches on the site. When you’re search is being used as a fallback to obscure site areas, this may not be particularly important. Search usually works very well for this sort of thing. But oddly enough, many internal search engines do a much worse job of steering visitors to the best pages when the topic is broad and central to your site.
For this reason, we’ve found that the biggest opportunity in improving search is in the customization of the returned template and results for high-volume search terms. When you customize the Search Template, however, you will almost certainly need to customize the measurement – since search results pages rarely make it possible to understand what area of the page visitors used. So if you are customizing Search, it’s essential to (once again) add OnClick handlers to record which part of the template the visitor actually clicked on.
Measuring Search is a big topic. I’ve barely scratched the surface in this post – but here’s a brief synopsis of the key points: search has some special attributes that are very worth measuring including failed searches, top searches and subsequent search terms; as a functional component, it is most fruitfully thought of and compared to a Router – both in terms of immediate metrics and eventual success; the high cardinality of search terms (the sheer number of different terms used) can make mapping search performance by site area tricky; and the biggest payoffs to search optimization are more often in the customization of templates for key results than in improved "organic" search results.
In my next post, I’m going to once again diverge from Functionalism and deal with a topic that (like epiphenomena) is especially interesting to ad-based sites – namely, how to measure the impact of Google Ads on visitors and revenue – along with some thoughts about how publishers should think about Google (or Yahoo, MSN, etc.) ads on their sites.