In my last post, I described an Analytic Roadmap – a very detailed plan of the analysis projects an organization will undertake – and why I think it’s such an important part of doing measurement well. I talked about it not only for its intrinsic importance but as a lead in to what will be a fairly sprawling series covering many of the key elements that might go into such a plan.
I wanted to go in that direction because it seemed to me that there is surprising little discussion of real analysis. Indeed, I was very struck by a comment I read recently in the Commerce360 blog after the Omniture Summit:
And another interesting note; while all the best minds will tell you it's not about the reports but rather about the analysis, insights, and actions that follow - there wasn't to my knowledge a single session or organized discussion on anything near analysis or the 'process' of gaining insight from of all this tracking. Everything was implied - do this, you'll be able to track that, then "you'll know".
So true. And the comment doesn’t just apply to Conferences. It applies to the vast majority of all our blogging as well. In fact, I find that even when most of us are talking how-to analytics that the discussion might be about some charting technique, data visualization or data manipulation. That’s all well and good, but it takes for granted that you know how to actually use that technique to approach an analysis.
My goal with this blog series is to cover a good swath of analytic projects – and to focus on the key how-issues for that particular topic. To provide direction not only about what goes into the Toolkit, but how to really tackle each item.
This ‘Analytic Toolkit’ is necessarily going to vary by site and tool, but many of the elements are quite common. I don’t really have a single starting place that I think is better than any other. When we build an Analytic Roadmap, we nearly always start with a Functional Analysis. But I've covered Functionalism in detail already, so for reasons largely arbitrary, my first toolkit item is going to be Internal Search.
Web Measurement and Analysis for Internal Search: Introduction
For many sites, internal search is the single most important navigational tool for users. Search is especially important on large publishing sites with lots of archival information and for retailers with a large number of products. For these sites, Search can account for more than 50% of all user page views on a site – and can dwarf in importance even the home page. What’s more, almost all retail sites find that Search is more effective as a tool for converting visitors than traditional directory navigation.
On the other hand, not all sites rely heavily on Search. Many smaller sites don’t support Search functionality. And for large sites with a narrow focus (a Mortgage site, a site for a Company with only a few products, etc.), Search is used more as a fallback – it’s an option for users who can’t find what they are looking for in the basic navigation scheme.
What makes this distinction (Search as a Preferred Navigation Option vs. Search as a Fallback Option) particularly important is that it directly affects the types of analysis that need to be performed. In the first scenario, the goal is to find ways to maximize search results, usage and navigation to increase conversion. But in the second scenario, much of the analysis is centered on keeping visitors from having to resort to Search at all.
What makes Search different than any other page? First, Search is a tool that changes with nearly every invocation – making it the most dynamic page on your site. So analyzing link paths from Search is especially tricky. Second, Search is usually accessible from every page on your site (all functions that are globally available share certain analysis problems). Third, Search has at least two additional data items of great importance: the search term entered and the # of results returned. These both play a significant role in understanding and analyzing Search. Finally, the Search Results page is typically refreshed in a fairly interesting fashion as visitors click on next pages of results – this is a behavior well worth tracking.
One other special note – since Search is most often a text box on every page, your analysis will typically be looking at the Search Results page. The Search Results page can be tracked in most tools Requested Pages, Pathing and From/Next Step reports.
Getting a Handle on how Search is Tracked
Before you dive into any statistics, your first step should be to make sure you understand how the Search Results page is being captured in your tool. Here is a little checklist to use when looking at your Search Results page and the corresponding numbers in the web analytics solution:
Check |
Method |
Page Name |
Check to see if a new Search, an advanced Search and browsing to the next page of results trigger different page names in the tag. When the name doesn’t change, most tools will record the action as Page Refresh. |
Search Term |
Check in the tag or your report suite to insure that Search Terms are being passed and populating. |
Search Results Count |
If your tool supports this or if you are passing a custom data item, check to make sure it shows up in the report suite. Check especially to see if searches that return zero results are handled correctly and identified as failures. |
When evaluating Search usage, you will always have to keep in mind how your site and your measurement tool are treating each key behavior – browsing pages of results, advanced search and repeat searches may result in distinct page views or, more commonly, Refresh views. It’s absolutely vital that you understand the difference.
If you are in the position of setting up tags for Internal Search, then it’s best to do yourself or your measurement team a favor and make sure that each of these behaviors DOES result in a distinct Page Name/Page View. My preferred method is to code browsing page results by affixing the Page Number to the Search Results Page Name. It isn’t always necessary to code every page this way. You can get most all of the analytic value by coding the first set of returned results as ‘SearchResults’ and all subsequent browsed pages as ‘SearchResultsAdditional.’ In our experience, it’s quite important to distinguish between Advanced Search (or types of Search if your site has special search categories) and basic Search. You’ll want to do this for both the first page ‘AdvancedSearchResults’ and all subsequent browsed pages – ‘AdvancedSearchResultsAdditional.’
Evaluating the Importance of Search
There’s a pretty good chance that you already know whether search is a primary navigation tool or merely a fallback mechanism on your site. Still, it never hurts to have real numbers to back up those intuitions.
There are several ways to think about tool usage – the most common being how many page views does it get. There are a couple of reasons why this isn’t ideal for Search. First, your basic Page View counts are often very misleading for Search because of the problems described above. Second, because Search is essentially a navigational tool, you are usually less concerned with its total page view usage. What’s most important with navigation tools is how often they are used by Session and by Visitor and, secondarily, how often per session they are used.
These first two numbers (usage by Session and Visitor) are usually pretty easily obtained. The third (how often per session) can be a real challenge. Almost every tool will give you the Views and Visits for Search (and for your site) right out of the box. Getting a Visitor usage rate may be trickier. First, you have to pick a time period. Most times, we’ll use a monthly number. Chances are your web measurement tool will provide you a monthly Uniques count for the site as a whole. It may or not provide you with a good monthly uniques count for the Search Results Page. There are too many tool differences and workarounds to delve into here, but usually this is an obtainable number.
When you have the usage rates (Search / Site), you’ll probably have a pretty good indication of how Search fits into your current site design. Typically, sites that treat Search as a Fallback Option will show visit usage well under 10% with 3%-5% being typical. Mixed mode sites typically range between 5%-20% - with the higher the number the more clearly is Search a Preferred Navigation Option. Above 20% and Search is clearly one of – and probably the – most important element on the site.
You’ll probably have noticed that how many times Search was used per Session wasn’t tackled in this discussion. Depending on how your tag is set up, it can be a snap to obtain (just divide Views by Visits) or nearly impossible (in situations where ‘repeated use of Search’ and ‘Page Browsing in Search’ are treated identically and recorded as Refresh Page Views).
Fortunately, the percent of visits that include Search is probably all you really need to know to decide how you are going to think about how Search is being used on your site.
But there is at least one other statistic worth thinking about at this level. Even when only a small percentage of visitors use Search, it may turn out that for those users it is the primary navigation option. If everyone thinks of Search as a kind of last resort on your website, this particular statistic can definitely raise eyebrows and is often used to help build a case for getting a better internal Search System. To find out if this is true, you can study when, in the Session, users visited Search.
Different tools provide very different ways to this information (and often not very simple ones). One of the simplest – but least informative – is to use the Avg. Page Depth statistic. While occasionally revealing, this number provides no distribution information and – even when it happens to provide an answer that turns out to be meaningful – you won’t know it’s right unless you do more analysis.
Usually, you’ll need to do that analysis using your tools Event or Path reports. A couple tips to keep in mind – pathing is often either trimmed data or very difficult to consolidate. For this type of analysis, you don’t need big chunks of data. If your site generates lots of search activity you can restrict this analysis to a single day and still get perfectly useful results. Even on Fallback sites, we often see that 30%-60% of Search Usage is the first thing a visitor does (the relevant pattern is Entry Page then Search Results). Having this number can definitely influence perceptions about how and when Search is really used and how important it is to have a good internal Search system.
I’m just getting started on the web measurement of Internal Search. Next time, I’ll discuss optimizing Search Access and evaluating Search performance.
Comments