Measuring Search Engine Marketing (SEM) as a Channel

Part 7 in a Series on Web Analytics and Search Engine Marketing Programs

My last post - some thoughts about Google Analytics inspired by Eric Peterson's thorough demolition of Brandt Dainow's post about GA - garnered me nice comments from both Eric and Google's Analytics Evangelist Avinash Kaushik. Which, frankly, seems wrong. At least ONE of those two really should have hated it.

I've been posting quite a bit recently (just seems like there's a lot to talk about) so if you are used to my "Sunday" cycle make sure you check out the recent posts. I actually have a bit of a back-log of topics to cover - very unusual! For now, I'm returning to my series on SEM Analytics - which is a lot less controversial than some recent topics but is, I hope, correspondingly more useful!

The first six parts of this series dealt largely with setting up and understanding the framework for doing Search Engine Marketing analytics. In this and upcoming posts, I’ll tackle some real-world SEM analytic studies. A previous post already covered analyzing overall SEM traffic (since that’s important for understanding the role Search plays on a site). I'm going to take something of a top-down approach; starting with some of the key studies for understanding Search as a channel. Next, I’ll cover time-based studies – looking at the longer term impact of Search sourcing. Following that, I’ll cover optimization analysis; these studies focus primarily on measuring the quality and nature of SEM visitor interactions with a site. After that I’ll cover some types of analysis that can help support SEO engagements. I’ll try to finish up with a nice summary of SEM Analytics – and maybe some further thoughts on the role of web analytics in Search Engine Marketing.

At Semphonic, we first started thinking seriously about Search as channel quite awhile back when one of our clients asked us to measure the impact of a "dark" period in their PPC program. For various reasons, their PPC program was shut-down for an otherwise typical two-week period. It was frustrating, but as it turned out, some good came of it. Because this two-week window gave us a very nice, unplanned view of what happened with traffic and conversions when the PPC program wasn’t running. Pay-Per-Click advertising has become such a significant driver of traffic for many sites that marketing people are afraid to stop doing it. That makes it difficult to measure its impact unless you assume that all of the traffic you get from Search is unrelated to other sourcing mechanisms.

When we looked at the traffic and conversions for the site in question, we found that their volumes had indeed dipped during the dark period. However, the drop was only about half what we would have expected. We checked direct sourcing and other campaigns to see if there was other activity compensating for the loss of Pay-Per-Click. What we found was that Organic Search Traffic had gone up; and it was up significantly – in some cases by more than 50%.

When the PPC program came back online, the Organic traffic went right back down to historical levels.

This was our first experience with organic cannibalization. And it opened our eyes to the extent to which PPC programs CAN interact with other channels. It also led us to question the conventional wisdom that PPC campaigns SUPPORT organic traffic. What we’ve found in the intervening years is this: adding a PPC campaign will always drive at least some INCREMENTAL volume. We’ve never measured a case where this wasn’t true. This fact is the basis from which many of the Search Engines claim that PPC traffic supports organic traffic. However, the actual effect of a PPC program on an organic program varies widely. We’ve measured everything from significant cannibalization of organic clicks to virtually no interaction to actual, real support for organic traffic.

The most common case we’ve measured seems to be mild to moderate cannibalization. Meaning that in most, but not all cases, you might expect your organic traffic on shared-terms to decline modestly when you add a PPC program.

What variables affect this outcome? We’re not really sure. Some that definitely matter are the positions of your organic terms (if you don’t have high-ranking then you should expect a very low level of interaction – positive or negative); degree to which you are buying Brand Terms; clarity of your organic listings, extent to which your organic and paid listings are "coordinated" (I got this from Bill Hunt at Searchnomics), and the strength of your brand. Piling all of these factors (and probably ten other ones) together makes it pretty much impossible to know how you’re going to come out – which is why it’s important to measure.

How do you measure organic cannibalization? It’s pretty trivial. The easiest way is to have a fairly extended dark period. Ideally, the dark period should be somewhat longer than the sweet spot for prospect conversion. You need to make sure you aren’t changing other campaigns during the dark period and you should also pick an historically flat time of year. You don’t want to pick March if you’re a tax preparation firm.

Measure organic traffic and conversion by engine for a period similar to the dark period. Then repeat the exercise during the dark period. It’s a good idea to also measure direct traffic and other online campaigns. Paid Search doesn’t just interact with organic – it can effect direct traffic and even banner traffic. You’ll also want to look at changes in response rate for these channels where possible. This will guard against hidden changes in total impressions that can bias the results.

By comparing the rate of Organic and other source traffic before and during the dark period, you can get a solid measure of potential cannibalization/support. If organic traffic increases significantly during the dark period, you’re likely cannibalizing organic traffic with paid. If organic traffic declines, you’re supporting organic traffic with paid. Understanding which is your real situation can make a pretty big difference in evaluating the effectiveness of your Paid program and in making allocation buying decisions within your Paid Keyword portfolio. Be sure to track conversion rates for channels as well. Some studies suggest that combining programs can boost conversion rates.

You don’t have to have a full dark period to test for cannibalization / support. You can do the study on a single PPC Campaign, Ad Group or significant Keyword. Just make sure you are setup to track at the appropriate level.

Of course, interactions between channels aren’t limited to PPC and Organic sourcing. I hinted at that above – any online channel may be either cannibalizing or supporting (or even some of both) others.

Of great interest in the last year or so has been the interaction within a single channel. Specifically, how users use Search as a process. A number of studies have documented a general tendency for searches to shift from sourcing on non-brand terms to brand terms as they move through the sales cycle. This can have a dramatic impact on your understanding of Ad Group and Search Effectiveness if you are using visit-based or "last-source" based campaign tracking.

Tracking search as a process is a non-trivial exercise and often requires some special setup. If you’re not concerned about the order of events, it is fairly easy in most tools to create a visitor segment based on one or more Search sourcing events. Apply this segment and you can see the number of times visitors sourced from other channels as well as Search.

This same segment will allow you to calculate the number of Search Sourced visits per visitor. These two metrics (non-search visits per Search Source visitor and search sourced visits per Search Source visitor) will give you a much better understanding of search as a process.

But to really understand what’s happening, you need to attach a time component to either your conversion or response measurement. There are many different ways to do this, some tool dependent. If you can create campaigns in your tool, you can setup a campaign for a specific time period. You can then use Response to that campaign as the criteria for a VISITOR segment that includes significant chunks of time before and after the Response. Now, you can compare the types of Ad Groups, Search Terms and Sourcing mechanisms before a particular campaign response (or conversion) AND after.

If you can’t create ad hoc campaigns in your tool, we suggest tagging key events (like campaign sourcing) with a date in a custom variable. Adding a date in a custom variable allows you to use that date or a date range to build a visitor segment identical to that discussed above. This is a very powerful, general tagging strategy that we almost always recommend to our clients.

This is an important technique and, from past experience, I know it isn't always clear to people. So here's a step-by-step recap of what you do:

1. Create a custom variable that contains the date YYYYDDMM when a conversion occurs (for example).
2. Now create a VISITOR-based segment where the criteria is that they have a value for that Custom Metric between two specific dates (say April 1-7th).
3. The timeframe for the segment should include a much wider range of data (say Feb. 1st to May 1st).
4. Because of the custom variable criteria, you know that all the visitors converted in a specific week (1st week of April). By applying the segment and studying other time periods, you can see how they sourced in the week of conversion, the week before conversion, the month before conversion the month after conversion, etc.
5. This technique can be extended by combining criteria (PPC sourced during period and converted during period) and using different dates.

This simple trick lets you generate all of the following: # of visits and # of visitors who sourced multiple times from PPC prior to a conversion or response; # of visits and clicks by Keyword and Ad Group from PPC prior to conversion or response; # of non-PPC source visits prior to a conversion or response; and, of course, all of these metrics for what happens after a conversion or response as well.

There is a boat-load of stuff you can do this with data. First, you can figure out whether your conversion optimization needs to be web analytic based not SEM tool-based. SEM tools will almost always use last source as the conversion optimization strategy. That may be fine but it may also be badly flawed. By looking at pre-conversion sourcing, you can also gauge the extent to which a channel is an ORIGINAL source. Let’s face it, it’s harder to get someone to your site the first time – and they are much less likely to use an alternative source. But subsequent visits have, at least, a better shot of being sourced via other channels. So understanding which channels drive the most original sourcing is vital for optimizing your channel balance.

These numbers can also be used to help understand what happens after a conversion or response. PPC Buyers tend, erroneously, to assume that everyone they source is a 1st Time Prospect. You may find, instead, that most of your PPC sourcing is existing customers looking for support or job-seekers looking for careers.

You might also find, for instance, that PPC prospects, once they convert, always visit your site via PPC. Depending on the circumstance, that may be a minor nuisance or a significant cost factor. If you have tight margins (as many publishers who arbitrage PPC do), then a visitor who always sources from Search is much less likely to be profitable than a visitor who only sources from Search initially.

If you’ve coded time as a variable and you have one of the more powerful current generation analytics tools, you can map out the "search process" with a fairly high degree of accuracy. But even if you’re limited to a basic "before" and "after" segmentation, you can get a much improved picture of the degree and manner in which a channel is used repeatedly by single visitors.

In Part 8, I’ll cover two more types of channel analysis – studying the over-time performance of channels to get clues about the potential opportunity and measuring mass media advertising using web sourcing and usage.

Other Posts in this Series: Introduction, Searchnomics Issues, Getting Setup for SEM Analysis, SEM Data vs. Web Analytics Data, High-Level Search Engine Reporting, Analyzing Search Traffic in more Detail, Measuring Search Effectiveness for eCommerce Sites and Measuring Search Effectiveness without Conversions.