Modeling Traffic

A Different Approach to Good Reporting

I’ve written before that Management Reporting is one of the most difficult tasks in all of web analytics. It’s an area where – as a company – we’ve struggled mightily. Nor is it the case that when I see other reporting styles and implementations I’m very impressed. Mostly they are worse, often much worse, than Semphonic efforts that I judge failures.

I’ve also written previously about why I think so many efforts at reporting are misguided. They tend to over-focus on identifying individual KPIs instead of describing real-world situations. And when they do describe a real-world situation, they generally lack the intelligent context to make clear to the decision-maker what’s driving change.

We’ve done some work recently that I think addresses both these concerns and has produced some reporting that is significantly more interesting.

I’m going to illustrate this work with a report on Traffic. Yes traffic.

Traffic is nothing to sneer at. Visits to a site may not be the type of KPI that you are supposed to focus on. But in my experience, it represents a crucial real-world fact about your online business opportunity. In addition, I get more questions about changes in traffic than almost any other single variable.

Suppose your report set includes a nice chart like this:

When a marketing manager sees the July to August, August to September, or September to October change, a perfectly reasonable question is generated. Why?

It’s possible that the consituents of an answer are already present in the report set.

Changes in visit traffic may well be driven by improvement or decline in various sources. If the report set also contains a report on marketing channels, the seeds of an answer might be there:

A chart like this contains a chunk of the data necessary to undertand the traffic variation shown in the 1st chart. But it certainly doesn’t make it easy to answer the question. Nor is all of the necessary data contained in this one place.

We can see, for example, that the Direct Traffic is mirroring the overall pattern – two months of decline and then a recovery. Perhaps it’s the causal cuplrit. But what’s causing the Direct Traffic to change?

To understand that, a manager might look for a different chart somewhere else in the report. Making the assumption that direct traffic is driven by offline marketing and repeat visits, the manager might look for something like this:

We can see that there was indeed some decline in visits per visitor in August. But that September actually went up instead of down. So changes in Loyalty may have been responsble for some (how much?) of the decline between July and August but can’t possibly have contributed to the decline between August and September.

Going back to the source chart, we can see that the decline from August to September looks to be biggest in Organic Search. So perhaps we would drill down into a detailed report on Search to see a comparison of August and September.

Google and Yahoo are both down - so that may be the driver for the monthly change. But it’s interesting to note that some other engines are actually up – forming a small counter-trend.

Now, we just have to explain the upward tend in October!

You can see that this is getting to be a lot of work. You can also see that it would require lots of charts and tables and fair amount of smarts to get an answer about what’s driving traffic change. That will lead to reporting bloat – and most users will wonder why they are drowning in so much apparently useless data.

To combat this, takes a very different approach. The essential idea is to build an analytic model of a concept like Traffic (or Conversion or Satisfaction or Efficiency or Engagement) and then embody that model in a report.

To do this we dump all of the relevant data for the model into a working spreadsheet. Then, we create a (fairly complicated) Excel Macro that processes the data using the analytic model. It then populates the report spreadsheet with the core numbers being tracked along with the analysis of the key causal factors and counter-trends.

The beauty of this approach is two-fold. First, a good deal of analytic complexity and intelligence can be built into the model. This can prevent a decision-maker from misunderstanding or missing the contribution of key factors (like visitor loyalty) on traffic. Second, it allows the report to encapsulate all of the key information in one simple presentation that provides actual immediate and well thought-out answers to the inevitable questions.

Here’s an example of the output for the Traffic Model from this situation:

The model captures not just the trend in traffic; it immediately identifies the factors that drove the trend, the extent to which they contributed, and any counter-trends that were also in-place. To get the same intelligence from any non-model based report set would take big chunk of work and a fair amount of knowledge about web analytics. The two things that a good report set is supposed to make unnecessary!

The more I work on reporting, the more I think that the distinction between analysis and reporting is meaningless. Because unless you embody good analysis in your reporting, you’ll never deliver a good report set.

Comments

[Ben Krull]

I do a lot of reporting like this and have found myself using graphs with bullet points of analysis to explain the changes. It often ends up being very wordy because of all the factors involved.

I like your usage of the "% Change Explained" field. As a web analyst, it is easy to get caught up in looking for all the reasons that a metric has changed, but it is often not worth the time to find the last 10%.

[Florian Pihs]

Thanks Gary, This is very helpful. I have to agree with Ben, graphs and bullet points are standard. Bloated reports, too ;) One question and one recommendation.
a) Any way to get a sample spreadsheet?
b) I was reading 90% of the post thinking "same old, same old..." Why hide all the good stuff in the end? Putting the first graph in front would have hooked we from the first second. Then you could have gone into details.
Regardless. Great post. I will have a try at creating a similar chart :) Oh, the beauty of correlation.

[Nate]

Great post! I really liked the Traffic Model that you've created . It makes the report much easier to read and understand then if you show endless charts that explain the changes.

It would be great if I could see an example of what you've created. Any chance you could send it to me?

[Paul Betts]

Great article distinguishing the real information that can and should be gained. Educating the consumer is such a large part of the reporting.
I also would be interested in receiving or hearing about getting that data out of the server stats (or client side?) and your preferred analytical package that provides that (by article or email)... and of course if it is not too much to ask that 'fairly complicated' macro spreadsheet. I guess I believe in Father Christmas. Have a good holiday break if you take one.

[Matt Gershoff]

When you speak of models do you mean probabilistic modeling? How do you factor in one off events (media event, competitor actions, ect.) that might be either directly or indirectly (through your explanatory variables) influencing your visits?