SemAngel

No West Coast Bias Here

After my previous post about a pretty sweet deal – a free upcoming Unica Breakfast in San Francisco with Akin Arikan, Anil Batra and myself presenting on advanced visitor segmentation and targeting, Jacques Warren left a little comment wondering whether there were sweets for anyone outside the Bay Area. For breakfast, you’d probably have to talk to Unica, since they are paying for the event. And I’m not sure if I can do much for Jacques. But if you live in Washington, D.C., I can offer something almost as good. I’m presenting with Alex Langshur on understanding and segmenting your visitor audience at the Web Managers Roundtable in early March.

Early in 2008 I presented to the Roundtable in Washington D.C. on web analytics reporting. That was a long and intensive session in which I laid out many of the problems we see and experience with online reporting and showed how we at Semphonic try to do better (not always successfully).

I’m going to be presenting there again on March 11th and this time I’ll have it quite a bit easier - being partnered up with Alex Langshur. Alex is one of the leading figures in public sector analytics and he and I will be giving our thoughts on understanding and segmenting your online audience. There’s a very small fee for the event but you will still get your choice of croissants or muffins! 

Here’s the short description of the presentation and the link to register.

Knowing your audience is the cornerstone of all market research, and it is the one task that lies at the heart of all meaningful attempts to measure your success. In the anonymous environment of the web, however, your audience can remain invisible and unknown. And when you don’t understand who they are or what they’re trying to do, designing an effective site, delivering appropriate service, and judging success can become all but impossible to do.

In this program, Gary Angel of Semphonic and Alex Langshur of PublicInsite will show you how to overcome this fundamental dilemma. They’ll walk you through the key steps in building hidden audience segmentation, cover all the tools you need to do the job, and show you how and when you should be using each.

I’ve never presented with Alex before – but I’ve seen his eMetrics presentations several times and enjoyed them (and our conversations afterward) very much indeed. If you live in the area, it would be great to see you there!

The Omniture API – Putting Your Own Spin on the Data

At last year’s Omniture Summit – yes the 2008 version – one of the most interesting announcements was the release of a Reporting API for SiteCatalyst. Like red wine, technology announcements generally require a certain amount of time to age into a respectable level of maturity and the Omniture Reporting API was no exception. But late last year, after playing on-and-off with the API, we at Semphonic decided it had reached a level of maturity where it could be successfully used.  It’s still early days – the API is beta and its deployment model a little fuzzy – but you can work with it reliably and produce interesting products. This blog isn’t about programming, and I don’t want to get ridiculously technical. So what I’m going to do is provide some background on the API. What is an API, when you might want to use the Omniture API, how it works from an implementation perspective and a brief primer in how the Reporting API, in particular, is actually used.

So what is an API?
API stands for Application Programming Interface – and it’s really nothing more than a way for one program to communicate with another. APIs are common in all types of technology applications and programmers use them all the time.

So the very first thing to understand about APIs is they are for programmers. If you aren’t a fairly technical person, you won’t want to use an API yourself though you may still want to know what they can do.

In these internet everywhere days, most APIs are a special type of API called a web service that is used to connect one program to another across the internet. Most of the Omniture APIs are web services. Notice that I used the plural tense there – because the Omniture API is actually several very distinct pieces and they don’t all function quite the same way.

What can the API do?
Here’s where we get into the different pieces of the Omniture API. The Omniture API includes web services for doing administrative tasks (essentially anything you might do from the SiteCatalyst Admin Console you can also do programmatically), accessing most SiteCatalyst data (this is the Reporting API I’ll talk about in my next post), accessing some Discover data, and for accessing the Data Warehouse (submitting or monitoring requests). There are also APIs for SAINT and Data Services that are very early days. In addition, Omniture has a long-stranding Data Insertion API. This isn’t a web service – it’s a simpler interface that allows you to use XML Post requests to send data to a report suite.

You should think about using an API whenever you need to either get Omniture data into some other program/system or you need to provide additional automation around some common Omniture task.

Let me give you an example. When we perform data quality audits for our customers, one of the things we check is how the Administrative items are configured. If you have a single report suite, that’s easily done in the Admin Console. But if a client has twenty, fifty or five-hundred report suites, it’s a lot of navigation time wasted just getting their information.

You can use the Omniture Administration Web Service to dump all of the settings for every report suite in an account to an Excel spreadsheet. It’s handy. And, in fact, there’s a sample application on the Omniture Developer site for doing just that.

Here’s another common problem. You have a system to report on that includes both offline and online data. You need to send out reports daily. Since the information you want is in Omniture and some other system (probably a database), you have to manually combine the two.

But now, you can pull the data from your internal system first, then use the Omniture Reporting API to get the online data you need. Your program can combine them and automatically send out a report that contains both types of data.

So the basic rule of thumb is this: if you need to combine data from two or more systems including Omniture Suite, if you need to move Omniture information into some application other than Excel, or if you need to automate some task that isn’t well handled by the Suite, then the APIs might be a good bet.

What Programming Languages can you use?
Since the Omniture APIs are mostly web services, you can use any language that will consume a web service. These days, that means you can use pretty much anything.

If, like Semphonic, you are Windows Shop, I’d strongly advise you to use VS 2008 languages. The security model used by Omniture is not well supported in earlier versions of the VS .NET languages.

How much does it cost?
Here lies the rub. Nobody knows. Right now, Omniture is using a token system. Each account gets a certain fixed amount of tokens free each month. Each request takes up a certain number of those tokens (based on the request). When you run out of tokens, your requests just start to fail.

I know of no policy for buying tokens and I don’t think Omniture has fixed the actual number of tokens that will be available or what the price will be. I’m going to devote a whole post to the “token” issue because it’s really the biggest problem with the API.

Is it hard to use?
Not really. Given that you are programmer, it’s very similar to most other web services in conception and execution. Most programmers will be able to download a sample application and begin using it almost immediately.

Does this mean I can get all my Omniture data or can access stuff I couldn’t before?
No. This is probably the biggest misconception about the API (I get this about the Data Warehouse too). You don’t suddenly have access to every record and you can’t run SQL like queries against the data. The API is NO substitute for the data feed. Using the Reporting API, you can get access to most (but by no means all) of the data available in SiteCatalyst. It’s pretty much like Excel Integration. So if you are moving data to Excel, it’s unlikely you’d get any benefit from the API and you have a much worse usage policy (tokens) to deal with. Nor does the Data Warehouse API somehow open up any capabilities – you just use it to automatically submit or check the status of requests. That’s it. The API is not equivalent to warehousing the information and it will generally give you less not more access to data than SiteCatalyst, Discover or the Data Warehouse.

Where can I find out more?
The first, best and pretty much only resource for Omniture API information is the web site http://developer.omniture.com. You will need an Omniture account login to access the documentation there.

I think that probably covers the most common questions about the Omniture APIs. In my next post, I’ll walk through using the Reporting API and show how it works.

Advanced Visitor Segmentation or Do you prefer muffins or croissants for breakfast

Late last year, I teamed up with Akin Arikan and Anil Batra to present a Web Analytics Association webinar on Advanced Visitor Segmentation. For that 50 minute broadcast, we each trimmed, pounded, massaged, and perhaps even butchered what amounted to three hour-long presentations into 15 minute segments. But even if the resulting presentation was denser than a triple chocolate flourless cake, I thought it worked reasonably well. 

I’m particularly gratified to get an opportunity to reprise that topic with the same team (I suppose you could call us “Quad A”) in more extended and interactive forum. On March 9th, Unica is sponsoring a breakfast here in San Francisco where the three of us will do less hurried versions of our topics followed by an interactive panel session. The breakfast is free (so the economy is not an excuse – accept maybe to attend) and each of these presentations is truly rewarding.

My section, on developing online behavioral segmentations, is one I enjoy presenting and I think it breaks some new ground in web analytics. Anil’s presentation extends the topic to behavioral targeting – a subject in which he is surely one of the world’s leading authorities. Akin’s presentation builds on each of these and extends the view of the visitor across channel and time.

These are important, interesting topics. The venue is great. The event itself should very pleasant and it’s always a pleasure to meet and talk with people who are passionate about web analytics. So please come out and enjoy! The link for the event just went live on the Unica site and you can register here.

Web Analytics Process : Do the Right Thing

[Just as an aside, I'm going to be out at the Omniture Summit from Tuesday afternoon on. Happy to break-out of the sessions and chat with anyone who has a mind - just drop me a line!]

I started this series on web analytics process with a post about some cases where our propensity to misuse information is so high that we may be better off without it. I closed my last post with the simple observation that although good measurement is much better than no measurement, no measurement is preferable to bad measurement.

If that wasn’t true, then perhaps it wouldn’t make sense to build up a set of web analytics processes from a theory of web analytics error. But after ten plus years in web analytics, I have seen, caused, and suffered from every error I enumerated and I know they can be devastating to web analytics efforts.

It isn’t enough for individual practitioners to learn from our own mistakes. An organization cannot afford to rely on our individual willingness to learn – we often disappoint even ourselves in that respect and organizations cannot count on always having the same practitioners. Change is as much a part of our world as error and the two are inextricably bound together.

So the defense against error needs to be embedded in the processes an organization uses to collect, refine, analyze and distribute information.

I’m sure that in these posts I have managed to touch upon no more than a small slice of a very large question. But for all that I have collected a goodly set of recommendations. As promised, they are grouped and summarized here without all the attendant explanation:

Organizational Setup

• Separate your measurement department and make it at least quasi-independent
• Measurement should report to a level above day-to-day consumption
• Vendors should never be allowed to self-measure – performance of individual channels should be centrally assessed.
• Make sure IT and Marketing have regular/defined communication of changes to the measurement team
• Give Measurement teams a say in the selection of survey and test tools
• Give Measurement teams access not just to the information but to the actual use of these tools
• Blend elements of offline research with online research in your organization
• Force some personnel exchange between offline and online research where you cannot blend

Resources

• Have at least one statistical gatekeeper to monitor quality and ensure statistical significance

Analytic Processes

• Force designers and marketers to pre-commit to measures of success for any site change / tool / addition.
• Use Functional methods to prevent undue site navigation and structural influence
• Use Testing instead of analysis where self-selection issues are unavoidable
• Never rely on “point optimization” for any important process or channel
• Your standard analysis template should include a place for “offline learnings”
• You should schedule at least one web analytics project targeted toward offline learnings every year
• Analysis and reporting should focus on marketing learnings – including failures. Don’t just talk about successes. Every campaign and multivariate test should have a “failed but we learned” portion of an analysis or report
• Force analysts to begin every project with a careful and complete walkthrough of the pages / tool they are studying

Reporting

• Never send out reports where you know the threshold of significance isn’t met - this means either building analytic reports or reviewing reports prior to distribution
  
• Build measures of variance and significance directly into your reporting
• Use Analytic reports to ensure some level of measurement sophistication
• Create standardized measures of value that can be used to measure the impact of every channel
• Standardize attribution (and pick a reasonable model) – never attribute simply to first or last
• Don’t over-focus on KPIs – use models to represent complex systems not metrics
• Don’t rely on stakeholders for reporting requirements – requirements must be generated by a combination of high-level stakeholder input and significant subject-matter expertise in web analytics.
• Limit direct access to the tool unless you’ve created a powerful and consistent culture of web analytics.

Infrastructure

• Create regular data auditing processes
• Standardize infrastructure for easier and more reliable maintenance
• Use diverse collection techniques to compare and check data quality
• Don’t rely solely on reporting requirements to drive infrastructure – use a systems based approach to deliver requirements

You may not agree with every one of these recommendations. You may believe that there are good and sufficient reasons for doing quite the opposite. Even if you agree that a practice might help prevent error it may prove too limiting on the productive use of data to work well. A friend of mine has often insisted that the modern role of IT departments is to say “no” to everything. There’s enough truth in that to make me dread a similar fate for web analytics. No one will ever misuse data if no one ever uses data, but that is hardly a goal to strive for.

I hope that I have at least convinced you that paying attention to when things go wrong in your use of data and searching for practical preventatives – having your own theory of error - is an essential part of thinking about your web analytics processes. 

Getting it Right: Over and Over and over And over

It's hard to get analysis right. Even when you do, it's hard to get it consistently right. Good process is very much about protecting ourselves from the things that cause mistakes so that we have a chance to be consistently correct. In two previous posts, I’ve listed eight different common sources of error in web analytics:

1.    Self-Interested Measurement: Finding what you expect in the data
2.    Lack of Statistical Significance: Believing small variations or tiny samples to have much more significance than they really do.
3.    Unreliable data and what to do about it: Trending bad data or “getting over” critical data quality issues
4.    Siloed Optimization: Improving one channel at the expense of others
5.    Metric Monomania: Over-reacting to changes in individual KPIs
6.    Tactical Focus: Concentrating on micro-analysis and missing the important information.
7.    Self-Selection: Reversing cause-and-effect and thinking related items are necessarily causal.
8.    Navigation Structural Influences: Reversing cause-and-effect and evaluating content performance without factoring in the influence of site structure.

In my last posts, I covered the first three topics and topics 4&5, today I’ll pick up from there.

Tactical Focus
Analytics is a very tactical discipline. Numbers most often suggest very discrete problems and they often get us thinking about small details. That’s actually a good thing in most cases. While you may not be able to see the forest for the trees in most web analytics, at least you can avoid running into tree after tree!

But if a strong tactical focus can be a strength of web analytics, an overweening concentration of small details can often contribute to huge missed opportunities. With web usage now so broad and representative, the web is a powerful vehicle for understanding your customers in every channel. And if a significant portion of your business is offline, failing to use web analytics to obtain a broader understanding of your business is a terrible waste.

Even within the context of the online business, too much tactical focus can often lead to siloed optimization and missed opportunities. Testing hundreds of small variations with a multivariate test tool is great. But if you’re not testing major creative alternatives and fundamentally different messaging strategies, you may only be finding the best variation within a basically unsound approach.

There is no guaranteed way to get the right balance between a healthy concern for details and a broader strategic perspective. There are, however, a couple of approaches that may help.

First, don’t completely silo your online effort – especially when it comes to consumer research. Traditional primary research materials should always be distributed within the online environment and online research should be regularly pushed back into the offline channel world. Making sure you have some cross-organizational meetings to focus on the combined research can greatly facilitate this. It’s also important to cross-pollinate people. Forcing some movement between the online/offline marketing and measurement organizations is extremely valuable. Creating organizational paths that facilitate this from a career perspective is absolutely necessary.

In planning your analytics, it’s also worth making sure that you have at least one effort specifically dedicated to offline learnings. So if you’ve mapped out an analysis plan and everything is about the web site, go back and try again!

If you’re creating an analysis template, always include a slide dedicated to offline/operational learnings. It may end up being discarded a fair amount of the time, but it will keep your analysts thinking about the possibilities.

Lastly, make sure you are disseminating your real learnings and not just bragging about your successes. I’ve written before that one of the biggest learnings for marketing people is what didn’t move the dial. If you are just reporting about how much of a lift your testing or analysis got – and not helping marketers understand what worked and what didn’t, you are missing out on the most important parts of the story.

Self-Selection
Web Analytics is greatly complicated by the fact that most correlations between pages and events are less a matter of content impact than visitor self-selection. Your “Sign-up Now” link is effective because the group of visitors who clicked on it were (mostly) ready to “sign-up now.”

The confusion of correlation with causation is an incredibly common problem with nearly all types of analytics – not just with web analytics. But we certainly manage to make the most of our opportunities for error.

Building good analysis processes is never going to fully prevent this type of error – nothing will – but it sure can reduce the number of your mistakes.

The most important counter-balance to self-selection in analysis is appropriate use of testing – especially controlled testing. There is no better way to eliminate self-selection from an analysis than to run a test that controls for the population in question. In some cases, this can be achieved with rigorous use of segmentation within the analytics tool. But nine times out of ten, the easiest way to ensure that Self-Selection isn’t driving your conclusions is to run a controlled test.  You can spend many valuable analyst cycles trying to protect an analysis from self-selection or you can just build a quick test and measure the actual results.Once you’re set up for it, testing is nearly always a lot easier and more reliable.

Similarly, integrating your Voice of Customer(VOC) data into your behavioral stream can help protect against errors of self-selection in each. Many an analyst can avoid basic errors in self-selection by beginning with an understanding of the intent and characteristics of the visitors who are being studied.

Sadly, most organizations have chosen to isolate multivariate testing AND voice of customer research from web analytics. The measurement group should have a significant say in how each is used – and should be able to organize both tests and surveys in conjunction with marketing. What’s more, the capabilities of each of these tools (especially VOC tools which are often delivered in shockingly limited configurations that make their use in analysis almost impossible) should be evaluated from the perspective of the measurement /analytics team not just the marketing team.

Analytics professionals need a fundamentally different set of survey and test & target tool features than do designers. Failure to recognize this and integrate their requirements into the tool acquisition process will almost certainly lead to critical missing capabilities.

Navigational and Structural Influences
Your web site isn’t a pure research vehicle designed to neutrally solicit customer interests and attitudes. It’s designed to influence, motivate and drive the visitor to a specific set of actions. In practice, of course, that’s a good thing. But it means that the web site cannot be treated as a pure research vehicle – it has a powerful effect on what visitors can and choose to do. This simple fact is often missed in analysis.

One of the most important process steps in protecting against structural influences is to have a sound analytics methodology. We use a Functional approach for many basic site analysis projects – and one of the virtues of the Functional approach is that it often helps protect the analyst from errors caused by navigation and site structure.

By grouping pages into types that often capture the navigational distance between outcomes and by focusing the measurement on type-specific metrics, Functionalism greatly reduces two of the most common sources of web analytic error.

Almost as important is making sure that your analysts don’t just look at the numbers. Every analysis should begin with a careful examination of the pages / tool to be studies. Again, Functionalism encourages this step by forcing the analyst to classify pages by their type – a step that greatly improves the analyst’s understanding of each page.

And, as with self-selection, use of controlled testing and integration of VOC data can help the analyst resolve potential questions and illuminate potential problems.

Summing Up

Well, that’s about it. As promised, I’m going to distill my various process recommendations down into something much shorter in my next post. I’ll try to just outline the various process and organizational structures/processes I’ve suggested and group them into categories for easier consumption. 

As I’ve said all along, building organizational processes from an error theory is only going to yield part of the necessary machinery for success. There is more to success than avoiding mistakes! But I do believe that taken together, the process suggestions I’ve outlined would greatly improve most organization’s ability to do quality analysis and measurement. Because when you get right down to it, no measurement is better than bad measurement and good measurement is much better than no measurement.

Getting it Right: Over and Over AND over

It's hard to get analysis right. Even when you do, it's hard to get it consistently right. Good process is very much about protecting ourselves from the things that cause mistakes so that we have a chance to be consistently correct. In two previous posts, I’ve listed eight different common sources of error in web analytics:
1.    Self-Interested Measurement: Finding what you expect in the data
2.    Lack of Statistical Significance: Believing small variations or tiny samples to have much more significance than they really do.
3.    Unreliable data and what to do about it: Trending bad data or “getting over” critical data quality issues
4.    Siloed Optimization: Improving one channel at the expense of others
5.    Metric Monomania: Over-reacting to changes in individual KPIs
6.    Tactical Focus: Concentrating on micro-analysis and missing the important information.
7.    Self-Selection: Reversing cause-and-effect and thinking related items are necessarily causal.
8.    Navigation Structural Influences: Reversing cause-and-effect and evaluating content performance without factoring in the influence of site structure.

In my last post, I covered the first three topics, today I’ll pick up from there.

Siloed Optimization
In discussing the many problems created by self-interested measurement, I suggested that letting vendors measure their own performance was nearly always a bad idea. It turns out to be bad not simply because of self-interested measurement, but because vendors are usually focused on just one small piece of your business – your SEO or your PPC campaign or your Display Advertising.

Measuring success when focused on a single area can often cause significant problems in optimization. The thinking often goes like this: “I’ll optimize my PPC program to get traffic to the web site. Then I’ll optimize my landing pages to get the best click-thru. Then I’ll optimize by order processes to get the best conversion rate.” But treating each of these steps as if they existed in isolation is a fundamental mistake. Nor is this problem unique to vendors. Even where companies are doing all of these steps internally, the likelihood of siloed optimization is very high.

The reason it’s a bad idea isn’t hard to understand. If you optimize your PPC program to drive traffic, you aren’t measuring the quality of traffic you are getting (it’s the landing page and web sites job to convert). When you optimize to traffic, you’ll almost invariably optimize to bad traffic. So your landing pages and web site are getting much worse traffic from your PPC program than they should.

The same process repeats itself with Landing Page optimization. If you tune the landing pages for click-through, you’ll probably find ways to get the most and not necessarily the best visitors to click. That makes the rest of the web site perform worse.

With siloed optimization, you can optimize the heck out of every step in the process and still have amazingly bad results. Indeed, it’s the optimization that’s creating the bad results.

All this is pretty bad, but the drawbacks to siloed optimization don’t end there. It today’s world, most marketing channels interact – for example, PPC and SEO simply cannot be treated as independent. If you treat them as independent channels, your budget allocations in each area will miss potential opportunities to arbitrage one channel over another. We’ve seen plenty of PPC programs than simply cannibalize other channels. It looks like the PPC program is performing well until you realize that most of what it’s doing is simply borrowing visitors from other – potentially less expensive – channels.

So what’s the cure for siloed optimization? First, never use proxies for success that don’t really translate into your real business goals. In SEO, measuring how many words you have in the top 10 is NOT a good proxy for success. It says nothing at all about how much quality traffic your SEO effort is actually driving.

For any marketing channel, you can avoid siloed optimization by focusing on the overall business success of that channel when all the other factors are held constant (like landing page and web site). You should also think about attribution carefully and make sure you use an attribution model that is at least reasonable in terms of crediting channels for success.

In general, getting out of siloed optimization means rejecting any point measurement solutions (like Spotlight tagging or Top Word Counting) and using your web analytics solution or back-office tracking as the ultimate arbiters of success across all channels. It also means that your success measurement across all channels should be centralized in a single place – not distributed out to each organization and certainly not entrusted to self-interested vendors.

You should also be sure that every marketing channel is measured in identical terms and against the same overall measures of success that have been agreed upon throughout the organization. This is best achieved via a formal effort as part of your online measurement to develop and standardize a set of value measurements for everything you want your online properties to achieve (branding, corporate communications, lead generation, ecommerce, customer support, etc.). This standardization of value and attribution should almost always be one of the first things you do when setting up online measurement.

Metric Monomania
It is our business, as analysts, to reduce complexity down to some level of simplicity and to distill valuable information from enormous quantities of data. But in our quest to achieve that distillation, there is always the possibility of over-reaching – of reducing a problem to such simple terms that the information we provide is no longer capable of correct interpretation. In the world of web analytics, Key Performance Indicators (KPIs) have been one of the main tools for distilling valuable information from data. And they've also been consistently one of the most abused techniques - resulting in a mono-maniacal focus on the actionability of single numbers. As I’ve argued extensively and I think conclusively, single metrics almost never have that inherent actionability. When we pretend that we do, we encourage misinformed and nearly always misguided decision making.

What is the solution? Instead of trying to capture complex systems in a single metric, analysts should focus on building reports that help decision-makers understand the important factors driving system performance AND their relationship to each other.

From a process perspective, this changes many of the most common tasks in web analytics. In building implementations, the focus is no longer on finding KPIs and then ensuring that they are supported by the implementation for reporting (a disastrous strategy in general and one guaranteed to produce a poor implementation). Instead, the focus is placed on understanding what key systems need to be described and ensuring that all the metrics important to understanding that system are captured.

In reporting, this means that the onus of requirements gathering is removed from stakeholders. No process is less productive than the misguided solicitation of “reporting requirements” from stakeholders. Based on a ruinous and deeply flawed analogy with traditional BI systems, web analysts have adopted the idea that stakeholders are supposed to be able to tell them what they want in reports. These requirements are then baked into the implementation and used to produce reports.

It’s all wrong. Individual metrics (no matter how “Key”) aren’t all that is significant. Stakeholders don’t and shouldn’t be expected to understand whole systems – that’s the analyst job. So instead of trying to drive reporting requirements by stakeholder knowledge of KPIs, web analysts should drive reporting requirements by understanding what systems decision-makers need to understand.

This is a fundamental change in the organizational flow of requirements to web analytics practitioners and it considerably raises the stakes on what’s required of those practitioners. It means that the subject-matter experts driving reporting requirements for web analytics are analysts – not stakeholders.This has deep implications for the type of resource necessary to generate web analytic reports; instead of a report geek, you should be using real business analysts for this work.

In practice, I also think that this means organizations should concentrate on Analytic reporting. Throwing out all those pimped up dashboards and pretty tabular reports may be a hard thing to do. But instead of facilitating good thinking about web analytics and decision-making, they are more often a fundamental part of the problem – encouraging error instead of protecting against it.

Finally, and as a corollary to this, organizations should think very carefully about who they provide access to the analysis tool. The tools available today are very metric focused. They demand careful analysis and considerable knowledge if you are trying to understand and properly evaluate any real-world system (be it traffic, conversion, engagement, etc.).

Because of this, I believe that access to the tools should be restricted primarily to analysts and true power-users. What’s more, power-users should only be given access after you’ve created a reporting framework that encourages and disseminates good-thinking about web analytics. A fundamental goal of reporting should be to provide a truly useful framework for talking about and using measurement in the online world.

Until your organization is mature enough to have produced a system-based framework and made it a part of the conversation and culture, it’s imperative that tool access be very limited.

In my next post, I’m going to tackle the last three issues on the list. After that, I think it will be worthwhile summarizing all of the process suggestions I’ve made into a single list that can be incorporated into a broader process initiative!