SemAngel

How do you Measure Social Media ROI?

It’s oh so typical that just when most of us start to feel pretty good about our knowledge and methods for measuring our own web sites, everyone starts asking us about what’s going on in the greater online space. Measuring communities, conversation, and social media is at once fascinating and absurdly challenging. The tools are primitive, the infrastructure almost non-existent, the questions largely undeveloped and even the business goals are poorly understood. It’s just like old times! So I’m delighted to be doing a webinar on measuring social media with my friend Scott Wilder.

Scott and I worked together for the last few years as he built-up Intuit’s impressive online communities. He’s that rarest of breeds – someone who uses social media and technology with true passion and combines that with real business acumen. Like lovers of ballet and football, neither one nor the other is particularly rare, but the two almost never to comingle!

Our plan for the webinar is to start with an examination of online communities and measurement. We’ll cover some of the basics of community measurement then dive into measurement based on both audience and organizational segmentation. This is interesting stuff. If you have or are thinking about a community, it should really help you understand how to measure your success. We also hope to provide some perspective on whether – if you are on the outside looking in – building a community is right for your business.

For many organizations, it’s cheaper and more practical to take the conversation to where your customers already live. The second part of the webinar will focus on the different uses of measurement in this broader online space. Scott and I will talk over how to find your audience online, how to measure the success of your brand-monitoring and customer support efforts, how to measure influencer programs, and how to track impacts on your own sites. Don’t expect miracles – this stuff is hard. But we’ll try to provide real solutions and along the way we’ll cover everything from old-school techniques like opinion research to new fangled tools like Radian6.

At this early stage in the game, it isn’t really sensible to talk about measurement without regard to strategy. Social Media strategies haven’t developed to the point where tactics are the primary focus of measurement. So our focus will be on how to use analytics to target your social efforts and establish whether you’re effective enough to continue.

For a take-way, we’re putting together a “Metrics Menu” that captures what we think are the most important types of measurement to focus on when you’re getting started.

Scott is the most effective and thoughtful strategist and practitioner in this space that I’ve yet encountered. I think you’ll really enjoy hearing him. If you’re thinking about a social marketing strategy or are responsible for social media measurement, do join us!

Measuring Online Applications: A Simple Primer on What to Analyze

Online Applications and highly-interactive web sites written in full-on programming languages are becoming the norm on the web, not the rare exception. Measuring these applications is challenging for a whole host of reasons. In my last post, I talked about the inevitable trade-offs between the desire for extensive measurement and the cost-of-measurement in a SaaS environment. In today’s post, I’m going to discuss some of the ways that analyzing applications is a little different than analyzing more traditional we sites.

One of the biggest differences I see between measuring applications and measuring websites is that application developers are far more interested in measuring the efficiency of their GUI. Traditional web sites have far fewer moving pieces – and the main concern is how well each page (as a unit) functions. But applications have many more options to consider and GUI developers are almost always deeply interested in understanding what users need most to accomplish any given task.

Much of the framework I suggested for measuring applications is driven by this recognition that applications can’t be measured quite like web sites. In my framework, units-of-work replace page views as the “core” ingredient in measurement. In addition, I added the concepts of “state” and “performance” to the unit of work.

From these concepts, a fairly powerful set of GUI analysis reports fall out quite naturally. It’s easy to report on which tasks are most commonly used; which tasks are done first in an application; the order of tasks and their associations (which are done together). In addition, the analyst can easily tell which application options are used most often (from application states); how application states are changed for each task; and which application states are most likely to result in task success. Finally, the performance measurements lets application developers compare how much time each task takes in total to how much of that is human time vs. machine time. It also makes it easy to see if time is impacting task success rates.

This is all good stuff and tremendously useful for GUI designers. And as I’ve said before, this type of measurement CAN be done with current web analytics systems that lack my suggested framework; it just requires quite a bit more work setting up the measurement infrastructure and hacking the reports.

There are some other types of application analysis that fall out a little less obviously from the framework and are distinct from traditional web analysis.

Unnecessary Clicks: Part of improving any GUI is making the path to task completion as swift and short as possible. The same forces are at work in traditional web sites. In a Functional analysis, we’re always looking for unnecessary pages – things like Router pages where the vast majority of users choose a single route. But use-cases are harder to apply to the open-ended non-task based paradigm that most web sites live in. With applications, however, use-case analysis (as embodied in my units-of-work concept) is probably the core methodology. For this analysis, the goal is to understand the swiftest path through any task and then identify and count all of the types of actions that deviate from that path. By identifying common deviations, designers may be able to bake-in settings, information or functionality to a process so that the user doesn’t have to – improving the overall performance of the application.

Human Performance Curves: One of the most difficult trade-offs an application designer faces is structuring a GUI that works well for beginners and experts alike. There is almost never a perfect solution – most interfaces will tend to optimize one experience or the other. But understanding the trade-offs doesn’t guarantee that your design matches your expectations. Analyzing this is non-trivial and quite different than most web site analysis. The best way to approach the problem is with performance curves. In the framework I suggested, performance times are automatically captured for each unit-of-work. These performance times measure both human time and machine-time. For this analysis, you need to plot the performance times against the number of times the user has done the task. This will yield a curve of times by usage. In an application that draws lots of regular users, you’d probably hope to see times drop sharply on repeat usage since the application should be designed to reward regulars. For infrequently-used applications, you’d probably hope to see a much flatter curve – the GUI should focus on getting all visitors through the process and not so much on speed improvements and options for heavy users.

Onboarding Success / Time: This is an analysis closely related to Performance Curves – the idea is to focus on the success of first time users. First use is critical to any application – first impressions do matter – and converting triers to regulars is vital. But the performance of first time users can be obscured if you focus on averages; especially if you have a significant population of regular users. These regulars are likely to be so good at the application that they make every measurement look good – performance times and completion rates will all be outstanding. By focusing on only first-time user performance, you can correct against that bias and better understand potential usability issues.

Help Rates: In traditional web analytics, people rarely go look for help. Well, they don’t do it that often in applications either! But there is a traditional analysis equivalent – we often analyze the internal search rate from pages, especially router pages. This rate is nearly always fairly low. But there are significant variations by page that highlight usability issues. Pages with high-rates of internal searching often lack pointers to types of content the user is expecting. "Help" usage in applications is similar. Tracking context sensitive help by unit-of-work is a good way to identify trouble spots. Remember, it’s quite unlikely that 50% of visitors are going to look for help ANYWHERE. You’re looking for tasks where the help rate is significantly higher even if the difference is between a rate of 3% and a rate of 6%. It’s almost like counting Voice of Customer complaints by task – the evidence is anecdotal but each complaint is likely a stand-in for many other users. 

Common Variations Analysis: Applications are built because they provide a user with the flexibility to try different combinations of factors. It doesn’t matter whether you are talking about configuring options on a car, selecting a life-insurance policy, designing a portfolio, or choosing a vacation – the underlying goal of a good application is to make it as easy as possible for users to explore the options that might interest them. Out of any set of options, however, there are bound to be strong patterns and relationships between individual options that the application designer might take advantage of to make the application more intuitive and powerful. If you know which combinations users tend to explore in sequence or together, the application can be designed to automatically create comparisons or at least to suggest and short-cut navigation to these associated options. In my measurement framework, applications states would be used to capture this type of information. But the same analysis can be done within current measurement frameworks though you’ll probably have to export the data to a SQL-Database or statistical package after capturing use states in custom variables.

There are, of course, an unlimited number of different and often highly specific analysis methods for applications. But just as certain methods of analysis have proven routinely fruitful for web sites (functional analysis, real-estate analysis, funnel analysis, etc.), these analysis methods fit many a different application. People often assume that having a good tool is the same as having good measurement. But as I've said repeatedly in this series, tight integration of measurement into an online application development environment is just a piece of the solution.

I’ve written this series as a sort of open-letter about what I’d like to see companies like Adobe, Google and Microsoft do to integrate measurement with online applications. But along the way, I’ve tried to point out how most of what I’m recommending can be done in today’s measurement system. It takes a lot more work than it should, but it’s still quite possible. Since nobody ever did useful work by waiting for tools to improve, I want to close by emphasizing this last point. People are already building application and dynamic web sites. Nobody’s waiting for good measurement paradigms to emerge and be incorporated into their tools.

By getting involved with the application development cycle early on, understanding what kinds of concepts are important to application measurement, doing your best to make your measurement infrastructure clean and testable, and adapting your analysis plan to meet the needs of applications, you can bring measurement value to the table right now.

Measuring Online Applications: Trade-offs between Measurement and Cost

Online Applications and highly-interactive web sites written in full-on programming languages are becoming the norm on the web, not the rare exception. Measuring these applications is challenging for a whole host of reasons. In my last post, I talked about the importance of automated testing to bring measurement fully into the application development life-cycle. In today’s post, I’m going to cover an issue that plagues application measurement, video and most other rich media measurement – how to balance cost vs. coverage when it comes to measurement.

The problem of cost in measurement springs from our basic model. With SaaS implementations, each server call you make to your measurement system costs money. Not a lot of money, of course. And for web sites, there has been little doubt that the knowledge gained from page measurement more than offsets the per-page cost of collection. On most web sites, a typical visitor might view six to ten pages. A really engaged visitor might view twenty to thirty pages. In an online application, however, the number of user interactions (every click, zoom, and setting) can be much, much higher.

Without a clear paradigm for a unit of measurement (such as a page view), it’s unclear which of these interactions should be measured. Capture every one, and you’ll likely find that the cost of measurement outweighs the benefit.

As I mentioned, we’ve seen similar issues with video measurement. Many of our media clients went from measuring almost nothing about video to detailed measurement of how far viewers penetrated into each and every visitor played. But to get that measurement, they usually had to make multiple server calls per video – sometimes as often as every 10 seconds. For a 2 minute video, that means a full-play results in 12 server calls. That’s a lot of measurement expense relative to the value of tracking abandonment per 10 second interval.

In fact, a lot of that more detailed measurement ended up getting pulled.

We’ve seen similar pull-backs when clients over-measured flash experiences – tracking roll-overs for example. After awhile, most companies find they aren’t using that level of detail about user interactions very effectively and the cost of measurement isn’t justified.

In an earlier post, I described a framework for measuring applications that included four core concepts: units-of-work, application state, rest-states, and performance. Part of the reason I settled on these concepts is that they provide a reasonable balance of measurement vs. cost.

A unit-of-work is a function within the application. The unit-of-work would typically encapsulate a single use-case (like getting driving directions for a mapping program or paying a bill in an online bill-pay system). The measurement system would make a call whenever a unit-of-work is initiated. With this call (and all calls) it would pass an application state.

The application state is a set of variables that capture all the important settings a user might have (like sort-order, filters, zoom level, tab open, etc.). The goal of application states was to be able to tie application settings to important actions (units-of-work) without having to pass measurement calls every time a state is changed (like a sort).

Now I’m going to add a new concept – milestones. Milestones would be way-markers within units-of-work. A milestone could be named and sequenced if desired. Whenever a milestone within a unit-of-work occurs, the measurement system would fire off a call. Every unit-of-work would have at least two associated milestones – an abandon milestone and a completion milestone.

Using the milestone concept, developers could control how much measurement is being done within a unit-of-work.

When a milestone is reached, the measurement system would automatically get a new application state and it would generate and pass a performance measurement for the previous milestone.

With this type of framework, there would be little need to track every user change or setting with a server call. Each measurement call would capture some significant action along with the associate states. On the measurement end, comparing states between milestones would make it easy to produce reports about what settings users changed when performing a unit-of-work.

The unit-of-work/milestone combination provides developers with a very granular way to control the level of measurement in the application (that’s important) and, potentially, to be very sparing about the cost of measurement.

It’s quite possible to duplicate some of this functionality in existing systems today. In Omniture, for example, you can code units-of-work using an s.prop variable. By turning on pathing, you can track movement between units-of-work. By using a combination of unit-of-work and milestone in a second prop, you can enable pathing within milestones. You can use a list-prop to capture application state or you can use a carefully formatted and delimited string in a normal string s.prop.

If you correlate that application state variable with the units-of-work s.prop and the milestones s.prop, you have basic reporting on how application states changed across milestones. Of course, you have to do most of the work to use those reports since the application isn’t going to really help you any. And you also have to do ALL the work to setup and capture this stuff.

Of course, that’s the whole point of this series. You CAN measure applications right now, it’s just not nearly as easy or powerful as it ought to be.

With that in mind, my next (and final) post in this series will tackle the last and probably the most difficult part of measuring applications – figuring out how to analyze the information to make it useful!

Measuring Online Applications : Automated Testing for Web Analytics

Online Applications and highly-interactive web sites written in full-on programming languages are becoming the norm on the web, not the rare exception. Measuring these applications is challenging for a whole host of reasons. In my last post, I talked about what a real application measurement framework might look like. In today’s post, I’m going to cover an issue that increasingly plagues measurement efforts – how to integrate testing of the measurement system into your application development process.

When we talk with the development teams these days, I’m often the one who has to break the bad news that there is no reasonable automation strategy for testing measurement. Not only that, but the testing is extraordinarily difficult to integrate into basic development cycles. Someone has to run through a test plan by hand, then go to the web analytics tool and try and shift through the data to isolate the test results. It’s kludgy, manual, slow and error prone. A perfect storm for testers.

This is especially frustrating because measurement isn’t “critical” functionality. Nobody wants to slip deadlines or delay release for measurement.

The changes I’ve talked about so far – especially in the measurement interface – would greatly reduce the burden on programmers to get measurement INTO the application. But they really wouldn’t do much to make it easier to test.

I’d like to see Omniture and Microsoft and other application and measurement vendors integrate testing functionality directly into the measurement system. Ideally, developers should be able to configure “testing” as an overlay on the existing measurement interface. In other words, I wouldn’t add any code to test, I’d simply set a variable or an object that controlled the “testing” state.

When testing was turned on, every measurement call would be mirrored to an additional destination. Since online applications typically run locally, I see several ways this might be handled. First, the application could potentially save the measurement in a flat file on the local system (given permission). Alternatively, the application could save the measurement calls to an interface object in some easily parsed format. Finally, the application could pass the calls back to a secondary server that simply recorded them in a flat file on the server side or was part of a testing automation suite.

With this type of setup, testing automation tools could drive through a series of application steps which would produce a measurement file that could be quickly “diffed” with previous versions to establish correctness. This would dramatically reduce the amount of time new testing cycles took while greatly improving accuracy. Even if each test cycle had to be done manually, programmers would only have to worry about checking the measurement file once and getting a correct version.

Even without the automation aspects, this type of system would make testing applications in secure and mobile environments much easier. We often do the bulk of our testing using HTTP sniffers like Firebug, Charles and Fiddler. But these sometimes don’t work or conflict with the application on development networks and in secure environments.

As I think about this, I realize that in some ways this “testing” capability mirrors a universal tag. If the measurement library makes it easy to reflect measurement calls to multiple locations, you have the basic infrastructure for universal measurement.

That seems like a pretty good idea – something that would be almost trivial to add into a measurement framework of the sort I described in the first two posts. Pegging testing on a universal tag capability has the additional benefit that people may pay more attention to it that way. Testing is one of the most time-consuming, manual, and expensive tasks in software development. Unfortunately, as the least sexy task is a discipline with very little sexiness in general, measurement testing gets less real-world attention than the “best script” honors at an adult video awards program.

But if you’re going to build a really good measurement interface, building support for automated testing right into the system would go a long way toward making the incorporation of measurement into online applications routine instead of esoteric and painful.

Measuring Online Applications – Creating a Measurement Framework

Online Applications and highly-interactive web sites written in full-on programming languages are becoming the norm on the web, not the rare exception. Measuring these applications is challenging for a whole host of reasons. In my last post, I talked about what a good programmatic interface might look like. But even with simple, logical calls to the measurement service, measuring applications won’t be a slam dunk. Applications simply don’t adapt well to the page/link view paradigm deeply embedded in web analytics systems. In today’s post, I’m going to cover some of the most important topics a measurement framework for applications should address.

It’s obvious that online applications don’t easily fit the page view paradigm of static web sites. What’s less obvious is what should replace that paradigm. I’m not prepared to give a complete answer to that question. However, in our work actually building measurement into applications, there are four themes that have emerged and seem necessary to any good framework.

Measuring the Application State

When we build application measurement into traditional measurement systems, one of the most important concepts we’ve evolved is the idea of an application state. The idea behind an application state is that it captures all of the “settings” that are in place while a user accomplishes some action. Typical elements of an application state are things like “sort type,” “filter-type,” “zoom-level,” and “view-type.” In my last post, I used the example of a mapping application to illustrate some of the concepts involved in a good programmatic interface. It’s a useful example, because it’s a type of online application that virtually everyone is familiar with. In a mapping application, the application state is likely to include variables like “map type,” “overlay-type,” “zoom-level,” and “route-type.”

The idea behind an application state is simple. It allows you to track key combinations of interface elements by application function. It can also be used to reduce the number of calls you need to make to the measurement service (a critical factor for any high-volume application). You don’t need or want to capture every single mouse click in an application.

If a user clicks on zoom or sort, sending a measurement click is generally too expensive. Instead, we capture the entire set of application variables whenever key actions happen. In the map example, this would allow us to say what the most popular zoom levels are when viewing a route or adding a restaurant overlay vs. a hotel overlay. That’s incredibly useful information for GUI designers and its captured much more effectively in an application state than in any other fashion.

When we build an application state into a current measurement system, we generally pass it as a List Prop or a simple string variable. However, it takes some work to get it to tie to application functions in an intelligible manner. I’d like to see an application framework capture the state as a type of thing and allow it to be viewed and counted and analyzed in conjunction with different application functions.

Rest States

One of the trickiest aspects of application measurement is what to do about applications where the interaction isn’t discrete. In our mapping example, panning and zooming are a good example of what I mean when I say not discrete. It’s easy to fire off a measurement call when someone clicks a button to change from a street view to a topo view. But when do you fire off a measurement call if someone is zooming? How do you know they are done?

Ignoring these non-discrete interactions works for some applications but cripples measurement for others. So what I’d like to see is a measurement framework that includes a threaded callback mechanism that can be attached to various interactions and basically just checks to see if the interaction has stopped for some period of time. I call this stop-time a rest state.

Using my map example, when a user clicks on the zoom control of a map, it would fire a measurement function that started a thread (call it zoomwait). The function would be passed three things – the action it’s checking for, the wait period required, and the measurement function to call when the wait period has been realized. Let’s say we are checking zoom and we have a 3 second wait period. The measurement thread would check to see if a zoom had been done in the last 3 seconds. If it had, it would until at least 3 seconds after the last registered zoom time and try again. It would do this until it detected that 3 seconds had passed since a zoom. It would then call the measurement function specified which would make the actual call to the measurement system and pass the new application state.

The idea behind rest states is to make it possible to measure the application state AFTER the user has finished some set of actions. This makes it possible to capture many key application states without necessarily trying to measure every application state when a user is zooming, panning, or doing some other non-discrete interaction.

Measuring Performance

The current static web site measurement framework largely ignores performance. That’s a serious drawback, but one that’s hard to solve given the limitations on measurement with our current infrastructure. But for applications, measuring app performance is really, really important.

Application frameworks like .Net already include some low-level calls designed to make performance tracking around key functions possible. But not every development framework has this (many do not), and I think it’s a natural part of a measurement system. So I’d like a measurement framework to include this functionality.

To measure performance, the system should have a way of starting a unit of work and defining it as being open-ended (not terminated by another start) or closed (terminated automatically if another unit of work starts). Units of work would be named and could be closed explicitly by the application sending a close call with the corresponding name.

The measurement system could use this concept in a variety of ways including measuring total time in application (by firing off an open ended unit of work named application start that is only closed  on app clean-up), measuring total time in app areas, and measuring specific application times for functions like “sort” or “synch.”

Depending on how this was used, it could just as easily measure how long a user takes for an application (like filling out a form) or how long an application takes to do processing (like placing an order). In the current web environment, these two numbers are always conflated into page times. Providing the capability to break them out would be significant win in application measurement.

Units of Work

In my discussion on performance, I touched on “units of work,” the last and, in some ways, the most important of the application measurement framework themes I’d like to see a measurement system address.

When we measure static web sites, we often employ a methodology that we call functionalism. It works by assigning specific functions to each page and then analyzing their performance relative to those functions. It doesn’t work well for applications, however. The functional layer in an application isn’t page based at all.

I’d like to see this functional concept extended directly into the measurement framework. The idea would be to encapsulate all of the major application use-cases as units of work. The framework should make it easy for the developer to kick-off a “unit-of-work” and, as with performance, control the interactions that can occur (in other words, to be able to write logic that determines what to do when one unit of work interrupts another or takes place inside it).

Instead of page-based tracking, I think the basic level of tracking in an application should be at the unit-of-work level. Functions inside the unit-of-work could be tracked as “steps” or “tasks” depending on whether the unit-of-work was ordered or unordered.

How would this work? Going back to our mapping application, a unit-of-work might be triggered when a user does an address search.

This “search unit” would persist as a user pans, zooms, and adds an overlay for hotels, and then clicks on a hotel or two. The unit would end when the user enters a new search or exits the application (and possibly on other actions).

The measurement system would track how often a visitor did each type of unit of work (how often the “search unit” was triggered, how long the user spent in the search unit, how long key actions (like the overlay fill) took to perform, the use-state at each point in the unit or work, and what the visitor did to exit the unit.

Summary

I’m not sure that these four concepts (unit of work, performance, rest state, application state) are a complete description of an application measurement framework, but they would go a long way to providing logical, rich, and appropriate measurement to applications. Some can (and should) be done even if you are coding in current measurement systems – though all require significant hacks to do right now. But if Microsoft and Omniture are working to build robust application measurement, creating a measurement framework that embedded these concepts would truly advance the state-of-the-art. In my next post, I’ll take up one of the true pain-points in application measurement – testing.

Measuring Online Applications – Structuring a Measurement Interface

Online Applications and highly-interactive web sites written in full-on programming languages are becoming the norm on the web, not the rare exception. Measuring these applications is challenging for a whole host of reasons. But one of the biggest problems right now is how clumsy the javascript tagging method is when applied to applications.

Most measurement systems now work similarly – they rely on a small javascript program (the tag) that allows a coder to customize the values for certain variables and then call a function that assembles those values, adds a set of environmental variables (like browser and referring site), creates an image request and then executes that image request. All of the custom and environmental variables are passed as parameters in the image request and are decoded by the vendor when the request is logged on their end.

Think about it – all the fuss and muss of tagging – just to make a call to another server and pass a few values. In the world of applications, there are many ways of handling this basic task, by far the most common being to use a web service. Web services are vastly more convenient for the application developer than the current method (which is well suited to static HTML). Indeed, it's hard to imagine a more primitive interface for web communications than passing parameters in image requests.

The virtues of web services include the fact that they are directly supported within every significant programming language (including direct support within the programmers development environment), they provide methods that allow for back-and-forth communication and error-handling when necessary, they provide more control over the nature and amount of information passed, and they typically encapsulate the information passing into much more logical and understandable programmatic units.

It’s true that some measurement vendors provide data insertion APIs. Omniture, for example, has one. But the Omniture Data Insertion API is not a real web-service. It’s just a ”post” based mechanism for sending data. It’s might be the right method to select if you’re coding SiteCatalyst measurement in an application right now – but it’s hardly a robust interface.

But to me, the most important part of building a good interface between application system and measurement system isn’t the handshake mechanism, it’s the structure of the programmatic interface.

Here’s some sample SiteCatalyst code that might get attached to a click-handler when an event is firing:

var s=s_gi('rsid');s.linkTrackVars='eVar34,prop8,eVar8,prop9,eVar9,prop32,eVar32,eVar50,events';s.linkTrackEvents='event3';
s.eVar34='+1';
s.prop8='[Section]:upload files';           
s.eVar8='[Section]:upload files';
s.prop9=s.pageName+':[Section]:upload files';
s.eVar9=s.pageName+':[Section]:upload files';
s.prop32=s.eVar32='upload files';
s.eVar50='+1';
s.events='event3';
s.tl(this,'o','upload files');

These 13 lines of code create the basic SiteCatalyst measurement object (s), tell SiteCatalyst which variables to pass (linkTrackVars), which events to pass (linkTrackEvents), assign seven different variables and one event, and then call the s.tl function that generates and fires the image request.

Strewing chunks of code like this – even if it’s translated into C# or java - all over your application makes for an incredibly messy situation: it’s hard to document, hard to code, hard to test, and very hard to maintain. And like most poor coding mechanisms, it means that if you want to change measurement systems, you’re faced with a significant re-wiring task.

If this is the wrong way to integrate measurement into an application, what’s the right way?

Well, it isn’t to give the programmer a function call like this:

doPageView(“pagename”, “events=1,3,5”, “evar1=test;evar2=test2”, “prop1=red”);

In truth, even this would be better than what we have now (though it’s easy enough to create this sort of wrapper). But this simple function library approach doesn’t create a structure that embeds measurement logically inside the application.

What I’d like to see is a system where you started with a basic measurement object. This object could be instantiated once by the application (though applications could use more than one) and would contain, at minimum, the target account information and the configuration of environmental variables (what needs to be pulled and passed).

Ideally, environmental variables would only be passed once on invocation of the application (since apps aren’t stateless like the web, the challenge of inferring sessions and the necessity for re-passing environmental information with every request doesn’t exist). This would dramatically reduce the amount of information required with each call and the corresponding network bandwidth required.

Underneath this global measurement object would exist one or more measurement objects that inherit the global state and allow for custom mappings of application variables. The custom mappings would let the developer populate measurement variables from object properties (like Name, ID, Value or isVisible), application variables, static values or special function.

In this way, a measurement variable could be populated with the Name Property (for example) of any object to which to which it was attached. If attached to a Form Object, it could automatically create something like a pagename variable for an application by mapping the Form's "name" property to the measurement pagename variable.

The measurement object could also be configured to support any or all of the default or specific methods that objects in the development environment have. In other words, a measurement object could be configured with a specification for “load”, “unload”, “click”, etc. This would allow the developer to automatically fire a measurement call in response to either specific functions (like a search) or general functions (like a form load).

With this approach, the developer can take advantage of built-in UI constructs and also manage a small set of application specific variables that will automatically be translated into the measurement system equivalents whenever an actual measurement call is made.

When I create an object in the interface, I would then attach my measurement object to it. When the object is loaded, any handlers for the measurement object would be fired if present. It would automatically set its values, give the programmer an opportunity for a call-back (which should be rare) and then fire the request.

This is rather abstract, so let me show how this work in a more concrete fashion.

I’d start with a measurement object in which I would set the global information about the account (and maybe the vendor) I’m sending information to along with the environmental variables I want to pass on invocation.

Next, I’d create a custom variable object that described the specific variables I want to pass. Let’s suppose that I’m working on a Mapping Application and I want to collect the viewed area (the region/country/city shown in the Map Title), the map view (road, topo, satellite, etc.), the map state (overlays, zoom level), the language of the map, and the visitor identity. I would map each of these application variables to a specific measurement object variable. Next, I’d assign the click-handlers I want the measurement to fire on (I may want it on every re-draw but I might also want it on a more limited set of actions such as onSearch and onMapChange). Finally, I’d attach the measurement object to the underlying UI object.

That’s pretty much it. Once the object is attached, the programmer is done. When the object to which the measurement is attached fires an appropriate handler, the object would inherit its default state, populate the variables (like viewed area, map state, etc.), and then fire the measurement request.

Depending on my application, I might need only one custom measurement object or I might need quite a few. But with this type of system, the measurement objects would be embedded into the underlying GUI objects and their handlers. The measurement layer would deeply in-grained, it would provide inheritance, and it would provide configurability. It would be much, much easier to maintain such a system. The measurement objects themselves would be compact and discrete and they would be directly attached to the objects in which they reside.

Having a measurement integration at this level inside an application development framework would make the development process dramatically easier. Indeed, it would be possible to achieve a very high-level of measurement with a simple default configuration of the measurement object that might simply “fall-out” of a single drag-and-drop of the measurement object into the application. Such a default configuration might capture the object name and title of every form on load and unload, the object name and title of every “play” movie action, and the selected value from any Combo-Box. But it would also provide a high-level of control within the application for measurement at a very fine-grained object level.

As I mentioned in my first post, there really aren’t any serious barriers to building this type of system. There are probably different ways of achieving a deep measurement integration or designing the measurement objects – and I’m not confident that what I’ve suggested is the exactly right answer. But some form of deeply embedded measurement at the object level within the app framework seems both inevitable and right. Having an embedded measurement system won’t solve all the problems in application measurement, but it would be a big step in the right direction.

In my next post, I’ll take up a thornier issue  – the right measurement paradigm in a world where page views and link clicks aren’t the ultimate measurement objects!

Measuring Online Applications

I’ve been doing a series of posts on topics that came up at X Change but today’s topic is even more timely than it was just one month ago. With Adobe’s acquisition of Omniture, the goal of increasing the measurability of web applications has really come to the forefront. It seems to be the core strategic rationale for the whole Omniture deal; making it an issue that’s really gotten everyone’s attention.

When I blogged on the deal, one of my main points was that while the strategic rationale makes sense, capturing the benefits will not be a slam dunk. When analysts think about this deal, they tend to assume that Adobe and Omniture will be able to deliver significantly enhanced application measurability. The question for them is how big a win is that? But I don’t think the barriers to measuring applications are necessarily simple or exclusively technical.

I had the good fortune to meet Michael Scherotter of Microsoft at X Change and we got a chance to talk in-depth during what I think was the smallest Huddle of the entire Conference in that brutal late-Friday-afternoon close of conference time slot. I don’t blame anyone who split out to enjoy some pretty spectacular San Francisco weather and it ended up working out great for me. It was a really interesting wide-ranging conversation but of particular interest was Michael’s current focus on creating better and more integrated measurement for Silverlight – the development environment that may end up being Adobe’s most serious competition.

Application measurement has been on mind in our work this past year; partially because of the work we are doing in mobile. For the mobile space, applications have become a huge focus – and while mobile apps have some unique measurement challenges, they share many of the same challenges as fixed web apps. In addition, we tend to work on implementations for more complicated web sites, and in the past year we’ve tackled a range of implementations that were really applications not web sites. We even did a whole class at Think Tank on measuring applications, and though I didn’t teach that class, I wrote a goodly chunk of it.

So we at Semphonic have had lots of recent practice experience in measuring applications and how frustrating it can be. In today’s post, I’m just going to outline what I see as the core problems and then I’m going to follow-up with a more in-depth look at each.

Building a framework for application measurement involves at least five (five!) significant problems.

Of these, the most obvious and maybe the most important is the mechanism for incorporating measurement. The current art of web analytics has been built around javascript tagging – with programmers doing page customizations and writing all sorts of little functions for click-handlers. This style doesn’t work for applications. Not only is javascript the wrong tool for the job, but this approach leads to far too much spaghetti code and far too much work for each measurement call to be appropriate in a development environment.

So the first task for measurement is to create an approach that lets developers integrate measurement quickly and easily. Programmers expect to be able to accomplish 3rd Party functionality like this with APIs and Function Libraries that require no more than a single call per invocation and encapsulate measurement variables in some logical structure.

Creation of a modular API or function-based approach to measurement is largely a technical challenge and really should be fairly straightforward. But the second major problem is more challenging. Application measurement has been shoe-horned into a measurement system designed for web sites based around page views. When we do application measurement, we have to spend a significant amount of time trying to come up with a translation between the way the application works and measurement concepts like page views and paths. It’s usually a clumsy fit. Omniture has done a decent job building a framework for video measurement – but I think that’s a much easier problem to solve. Useful analysis is going to require a real framework for application measurement and that doesn’t currently exist.

Problem three is one of the most neglected aspects of tagging in general – testing. We see lots of implementations where the level of testing on rollout is just pathetic. That’s a bad idea even when you’re tagging a traditional website. But you can’t get away with sloppy or non-existent testing when measuring applications – not least because your development groups won’t let you. Applications are built in formal stages with significant and carefully controlled testing.

There is no automated testing system for web analytics right now (though Observepoint moves in that direction) and it is much, much too hard to actually test. Having to integrate all sorts of new testing steps into an application process sucks down time and resources and makes getting measurement much more challenging than it should be.

Problem four is a combination of cost & performance. The traditional web analytics cost model is based on server calls. The cost of measuring web sites is generally considered fairly reasonable. But for many of our clients, trying to micro-track inside video has resulted in lots of server calls – more than the information is sometimes worth. For applications, the problems are likely to be similar. Will the server call model really hold up in application measurement and how much will it impact the style of measurement called for?

Lastly, we have the problem of analysis. Just as we don’t have a good framework for capturing and showing application usage, we don’t really have an analytic paradigm for using the data. Without such a paradigm, the whole exercise to integrate more measurability in applications could easily be an expensive waste. Indeed, it’s almost guaranteed to be an expensive waste in most early attempts. One thing I’ve certainly seen in our application measurement is that in addition to the usual concerns of web measurement, when we code application measurement there are almost always a whole set of measurement requirements and questions generated by the GUI designers. You might think the same would be true of websites, but it really isn’t. GUI application measurement is generally a new class of analysis in our field and in our experience so far it changes many aspects of what you collect and how you use the data.

I wouldn’t classify any of these concerns as trivial – and only the first (modularization) and third (testing) are primarily technical. Adobe or Microsoft really don’t have to deliver on every one of these challenges to be successful. Solving the technical problems would, at the least, open the door to easier resolution of the other challenges. But a really good application measurement framework would solve all of the first four problems and might even advance a point of view on good framework for analysis.

Data Warehousing Analytics Data and the Universal Tag

The Really Big Topics at X Change

I’ve been doing a series of posts on thoughts from X Change Huddles and I started with two on behavioral integration of survey data – which was the very first Huddle I took part in. But this was by no means the hot topic at X Change. In fact, the Think Tank class on survey/behavioral integration was one of the least popular offerings we did (much to my chagrin). What were the really big topics? There were two – warehousing online analytics data and using a universal tag. Both came up over and over again. I was mildly surprised by how popular the first was – shocked by the amount of attention given the second. What’s interesting is that the two topics are more related that people might think – and that’s what I want to talk about today.

People have been talking about integrating online and offline data for years of course. But a couple of things this year felt different. Quite a few companies were actually doing real warehousing projects. Some of these were deep into production others felt more exploratory. Either way, people were clearly moving toward making this a reality. Given the broader economy, that’s impressive. Warehousing projects tend to be large investments – not easy to do in a seriously down economy.

The other thing that felt different this year was the number of companies that simultaneously feel confident of their basic web analytics infrastructure and reporting but dissatisfied with the maturity of their analysis. Achieving maturity in analysis has always been a problem – but in past years the big issue always felt like it was just achieving a quality infrastructure and reporting system. It would be an exaggeration to suggest that those issues have gone away but it felt to me like the balance has shifted. A lot more companies seem like they are getting quite a bit from their web analytics implementations and have reached the point where the existing tools just won’t give them any more.

Unlike data integration, the push for a “universal” tag is a lot newer. The idea is to implement a single point of data capture and then move the data to each of the analytics vendors who consume it. A single tag is appealing to many enterprises – after all, it isn’t unusual for a company to have five or six measurement tags on a single page. In addition to one or more full web analytics tags, ones for advertising, video, and various other niche measurement functions are all too common. A multiplicity of tags starts to be a performance burden and a nuisance from an IT perspective.

In some ways, the name “universal” tag may be a misnomer since not every universal tagging strategy even uses a tag. In some systems, wireline capture is used to grab the data which is then piped to the various information consumers. Alternatively, a single tag can capture the data to a central location which then moves just the necessary data to the consumers.

It’s a beautiful idea. The IT burden of tagging is dramatically lessened. Measurement across systems is unified with a single point of data capture (tag placement on existing pages is a common culprit in reconciliation projects). Performance on the page is no longer threatened.

Those are the immediate and obvious benefits.

But there are some less obvious but very real benefits that may be even more important.
X Change happened just before the Adobe deal, but many of those considering or using a universal tag approach named the ability to change analytic solutions as a key driver. For companies that may have migrated through two-or-three solutions in the last few years, the idea of being able to separate the capture layer from the reporting and analysis layer is pretty attractive. Let’s face it, one of the most difficult parts of switching vendors is re-tagging a site. And with the vendor landscape in web analytics looking rather less settled in some respects and the prospect of more acquisitions seemingly likely, the ability to swap software solutions without major infrastructure cost is undeniably appealing.

The last benefit I see is the one that may be the most significant – and it’s the fact that a universal tagging solution can enable warehousing with minimal additional fuss. After all, the whole point of a universal tag (however it’s implemented) is to create a single point of capture that can then move the data wherever necessary. Wherever necessary may well be your own data warehouse.  A universal tag solution eliminates the need for you to move data to a web analytics vendor and then pay them to move it back to you. Instead, your warehouse could potentially take advantage of the same underlying data stream as all of the other measurement vendors.

That’s why I started this post with the observation that warehousing analytics data and universal tagging are more connected than people may think. Universal tagging is a potential enabler for data warehousing. That’s a good thing in-and-of-itself and I think the goal of warehousing and the savings around eliminating a feed from your vendor might become a real enabler of universal tagging.

And think about this - data warehousing in conjunction with a universal tag might make some intriguing types of analytics systems (like Google Analytics for basic reporting plus a warehouse for deep-dive analysis) possible that would otherwise not be.

So far, one feature that you just can’t get out of Google Analytics is a true data feed. And that’s a feature that Google may not even have on its radar. So if you wanted to warehouse analytics data, you couldn’t use GA for data capture. With a universal tag, a system like that suddenly becomes possible.

No matter how you slice it, I expect both warehousing analytics and universal tagging to grow significantly in popularity in 2010. We may not be quite at the tipping point yet, but I wouldn’t be surprised if it’s all that far away.

Validating your Online Survey Sample

More thoughts from X Change on VOC Online Survey and Web Analytics Integration

[Before I dive into today’s topic, I wanted to highlight Jared Waxman’s comment on my last survey post. Jared made a really interesting point about a different way to think about sample size. I totally agree with his take. and it’s a different perspective than you usually get.]

My last post on online survey and behavioral integration made the point that the demands of behavioral analysis (which typically involve fairly small subsets of the total population) make large sample sizes an absolute requirement if you really want to take advantage of behavioral integration and not just say you have it. But the news about behavior integration isn’t all bad – even if you’re primary interest is on the survey side. Larger sample sizes are a burden. But one of the biggest concerns I heard during X Change was the validity of their online survey samples - and that's something that behavioral integration can help establish.

How do you know if your online survey sample really is representative?

From a pure survey perspective, you can take a methodological approach. A good methodology goes a long way toward getting you a good sample. But the online world is not as settled from a methodological perspective as the offline world and I heard lots of concern about the online surveys actually deployed at the enterprise level.

In the traditional survey world, key demographic variables can be used as a check on the validity of your sample. If you’re polling a presidential election and your gender, age or ethnicity break-downs don’t match the registered voter (or likely voter) population, you know you have issues.

But demographic variables in the online world provide no similar re-assurance. You have no independent measure of your actual demographics. Even if the online survey demographics match your broader audience profile, that’s no guarantee – it might even be suspicious. Of course, you may be able to match your online survey demographics with traditional research that includes a segmentation of online.

Even a good match here is no guarantee that you aren’t mis-sampling significant types of traffic. You may be missing out, for instance, on your natural search non-customer base and you’d never realize it with this type of check.

If you are doing behavioral integration, however, there is a simple and powerful way to validate your sample. Simply run a set of behavioral profiles against your survey population. Obvious variables to look at include pages viewed, acquisition sources, loyalty, and success event counts. With a large sample size, you can even check on geography.

In addition to the high-level behavioral variables, it's not a bad idea to check usage of key content types. It's possible, for instance, that your "job-seeking" population looks rather like your "shopping" population from a macro-perspective of sourcing, page views or time-on-site. But it would be distressing to find that your increased satisfaction scores are being driven by an increase of job-seekers coming to your site in a down-economy. Of course, surveys have an internal check (visit reason) that might catch this; but the behavioral measure is probably a more reliable indicator in this particular case.

If the behavioral profiles for your survey sample match that of your broader population, it’s an extremely powerful confirmation that your sample is valid. This is a great way to once and for all quash doubts in your organization about whether an online survey is really representative.

This works even if you are pulling a criteria-based sample. If you’ve made the decision that you’re only going to survey visitors with 5+ pages, you can compare the survey population to the total population of 5+ page viewers. As a bonus, you can also get a good sense of what you’re missing when you do that.

On the other hand, if you detect significant mismatches in the behavioral profiles, this can indicate problems in your sampling.

If you do find problems in your sample, you shouldn’t assume that you can just weight the population to fix the problem. Suppose that you find your survey sample consistently undercounts your natural search sourced population. It would be a mistake to simply weight what natural search respondents you have up to a representative number and assume you’ve fixed the problem.

You may only be sampling engaged natural searchers or natural searchers from brand terms. These populations may be fundamentally different than the broad natural search population.

You may need to weight by a combination of factors (e.g. pages viewed & source or pages viewed & source & visit number) or figure out how to actually adjust your sample.

Either way, using behavioral data as an independent confirmation of survey sample is a powerful tool. Not only can it help you figure out if you are getting a good sample, it’s a tremendous tool for PROVING to your organization that you are getting a good sample.

With organizations increasingly relying on the data collected from online opinion research, behavioral validation of the sample population is something you really should consider. After all, it’s really bad if your sample is wrong and people are relying on your numbers. It’s almost as bad if your sample is good and people don’t trust the numbers. And even if your sample is good and people rely on your numbers, an independent confirmation and the peace-of-mind it brings is still worth having.

Is it worth doing a behavioral integration just to validate your sample? If you take your research seriously (and you should), then I think it is. There are many other benefits to integration, but this is an oft-overlooked and quite substantial win that is available without all that much effort or cost. It’s not often I get to write a sentence like that!

Survey Says: Thoughts from X Change on Online Survey Sample Size

In Russ Rueden’s X Change Huddle on Survey and Web Analytics Integration, the topic of online survey sample size came up. It’s a big issue because many organizations find themselves deploying almost as many different surveys as tags and they don’t want to suffer from what I’d call “uncertainty principle syndrome” – damaging the user experience that they’re trying to measure.

But how many respondents are really enough?

There are two schools of thought about sample size – one school holds that as long as a survey is representative, a relatively small sample size is adequate. Perhaps 300-500 respondents can work. The other point of view is that while maintaining a representative sample is essential, having a large sample size is almost as important.

This is a big issue because it impacts all sorts of decisions including the length of your survey, your collection mechanism, and, of course, you’re sampling rate.

So what’s the right answer?

If you aren’t integrating your survey data with web behavioral data, then a relatively small sample size might be okay (I’d emphasize the might). But if you want to combine behavioral analysis and survey data, then forget a sample of 300 or 500 respondents.

Those numbers simply won’t work.

Let me give you a real-world example showing why that’s true. We’re working right now with a client that samples more than 1000 site visitors a month for their satisfaction survey. They asked us to do a study of the impact of using one of two alternative tools on their site on both overall site satisfaction and visit accomplishment.

On this site, the tools are used in about 10% of visits. Since the site gets more than 10 million visits a month, that still yields a heckuva lot of behavior to study – more than 1 million tool visits every month. No problem there.

But our representative sample only captured about 100 respondents who’d used either tool.

Between the two tools, one served about 70% of the queries. So for the second tool, we had about 30 respondents to deal with. Getting the picture?

For our analysis, we wanted to track visit reason vs. satisfaction vs. outcomes for tool users. With some visit reasons only accounting for about 10% of visits, there were cases where we were trying to analyze the outcomes for all of 3 visitors. Sorry – not possible.

And that’s with a survey size above 1000 and a simple cross-tabulation of visit intent and one fairly common behavior. Sure, we could add lots more months to the picture. But tracking behavior over extended periods of time adds all sorts of complications to the analysis. The combination of seasonality, site change and macro-economic change make this dangerous. Very few of our client sites remain constant for six months.

If doing behavioral analysis with 1000 survey respondents is challenging, imagine what it would be like with a sample size of 300. Impossible. I don’t believe that with a 300 sample size there is a single site behavior you could cross-tabulate with any survey variable and still have a significant population.

So what’s the right answer?

If I had my druthers, I’d recommend that large volume sites strive for a much higher sample size – 15K would be nice on a monthly basis. With the old constraints of cost gone, there’s really no reason not to go for as large a sample as your site volume will support without impacting your user experience.

Obviously, this upper-limit depends on your site volume and your take-up and completion rates.

You can’t do much about volume, but if your survey length is impacting your take-ups or completion rates, then I’d be willing to sacrifice a whole bunch of questions to get to the increased size. The fact is that on many 30-40 question surveys, we’d only expect to use at most 5-6 of those questions in a behavioral analysis. I’d bet even money that your analysts feel the same way and that a heavy majority of questions on many long surveys hardly ever get studied at all.

At some point you may have to make a decision: do you want a whole lot of really shallow information or do you actually want to do analysis on a narrower set of data?
When it comes to behavioral analysis combined with survey integration, the right answer is pretty obvious. A representative sample is essential, but size really does matter. Let yourself get talked into a 300 person sample, and you might as well throw all that work you did to integrate online survey data with behavioral data in the junk pile.

[Just a last reminder - I'm doing a free webinar on web analytics and realtime customer event marketing this Wednesday. Register here!]