SemAngel

Measuring Online Applications – Creating a Measurement Framework

Online Applications and highly-interactive web sites written in full-on programming languages are becoming the norm on the web, not the rare exception. Measuring these applications is challenging for a whole host of reasons. In my last post, I talked about what a good programmatic interface might look like. But even with simple, logical calls to the measurement service, measuring applications won’t be a slam dunk. Applications simply don’t adapt well to the page/link view paradigm deeply embedded in web analytics systems. In today’s post, I’m going to cover some of the most important topics a measurement framework for applications should address.

It’s obvious that online applications don’t easily fit the page view paradigm of static web sites. What’s less obvious is what should replace that paradigm. I’m not prepared to give a complete answer to that question. However, in our work actually building measurement into applications, there are four themes that have emerged and seem necessary to any good framework.

Measuring the Application State

When we build application measurement into traditional measurement systems, one of the most important concepts we’ve evolved is the idea of an application state. The idea behind an application state is that it captures all of the “settings” that are in place while a user accomplishes some action. Typical elements of an application state are things like “sort type,” “filter-type,” “zoom-level,” and “view-type.” In my last post, I used the example of a mapping application to illustrate some of the concepts involved in a good programmatic interface. It’s a useful example, because it’s a type of online application that virtually everyone is familiar with. In a mapping application, the application state is likely to include variables like “map type,” “overlay-type,” “zoom-level,” and “route-type.”

The idea behind an application state is simple. It allows you to track key combinations of interface elements by application function. It can also be used to reduce the number of calls you need to make to the measurement service (a critical factor for any high-volume application). You don’t need or want to capture every single mouse click in an application.

If a user clicks on zoom or sort, sending a measurement click is generally too expensive. Instead, we capture the entire set of application variables whenever key actions happen. In the map example, this would allow us to say what the most popular zoom levels are when viewing a route or adding a restaurant overlay vs. a hotel overlay. That’s incredibly useful information for GUI designers and its captured much more effectively in an application state than in any other fashion.

When we build an application state into a current measurement system, we generally pass it as a List Prop or a simple string variable. However, it takes some work to get it to tie to application functions in an intelligible manner. I’d like to see an application framework capture the state as a type of thing and allow it to be viewed and counted and analyzed in conjunction with different application functions.

Rest States

One of the trickiest aspects of application measurement is what to do about applications where the interaction isn’t discrete. In our mapping example, panning and zooming are a good example of what I mean when I say not discrete. It’s easy to fire off a measurement call when someone clicks a button to change from a street view to a topo view. But when do you fire off a measurement call if someone is zooming? How do you know they are done?

Ignoring these non-discrete interactions works for some applications but cripples measurement for others. So what I’d like to see is a measurement framework that includes a threaded callback mechanism that can be attached to various interactions and basically just checks to see if the interaction has stopped for some period of time. I call this stop-time a rest state.

Using my map example, when a user clicks on the zoom control of a map, it would fire a measurement function that started a thread (call it zoomwait). The function would be passed three things – the action it’s checking for, the wait period required, and the measurement function to call when the wait period has been realized. Let’s say we are checking zoom and we have a 3 second wait period. The measurement thread would check to see if a zoom had been done in the last 3 seconds. If it had, it would until at least 3 seconds after the last registered zoom time and try again. It would do this until it detected that 3 seconds had passed since a zoom. It would then call the measurement function specified which would make the actual call to the measurement system and pass the new application state.

The idea behind rest states is to make it possible to measure the application state AFTER the user has finished some set of actions. This makes it possible to capture many key application states without necessarily trying to measure every application state when a user is zooming, panning, or doing some other non-discrete interaction.

Measuring Performance

The current static web site measurement framework largely ignores performance. That’s a serious drawback, but one that’s hard to solve given the limitations on measurement with our current infrastructure. But for applications, measuring app performance is really, really important.

Application frameworks like .Net already include some low-level calls designed to make performance tracking around key functions possible. But not every development framework has this (many do not), and I think it’s a natural part of a measurement system. So I’d like a measurement framework to include this functionality.

To measure performance, the system should have a way of starting a unit of work and defining it as being open-ended (not terminated by another start) or closed (terminated automatically if another unit of work starts). Units of work would be named and could be closed explicitly by the application sending a close call with the corresponding name.

The measurement system could use this concept in a variety of ways including measuring total time in application (by firing off an open ended unit of work named application start that is only closed  on app clean-up), measuring total time in app areas, and measuring specific application times for functions like “sort” or “synch.”

Depending on how this was used, it could just as easily measure how long a user takes for an application (like filling out a form) or how long an application takes to do processing (like placing an order). In the current web environment, these two numbers are always conflated into page times. Providing the capability to break them out would be significant win in application measurement.

Units of Work

In my discussion on performance, I touched on “units of work,” the last and, in some ways, the most important of the application measurement framework themes I’d like to see a measurement system address.

When we measure static web sites, we often employ a methodology that we call functionalism. It works by assigning specific functions to each page and then analyzing their performance relative to those functions. It doesn’t work well for applications, however. The functional layer in an application isn’t page based at all.

I’d like to see this functional concept extended directly into the measurement framework. The idea would be to encapsulate all of the major application use-cases as units of work. The framework should make it easy for the developer to kick-off a “unit-of-work” and, as with performance, control the interactions that can occur (in other words, to be able to write logic that determines what to do when one unit of work interrupts another or takes place inside it).

Instead of page-based tracking, I think the basic level of tracking in an application should be at the unit-of-work level. Functions inside the unit-of-work could be tracked as “steps” or “tasks” depending on whether the unit-of-work was ordered or unordered.

How would this work? Going back to our mapping application, a unit-of-work might be triggered when a user does an address search.

This “search unit” would persist as a user pans, zooms, and adds an overlay for hotels, and then clicks on a hotel or two. The unit would end when the user enters a new search or exits the application (and possibly on other actions).

The measurement system would track how often a visitor did each type of unit of work (how often the “search unit” was triggered, how long the user spent in the search unit, how long key actions (like the overlay fill) took to perform, the use-state at each point in the unit or work, and what the visitor did to exit the unit.

Summary

I’m not sure that these four concepts (unit of work, performance, rest state, application state) are a complete description of an application measurement framework, but they would go a long way to providing logical, rich, and appropriate measurement to applications. Some can (and should) be done even if you are coding in current measurement systems – though all require significant hacks to do right now. But if Microsoft and Omniture are working to build robust application measurement, creating a measurement framework that embedded these concepts would truly advance the state-of-the-art. In my next post, I’ll take up one of the true pain-points in application measurement – testing.

Measuring Online Applications – Structuring a Measurement Interface

Online Applications and highly-interactive web sites written in full-on programming languages are becoming the norm on the web, not the rare exception. Measuring these applications is challenging for a whole host of reasons. But one of the biggest problems right now is how clumsy the javascript tagging method is when applied to applications.

Most measurement systems now work similarly – they rely on a small javascript program (the tag) that allows a coder to customize the values for certain variables and then call a function that assembles those values, adds a set of environmental variables (like browser and referring site), creates an image request and then executes that image request. All of the custom and environmental variables are passed as parameters in the image request and are decoded by the vendor when the request is logged on their end.

Think about it – all the fuss and muss of tagging – just to make a call to another server and pass a few values. In the world of applications, there are many ways of handling this basic task, by far the most common being to use a web service. Web services are vastly more convenient for the application developer than the current method (which is well suited to static HTML). Indeed, it's hard to imagine a more primitive interface for web communications than passing parameters in image requests.

The virtues of web services include the fact that they are directly supported within every significant programming language (including direct support within the programmers development environment), they provide methods that allow for back-and-forth communication and error-handling when necessary, they provide more control over the nature and amount of information passed, and they typically encapsulate the information passing into much more logical and understandable programmatic units.

It’s true that some measurement vendors provide data insertion APIs. Omniture, for example, has one. But the Omniture Data Insertion API is not a real web-service. It’s just a ”post” based mechanism for sending data. It’s might be the right method to select if you’re coding SiteCatalyst measurement in an application right now – but it’s hardly a robust interface.

But to me, the most important part of building a good interface between application system and measurement system isn’t the handshake mechanism, it’s the structure of the programmatic interface.

Here’s some sample SiteCatalyst code that might get attached to a click-handler when an event is firing:

var s=s_gi('rsid');s.linkTrackVars='eVar34,prop8,eVar8,prop9,eVar9,prop32,eVar32,eVar50,events';s.linkTrackEvents='event3';
s.eVar34='+1';
s.prop8='[Section]:upload files';           
s.eVar8='[Section]:upload files';
s.prop9=s.pageName+':[Section]:upload files';
s.eVar9=s.pageName+':[Section]:upload files';
s.prop32=s.eVar32='upload files';
s.eVar50='+1';
s.events='event3';
s.tl(this,'o','upload files');

These 13 lines of code create the basic SiteCatalyst measurement object (s), tell SiteCatalyst which variables to pass (linkTrackVars), which events to pass (linkTrackEvents), assign seven different variables and one event, and then call the s.tl function that generates and fires the image request.

Strewing chunks of code like this – even if it’s translated into C# or java - all over your application makes for an incredibly messy situation: it’s hard to document, hard to code, hard to test, and very hard to maintain. And like most poor coding mechanisms, it means that if you want to change measurement systems, you’re faced with a significant re-wiring task.

If this is the wrong way to integrate measurement into an application, what’s the right way?

Well, it isn’t to give the programmer a function call like this:

doPageView(“pagename”, “events=1,3,5”, “evar1=test;evar2=test2”, “prop1=red”);

In truth, even this would be better than what we have now (though it’s easy enough to create this sort of wrapper). But this simple function library approach doesn’t create a structure that embeds measurement logically inside the application.

What I’d like to see is a system where you started with a basic measurement object. This object could be instantiated once by the application (though applications could use more than one) and would contain, at minimum, the target account information and the configuration of environmental variables (what needs to be pulled and passed).

Ideally, environmental variables would only be passed once on invocation of the application (since apps aren’t stateless like the web, the challenge of inferring sessions and the necessity for re-passing environmental information with every request doesn’t exist). This would dramatically reduce the amount of information required with each call and the corresponding network bandwidth required.

Underneath this global measurement object would exist one or more measurement objects that inherit the global state and allow for custom mappings of application variables. The custom mappings would let the developer populate measurement variables from object properties (like Name, ID, Value or isVisible), application variables, static values or special function.

In this way, a measurement variable could be populated with the Name Property (for example) of any object to which to which it was attached. If attached to a Form Object, it could automatically create something like a pagename variable for an application by mapping the Form's "name" property to the measurement pagename variable.

The measurement object could also be configured to support any or all of the default or specific methods that objects in the development environment have. In other words, a measurement object could be configured with a specification for “load”, “unload”, “click”, etc. This would allow the developer to automatically fire a measurement call in response to either specific functions (like a search) or general functions (like a form load).

With this approach, the developer can take advantage of built-in UI constructs and also manage a small set of application specific variables that will automatically be translated into the measurement system equivalents whenever an actual measurement call is made.

When I create an object in the interface, I would then attach my measurement object to it. When the object is loaded, any handlers for the measurement object would be fired if present. It would automatically set its values, give the programmer an opportunity for a call-back (which should be rare) and then fire the request.

This is rather abstract, so let me show how this work in a more concrete fashion.

I’d start with a measurement object in which I would set the global information about the account (and maybe the vendor) I’m sending information to along with the environmental variables I want to pass on invocation.

Next, I’d create a custom variable object that described the specific variables I want to pass. Let’s suppose that I’m working on a Mapping Application and I want to collect the viewed area (the region/country/city shown in the Map Title), the map view (road, topo, satellite, etc.), the map state (overlays, zoom level), the language of the map, and the visitor identity. I would map each of these application variables to a specific measurement object variable. Next, I’d assign the click-handlers I want the measurement to fire on (I may want it on every re-draw but I might also want it on a more limited set of actions such as onSearch and onMapChange). Finally, I’d attach the measurement object to the underlying UI object.

That’s pretty much it. Once the object is attached, the programmer is done. When the object to which the measurement is attached fires an appropriate handler, the object would inherit its default state, populate the variables (like viewed area, map state, etc.), and then fire the measurement request.

Depending on my application, I might need only one custom measurement object or I might need quite a few. But with this type of system, the measurement objects would be embedded into the underlying GUI objects and their handlers. The measurement layer would deeply in-grained, it would provide inheritance, and it would provide configurability. It would be much, much easier to maintain such a system. The measurement objects themselves would be compact and discrete and they would be directly attached to the objects in which they reside.

Having a measurement integration at this level inside an application development framework would make the development process dramatically easier. Indeed, it would be possible to achieve a very high-level of measurement with a simple default configuration of the measurement object that might simply “fall-out” of a single drag-and-drop of the measurement object into the application. Such a default configuration might capture the object name and title of every form on load and unload, the object name and title of every “play” movie action, and the selected value from any Combo-Box. But it would also provide a high-level of control within the application for measurement at a very fine-grained object level.

As I mentioned in my first post, there really aren’t any serious barriers to building this type of system. There are probably different ways of achieving a deep measurement integration or designing the measurement objects – and I’m not confident that what I’ve suggested is the exactly right answer. But some form of deeply embedded measurement at the object level within the app framework seems both inevitable and right. Having an embedded measurement system won’t solve all the problems in application measurement, but it would be a big step in the right direction.

In my next post, I’ll take up a thornier issue  – the right measurement paradigm in a world where page views and link clicks aren’t the ultimate measurement objects!

Measuring Online Applications

I’ve been doing a series of posts on topics that came up at X Change but today’s topic is even more timely than it was just one month ago. With Adobe’s acquisition of Omniture, the goal of increasing the measurability of web applications has really come to the forefront. It seems to be the core strategic rationale for the whole Omniture deal; making it an issue that’s really gotten everyone’s attention.

When I blogged on the deal, one of my main points was that while the strategic rationale makes sense, capturing the benefits will not be a slam dunk. When analysts think about this deal, they tend to assume that Adobe and Omniture will be able to deliver significantly enhanced application measurability. The question for them is how big a win is that? But I don’t think the barriers to measuring applications are necessarily simple or exclusively technical.

I had the good fortune to meet Michael Scherotter of Microsoft at X Change and we got a chance to talk in-depth during what I think was the smallest Huddle of the entire Conference in that brutal late-Friday-afternoon close of conference time slot. I don’t blame anyone who split out to enjoy some pretty spectacular San Francisco weather and it ended up working out great for me. It was a really interesting wide-ranging conversation but of particular interest was Michael’s current focus on creating better and more integrated measurement for Silverlight – the development environment that may end up being Adobe’s most serious competition.

Application measurement has been on mind in our work this past year; partially because of the work we are doing in mobile. For the mobile space, applications have become a huge focus – and while mobile apps have some unique measurement challenges, they share many of the same challenges as fixed web apps. In addition, we tend to work on implementations for more complicated web sites, and in the past year we’ve tackled a range of implementations that were really applications not web sites. We even did a whole class at Think Tank on measuring applications, and though I didn’t teach that class, I wrote a goodly chunk of it.

So we at Semphonic have had lots of recent practice experience in measuring applications and how frustrating it can be. In today’s post, I’m just going to outline what I see as the core problems and then I’m going to follow-up with a more in-depth look at each.

Building a framework for application measurement involves at least five (five!) significant problems.

Of these, the most obvious and maybe the most important is the mechanism for incorporating measurement. The current art of web analytics has been built around javascript tagging – with programmers doing page customizations and writing all sorts of little functions for click-handlers. This style doesn’t work for applications. Not only is javascript the wrong tool for the job, but this approach leads to far too much spaghetti code and far too much work for each measurement call to be appropriate in a development environment.

So the first task for measurement is to create an approach that lets developers integrate measurement quickly and easily. Programmers expect to be able to accomplish 3rd Party functionality like this with APIs and Function Libraries that require no more than a single call per invocation and encapsulate measurement variables in some logical structure.

Creation of a modular API or function-based approach to measurement is largely a technical challenge and really should be fairly straightforward. But the second major problem is more challenging. Application measurement has been shoe-horned into a measurement system designed for web sites based around page views. When we do application measurement, we have to spend a significant amount of time trying to come up with a translation between the way the application works and measurement concepts like page views and paths. It’s usually a clumsy fit. Omniture has done a decent job building a framework for video measurement – but I think that’s a much easier problem to solve. Useful analysis is going to require a real framework for application measurement and that doesn’t currently exist.

Problem three is one of the most neglected aspects of tagging in general – testing. We see lots of implementations where the level of testing on rollout is just pathetic. That’s a bad idea even when you’re tagging a traditional website. But you can’t get away with sloppy or non-existent testing when measuring applications – not least because your development groups won’t let you. Applications are built in formal stages with significant and carefully controlled testing.

There is no automated testing system for web analytics right now (though Observepoint moves in that direction) and it is much, much too hard to actually test. Having to integrate all sorts of new testing steps into an application process sucks down time and resources and makes getting measurement much more challenging than it should be.

Problem four is a combination of cost & performance. The traditional web analytics cost model is based on server calls. The cost of measuring web sites is generally considered fairly reasonable. But for many of our clients, trying to micro-track inside video has resulted in lots of server calls – more than the information is sometimes worth. For applications, the problems are likely to be similar. Will the server call model really hold up in application measurement and how much will it impact the style of measurement called for?

Lastly, we have the problem of analysis. Just as we don’t have a good framework for capturing and showing application usage, we don’t really have an analytic paradigm for using the data. Without such a paradigm, the whole exercise to integrate more measurability in applications could easily be an expensive waste. Indeed, it’s almost guaranteed to be an expensive waste in most early attempts. One thing I’ve certainly seen in our application measurement is that in addition to the usual concerns of web measurement, when we code application measurement there are almost always a whole set of measurement requirements and questions generated by the GUI designers. You might think the same would be true of websites, but it really isn’t. GUI application measurement is generally a new class of analysis in our field and in our experience so far it changes many aspects of what you collect and how you use the data.

I wouldn’t classify any of these concerns as trivial – and only the first (modularization) and third (testing) are primarily technical. Adobe or Microsoft really don’t have to deliver on every one of these challenges to be successful. Solving the technical problems would, at the least, open the door to easier resolution of the other challenges. But a really good application measurement framework would solve all of the first four problems and might even advance a point of view on good framework for analysis.

Data Warehousing Analytics Data and the Universal Tag

The Really Big Topics at X Change

I’ve been doing a series of posts on thoughts from X Change Huddles and I started with two on behavioral integration of survey data – which was the very first Huddle I took part in. But this was by no means the hot topic at X Change. In fact, the Think Tank class on survey/behavioral integration was one of the least popular offerings we did (much to my chagrin). What were the really big topics? There were two – warehousing online analytics data and using a universal tag. Both came up over and over again. I was mildly surprised by how popular the first was – shocked by the amount of attention given the second. What’s interesting is that the two topics are more related that people might think – and that’s what I want to talk about today.

People have been talking about integrating online and offline data for years of course. But a couple of things this year felt different. Quite a few companies were actually doing real warehousing projects. Some of these were deep into production others felt more exploratory. Either way, people were clearly moving toward making this a reality. Given the broader economy, that’s impressive. Warehousing projects tend to be large investments – not easy to do in a seriously down economy.

The other thing that felt different this year was the number of companies that simultaneously feel confident of their basic web analytics infrastructure and reporting but dissatisfied with the maturity of their analysis. Achieving maturity in analysis has always been a problem – but in past years the big issue always felt like it was just achieving a quality infrastructure and reporting system. It would be an exaggeration to suggest that those issues have gone away but it felt to me like the balance has shifted. A lot more companies seem like they are getting quite a bit from their web analytics implementations and have reached the point where the existing tools just won’t give them any more.

Unlike data integration, the push for a “universal” tag is a lot newer. The idea is to implement a single point of data capture and then move the data to each of the analytics vendors who consume it. A single tag is appealing to many enterprises – after all, it isn’t unusual for a company to have five or six measurement tags on a single page. In addition to one or more full web analytics tags, ones for advertising, video, and various other niche measurement functions are all too common. A multiplicity of tags starts to be a performance burden and a nuisance from an IT perspective.

In some ways, the name “universal” tag may be a misnomer since not every universal tagging strategy even uses a tag. In some systems, wireline capture is used to grab the data which is then piped to the various information consumers. Alternatively, a single tag can capture the data to a central location which then moves just the necessary data to the consumers.

It’s a beautiful idea. The IT burden of tagging is dramatically lessened. Measurement across systems is unified with a single point of data capture (tag placement on existing pages is a common culprit in reconciliation projects). Performance on the page is no longer threatened.

Those are the immediate and obvious benefits.

But there are some less obvious but very real benefits that may be even more important.
X Change happened just before the Adobe deal, but many of those considering or using a universal tag approach named the ability to change analytic solutions as a key driver. For companies that may have migrated through two-or-three solutions in the last few years, the idea of being able to separate the capture layer from the reporting and analysis layer is pretty attractive. Let’s face it, one of the most difficult parts of switching vendors is re-tagging a site. And with the vendor landscape in web analytics looking rather less settled in some respects and the prospect of more acquisitions seemingly likely, the ability to swap software solutions without major infrastructure cost is undeniably appealing.

The last benefit I see is the one that may be the most significant – and it’s the fact that a universal tagging solution can enable warehousing with minimal additional fuss. After all, the whole point of a universal tag (however it’s implemented) is to create a single point of capture that can then move the data wherever necessary. Wherever necessary may well be your own data warehouse.  A universal tag solution eliminates the need for you to move data to a web analytics vendor and then pay them to move it back to you. Instead, your warehouse could potentially take advantage of the same underlying data stream as all of the other measurement vendors.

That’s why I started this post with the observation that warehousing analytics data and universal tagging are more connected than people may think. Universal tagging is a potential enabler for data warehousing. That’s a good thing in-and-of-itself and I think the goal of warehousing and the savings around eliminating a feed from your vendor might become a real enabler of universal tagging.

And think about this - data warehousing in conjunction with a universal tag might make some intriguing types of analytics systems (like Google Analytics for basic reporting plus a warehouse for deep-dive analysis) possible that would otherwise not be.

So far, one feature that you just can’t get out of Google Analytics is a true data feed. And that’s a feature that Google may not even have on its radar. So if you wanted to warehouse analytics data, you couldn’t use GA for data capture. With a universal tag, a system like that suddenly becomes possible.

No matter how you slice it, I expect both warehousing analytics and universal tagging to grow significantly in popularity in 2010. We may not be quite at the tipping point yet, but I wouldn’t be surprised if it’s all that far away.