It’s obvious that online applications don’t easily fit the page view paradigm of static web sites. What’s less obvious is what should replace that paradigm. I’m not prepared to give a complete answer to that question. However, in our work actually building measurement into applications, there are four themes that have emerged and seem necessary to any good framework.Measuring the Application State
When we build application measurement into traditional measurement systems, one of the most important concepts we’ve evolved is the idea of an application state. The idea behind an application state is that it captures all of the “settings” that are in place while a user accomplishes some action. Typical elements of an application state are things like “sort type,” “filter-type,” “zoom-level,” and “view-type.” In my last post, I used the example of a mapping application to illustrate some of the concepts involved in a good programmatic interface. It’s a useful example, because it’s a type of online application that virtually everyone is familiar with. In a mapping application, the application state is likely to include variables like “map type,” “overlay-type,” “zoom-level,” and “route-type.”
The idea behind an application state is simple. It allows you to track key combinations of interface elements by application function. It can also be used to reduce the number of calls you need to make to the measurement service (a critical factor for any high-volume application). You don’t need or want to capture every single mouse click in an application.
If a user clicks on zoom or sort, sending a measurement click is generally too expensive. Instead, we capture the entire set of application variables whenever key actions happen. In the map example, this would allow us to say what the most popular zoom levels are when viewing a route or adding a restaurant overlay vs. a hotel overlay. That’s incredibly useful information for GUI designers and its captured much more effectively in an application state than in any other fashion.When we build an application state into a current measurement system, we generally pass it as a List Prop or a simple string variable. However, it takes some work to get it to tie to application functions in an intelligible manner. I’d like to see an application framework capture the state as a type of thing and allow it to be viewed and counted and analyzed in conjunction with different application functions.
Rest StatesOne of the trickiest aspects of application measurement is what to do about applications where the interaction isn’t discrete. In our mapping example, panning and zooming are a good example of what I mean when I say not discrete. It’s easy to fire off a measurement call when someone clicks a button to change from a street view to a topo view. But when do you fire off a measurement call if someone is zooming? How do you know they are done?
Ignoring these non-discrete interactions works for some applications but cripples measurement for others. So what I’d like to see is a measurement framework that includes a threaded callback mechanism that can be attached to various interactions and basically just checks to see if the interaction has stopped for some period of time. I call this stop-time a rest state.
Using my map example, when a user clicks on the zoom control of a map, it would fire a measurement function that started a thread (call it zoomwait). The function would be passed three things – the action it’s checking for, the wait period required, and the measurement function to call when the wait period has been realized. Let’s say we are checking zoom and we have a 3 second wait period. The measurement thread would check to see if a zoom had been done in the last 3 seconds. If it had, it would until at least 3 seconds after the last registered zoom time and try again. It would do this until it detected that 3 seconds had passed since a zoom. It would then call the measurement function specified which would make the actual call to the measurement system and pass the new application state.
The idea behind rest states is to make it possible to measure the application state AFTER the user has finished some set of actions. This makes it possible to capture many key application states without necessarily trying to measure every application state when a user is zooming, panning, or doing some other non-discrete interaction.
The current static web site measurement framework largely ignores performance. That’s a serious drawback, but one that’s hard to solve given the limitations on measurement with our current infrastructure. But for applications, measuring app performance is really, really important.Application frameworks like .Net already include some low-level calls designed to make performance tracking around key functions possible. But not every development framework has this (many do not), and I think it’s a natural part of a measurement system. So I’d like a measurement framework to include this functionality.
To measure performance, the system should have a way of starting a unit of work and defining it as being open-ended (not terminated by another start) or closed (terminated automatically if another unit of work starts). Units of work would be named and could be closed explicitly by the application sending a close call with the corresponding name.
The measurement system could use this concept in a variety of ways including measuring total time in application (by firing off an open ended unit of work named application start that is only closed on app clean-up), measuring total time in app areas, and measuring specific application times for functions like “sort” or “synch.”
Depending on how this was used, it could just as easily measure how long a user takes for an application (like filling out a form) or how long an application takes to do processing (like placing an order). In the current web environment, these two numbers are always conflated into page times. Providing the capability to break them out would be significant win in application measurement.
Units of Work
In my discussion on performance, I touched on “units of work,” the last and, in some ways, the most important of the application measurement framework themes I’d like to see a measurement system address.
When we measure static web sites, we often employ a methodology that we call functionalism. It works by assigning specific functions to each page and then analyzing their performance relative to those functions. It doesn’t work well for applications, however. The functional layer in an application isn’t page based at all.
I’d like to see this functional concept extended directly into the measurement framework. The idea would be to encapsulate all of the major application use-cases as units of work. The framework should make it easy for the developer to kick-off a “unit-of-work” and, as with performance, control the interactions that can occur (in other words, to be able to write logic that determines what to do when one unit of work interrupts another or takes place inside it).Instead of page-based tracking, I think the basic level of tracking in an application should be at the unit-of-work level. Functions inside the unit-of-work could be tracked as “steps” or “tasks” depending on whether the unit-of-work was ordered or unordered.
How would this work? Going back to our mapping application, a unit-of-work might be triggered when a user does an address search.
This “search unit” would persist as a user pans, zooms, and adds an overlay for hotels, and then clicks on a hotel or two. The unit would end when the user enters a new search or exits the application (and possibly on other actions).The measurement system would track how often a visitor did each type of unit of work (how often the “search unit” was triggered, how long the user spent in the search unit, how long key actions (like the overlay fill) took to perform, the use-state at each point in the unit or work, and what the visitor did to exit the unit.
I’m not sure that these four concepts (unit of work, performance, rest state, application state) are a complete description of an application measurement framework, but they would go a long way to providing logical, rich, and appropriate measurement to applications. Some can (and should) be done even if you are coding in current measurement systems – though all require significant hacks to do right now. But if Microsoft and Omniture are working to build robust application measurement, creating a measurement framework that embedded these concepts would truly advance the state-of-the-art. In my next post, I’ll take up one of the true pain-points in application measurement – testing.