Online Applications and highly-interactive web sites written in full-on programming languages are becoming the norm on the web, not the rare exception. Measuring these applications is challenging for a whole host of reasons. In my last post, I talked about the importance of automated testing to bring measurement fully into the application development life-cycle. In today’s post, I’m going to cover an issue that plagues application measurement, video and most other rich media measurement – how to balance cost vs. coverage when it comes to measurement.
The problem of cost in measurement springs from our basic model. With SaaS implementations, each server call you make to your measurement system costs money. Not a lot of money, of course. And for web sites, there has been little doubt that the knowledge gained from page measurement more than offsets the per-page cost of collection. On most web sites, a typical visitor might view six to ten pages. A really engaged visitor might view twenty to thirty pages. In an online application, however, the number of user interactions (every click, zoom, and setting) can be much, much higher.
Without a clear paradigm for a unit of measurement (such as a page view), it’s unclear which of these interactions should be measured. Capture every one, and you’ll likely find that the cost of measurement outweighs the benefit.
As I mentioned, we’ve seen similar issues with video measurement. Many of our media clients went from measuring almost nothing about video to detailed measurement of how far viewers penetrated into each and every visitor played. But to get that measurement, they usually had to make multiple server calls per video – sometimes as often as every 10 seconds. For a 2 minute video, that means a full-play results in 12 server calls. That’s a lot of measurement expense relative to the value of tracking abandonment per 10 second interval.
In fact, a lot of that more detailed measurement ended up getting pulled.
We’ve seen similar pull-backs when clients over-measured flash experiences – tracking roll-overs for example. After awhile, most companies find they aren’t using that level of detail about user interactions very effectively and the cost of measurement isn’t justified.
In an earlier post, I described a framework for measuring applications that included four core concepts: units-of-work, application state, rest-states, and performance. Part of the reason I settled on these concepts is that they provide a reasonable balance of measurement vs. cost.
A unit-of-work is a function within the application. The unit-of-work would typically encapsulate a single use-case (like getting driving directions for a mapping program or paying a bill in an online bill-pay system). The measurement system would make a call whenever a unit-of-work is initiated. With this call (and all calls) it would pass an application state.
The application state is a set of variables that capture all the important settings a user might have (like sort-order, filters, zoom level, tab open, etc.). The goal of application states was to be able to tie application settings to important actions (units-of-work) without having to pass measurement calls every time a state is changed (like a sort).
Now I’m going to add a new concept – milestones. Milestones would be way-markers within units-of-work. A milestone could be named and sequenced if desired. Whenever a milestone within a unit-of-work occurs, the measurement system would fire off a call. Every unit-of-work would have at least two associated milestones – an abandon milestone and a completion milestone.
Using the milestone concept, developers could control how much measurement is being done within a unit-of-work.
When a milestone is reached, the measurement system would automatically get a new application state and it would generate and pass a performance measurement for the previous milestone.
With this type of framework, there would be little need to track every user change or setting with a server call. Each measurement call would capture some significant action along with the associate states. On the measurement end, comparing states between milestones would make it easy to produce reports about what settings users changed when performing a unit-of-work.
The unit-of-work/milestone combination provides developers with a very granular way to control the level of measurement in the application (that’s important) and, potentially, to be very sparing about the cost of measurement.
It’s quite possible to duplicate some of this functionality in existing systems today. In Omniture, for example, you can code units-of-work using an s.prop variable. By turning on pathing, you can track movement between units-of-work. By using a combination of unit-of-work and milestone in a second prop, you can enable pathing within milestones. You can use a list-prop to capture application state or you can use a carefully formatted and delimited string in a normal string s.prop.
If you correlate that application state variable with the units-of-work s.prop and the milestones s.prop, you have basic reporting on how application states changed across milestones. Of course, you have to do most of the work to use those reports since the application isn’t going to really help you any. And you also have to do ALL the work to setup and capture this stuff.
Of course, that’s the whole point of this series. You CAN measure applications right now, it’s just not nearly as easy or powerful as it ought to be.
With that in mind, my next (and final) post in this series will tackle the last and probably the most difficult part of measuring applications – figuring out how to analyze the information to make it useful!
Comments