Measuring Online Applications

I’ve been doing a series of posts on topics that came up at X Change but today’s topic is even more timely than it was just one month ago. With Adobe’s acquisition of Omniture, the goal of increasing the measurability of web applications has really come to the forefront. It seems to be the core strategic rationale for the whole Omniture deal; making it an issue that’s really gotten everyone’s attention.

When I blogged on the deal, one of my main points was that while the strategic rationale makes sense, capturing the benefits will not be a slam dunk. When analysts think about this deal, they tend to assume that Adobe and Omniture will be able to deliver significantly enhanced application measurability. The question for them is how big a win is that? But I don’t think the barriers to measuring applications are necessarily simple or exclusively technical.

I had the good fortune to meet Michael Scherotter of Microsoft at X Change and we got a chance to talk in-depth during what I think was the smallest Huddle of the entire Conference in that brutal late-Friday-afternoon close of conference time slot. I don’t blame anyone who split out to enjoy some pretty spectacular San Francisco weather and it ended up working out great for me. It was a really interesting wide-ranging conversation but of particular interest was Michael’s current focus on creating better and more integrated measurement for Silverlight – the development environment that may end up being Adobe’s most serious competition.

Application measurement has been on mind in our work this past year; partially because of the work we are doing in mobile. For the mobile space, applications have become a huge focus – and while mobile apps have some unique measurement challenges, they share many of the same challenges as fixed web apps. In addition, we tend to work on implementations for more complicated web sites, and in the past year we’ve tackled a range of implementations that were really applications not web sites. We even did a whole class at Think Tank on measuring applications, and though I didn’t teach that class, I wrote a goodly chunk of it.

So we at Semphonic have had lots of recent practice experience in measuring applications and how frustrating it can be. In today’s post, I’m just going to outline what I see as the core problems and then I’m going to follow-up with a more in-depth look at each.

Building a framework for application measurement involves at least five (five!) significant problems.

Of these, the most obvious and maybe the most important is the mechanism for incorporating measurement. The current art of web analytics has been built around javascript tagging – with programmers doing page customizations and writing all sorts of little functions for click-handlers. This style doesn’t work for applications. Not only is javascript the wrong tool for the job, but this approach leads to far too much spaghetti code and far too much work for each measurement call to be appropriate in a development environment.

So the first task for measurement is to create an approach that lets developers integrate measurement quickly and easily. Programmers expect to be able to accomplish 3rd Party functionality like this with APIs and Function Libraries that require no more than a single call per invocation and encapsulate measurement variables in some logical structure.

Creation of a modular API or function-based approach to measurement is largely a technical challenge and really should be fairly straightforward. But the second major problem is more challenging. Application measurement has been shoe-horned into a measurement system designed for web sites based around page views. When we do application measurement, we have to spend a significant amount of time trying to come up with a translation between the way the application works and measurement concepts like page views and paths. It’s usually a clumsy fit. Omniture has done a decent job building a framework for video measurement – but I think that’s a much easier problem to solve. Useful analysis is going to require a real framework for application measurement and that doesn’t currently exist.

Problem three is one of the most neglected aspects of tagging in general – testing. We see lots of implementations where the level of testing on rollout is just pathetic. That’s a bad idea even when you’re tagging a traditional website. But you can’t get away with sloppy or non-existent testing when measuring applications – not least because your development groups won’t let you. Applications are built in formal stages with significant and carefully controlled testing.

There is no automated testing system for web analytics right now (though Observepoint moves in that direction) and it is much, much too hard to actually test. Having to integrate all sorts of new testing steps into an application process sucks down time and resources and makes getting measurement much more challenging than it should be.

Problem four is a combination of cost & performance. The traditional web analytics cost model is based on server calls. The cost of measuring web sites is generally considered fairly reasonable. But for many of our clients, trying to micro-track inside video has resulted in lots of server calls – more than the information is sometimes worth. For applications, the problems are likely to be similar. Will the server call model really hold up in application measurement and how much will it impact the style of measurement called for?

Lastly, we have the problem of analysis. Just as we don’t have a good framework for capturing and showing application usage, we don’t really have an analytic paradigm for using the data. Without such a paradigm, the whole exercise to integrate more measurability in applications could easily be an expensive waste. Indeed, it’s almost guaranteed to be an expensive waste in most early attempts. One thing I’ve certainly seen in our application measurement is that in addition to the usual concerns of web measurement, when we code application measurement there are almost always a whole set of measurement requirements and questions generated by the GUI designers. You might think the same would be true of websites, but it really isn’t. GUI application measurement is generally a new class of analysis in our field and in our experience so far it changes many aspects of what you collect and how you use the data.

I wouldn’t classify any of these concerns as trivial – and only the first (modularization) and third (testing) are primarily technical. Adobe or Microsoft really don’t have to deliver on every one of these challenges to be successful. Solving the technical problems would, at the least, open the door to easier resolution of the other challenges. But a really good application measurement framework would solve all of the first four problems and might even advance a point of view on good framework for analysis.

Comments

[Michael Feiner]

I've recently sat through a demo of Speed Trap. This UK-based company is known locally for its web analytics tool. I've used it in the past to a good degree of satisfaction.
However, they now seem more focused on the data collection side rather than the reporting interface. Their measurement model tackles several of your points successfully.
They use a unique JS tracking code on each page that can track user events without having to tag each event separately. This significantly simplifies the tagging and testing processes.
The pricing model is based on number of visits rather than server calls so micro-tracking shouldn't necessarily increase cost.
Speed Trap recommend sending the collected data into a data warehouse for analysis and segmentation.
The data could also be used in conjunction with real-time decisioning tools such as Chordiant or with content and campaign management tools such as Alterian and Neolane.
Speed Trap have kept a low profile so far. But intend to ramp up their marketing in 2010. So might start making a name for themselves in the US.
I actually think that Adobe would have been better off buying Speed Trap rather than Omniture if the whole integration of measurement was at the heart of the deal. Would have been cheaper and more appropriate from a technological stand point (given Speed Trap's app tracking capabilities).

I still believe there was a market positioning element to that acquisition - Adobe is positioning itself as a potential acquisition target for Microsoft. It already has the application platforms competing with Silverlight. Now it has the measurement platform. At least that is my speculation.
Anyway, Speed Trap is certainly a company to watch in this space.

Michael
URL link: www.aepconvert.com

[Adam Greco]

Gary,

I agree with many of your points. At my current employer, our transaction volume and security concerns make tracking our application prohibitive. I think this is an area for a nimble vendor to exploit!!