Dylan’s an acknowledged leader in web analytics testing. He did a great speech at eMetrics last time around on the topic and he’s done deep work in it for TurboTax. And while I have to admit I’m not fond of the Huddle Title (“Test Yourself Rich” – given my pessimistic side I would have preferred “Testing’s a Bitch”), talking with Dylan about how to think about and run a testing program is always a treat. But it’s his second topic, Measuring Applications and Intranets, that really intrigues me.
I’ve felt for years that there is an untapped need for serious measurement and analysis of SaaS applications. SaaS applications are, after all, the most complex of web sites. And while they often receive more comprehensive usability design and testing, better performance monitoring, and much better QA than traditional web sites, they simply don’t get the equivalent treatment when it comes to measurement.
Internet SaaS Applications have benefited, in their creation, from the rich processes setup to build traditional software. But this didn’t carry over into measurement because traditional software never got well measured either.
A combination of factors is changing all this. First, there’s a growing realization by measurement teams working on the marketing side of these web sites that similar analysis opportunities must exist on the application web site. Second, of course, are mobile apps.
As I’ve blogged before, measuring applications is quite different than measuring traditional web sites. To do a good job of it, you have to deal with some fundamental challenges that either don’t exist or exist in much less serious form on traditional web sites:
1. What is the fundamental unit of measure (i.e. the Page View in Traditional Measurement)?
2. What happens to measurement if you’re offline (mobile) or in the background (a new twist as mobile operating systems get more sophisticated)?
3. What is the appropriate way to id users and what information can be collected?
4. How do you measure app performance (it isn’t page load time)?
5. How do you measure success?
6. How do you tell if a design is working?
And, because many of the basic questions are different, many of the analysis techniques that are most useful in traditional fixed web analytics simply don’t work or aren’t that interesting when it comes to apps. That’s not necessarily a big issue for the great majority of companies that haven’t done much interesting analysis on their fixed web sites either, but it’s a bit a shock for any analysis team that’s actually accomplished something on the fixed web.
Because of mobile apps, this is a much bigger topic than it would have been two years ago. Back then, it would still (or should still) have been of interest to companies like Salesforce and Intuit because SaaS applications are a big part of their business. Now, however, nearly every company we work with has launched or is launching some form of mobile application focused on marketing or sales.
So whether your interest is in traditional SaaS applications on the web, or in rich media experiences embedded in your traditional web site, or in intranet applications for your company or in mobile apps, getting a chance to sit down with Dylan and others and talk over the how, the what, and the why of application measurement should be pretty darn useful.