When we talk with the development teams these days, I’m often the one who has to break the bad news that there is no reasonable automation strategy for testing measurement. Not only that, but the testing is extraordinarily difficult to integrate into basic development cycles. Someone has to run through a test plan by hand, then go to the web analytics tool and try and shift through the data to isolate the test results. It’s kludgy, manual, slow and error prone. A perfect storm for testers.
This is especially frustrating because measurement isn’t “critical” functionality. Nobody wants to slip deadlines or delay release for measurement.The changes I’ve talked about so far – especially in the measurement interface – would greatly reduce the burden on programmers to get measurement INTO the application. But they really wouldn’t do much to make it easier to test.
I’d like to see Omniture and Microsoft and other application and measurement vendors integrate testing functionality directly into the measurement system. Ideally, developers should be able to configure “testing” as an overlay on the existing measurement interface. In other words, I wouldn’t add any code to test, I’d simply set a variable or an object that controlled the “testing” state.
When testing was turned on, every measurement call would be mirrored to an additional destination. Since online applications typically run locally, I see several ways this might be handled. First, the application could potentially save the measurement in a flat file on the local system (given permission). Alternatively, the application could save the measurement calls to an interface object in some easily parsed format. Finally, the application could pass the calls back to a secondary server that simply recorded them in a flat file on the server side or was part of a testing automation suite.
With this type of setup, testing automation tools could drive through a series of application steps which would produce a measurement file that could be quickly “diffed” with previous versions to establish correctness. This would dramatically reduce the amount of time new testing cycles took while greatly improving accuracy. Even if each test cycle had to be done manually, programmers would only have to worry about checking the measurement file once and getting a correct version.
Even without the automation aspects, this type of system would make testing applications in secure and mobile environments much easier. We often do the bulk of our testing using HTTP sniffers like Firebug, Charles and Fiddler. But these sometimes don’t work or conflict with the application on development networks and in secure environments.
As I think about this, I realize that in some ways this “testing” capability mirrors a universal tag. If the measurement library makes it easy to reflect measurement calls to multiple locations, you have the basic infrastructure for universal measurement.
That seems like a pretty good idea – something that would be almost trivial to add into a measurement framework of the sort I described in the first two posts. Pegging testing on a universal tag capability has the additional benefit that people may pay more attention to it that way. Testing is one of the most time-consuming, manual, and expensive tasks in software development. Unfortunately, as the least sexy task is a discipline with very little sexiness in general, measurement testing gets less real-world attention than the “best script” honors at an adult video awards program.
But if you’re going to build a really good measurement interface, building support for automated testing right into the system would go a long way toward making the incorporation of measurement into online applications routine instead of esoteric and painful.
Comments