the future of software testing (part 8)

Testing Beyond Release

This is the final part of my series on the future of testing. I hope you’ve enjoyed it. For this post I’ve saved what might be one of the more controversial of my predictions: namely that in the future we will ship test code with our products and be able to exercise that code remotely. I can see the hackers’ grins and hear the privacy advocates’ indignation already, but I’ll respond to those concerns in a minute.

I was in the Windows org when Vista shipped and I recall demonstrating it to my then 8 year old son at home one evening. He plays (and works if you’ll believe that) on computers a great deal and he really liked the Aero interface, the cool sidebar gadgets and the speed at which his favorite games (which at that time were line rider and zoo tycoon) ran really impressed him. I recall thinking ‘too bad he’s not an industry blogger,’ but I digress.

At the end of the demo, he hit me with the question every tester dreads: ‘Daddy, which part did you do?’

I stopped speaking, which is rare for me, and stammered something unintelligible. How do you tell an 8 year old that you worked for months (I had just started at Microsoft and only got in on Vista toward the end of its cycle) on something and didn’t actually create any of it? I tried my canned answers to this dreaded question (exclamation points required … they help me convince myself that what I am saying has some truth to it):

“I worked on making it better!”

“The fact that it works as well as it does … well that’s me!”

“If it wasn’t for us testers, this thing would be a menace to society!”

I am especially fond of that last one. However, all of them ring hollow. How is it that I can work on a product for so long and not be able to point to more than the absence of some of the bugs as my contribution?

I think that’s where this idea came from: that test code should ship with the binary and it should survive release and continue doing its job without the testers being present. This isn’t a lame attempt to give me and my compatriots something to point to for bragging rights, but to provide ongoing testing and diagnostics. Let’s face it, we’re not done testing when the product releases, so why should we stop?

We already do some of this. The Watson technology (the famous “send/don’t send” error reporting for Windows apps) that ships in-process allows us to capture faults when they occur in the field. The next logical step is to be able to do something about them.

Currently, Watson captures a fault and snaps an image of relevant debug info. Then some poor sap at the other end of the pipe gets to wade through all that data and figure out a way to fix it via Windows update. This was revolutionary in 2004, still is actually. In 2-5 years it will be old school.

What if that poor sap could run additional tests and take advantage of the testing infrastructure that existed before the software was released? What if that poor sap could deploy a fix and run a regression suite in the actual environment in which the failure occurred? What if that poor sap could deploy a production fix and tell the application to regress itself?

He’d no longer be a poor sap, that’s for sure. But since testing and test artifacts are lost when the final product is built and released this is all impossible.

In order to achieve these capabilities, it will be necessary for an application to remember its prior testing and carry along that memory wherever it goes. And that means that the ability to test itself will be a fundamental feature of software of the future. Our job will be to figure out how to take our testing magic and embed it into the application itself. Our reward will be the pleasure of seeing that sparkle in our kids’ eyes when they see that the coolest feature of all is the one we designed!

Oh, and to the hackers and privacy folks: never fear! Hugh Thompson and I warned about including test code in shipping binaries (see Attack 10 in How to Break Software Security) long ago. Since we know how to break it, we’ll be in a great position to get it right.

Comments

• Anonymous
  October 16, 2008
  PingBack from http://www.alvinashcraft.com/2008/10/16/dew-drop-october-16-2008/

• Anonymous
  December 20, 2008
  This is something I really look forward to! :)

• Anonymous
  February 03, 2009
  [Nacsa Sándor, 2009. január 13. – február 3.]  A minőségbiztosítás kérdésköre szinte alig ismert

• Anonymous
  February 05, 2009
  [ Nacsa Sándor , 2009. február 6.] Ez a Team System változat a webalkalmazások és –szolgáltatások teszteléséhez