prevention v. cure (part 2)
Ok, re-hire the testers.
Perhaps you’ve noticed but the whole prevention thing isn’t working so well. Failures in software are running rampant. Before I talk about where we should invest our resources to reverse this trend, I want to talk about why prevention fails.
I see a number of problems, not the least of which is that good requirements and specifications seldom get written and when they do they often fall out-of-date as the focus shifts to writing and debugging code. We’re working on that problem in Visual Studio Team System but let’s not get ahead of ourselves. The question in front of us now is why prevention fails. It turns out, I have an opinion about this:
The developer-makes-the-worst-tester problem. The idea that a developer can find bugs in their own code is suspect. If they are good at finding bugs, then shouldn’t they have known not to write the bugs in the first place? This is why most organizations that care about good software hire a second set of eyes to test it. There’s simply nothing like a fresh perspective to detect defects. And there is no replacement for the tester attitude of how can I break this to compliment the developer attitude of how can I build this.
The software-at-rest problem. Any technique such as code reviews or static analysis that don’t require the software to actually run, necessarily analyzes the software at rest. In general this means techniques based on analyzing the source code, byte code or the contents of the compiled binary files. Unfortunately, many bugs don’t surface until the software is running in a real operational environment. Unless you run the software and provide it with real input many bugs will simply remain hidden.
The no-data problem. Software needs input and data to execute its myriad code paths. Which code paths actually get executed depends on the inputs applied, the software’s internal state (the values of the data structures and variables) and external influences like databases and data files. It’s often the accumulation of data over time that causes software to fail. This simple fact limits the scope of developer testing which tends to be short in duration…too short to catch these data accumulation errors.
Perhaps tools and techniques will one day emerge that allow developers to write code without introducing bugs. Certainly it is the case that for narrow classes of bugs like buffer overflows which developer techniques can and have driven to near extinction. If this trend continues, the need for a great deal of testing will be negated. But we are a very long way, decades in my mind, from realizing that dream. Until then, we need a second set of eyes, running the software in an environment similar to real usage and using data that is as rich as real user data.
Who provides this second set of eyes? Software testers provide this service, using techniques to detect bugs and then skillfully reporting them so that they get fixed. This is a dynamic process of executing the software in varying environments, with realistic data and with as much input variation as can be managed in the short cycles in which testing occurs.
In part 3 of this blog series I will turn my attention to tester testing and talk about whether we should be doing this with automation or with manual testing.