explaining exploratory testing

I just got finished talking (actually the conversation was more like a debate) to a colleague, exploratory testing critic and a charter member of the plan-first-or-don’t-bother-testing-at-all society.

I am happy to say, he conceded the usefulness (he would not grant superiority, if he had I would be on my way to the pub right now with his credit card) of exploratory testing. Perhaps I have finally found a way to explain the utility of exploration. Here’s what I said:

“Software testing is complicated by an overload of variation possibilities from inputs and code paths to state, stored data and the operational environment. Indeed, whether one chooses to address this variation in advance of any testing by writing test plans or by an exploratory approach that allows planning and testing to be interleaved, it is an impossible task. No matter how you ultimately do testing, it’s simply too complex to do it completely.

However, exploratory techniques have the key advantage that they encourage a tester to plan as they test and to use information gathered during testing to affect the actual way testing is performed. This is a key advantage over plan-first methods. Imagine trying to predict the winner of the Super Bowl or Premier League before the season begins … this is difficult to do before you see how the teams are playing, how they are handling the competition and whether key players can avoid injury. The information that comes in as the season unfolds holds the key to predicting the outcome with any amount of accuracy. The same is true of software testing and exploratory testing embraces this by attempting to plan, test and re-plan in small ongoing increments guided by full knowledge of all past and current information about how the software is performing and the clues it yields in the testing results.

Testing is complex, but effective use of exploratory techniques can help tame that complexity and contribute to the production of high quality software.”

Comments

• Anonymous
  January 08, 2009
  PingBack from http://www.codedstyle.com/explaining-exploratory-testing/

• Anonymous
  January 08, 2009
  The comment has been removed

• Anonymous
  January 08, 2009
  I don't think I understand why there is a contest between structural and exploratory testing methods at all. You need a balance of both to test the product to a meaningful level. I know that traditionally at Microsoft, we begin with a structured test plan, but rely a whole lot on exploratory testing to find bugs as the product cycle progresses. I don't think I would choose one over the other, but quite simply a mix of the 2. Wouldn't you agree that's common sense?

• Anonymous
  January 09, 2009
  I guess the biggest question for me is what would lead a tester to taking an exploratory approach vs. say, an planned API level approach? Obviously, they're not mutually exclusive, nor can the tester not use them both to feed each other. It really comes down to a practical problem of time and investment. The more I read your blog the more I realize that there are so many "schools" of testing that you will find many testers are drawn to one (or at best two). The best testers can employ any technique necessary to get the job done. Approproateness of methodology is what (in my my view) makes the best testers - and that requires an agile and broad set of perspectives and varied skills. For exploratory testing, what I struggle with is that it could be applied to any kind of testing task. Should it be? Or are there times when it's too costly and a more statistical approach (if possible) is the better bet.

• Anonymous
  January 09, 2009
  The comment has been removed

• Anonymous
  January 10, 2009
  It's not a contest. It's a debate. A good and healthy debate. And if this debate ends with a better understand of when planning is superior and when doing is superior and how to find the optimal mix, then everyone wins. But I can't help my competitive nature ... I am an explorer and I will use every chance I have to promote it.

• Anonymous
  January 12, 2009
  I agree strongly with anutthara. Proponents of exploratory testing will always cite something along the lines of the argument you have given, and then use this as evidence of the superiority of the technique. But the reality is that ET is just one of many approaches that should be employed during testing, depending on the sdlc and restraints in the scenario. Too many times have I seen sloppy testing being defended as being "exploratory". I think it always makes sense to produce planning before hand when the necessary information is available - e.g. functional specs, prototypes etc. Exploratory approaches can then be applied when the product goes into test in order to build on this initial planning.

• Anonymous
  January 12, 2009
  James Whittaker say that testing is Hard and I quote: “Software testing is complicated by an overload

• Anonymous
  January 16, 2009
  The comment has been removed

• Anonymous
  February 03, 2009
  [Nacsa Sándor, 2009. január 13. – február 3.]  A minőségbiztosítás kérdésköre szinte alig ismert

• Anonymous
  February 05, 2009
  [ Nacsa Sándor , 2009. február 6.] Ez a Team System változat a webalkalmazások és –szolgáltatások teszteléséhez