Exploring “Exploratory Testing”

  • Article

My current team has been integrating this testing technique for a while now along with the traditional testing methods of manual testing as well as automation.  Most of the testing community that I have known seem to brush this testing technique off simply as “monkey testing” which doesn’t really provide any significant yield on bugs nor achieve high quality bugs.  However, I have been observing how powerful this testing method can be from my own perspective as well as by seeing the type of bugs that some of my team mates find. 

Recently, I went for a test talk on this testing topic and to my surprise there are a few other teams within Microsoft that actually use this testing technique as well.  Here are the short summary of what I did learn from that talk:

  1. In summary, many testers still under-estimate the power of this technique and simply disregard it as another “ad-hoc” testing.  They think that this is simply ad-hoc testing or monkey testing where tester just randomly bang and shake the product until bugs are found.  However, this technique is actually proven to find critical bugs, cool bugs, and more importantly the kind of bugs that customers would encounter as they use our products.
  2. On the other hand, if testers use Exploratory testing in unplanned, undocumented, random manner, then that can be a waste of time and energy. 
  3. In order to maximize Exploratory testing, testers should sit down and come up with a strategy or plan on what testing techniques to use.  This helps keep the testers stay in focus, and also apply the right techniques to get the most out of an Exploratory testing session time.
  4. Knowledge sharing is key to Exploratory testing.  Make use of historical data such as the type of bugs found in previous versions of the product, which features are buggy, customer-reported bugs, even knowing your own devs and their way of coding can also help uncover holes and bugs (human tend to repeat the same mistakes).
  5. It’s quite difficult to determine what is the right balance between Exploratory testing and the traditional testing methodologies.  How much exploration should a tester execute in order to get the biggest bang for the buck.

Exploratory Testing Techniques

According to the speaker, here are the 4 key techniques that any testers can refer to and use for their exploratory testing.  Each component has its own strengths/weaknesses.  Tester do not have to stick with a single component, but should spread out and make use of all of them or a combination of them.  The trick is to know which one to choose and apply and when. 

1. Freestyle

This is basically Ad-hoc testing.  Hit the product from black box standpoint.  From the customer’s perspective.  Pros: Low cost.  Anyone can do it.  Encourage creativity.  Cons: May not yield the critical bugs.  Low quality bugs are typically found with lots of dupes.  Difficult to determine coverage.

2.  Scenario based

Using a pre-defined end-to-end scenarios, complete with actual reproducible steps and validation methods prior to testing.  Testers can use these scenarios as the base, and figure out new end-to-end path that will achieve the same goals.  Pros: Easy to trace the repro steps and follow.  Can determine test coverage.  Cons: Pre-defined scenarios may actually not represent what the customer really wants. 

3.  Strategy based

Testers do have some background of the product.  Thoroughly understand the architecture and flow of the product.  Tester can then leverage the product knowledge and combined with well-known testing techniques to go about testing.  More to come on the techniques later.  Pros: Test techniques are proven to be effective.  Testers just need to master them and apply them in the right situation.  Testers are also pretty much free to use their own instinct and creativity to go about testing the product.  Focus on learning.  Cons: Newer testers on the team may not have the sufficient experience or knowledge of the product to be completely effective. 

4. Feedback based

Start out pretty much same as Freestyle testing.  But as testing sessions are conducted, testers build up a history of test executions, areas covered, bugs identified, code churns, etc.  Very similar to our session notes.  Testing from historical data such as bug records, data from automations and scripts, application logs, customer reported issues, etc.  Use these data as input so tester can identify which areas may yield the most bugs.  Pros: Knowledge is not lost.  Make use of previous knowledge is very powerful.  Cons: Sometimes it’s very difficult to make use of existing data effectively (especially if there are a lot).  May actually spend more time gathering and processing these data rather than actual testing.