This year fellow Microsofties who are photographer nuts from around the world spent five months assembling and publishing a world-class, fine art photo book.  The time and talent that went into this project makes every copy of the book a gift to its owner.  And guess what, one of my photos actually made it into the book. 

Here is the full book preview online (160 pages of preview!).  I should mention that if you purchase the book, 100% of the profit from sales goes directly to the United Way and you get a valued and timeless reminder of how you too make a difference for those in need.

You can preview the book, or order the book, from your desk.  Not only does this book make a wonderful addition to any collection, and a thoughtful gift, each copy you purchase contributes directly to the ongoing, important work of United Way.  This book is offered for purchase at the cost of printing plus $25.00. This additional amount is contributed directly to United Way in your name and is fully tax deductable as a charitable contribution.

Even if you don’t think you’ll buy a copy of the book, go take a look at the online preview anyway. It really is amazing!

Oh, and don’t forget to turn to page 114.  :)

What is Session Based Testing?

Session Based Testing (SBT) is a form of Exploratory testing that is used for rapid testing, on project with tight schedule, or informal/incomplete/changing specifications.  It was developed by Jonathan and James Bach, and now being used in many QA organizations, including Microsoft.  Here is the original paper (PDF) written by Jon Bach defining SBT.  For our test team, Session Based Testing represents an “optimized, ad-hoc E2E testing, but at the same time very disciplined and focus”.  For us, it is being used to find bugs quickly and early in the product cycle.

Finding bugs very quickly, and very early is extremely powerful.  Let’s think about that for a minute.  Tradition test process which test plan needs to be written, follow by test cases, follow by numerous test passes, and then finally we start to see bug count going up.   Session Based Testing has a much more lower “cost of entry”.  It pretty much short-circuits the whole testing process with the goal of finding bugs.  The instantaneous feedback to the development team is extremely beneficial.  Now, everybody can gauge how well the product is performing event though it can still be under development.  Any bugs identified may also help shape the way code is written, or how the spec is defined.  It’s okay if developers do not want to fix these bugs right now (their usual counter-argument is always we are not code complete yet!).  But with these bugs staring at their faces everyday, it’s easy for them to realize that they need to crank out better code. 

I would want to point out that Session does not and should not replace the traditional test process.  SBT’s goal is to lower down the initial cost of  testing, and also increase flexibility and serendipity that makes Exploratory so useful.  IMO, the combination of Session Based Testing AND a well thought-out test planning is definitely a one-two knock out punches to the major bugs hiding within the code.  Every tester should definitely try it.

So, how can it be that SBT is ad-hoc and at the same time focused testing methodology?  This more or less bring us to the next topic – the difference between Sessions and Bug Bash.

Sessions VS Bug Bash

As I talked to a few people about SBT, the #1 question that everybody asked is -- “So… what is the difference between Session and Bug Bash?”  To the best of my knowledge, here is the similarities and differences between the two:

I am not saying that holding Bug Bash is a bad thing.  Bug Bash (and dogfooding) has been one of the staple diet for every testing organizations.  The more eyes looking for bugs, the better.  However, one of the main weakness of Bug Bash is that the whole process is very random and executed in somewhat ad-hoc manner.  Many times I would see everybody pretty much find the same bugs over and again.  In addition, since everybody is banging on the product at the same time, there is no good way to tell if the product has really been tested as thoroughly as it can, or if test executions are evenly spread out for all the features.  And in my humble opinion, the greatest sin of all is that testing knowledge is lost the moment Bug Bash ends. 

Session Based Testing is built on top of this same principles, but take it one step further.  Test owner should do a little bit of homework on defining what are the test areas that he or she would like to be tested.  And then the test owner will create a number of “sessions”, which typically last about 60-90 mins.  There is a clear goal or mission of what each session aims to do (e.g. “Test open file dialog in Notepad”).  But there is no execution steps defined on what steps to do, or how to go about doing that.  Tester essentially becomes monkey tester (with a focus) and let the magic of Exploratory testing kick in.

During session, the tester would need to note down what he or she had done.  More or less like micro-blogging or “tweet” if you know what I’m talking about.  :)  Could be something like this:

• I tried launching the Open File dialog using keyboard shortcut, and I found out that the hotkey to Open is “n” instead of the more traditional “o” (I just made this up).  We should keep the hotkey consistent with other applications.  Bug #123 filed.
• Dragging the Open File dialog window around, and this seems to work fine as expected.  Also drag the window outside of the screen and the window redraws itself back correctly.  (again just made this up)
• Tried opening file without extension, and Notepad crashes!  Bug #456 filed. 

Information such as these are quite critical because it provides a lot of insights into how well the product is doing.  Tester can report not only on just the issues that were identified, but also can report on the general feedback of the product itself.  This can even be opinions on what the tester thought, whether it works great, intuitive to use, great experience, or just plain out sucks.

Learning through Sessions

I think this is probably the biggest advantage that SBT provided – the knowledge that is associated with it.  Since every executed session will produce some form of data, we can then make use of those information to help us increase the quality bar.  Information gathered from sessions are really quite simple.  Here are some of examples:

• Number of sessions run
• When were they run
• Which target area were executed
• Number of bugs filed
• Any issues raised (that are not considered to be bugs)
• …

With this set of inputs, we can easily perform the following simple analysis.  In my team, we just plug the data into Excel and generate a Pivot Table out of it.  This way, slicing and dicing the data is just one click away.

• Bug density per each target area.
• Average bug yield per sessions
• Bug trend over a given period of time (Example: Total # of sessions and total # of bugs week by week for the past 3 month).  This is another way to pulse check on how good/bad the product is doing.
• Who ran the most sessions/who filed the most bugs

Somebody once said “A picture is worth a thousand words”.  So in order to save myself 2K words, let me present to you 2 pictures.  :) 

Analysis 1: Bug density per each target area

Row Labels Total Sessions Total bugs Feature 1 20 4 Feature 2 10 8 Feature 3 31 86 Feature 4 29 79 Feature 5 11 22 Feature 6 14 37 Grand Total 115 236

By looking at the graph, you can immediately tell that Feature 3 and Feature 4 are somewhat in trouble.  The total number of bugs are pretty high compared with the other features.  Using this data, you can start off investigating on why quality is lower than other areas. 

In addition, a lot of sessions were executed for Feature 1, but the bug yield is extremely low.  This might be a good indication that you might want to pull back on running more sessions this feature, and spend time on other area instead.

Okay, let’s do one more:

Analysis 2: Bug trend over a given period of time

Time Avg Bug Week 25 5.60 Week 26 4.31 Week 27 2.86 Week 28 2.68 Week 29 2.42 Week 30 1.32 Week 31 1.13 Week 32 1.50

This example just show the average number of bugs filed per week over a 2 month period.  It’s easy to see that the # of bugs are trending downwards, which is a good thing assuming that sessions are running constantly.  If you see spikes in the trend which is out of place, then it can be due to poor check in or some form of regression.  Inversely, if you see a sharp drop, it is something to look at as well.

I hope these 2 analysis help to illustrate some forms of measurement that QA team can derive and use.  Session Based Testing has certainly helped our team.  First and foremost, we are able to test creatively using Exploratory techniques, but at the same time, we are also disciplined in terms of relying on information obtained by running sessions to help us to be even more productive and efficient. 

Here is the front page search results.  Seems pretty plain old Googley.

So then I got curious and want to see how Live Search is doing.  And here is what is shown on the front page.  Immediately, all the major winner results are displayed, which is exactly what I wanted to see.  Impressive…

Oh by the way, congrats to Slumdog. 

Here is the list of valid and invalid email format which can be used for testing.  The list is not comprehensive by any means, and is probably missing a bunch of edge cases.  However, it provides enough coverage for the majority of the allowed/disallowed email address according to RFC 2822.

BTW, if anyone else would like to contribute more cases to this list, or spot errors (fingers crossed), please feel free to let me know and I'll make sure the list is up-to-date.

List of people who help contribute to help solidify the cases

gabriel.lozano-moran

bernie halpin

Avdhut

1) Order t-shirts for the development team

2) Announce the release date

3) Write the code

4) Write the manual

5) Hire a PM

6) PM spec the software based on the code (to ensure the software meets the specifications)

7) Ship/Launch

8) Test with real users (free)

9) Bugs are potential enhancements

10) Announce the upgrade

Is your team practicing this?

My current team has been integrating this testing technique for a while now along with the traditional testing methods of manual testing as well as automation.  Most of the testing community that I have known seem to brush this testing technique off simply as “monkey testing” which doesn’t really provide any significant yield on bugs nor achieve high quality bugs.  However, I have been observing how powerful this testing method can be from my own perspective as well as by seeing the type of bugs that some of my team mates find. 

Recently, I went for a test talk on this testing topic and to my surprise there are a few other teams within Microsoft that actually use this testing technique as well.  Here are the short summary of what I did learn from that talk:

1. In summary, many testers still under-estimate the power of this technique and simply disregard it as another “ad-hoc” testing.  They think that this is simply ad-hoc testing or monkey testing where tester just randomly bang and shake the product until bugs are found.  However, this technique is actually proven to find critical bugs, cool bugs, and more importantly the kind of bugs that customers would encounter as they use our products.
2. On the other hand, if testers use Exploratory testing in unplanned, undocumented, random manner, then that can be a waste of time and energy. 
3. In order to maximize Exploratory testing, testers should sit down and come up with a strategy or plan on what testing techniques to use.  This helps keep the testers stay in focus, and also apply the right techniques to get the most out of an Exploratory testing session time.
4. Knowledge sharing is key to Exploratory testing.  Make use of historical data such as the type of bugs found in previous versions of the product, which features are buggy, customer-reported bugs, even knowing your own devs and their way of coding can also help uncover holes and bugs (human tend to repeat the same mistakes).
5. It’s quite difficult to determine what is the right balance between Exploratory testing and the traditional testing methodologies.  How much exploration should a tester execute in order to get the biggest bang for the buck.

Exploratory Testing Techniques

According to the speaker, here are the 4 key techniques that any testers can refer to and use for their exploratory testing.  Each component has its own strengths/weaknesses.  Tester do not have to stick with a single component, but should spread out and make use of all of them or a combination of them.  The trick is to know which one to choose and apply and when. 

1. Freestyle

This is basically Ad-hoc testing.  Hit the product from black box standpoint.  From the customer’s perspective.  Pros: Low cost.  Anyone can do it.  Encourage creativity.  Cons: May not yield the critical bugs.  Low quality bugs are typically found with lots of dupes.  Difficult to determine coverage.

2.  Scenario based

Using a pre-defined end-to-end scenarios, complete with actual reproducible steps and validation methods prior to testing.  Testers can use these scenarios as the base, and figure out new end-to-end path that will achieve the same goals.  Pros: Easy to trace the repro steps and follow.  Can determine test coverage.  Cons: Pre-defined scenarios may actually not represent what the customer really wants. 

3.  Strategy based

Testers do have some background of the product.  Thoroughly understand the architecture and flow of the product.  Tester can then leverage the product knowledge and combined with well-known testing techniques to go about testing.  More to come on the techniques later.  Pros: Test techniques are proven to be effective.  Testers just need to master them and apply them in the right situation.  Testers are also pretty much free to use their own instinct and creativity to go about testing the product.  Focus on learning.  Cons: Newer testers on the team may not have the sufficient experience or knowledge of the product to be completely effective. 

4. Feedback based

Start out pretty much same as Freestyle testing.  But as testing sessions are conducted, testers build up a history of test executions, areas covered, bugs identified, code churns, etc.  Very similar to our session notes.  Testing from historical data such as bug records, data from automations and scripts, application logs, customer reported issues, etc.  Use these data as input so tester can identify which areas may yield the most bugs.  Pros: Knowledge is not lost.  Make use of previous knowledge is very powerful.  Cons: Sometimes it’s very difficult to make use of existing data effectively (especially if there are a lot).  May actually spend more time gathering and processing these data rather than actual testing.

So, to welcome myself back to blogosphere, what better way than to show off a bug in my code?  I just found it this morning, in plain sight. 

Okay, I know this is pretty lame, and probably not the first picture of a “bug” in the code.  In fact, the first ever recording of a bug in the software is probably this one here (according to various sources). 

This moth is probably the God of all bugs (well… it’s where the term bug came from).  Mine is just a tiny, teeny little baby spider, whom my next door office mate has now kept it in his jelly bean jar as the team mascot. 

Cheers, and hopefully, I’ll keep on blogging this time.

However, I noticed something different today on that greeting.  "OH HAI ch4n!".  I just subconciously thought at first that's how Chinese greet one another, but then on second thought, that's actually not Chinese.  That got me to look closer at the text which told me what language this was and wham! it hit me.

It's LOLCAT language!!!  (not sure if the screenshot is clear enough)

What?!  The internet has evolved so fast that I didn't realize a new language was born completely out of weird cat pictures with funny-talk subtitles.   Happycat must be proud feline today.

Curiousity got a better hold of me and I start to Live search to see if this language really existed or was it just a bunch of cat-loving Flickr devs' doing.

That's when the second wave of shock hit me.  LOLCAT is truly a language on its own!  There's a English-LOLCAT translator, a CPAN Perl module which translate English to LOLCAT, and last but not least, a LOLCAT bible!!!

Incredible...

BTW, if you are wondering, this blog title is "Hello World" in uh.... LOLCAT.

2¹ => 2

2² => 4

2³ => 8

Putting together make 2/4/8 => 2/4/08.

Of course, back home in Asia and possibly in other places around the globe today is actually 2²/2¹/2³ which isn't as interesting.

k.  'nuff nerdinous for the day.