Like most other frequent speakers, I usually construct a quick mental map on what I want to talk about in any given event and fill that out in my talks. After scores of presentations, I noticed there were a few patterns that were recurring in the feedback I was providing to the presenters. So, I wanted to whip up this quick cheat sheet that could provide submitters an overview of some key things to consider when submitting an entry into a software engineering conference. Many of these are generic tips as well, but I have tried to give specific examples relevant to software testing in most places.

1. Is the core idea that you want to share exciting, practical and have market demand?
• It needs to attract an audience, has to be useful to most of the attendees and preferably in an area that there is a reasonable market demand for. Having said that, if your talk caters to a niche audience, call it out upfront so you have an interested and targeted audience. Good examples are “Making the next evolution in software testing!”, “Cloud testing – the new frontier” or “Testing Natural UI applications”
2. Have you articulated the crux of your content in a clear one liner?
• Ideally that is your title. Anything more than one line needs further refinement. Good examples are: “Beta testing for large scale web services”, “Testing in production for composite apps”, “How to delight customers with your design?”
3. Do you have an innovation or invention you want to talk about?
• If yes, that would be great – make sure this clears the criteria in the first question. Good examples are: “MyTool: A way to speed up testing on rich client apps”, “Using MyPractice to deliver customer value faster on remote teams”.
4. If not, what is the new unique perspective you have on an existing idea?
• Remember, someone will attend your talk to hear your unique perspective unless this is a purely informational basic talk. Avoid obvious details and instead focus on a specific set of value propositions. Good examples are: “Scenario Focused Engineering – what does it mean for UX?” or “Lessons learned in testing cloud apps” or “A new model for test optimization”
5. What are your key takeaways for the audience?
• What is in it for the audience? Delineate your key concepts/ideas or call to action clearly to make your content useful to apply. The audience will likely not remember more than 3 key ideas – prioritize what you want to say. Trying to talk about too many things will dilute your message. Good examples are “3 best practices for testing device drivers”, “Avoiding the 3 common pitfalls in usability testing”
6. If you are presenting a practice or a case study, do you have appropriate data?
• Telling someone that your practice increased team productivity is a lot more impactful if you have metrics that led you to that conclusion. Of course, this may not be applicable in every situation but it is a good idea to lend credence to your content with adequate examples or data.
7. Have you chosen a suitable vehicle for the presentation?
• If you are doing a talk, make sure the audience gets time to understand your key ideas. Publish a paper if you have comprehensive content, details and data that absolutely need to be available to understand your idea that cannot be done in a short duration of a talk. Consider a tutorial if there are walkthroughs and exercises you would want to do over a half day to a day.

Myth 3: Bugs found through exploratory testing are hard to reproduce

Testers have to remember exactly what steps they did and note it down in detail when they find a defect. The stated problem is that with no pre-scripted steps in exploratory testing, the onus on the tester is too huge and might lead to poorly documented bugs that cannot be reproduced later on. There is also an additional problem that there could have been a state based defect where the step that triggered the bug could have been done several minutes earlier but the defect showed up later, leading the tester to delink the earlier action and the bug.

This is where tools can help ease the tester’s life. One of the most appealing aspects of exploratory testing is that it lets testers focus on the creative aspects of testing and not get bogged down by the tedious job of writing detailed documentation for test cases or bugs. In order to keep the tester’s time maximized for the actual job of breaking software, tools can help do the defect creation automatically with the right data.

For instance, we dogfood our own Microsoft Test Manager to file actionable bugs with the right information automatically populated so all the tester needs to do when she finds a defect during testing is to just type the title of the bug. The steps that the tester performed before hitting the defect, desktop recording of the actions until then, event logs, historic debugging logs and snapshots of the virtual machine that the tester performed her testing on – all of this is stored in the defect automatically. Check out the tutorial here that shows how to do exploratory testing with the newly released CTP of Microsoft Test Manager in VS 11 and here is a list of all the different kinds of data that can be captured in the tool. We also use the tool to automatically create test cases when there are tests worth rerunning as part of the regression test suite.

Download the free Visual Studio 11 Developer Preview and check out how our exploratory testing tool captures actionable bugs:

In summary, exploratory testing done with the right tools enables you to:

• Define and vector testing as appropriate for your app
• Communicate your testing goals and progress to the team
• Test more frequently and file comprehensive bugs faster

I hope this series has enthused you to try exploratory testing if you haven't already - let me know your experiences...I'd love to hear from you.

So, go on, be a fearless explorer!

Myth 2: There is no way to measure exploratory testing

I have heard this often from test managers – “how do we measure testers’ productivity in exploratory testing?” Scripted testing allows for measuring rate of tests executed per hour, test pass progress percentage etc. but how do we measure progress or productivity in exploration?

First up, there are no absolute metrics that is a silver bullet for everyone. One could easily argue against any metric if the situation does not suit measuring that particular component. For instance, I could say number of bugs found per hour in exploratory testing is a metric, but it is a function of how buggy it originally was, what kind of software this was, the amount of churn went into it etc. So, first we need to come up with metrics that are meaningful to measure on a specific team. Once you determine that, it should be easy to wire those metrics up with the tools you use to do exploratory testing.

For instance, on my team we measure progress in exploratory testing like any other task in the daily standup meeting. We report on time spent, time remaining, adjustments made if any based on bug density or risk. We look at metrics like tour effectiveness based on bug priorities or numbers found per tour, code coverage across tours to assess if we have holes in testing, user story complexity to bugs ratio to see if we have spikes in certain stories etc. Since TFS is the underlying data repository for all our testing, it is easy for us to pull this data in reports from TFS.

An interesting visualization we tried was built up on code coverage was to show a heat map that looked something like this:

The colors in the components indicate how well covered a specific component is in terms of code coverage. You can track coverage while exploring specific requirements or user stories of your application and figure out where there is a lack in code coverage to direct where your future testing efforts should be. There could be similar heat maps built on bug density or bug numbers for instance. Your map could show those stories in red that have a high bug density in terms of bugs per line of code to inform future testing.

In summary, there are various ways that you could measure and direct your exploration efficiently. Choose metrics wisely!

1. Deployment binaries - the pdbs and dlls that you use to deploy during your test run are all stored in TFS each time you kick off a test run. Typically, this data is needed only if user downloads the test result's trx file , associated test binaries and attempts to "rerun" the test in the same test result
2. Test attachment data from diagnostic data adapters - Diagnostic data adapters like intellitrace, video recording, event logs etc. collect data during your test execution and store this data in the test run if you file bugs during the test run. This data is usually heavy in manual test runs since it is uncommon for automated test runs to capture all of this data each time a set of tests is run
3. Code coverage files - The code coverage data diagnostic adapter when used in automated runs, generates a fairly heavy coverage file and also uploads all the pdb files of your app under test since the coverage file needs those. This can tend to be fairly heavy

After analyzing several customer TFS databases, we realized that 1 was contributing to a significant chunk of the data being accumulated in the TFS database - anywhere from 30% to 80% of all test data. Most users did not however want these binaries in TFS since the rerun operation was very rarely done. Given this, we released a QFE last week, that would stop uploading these deployment binaries into TFS every time a test run is published into TFS: Team Foundation Server 2010 Service Pack 1: Reduce the size of the test data saved to the TFS database

 More details on the specifics of the fix are in the KB article from where you can also download the hotfix by simply typing out your email ID where you will receive the hotfix link. Sweet! I tried this with my mail id and here is the link I was sent.

To deal with the remaining test data though, we recommend running the Test Attachment Cleaner to routinely clean up the TFS database of unwanted or stale data. The tool is recommended to be run on the application tier of your TFS and has several modes that it can be executed in - see the description here for more details on the tool. The tool generates a whole lot of database records when you clean up your database and we recently learnt of a couple of SQL bugs that would cause these deleted records to persist as ghosted records instead of getting cleaned up automatically. To address this, SQL shipped 2 fixes in their Cumulative update package 10 for SQL Server 2008 R2 - it is recommended you install this update on your TFS before running the test attachment cleaner. In addition, we are also working on reducing (either in space or the number of db records) other kind of test data published to TFS.

In the meantime, if you are experiencing growth in your test data size in TFS, please download and install this hotfix and send your comments my way :-)

Myth 1: “Exploratory testing is ad-hoc, monkey testing”

Exploratory testing is NOT random, undirected testing. On the contrary, unlike scripted testing, you cannot just follow a set of pre-determined steps that might arrive at a defect. Testers often have hunches about where defects could be – a particular dev tends to be bad at exception handling, or an app that is weak in dealing with frequently disconnecting connections, or a website doesn’t do effective load balancing when stressed. As you learn more about the system under test, you get new hunches, new test ideas that could lead to unfound defects.

Vectoring exploratory testing with some of these test ideas will help explore the system more effectively and completely. Two great ways of vectoring exploratory testing can be found in session based test management by James Bach and exploratory testing tours by James Whittaker. SBTM will help direct exploratory testing by getting the tester to define a mission upfront and help keep exploration focused in that area. Testing tours help define various paths to explore in the system with each tour leading to different kinds of defects in different areas.

We use the concept of testing tours extensively in our in house testing – check out how our testing tours look:

Tours are essentially used to define a theme for exploration. If you were a tourist in a new city, how would you explore the city? Perhaps you would visit the famous landmarks in a quick tour, perhaps you would do a detailed walking tour of just one of your favorite sites, perhaps you would visit the lesser known attractions and avoid the touristy places. Likewise, if you were a tester testing a new app, how would you test it? Perhaps you would do a quick sales demo walking through the primary scenarios, perhaps you would test one scenario in great detail, perhaps you would test some of the lesser advertised features. Any theme that you choose to apply to your exploration is informed by the learning you have had from your previous explorations. You know your dev is lousy at exception handling - a tour like "Intelligent tourist tour" (where the tourist asks hard questions to the guide :D) would help - you could try all the hard boundary value inputs or rarer scenarios bound to produce a boundary condition. You know the app has perf issues when more than a few users hit the server - you could try the "Carnival tour" where your entire test team logs on to the client altogether and parties on it! :)

My favorite tour is the Back Alley tour where you exercise some of the lesser known paths in the application, the Fedex tour where you can track a data structure from point to point to see how it gets transmuted, the All Nighter tour that has caught some nasty memory leaks we fixed recently. More mythbusting in the next part...

What is exploratory testing?

James Bach defines it succinctly as simultaneous test definition and execution. Of course, exploratory testing has been very popular and most testers do exploratory testing, perhaps calling it by another name or not calling it anything at all ;-) What pops in your mind as you think about exploratory testing?

So, why care about exploratory testing?

There are many reasons why exploratory testing might be interesting to testers in today’s world:

• Find bugs fast and early without test overhead – exploratory testing is lightweight and reasonably good test hunches will let you find defects quick and early without the documentation time overhead
• Centered on customer value rather than spec compliance – Tester is an end user proxy on the team and is key to the product being useful to the end user. Exploratory testing is done on working software and focused on customer value.
• Crucial component of Agile testing – Everyone tests on an Agile team : the developers, Product owner, SCRUM master. The iterations are typically short and defects need to be found and fixed to deliver optimal customer value within a short turnaround time. This is an ideal situation to use a lightweight and effective style of testing like exploratory testing

So, despite so many advantages, why do many people still have reservations with exploratory testing? In the next part, we look at some of the myths surrounding exploratory testing and do some mythbusting. Stay tuned…

There have been some pretty good books released recently that discuss software testing with Visual Studio ALM 2010 in detail. I thought I would point out 3 noteworthy books in my post today.

Software Testing with Visual Studio 2010 (Microsoft .NET Development Series)

This book is a great in depth analysis of the testing capabilities in Visual Studio ALM 2010. The author, Jeff Levinson is an ALM MVP and has been involved with us in the product group reviewing our specs, early builds and docs from the beginning of the product cycle. The insight he brings from this experience really sets this book apart in terms of content and in the ability to capture the essence of the release. Jeff covers the new capabilities of Microsoft Test Manager and Lab Management in great depth and in a seamless flow from beginning to end. The book also includes nuggets on practical info otherwise hard to find. Overall, I would recommend this book for anyone looking for a practical guide to software testing with Visual Studio.

I remember doing a review on the first draft of this book on a 22 hour air journey where I also included “Airplane food sucks for vegetarians” comments in my notes for Jeff! LoL!

Software Testing using Visual Studio 2010

I just finished reading this book from Subhashni and Satheesh Kumar over the last weekend, thanks to Packt publishers that sent me a copy to review. The authors have run through the entire set of features for software testing in a lot of detail, replete with good illustrations and examples. This is a refresh of their earlier book, Software Testing with Visual Studio 2008. The tutorials have a lot of detail and will help newbies start and learn about Visual Studio 2010 easily.

The authors take more of a documentation approach rather than have a narrative in the book which makes it hard for new users to obtain initial context. I am also slightly disappointed with some obvious missing content like using data diagnostic adapters to file actionable bugs, use of tcm.exe, migrating tests etc. On the positive side though, this book contains a lot of detail on web performance testing and load testing which is unique to the book as compared to other books on the market. If you are looking for a set of tutorials covering all testing types in Visual Studio 2010, this book is a good pick.

Professional Application Lifecycle Management with Visual Studio 2010

This book is a 700 page treatise that deals with a superset of the testing capabilities - all of Application Lifecycle Management in Visual Studio 2010. 3 of the authors – Brian, Ajoy and Martin are Microsoft FTEs while Mickey is an ALM MVP. The book is a detailed reference for the complete lifecycle management system in Visual Studio 2010 covering modeling, testing, code analysis, collaboration, build and deployment tools.

Brian Keller, our very own star evangelist, has authored the testing chapters. Readers will love the simplicity of his writing – uncomplicated and effective, he covers the gist of the testing solution beautifully. The 5 chapters dealing with software testing are well structured and deals with both the test and lab management pieces in a fair amount of detail. Brian zooms in and out from the big picture to the specifics of the tool with ease and provides a clear walkthrough of the different test types and test management toolset in context of the overall ALM solution. This is a great book for folks that are looking to understand the big picture of ALM with Visual Studio 2010 and also drill down into testing in particular. 

Let me know what you guys think once you check out any of these 3 books.

I met him in June last year to seek help managing my work schedule. It just seemed like I was putting in endless hours at work answering zillions of mail, attending hours of meetings – but it felt like I was getting lost in a sea of info and drifting endlessly without a feeling of achievement. JD introduced me to his system of Getting Results the Agile Way as a framework to help me organize my work. I LOVE agile. So, the idea of using agile principles beyond software engineering had instant appeal for me. But boy –it needed a lot of discipline to practice this each and every day! However, as I put this into use in my daily life, I realized how effectively it was helping me find my way through my amorphous barrage of work. And I thought I would share it on my blog.

There are 3 principles that I adopted from the system and snapshots from my One Note notebook that I use to track them:

1.       Rule of 3

2.       Monday Vision, Daily Outcomes, Friday Reflection

3.       Hot Spots

The Rule of 3 helps me focus. It helps me identify the top 3 results I want out of any activity(ies) and provides direction. I use this to define 3 primary outcomes I want off each day and each week. At the start of every day, I list these 3 outcomes and direct my day accordingly.

Monday Vision helps define 3 outcomes I want for the week, daily outcomes track 3 key results for the day and each Friday, I update my weekly results based on how well I did on my outcomes through the week. Since I have a backlog of outcomes built up already, each day, I can pick the top 3 prioritized backlog items to attack in a day. It’s simple agile! Friday reflection functions as a retrospective on what went well and where I can improve things for the next week. This provides a clean start each week.

The 2 items below the blue line are not really outcomes – they are tasks. So, they are not part of the big 3 for the day, but I put them down as a todo list so I make sure to complete those to achieve the outcomes listed

Monthly results are a great way to track your outcomes each month so you don’t have to scramble at annual review J It helps if you pick a theme for each month so you run that as a background thread in all the activities you do that month. Here is a partial snapshot of one of my monthly results with some confidential info blurred out

Hot spots are useful to identify the key areas of your life/work that you want to focus on. It’s useful to list this to ensure you are not working on too many things( which is one of my chronic problems). I currently classify this as work-in-progress :D

Surprisingly, all the practices in the system take very little time to do. For instance, I spend barely 10 minutes in my One Note each day tracking this. Also, a very pleasant side effect of this was that it stopped me being constantly driven by Outlook. I regularly hit zero mail on my inbox every day, despite getting over a 150 pieces of mail explicitly directed to me! People at MSFT especially, will realize what a wonderful feeling this is! J All my pending items are tracked in my unified prioritized backlog and other mail is acted upon immediately.

JD received so many positive responses to his system that he has now converted this into a book. I strongly recommend the book to anyone that wants to achieve more effective results. Bonus points if you already believe in agile development. There are more aspects of the book like planning your week, managing energy and working by themes that I haven’t tried yet, but I am hoping to try sometime soon.

Do you have practices that make you more effective? I’d love to hear from you!

The feature pack contains some much awaited bits with which you can:

• Use Microsoft Test Manager to capture and playback action recordings for Silverlight 4 applications.
• Create coded UI tests for Silverlight 4 applications with Visual Studio 2010 Premium or Visual Studio 2010 Ultimate.
• Edit coded UI tests using a graphical editor with Visual Studio 2010 Premium or Visual Studio 2010 Ultimate.
• Use action recordings to fast forward through manual tests that need to support Mozilla Firefox 3.5 and 3.6.
• Run coded UI tests for web applications using Mozilla Firefox 3.5 and 3.6 with Microsoft Visual Studio 2010 Premium or Visual Studio 2010 Ultimate.

I know a lot of you have been waiting for the Silverlight automation functionality to be released – give this a spin and let us know your feedback!

For more details, see  http://go.microsoft.com/fwlink/?LinkID=192777