As testers, we're often incredibly busy with the wide variety of work we do (spec feedback, test plan writing, bug-finding, automation writing & run analysis, etc.).  In fact, we're often so busy that when new work comes up, it's incredibly tempting to fall back on patterns that have brought us success in the past without thinking through whether or not the actual problem or situation surrounding the problem has changed.  

However, the fact that we're so busy is exactly why we should take a moment to make sure we're not doing unnecessary work or worse, causing unnecessary work for others!  

Even if the problem sounds similar to something you've solved in the past, taking just a couple of minutes to answer the following questions can literally save hours for you or your co-workers:

• Can you clearly state the problem you're solving in 15 words or less?   
• e.g. "We do not know if we have enough QA capacity for this sprint."
• Can you clearly state the goal you're trying to reach?   
• e.g. - "Ensure our team is appropriately staffed for the QA work required in this sprint."
• Can you separate out actions from your end goal?  
• e.g. "Discuss options with feature stakeholders" is an action towards solving the problem "Stakeholders are not all in agreement on the strategy for ensuring the new Whizbang Feature ships with high quality" or the goal "Stakeholders are all in agreement on our testing strategy"

Team X:  “Feature Y is completely broken, but we don’t want to fix it because we don’t think any users will actually use it this way.”

My first instinct was, “Are you crazy?  The fix is so easy and low risk!  Of course you should fix it!”  After all, if we as testers can help our teams validate fixes with high confidence, why wouldn’t we?

My first instinct was dead wrong.

This is a trap I still occasionally fall into, admittedly.  But it’s a dangerous one that can have repercussions long after the decision is made.  Essentially, my first instinct would have signed up Team X to spend the next decade supporting, manually testing, automation testing, statically analyzing, re-testing, servicing, and likely re-engineering several times a feature that nobody would ever use.

In cases like this, the right question to ask is “why is that feature even in the product?”  If you wouldn’t fix a bad bug in a given feature, then why bother building that feature in the first place?

As testers, we need to change our basic instincts from “fix every bug” to “what’s in it for the user?”  If we aren’t getting enough value for the long-term cost of building and maintaining the feature, we should negotiate to get those features removed from our products. 

Complexity leads to cost, cost leads to fatigue, fatigue leads to shortcuts, shortcuts lead to holes, holes lead to dark places you don’t want to find yourself in, lest you be eaten by a Grue.

As smart as we think we all are, no reasonable amount of brainstorming and careful test planning can discover all possible tests that should be put in place for a given system.  Even for seemingly simple systems, “test space” can be huge.  I say “reasonable”, because at some point every test planning process, you cross the cost/benefit threshold where further planning approaches pure guesswork as to what will actually pay off.

So what do you do next?  As an alternative to exhaustive brainstorming, consider using “canaries” and “scouts” in your testing to help you quickly discover which parts of uncharted test space are worth venturing into.

The term “canaries” comes from the old practice of placing canaries in cages in underground mines.  Birds’ bones are hollow, and what they breathe permeates their entire body nearly immediately.  Unfortunately for the canaries, this made them a great early-warning system for miners.  If dangerous gas was present in a mine, the canary would breathe it in and would quickly die, and the miners would be alerted to leave the area as fast as possible.

In software testing, canaries are mechanisms that provide an early indication of impending problems.  They can take any number of forms, and they don’t have to be automated.  For example, getting a couple of customer bug reports around the same issue can warn you that you have a test hole in a scenario that’s more common than you thought.  An automated canary might look like a set of automated Pri0 integration tests that runs on every daily build before anyone else picks it up for deeper testing. If any of the tests fail, it’s a good sign that the build has major flaws and shouldn’t be evaluated further.

“Scouts” venture forth into uncharted territory looking for something interesting to report back.  It could be an “all clear” report, or it could be along the lines of “danger approaching – to the battle stations!”.  I consider automated exploratory tests using real-world data to be scouting tests. For example, tests that leverage actual customer documents or databases can help you discover test holes around specific data even if you have great code coverage for your system under test.  Beta testers are also great scouts.

In summary, canaries alert you to impending danger, and scouts forge ahead to help you decide where to look next.  Both are invaluable in the testing process. 

So what canaries and scouts are you using?

So what went wrong?

The problem was that running the scenario once worked fine, but it left behind some side-effects in the system that caused it to fail it you executed the scenario again.  So the next time you ran it (and every time after that), the user was presented a nice unhandled exception message rather than completing the task they were attempting.

How did we find the bug?  Since the daily build had passed initial tests, one of our engineers picked it up to do some “real world” exploratory testing.  As soon as he tried to do something “real” with the build, he quickly ran into the problem.  This scenario was one that any real user would almost certainly execute multiple times under normal operating conditions.

So how could we have found this bug sooner?  I’d like to say we discovered some key indicator in our retrospective that should have warned us we had a hole, but the truth is the scenario was covered well and passed the sniff test.  What was missing was an automated repetition test.  As our real world testing illustrated, this was a “high repetition” type of scenario.  We should have had a test in place ensuring that the scenario worked reliably over and over.  This particular scenario was difficult to automate because it actually required two computers.  That’s not impossible, and in fact a task for adding that automation was on our backlog.  However, the fact that we had a highly repetitive user scenario that wasn’t automated should have been our indicator to focus on the scenario with manual testing more frequently until we had the automation in place.  It turned out that a simple tweak to our existing automation was all it took to put this test in place. As an added bonus, we can now also use this automated test as a canary for finding memory leaks and performance degradation issues over time.

Obviously this isn’t the only way to catch this type of bug.  Additional code reviews and pre-commit unit tests might have also helped.  But the code was code reviewed, and our unit tests also had high code coverage for this scenario. So this was the next most obvious and easiest way for us to ensure the right coverage.

So, next time you’re scratching your head thinking of ways to find more bugs in your system under test, maybe try running your existing tests… again (and again, and again, and again…).

I’m replacing the term “Test Plan” with “Quality Plan”.  This is a plan that explains how various people working on a feature will work together to ensure it’s delivered with high quality.

Quality isn’t just the responsibility of a single person, and as such, Quality Plans don’t have a single author.  Rather, multiple people will contribute to them at various points throughout the feature development cycle.  For practical purposes, I’ll have a tester being the "guardian” of the document, making sure it’s complete, up to date, that everyone is bought into the plan, etc.  Developers, program managers, and testers will work together on a unified testing strategy for the feature:

• The developer will describe how she’ll validate the code before she commits changes, which scenarios will be covered by pre-check-in tests, any testability considerations in the code, etc. 
• The tester will describe the additional testing he’ll do post-check-in and any other processes, procedures, test tools, etc. he’ll use to get the job done. 
• A program manager might contribute by ensuring the plan covers validation of end-user scenarios, making sure those scenarios are well documented, and tracking progress during feature development.

It may sound like a subtle difference in terminology, but I think it encourages a much bigger change in how a team thinks about designing in quality from Day 1 as opposed to the extreme opposite: “testing in quality” after the feature is “complete”.

However, I’ve found an opportunity I consider to be very exciting and one I couldn’t pass up.  I’ll be switching back from a Test Architect to a Test Manager position for one.  Among other things, one feature I’ll own testing for is the F12 debugging experience inside Internet Explorer.  If you’ve tried the IE9 preview, you’ve seen that there’s a lot going on here, and I’m really looking forward to being a part of that.

This also means I’ll be moving my family from Chapel Hill, NC back to Redmond, WA.  Our exact move date isn’t settled yet, but we’re planning to be there before September so we can get our kids registered in school before the start of the year.  Leaving North Carolina is also a very difficult decision, as we have family and some very good friends here.  We also left some great friends in Redmond when we moved to NC, so it will be fun getting re-acquainted with them.

So… that’s my news.  Here’s to change.  Cheers!

One of my first projects is to find ways to improve the return on investment (ROI) we’re getting with our test automation.  In general, there are 2 options to improve ROI: decrease investment or increase return (or both).

Left unchecked, test automation can be like a horde of Tribbles.  Sure, in the beginning it’s cute and fuzzy and makes you happy, but when your automated tests multiply and grow out of control until you’re up to your neck in them, you suddenly realize perhaps automation is not the silver bullet you may have initially thought it was.

I’ll attempt to cover in this post what we’re doing to improve test automation ROI, and depending on interest in the comments, I may go into more detail in follow-up posts.

Costs of Test Automation

Often, product teams are so focused on the product lifecycle that they forget about the lifecycle for test automation.  Like any other software project, writing code for automated tests has certain costs.  For example:

• Initial coding and testing time (including time spent developing any “helper” libraries or layers)
• Maintenance – keeping the test code up to date as the product under test changes over time
• Investigation time when a test fails – is it a product bug or a bug in the test itself?
• Test execution time – think of it as an opportunity cost; what other processes are blocked pending the outcome of the automation results?
• Infrastructure, CPU cycles, electricity, lab maintenance costs, licenses & fees, etc.

There are many more costs, of course (see here for a good list including fixed and variable costs), but I think this short list represents some of the more impactful among them.

Taming the Automation Cost Beast

In our effort to improve automation ROI, we’re looking at several aspects of our automation.  I’ll walk through some of them and explain how we’re thinking about them in terms of ROI.  As we look at these metrics, we consider them all in concert and then decide on an appropriate action to take.  This could be the merging of 2 or more tests into a single case, making improvements to the test automation, or even deleting the automation altogether!

1. How long does the test take to run on average?

Tests that take a long time to run may be more susceptible to being broken by changes in the product because they’re touching a larger surface area of the product.  Consider whether these are truly worthwhile to be running at their current frequency and whether it might be more prudent to factor these into multiple smaller tests that could be run more often.  This might not be possible if the test is an end-to-end scenario and each step depends on the outcome of the previous step.  This type of scenario is sometimes more beneficial to run by hand though, since testers acting as customers would get a true feel for the flow through the code under test. 

Other times, tests that take a long time to run may be hung waiting for an unexpected modal dialog to be dismissed.  To give you an example, some of our integration tests specified a 10 minute default timeout before our test harness would kill them.  If the test timed out 15 seconds into the scenario, it would sit there doing nothing for the remaining 9 minutes and 45 seconds until the harness stopped the test process and moved on to the next test.  Clearly this time could have been better spent!

Another related metric is to look at the standard deviation of runtimes for the test.  If the test runtime varies greatly over time when you look across all runs but is relatively consistent when the test passes or fails, you might have found a test that hangs on failure until it times out and otherwise passes very quickly when it succeeds

2. How often did the test fail for test-related problems vs. product bugs?

Did the test find more test problems or product problems?  More test bugs means the test is either poorly written or it’s too dependent on product behaviors and interfaces that are changing over time.  For example, if the test is driving the product via UI automation and someone redesigns a particular dialog box or web page to use a checkbox control instead of a radio button, the test will likely break.

After analyzing each test failure, we use a custom field in our TFS Bug work items to indicate whether the test failure was due to an intended product change or a flaky test.  If analysis shows us a test failure is because of a test bug, we will often either fix it immediately or disable the test so it doesn’t get run again until it’s fixed.  Otherwise, the test will continue to fail in runs for the same issue until it is fixed, thus wasting time.

If the test breaks most often because of test issues, perhaps either it or the product code should be re-factored to be more testable.  There is a wealth of information on the web about coding for testability.  For starters, have a look at the SOLID principles. 

3. How many unique test and product failures did the test find?

Closely related to the previous metric is the concept of unique number of test bugs and product bugs found.  A test that’s finding a lot of unique product bugs might be too big a test.  That is, while it’s failing often for valid product problems, it might be failing before it finishes even setting up everything it needs to test a specific scenario.  Alternatively, if a test fails in 9 runs out of 10 for a test bug, it would be a lot more interesting to know the test found 9 unique issues than the same issue over and over.

4. What is the average pass % for the test?

The pass rate for a test is the number of times it passed divided by the number of times it was executed.

Now, this may seem counter-intuitive at first, but if you have a test that has always passed, that test may be providing little value.  Sure it’s telling you something is working, but it’s likely that’s because it has always worked and the feature hasn’t been changed.  The test impact analysis feature in Visual Studio 2010 is a great way to make sure these tests aren’t executed when the code they test hasn’t been modified.  Another possibility is that the test doesn’t actually verify anything!  You might be surprised at what you find when you start investigating these tests.

If a test is failing a large percentage of the time for test bugs, say > 50%, then you should probably remove it from your runs and figure out what needs to be fixed.  If it’s failing a high % of the time for product bugs, ask yourself if the test could be executed further “upstream” in the production pipeline (say, as part of a gated checkin?) so these bugs are caught before the code that cause them is committed.

Wrapping Up

By looking at test metrics over time, you can find candidate tests to delete or merge into other tests.  I’ve covered just a few of the ways we’re combing through our test automation to determine which tests are useful and which are candidates to be deleted or merged into other tests.  By keeping our automation arsenal lean and mean, we’re optimizing the automation to test what it’s good at testing.  When we delete a test, we keep the scenario around to cover through exploratory or strict manual testing, usually on a much less frequent basis than it would execute in automation runs.

To help you become more familiar with it, I’ll be blogging a short series of katas – small exercises you can repeat over and over until you’ve mastered the interface.  Today, I present the first kata – basic tf.exe commands.

Getting Started:

In order to use these katas, you’ll need to first do the following:

1. Open the Visual Studio command prompt: Start –> All Programs –> Microsoft Visual Studio 2010–> Visual Studio Tools –> Visual Studio Command Prompt.  This will ensure that tf.exe is in your %PATH% environment variable so you don’t have to type the full path to it each time you use it.
2. cd into a folder where you have an existing TFS workspace mapped. This is how tf.exe will know what workspace and server to work against.

TFS Command Line Kata #1 – TF.EXE Basics

My first kata introduces some of the most used commands in the command line client.  In particular, all of these can work without any additional parameters other than the command name.  I’ll address more complex commands in future posts.

1. tf /? – list all available tf.exe commands
2. tf workspace – view details about the current workspace
3. tf workfold – view a list of server folders and the local folders they’re mapped to for the current workspace
4. tf workspaces – view a list of all workspaces on the current machine
5. tf dir – view a listing of files & folders in the source repository folder mapped to the current local folder
6. tf status – view pending changes in the current workspace

That’s enough for now.  Remember, katas are short and simple, easy to memorize and repeat daily.  Work on these for a few days while I prepare the next post in the series.  If you have questions about the command line interface to TFS, please post comments.

More info here.  If you're in the area, I hope to see you there!

Many applications today consist of multiple layers.  For example, a typical ASP.NET MVC web application may have views, controllers, services, data repositories, a domain model, etc.  A typical IT type application might have a WinForms or WPF GUI, some client side logic, web services, and a database layer.

The question is, when testing a layered application, should we test each layer independently or just test the product from an end-to-end perspective?

My answer (and yours may vary), is “both”.  The key goal in my mind is to find each class of bug in the most efficient way possible while keeping the efficiency of your overall quality process in check too.

Why test layers?

Often, the argument against testing layers sounds something like “we’ll end up with the same test written N times” or “it’s too inefficient to test each layer independently”.  While I completely disagree with the 1st point, there may be validity to the 2nd.  I’ll tackle each individually.  I’ll point out here that I’m mainly talking about the layers that your product has created (i.e. code that you own).  If you depend on 3rd party products such as a database or web server or user controls, you may or may not want to do some testing to ensure they meet your needs.

Argument #1 - We’ll end up with the same layer N times!

If your app is layered, there’s a really good chance each layer does something slightly different (why else would there be layers?).  If you target your test at each layer and mock away the layers below it, then you can write tests to each layer that target the specific behavior of that layer and nothing else.  This is typically what unit testing covers.

In my experience, people who use point #1 against testing layers often misunderstand the point/value/cost of mocking and they’re thinking about the problem as testing the whole stack in increments.  I agree this would probably not be an efficient use of time and resources.  Imagine testing the database, then the database + the data access layer, then the database + the data access layer + the business logic, then finally the database + the data access layer + the business logic + the client code that sits on top of the business logic…  you get the point.  In this approach, you’d be testing the bottom-most layer 4 times, the DAL 3 times, the business logic twice, etc.  I don’t think that’s efficient or necessary at all in most cases. 

Argument #2 – It’s too inefficient to test each layer independently

This is where mocking comes in handy, and in particular mocking frameworks.  The common misperception is that you have to write and maintain a bunch of code to create a mock for things your code depends on.  Mocking frameworks such as RhinoMocks, TypeMock, Moq, etc. help you by automatically creating the mocks for you at runtime.  You simply tell them how to respond when certain methods are called or properties accessed.  For example, if your code under test depends on a class with 50 methods, but only 1 is called by the code under test, you only have to write about 2 lines of code to create the mock object and tell it how to behave when your code calls it.  That’s all!  How is that expensive?

A variation on this argument is the classic “why automate testing at all?” argument: “it’s too expensive to maintain test automation”.  My response is, “it’s usually too expensive not to”.  Sure, if you change the product code to do something differently, then you should change the test code accordingly. In fact, you should want to.  Otherwise, how do you know the code you just wrote even works?  You could go as far as writing the tests first in fact, but since I’m trying to convince you to write tests at all, I’m going to stop there for now :)

Argument #3 – We’ll end up wasting time fixing bugs that would never impact our users

Are you sure? How can you be certain that that particular code path will never be hit by customers using the product from a layer higher up or possibly by a change in a dependency further down the layer stack?  Why would you knowingly write code that’s full of bugs and rely on the fact that what’s true at a specific point in time will always be true?  Think about all the time you’ve saved by testing in layers and finding bugs early instead of late in the development cycle when integration testing usually happens. Why not use a little of that time to fix this type of bug?  I’m speaking in broad terms, of course… I’m sure everyone can come up with specific examples where this doesn’t make sense. 

Why test end-to-end?

Testing layers doesn’t give you an excuse to not test from an end-user/end-to-end perspective.  It doesn’t matter if all the parts of a car are tested independently and work fine if they don’t fit together in the end.  It’s critical to understand how components work together in a system and validate that assumptions made about the interfaces between them align from both sides of the fit.  If you’re having the “no testing of layers” argument with me, chances are you’re more interested in end-to-end testing anyway, so I’m not going to spend a lot of time on this part.

Why test both?

I think of it this way: when building a house, you wouldn’t just get an inspection after the whole house is finished just to discover that the electrical wiring was done incorrectly, would you?  Nor would you only test the electrical wiring once before all the drywall was up and forego the final walkthrough. To really have confidence that your house is built correctly, you have inspections for each major sub-system along the way (framing, plumbing, electrical, insulation, drywall, etc.) and then you do a final walkthrough for an integration/end-to-end kind of test. 

By testing all the components/layers independently, you’re finding bugs in those components more efficiently than waiting until the whole system is put together.  By testing from an integration perspective, you’re testing the coupling of those individual components and their interactions.  In both cases, you’re using the right tool for the right job.

So that’s my pitch! Convinced?