But the tours will continue. My book Exploratory Software Testing: Tips, Tricks, Tours and Techniques to Guide Manual Testers is in press and will appear through Addison-Wesley sometime this summer. I am truly thankful for the many wonderful testers at Microsoft who contributed wisdom, thoughts and even case studies to the effort. Special thanks go to Nicole Haugen, Geoff Staneff, David Gorena Elizondo, Shawn Brown and Bola Agbonile. Microsoft is full of great testers and even here, these guys manage to stand out.

I imagine that I will not be long in setting up a new blog as I have very much enjoyed this experience and being the only tester in Developer Division's top ten bloggers was quite an honor. For that I thank you.

In case you are interested in my landing place, I can imagine that one or two of the more popular testing blogs around town will be talking about it.

Wish me luck ...

Software is much the same – there has to be some reason for users to buy it. Some landmarks that make it what it is. If you identify the landmark features that draw users, that’s where you need to concentrate a lot of testing. You can tell where the tourist landmarks are by noticing crowds and watching where they spend their money. For exploratory testers finding the landmarks also means following the money, literally. And money usually leads directly to the sales force.

Sales folk spend a great deal of time giving demos of applications. One might imagine that since they get paid based on fulfilling their sales quota, they would be very good at it and would include any interesting nuances of usage that make the product look its very best. They also excel at shortcuts to smooth out demos and often come up with scenarios that sell the product but weren’t part of any specific requirements or user story. The long and short of it is that sales people are a fantastic source of information for the landmark tour.

Testers performing the landmark tour should sit in on sales demos, watch sales videos, and accompany salespeople on trips to talk to customers. Each demo or demo script identifies the landmark features and can even give you insight into their relative importance by how often those features appear.

To execute the tour, you could simply run through the demos yourself and look for problems. As the product code is modified for bug fixes and new features, it may be that the demo breaks and you’ve not only found a great bug, but you’ve saved your sales force from some pretty serious embarrassment (perhaps even salvaging a sale). I have found enough bugs this way to privately wonder if there is a case to be made for testers sharing in sales commissions!

But we’ve found a more powerful way is to inject variation into the demo scenarios by “landmark hopping.” Ever use a compass in the woods? That’s the effect we are going for here. My army-trained older brother taught me to locate landmarks (rocks, trees, hillsides) in a compass scope and then to make a path from landmark to landmark in the general direction you want to go. It kept me from getting lost in the woods of Kentucky where I grew up and it creates excellent variation in demo scripts and end-to-end sceanrios. Pick the landmarks in advance, scramble them up and tour from one to the next. Rinse and repeat trying to use the landmark feature in a different way every time.

A powerful variation of this tour is the skeptical customer tour in which you execute the landmark tour but pretend there is a customer constantly stopping the demo and asking ‘what if’. ‘What if I wanted to do this?’ they may ask, or, ‘how would I do that?’ requiring you to go off script and include a new feature into the demo. This happens a lot in customer demos, especially the serious ones where a purchase is imminent and the customer is kicking the tires one last time. It’s a powerful way to create test cases that will matter to end users.

Landmarks can be "back of the box" features, features used as part of sales demos, features popular to users (in the event you collect such statistics), features prominently displayed in the UI and so forth. Clearly any bugs you find on this tour or any of its variations are very important ones as they are likely to be seen by real customers.

Enjoy the tour and don't get lost.

Bet that got your attention. It's true, but let me qualify it: Running test cases over and over in the hope that bugs will manifest sucks. It’s boring, uncreative work and since half the world thinks that is all testing is about, it is no great wonder few people covet testing positions. Testing is either too tedious and repetitive or it’s downright too hard. Either way, who would want to stay in such a position?

What is interesting about testing is strategy, deciding what to test and how to combine multiple features and environmental consideration in a single test. The tactical part of testing, actually running test cases and logging bugs is the least interesting part.

Smart test managers and test directors need to recognize this and ensure that every tester splits their time between strategy and tactics. Take the tedious and repetitive parts of the testing process and automate them. Tool development is a major creative task at Microsoft and is well rewarded by the corporate culture.

For the hard parts of the testing process like deciding what to test and determining test completeness, user scenarios and so forth we have another creative and interesting task. Testers who spend time categorizing tests and developing strategy (the interesting part) are more focused on testing and thus spend less time running tests (the boring part).

Testing is an immature science. There are a lot of insights that a thinking person can make without inordinate effort. By ensuring that testers have the time to take a step back from their testing effort and find insights that will improve their testing, teams will benefit. Not only are such insights liable to improve the overall quality of the test, but the creative time will improve the morale of the testers involved.

So all the managers out there need to ask themselves what they've done lately to make their testers more creative. If you don't have an answer, then testing isn't the only thing that sucks.

The link to register is https://blog.utest.com/?p=122. Note the use of the words 'sucks' in the blurb. I love it when I get marketing people to say sucks...

Enjoy!

I was once on a walking tour of London in which the guide was a gentleman in his fifties who claimed at the outset to have lived in London all his life. A fellow tourist happened to be a scholar who was knowledgeable in English history and was constantly asking hard questions of the guide. He didn’t mean to be a jerk, but he was curious and that ended up being a dangerous combination … at least to the guide.

Whenever the guide would talk about some specific location on the tour whether it was Oscar Wilde’s former apartment in Chelsea, details of the great fire, or what life was like when horses were the primary mode of transportation, the scholar would second guess him or ask him some hard question that the guide struggled to answer. The poor guide had never worked so hard on any past tour. Every time he opened his mouth he knew he was going to be challenged and he knew he had to be on his toes. He wasn’t up to the task and finally admitted that he had only actually lived in London for five years and he had memorized the script of the tour. Until he met the scholar, his ruse had worked.

What a fantastic bug! I was so impressed I bought both the guide and the scholar a pint when the tour ended at a pub (a place, incidentally, where the hapless guide was infinitely more knowledgeable than the scholar).

When applied to exploratory testing, this tour takes on the approach of asking the software hard questions. How do we make the software work as hard as possible? Which features will stretch it to its limits? What inputs and data will cause it to perform the most processing? Which inputs might fool its error checking routines? Which inputs and internal data will stress its ability to produce any specific output?

For folks who test word processors this tour would direct them to create the most complicated documents possible, ones full of graphics, tables, multiple columns, footnotes and so forth.  For folks testing online purchasing systems, what is the hardest order we can place? Can we order two hundred items? Can we place multiple items on backorder? Can we keep changing our mind about the credit card we want to use? Can we make mistakes on every field in the data entry forms? This tour is going to be different for every application, but the idea is the same: ask your software hard questions. Just as the scholar did with the London guide, you are likely to find gaps in its logic and capabilities.

A variation of this tour is the arrogant American tour that celebrates a stereotype of my countrymen when we travel abroad. Instead of asking hard questions, we ask silly questions otherwise intended simply to annoy or impede the tour and draw attention to ourselves. We intentionally present obstacles to see how the software reacts. Instead of the most complicated word processing document, we make the most colorful, invert every other page, print only prime number pages, or place something in a location that makes little sense. On a shopping site we’ll seek out the most expensive items only to return them immediately. It doesn’t have to make sense … we do it because we can. 

A video interview on Channel 9 about software testing

A video interview on Channel 9 about Visual Studio Team System

A radio interview on Dot Net Rocks with two funny guys (I had a blast with this one and it will be one reason I never run for political office in the future)

The Future of Testing webinar for uTest

No, “Piece of Crap” is not included. Sorry but it’s time we put that one to sleep. Lee Copeland is widely known to play it at his lectures so try and catch him on his speaking circuit if you just can’t resist. That video is another reason I can’t run for political office. Neil Young has my sincerest apologies.

Some internal links for my MS readers:

The Future of Testing

An Update on the Future

Online Security Testing Course (this is officially a classic since it is the last time I taught this material)

When I think of software, I naturally think of it in testing terms. In my mind software is made up of components which are defined by structural boundaries (code files/assemblies) where multiple components communicate through some defined interface, a set of features that define the functionality for each component, and capabilities that produce behavior that the user desires. It’s extremely useful to me to think about it this way because I can now more purposefully explore my application.

For example, I can choose a set of capabilities to test as an ensemble or focus on a single one. I can purposefully explore the capabilities of many features or a single feature. I can exercise capabilities that cross component boundaries or choose to stay within a component. I can use my component/feature/capability map to guide my testing and ensure that I cover the interesting combinations to the extent possible.

I call these tests tours and think in terms of a tourism metaphor when I test software. Ideas from tourism fit well into testing. If you think about a big, complicated tourist destination like New York or London you realize that wandering aimlessly around its streets is a pretty lousy way to tour. If you understand a little bit about its districts and destinations (components and features) you have a better feel for how to get around. If you go further and use published guidebooks, consult locals and read tourist maps you’ll discover the best places to go and things to do (capabilities) a lot faster. Travelers want to get the most out of their brief vacation time and as testers we can learn from them.

At Microsoft a group of us test folk from around the division and around the company are experimenting with tour-guided testing. We have the Supermodel Tour in which all we do is window shop … look at the interface and find superficial (no intended disrespect to any actual supermodel who might be reading this blog) bugs. We have the Money Tour where we watch salespeople demo the product and create variations of those demos as test cases. We have the Intellectual Tour where we ask the software hard questions. We have the Garbage Collector’s Tour, the Competitor’s Tour, the Museum Tour, the After Hours Tour and a host more that we are experimenting with. All of these tours represent specific, focused testing advice that guides the tester through paths of the software. The tours help a tester think through inputs, data and environment settings that satisfy the goals of the tour. It's a more methodical and purposeful way to test.

We created the tours as exploratory test guidance and even more as a way of partitioning test concerns. Each tour is guidance about a very specific type of capability. Add all the tours together and they collectively should encompass most all interesting scenarios, but there is more to them that that. They can also be associated with the component/feature/capability map to help testers associate a tour with a specific capability. Want to test the interface between a database component and the application? Run the FedEx Tour. Want to test legacy code? Run the Museum Tour and the Prior Version Tour. Want to test how two features interact? Run the Guidebook Tour between them. The touring tests take the discussion of how to test up to a higher level. Instead of talking about specific test cases, they guide us toward talking about test concepts because each tour embodies testing advice for a very specific capability. Such higher level concepts have been missing from our testing vocabulary for far too long.

Tours also take the discussion of bugs to a higher level. Want to find UI bugs? Run the Supermodel Tour. Want to find stress bugs? Run the Saboteur (get it?) and the TOGOF Tour. Want to find timing bugs? Run the Rained Out Tour.

Want to learn more about the details of these tours? Stay tuned to this site, I am going to start running a tour-of-the-month post to explain these concepts in more detail. Perhaps your own team will join us and pass the touring test!

Now clearly peanut butter is not software. I very much doubt that the processes for making peanut butter have changed much over the past few decades. I also imagine that one batch of peanut butter is about the same as the last batch. I concede that we have a harder problem.

But, the term ‘long time habit’ really caught me. Because I haven’t seen too many long term habits established in the testing industry. We plan tests, write test cases, find bugs, report bugs, use tools, run diagnostics and then we get a new build and start the process all over again. But how much do we learn in the process? How much do we retain from one build to the next? Are we getting better each time we test? Are we getting better purposefully or are we just getting more experienced? In many ways, the only real repository of historical wisdom (those long term habits of Jif) is embodied in our tools.

My friend Alan Page likens testing to playing whack-a-mole. You know the one: chuck a quarter in and plastic moles pop up through a random sequence of holes and you whack them on the head with a mallet. Whack one, another appears and even previously-whacked moles can pop up again requiring additional mallet treatment. It’s a never ending process, just add quarters.

Sound familiar? Testing is whack-a-mole with developers applying the quarters liberally. Now, defect prevention notwithstanding, we can take a lesson from Jif. They understand that certain risks are endemic to their business and they’ve designed standard procedures for mitigating those risks. They’ve learned how to detect salmonella and they have wired those tests into their process.

Have we paid enough attention to our risks that we can codify such routine test procedures and require their consistent application?

Clearly software is not peanut butter. Every piece of software is different; Office’s salmonella is likely irrelevant to Windows and vice versa. But that is no excuse to play whack-a-mole with bugs. We have to get better. Perhaps we can’t codify a salmonella testing procedure into a cookbook recipe but we can start being more proactive with the whole business of learning from our mistakes.

I propose that testers take a break from finding bugs and just take some time to generalize. When the bug pops its head from some coded hole, resist the temptation to whack it. Instead, study it. How did you find it? What was it that made you investigate that particular part of the app? How did you notice the hole and what was happening in the app that caused the bug to pop out? Is the test case that found the bug generalizable to find more bugs similar to the one you are ready to whack? Is there some advice you could pass along to other testers that would help them identify such holes?

In other words, spend part of your time testing the current product you are trying to ship. Spend the rest of the time making sure you learn to test the next product better. There is a way to do this, a metaphor we use here at Microsoft to help.

I’ll discuss the metaphor and how we apply it to form long time habits, peanut butter style, in my next post. 

Yet my mind wasn’t really away. In fact, I thought about work all the time. Given that software was everywhere I looked, it’s not hard to see why. My entire trip was booked online, even the taxi to the airport. Not a single person besides myself took part in the process. Just me … and a load of software.

The taxi cab itself contained software, as did the airplane. The baggage carousel, the espresso machine, the car rental counter (no person there, just a self serve terminal) and even the surveillance camera that watched my son juggle his soccer ball while I packed our bags in the trunk. All alone, except for the software. Even the frozen concoction machine had software that helped it maintain the right temperature. (It broke, incidentally, making me thankful that I am a beer drinker.)  

Is it possible for anyone in this field to really get away from it all? (Don’t get me started on the motion sensors that control the air conditioning in the hotel room. I’m all for turning them off when they are not in use, but apparently sitting still and being cool was not one of their end-to-end scenarios.)

The truth of the matter is that getting away from it all just isn’t necessary for me. I like seeing software in action and I enjoy brooding over problems of testing it. Vacations free my mind from the daily grind and leave my mind to question things that back home I might overlook. Does this make me work obsessed or just indicate that I really like what I do?

Vacations have always been like this for me. When I was a professor, two students who led my research lab, Ibrahim El-Far and Scott Chase, actually avoided me when I returned from a trip, afraid of the work my new insights would bring. They never quite managed to successfully do so.

Which brings me back to the motion sensor in my room. The problem isn’t so much a poor tester, rather poor testing guidance. The sensor does exactly what it is designed to do and testing it based on those requirements got me in the sit-and-sweat loop. The problem is that no one thought to give it a field try … what I call ‘day in the life’ testing. Had the tester thought to take the sensor through a 24 hour cycle of usage they would have identified that problematic ten hour period (yes, ten, it’s a vacation after all) when motion is low and the desire to be cool is high. But what tool gives such guidance? Modern tools help testers in many ways, but helping them think of good test scenarios isn’t one of them. They help us organize, automate, regress and so forth, but do they really help us to test?

That’s the tool I want. Tomorrow, when I return, I am going to direct someone to build it for me. Ibrahim and Scott, you are off the hook this time.

Instead of writing a test case for an application, we could move down a level and write them for features instead. There are any number of web applications, for example, that implement a shopping cart so test cases written for such a feature should be applicable to all such apps. The same can be said of many common features like connecting to a network, making SQL queries to a database, username and password authentication, and so forth. Feature-level test cases are far more reusable and transferable than application-specific test cases.

The more focused we make the scope of the test cases we write, the more general they become. Features are more focused than applications, functions and objects are more focused than features, controls and data types are more focused than functions and so forth. At a low enough level, we have what I like to call “atomic” test cases. A test atom is a test case that exists at the lowest possible level of abstraction. Perhaps you’d write a set of test cases that simply submits alphanumeric input into a text box control. It does one thing only and doesn’t try to be anything more. You may then replicate this test atom and modify it for different purposes. For example, if the alphanumeric string in question is intended to be a username, then a new test atom that encoded the structure of valid usernames would be refined from an existing atom. Over time thousands (and hopefully orders of magnitude more) of such test atoms would be collected.

Test atoms can be combined into test molecules. Two alphanumeric string atoms might be combined into a test molecule that tests a username and password dialog box. I can see cases where many independent test authors would build such molecules and then over time the best such molecule would win out and yet the alternatives would still be available. With the proper incentives, test case authors would build any number of molecules that could then be leased or purchased for reuse by application vendors that implement similar functionality.

At some point, enough test atoms and molecules would exist that the need to write new, custom tests would be minimal. I think that Wikipedia, a site with user supplied, policed and maintained content, would be what the industry would need to store all these tests. Perhaps such a community Testipedia can be constructed or companies can build their own internal Testipedias for sensitive applications. But a library of environment-carrying (see my last post) test atoms and molecules would have incredible value.  

A valuable extension of this idea is to write atoms and molecules in such a way that they will understand whether or not they apply to an application. Imagine highlighting and then dragging a series of ten thousands tests onto an application and having the tests themselves figure out whether they apply to the application and then running themselves over and over within different environments and configurations.