Share via

Testing Complex Software

  • Article

It's really hard to wrap your head around how complex software like Office or Windows is. These programs are huge, and have hundreds of millions of users. All of those users have different configurations and different settings and different ways they like to use their software.

It's only natural that with all those differences some problems will be unearthed.

This is one of the key problems of testing big software like this. You can test and test and test, but it's a bit frustrating to know that even with all your work someone, somewhere will have a problem with your stuff. I had a great conversation with two developers at lunch today about this. Once your software stabilizes testing becomes a low-yield game (tangent: at this point one of the developers cheerfully said “not when you're testing my stuff!”) You test a lot of surface area, and don't find many problems. This is one of the hardest times for testing, the temptation is to say “yep, it's done!”

This is a key dark art for shipping successful software. Everyone wants to be done, but someone has to have a good idea of when done actually happens. Ship too soon and you've got a buggy product out there. Ship too late and diminishing returns has kicked in so your quality isn't much better, but you just lost a lot of money by not being out in the marketplace. Every time software ships someone, or hopefully a group of someone's, thought about these issues an made a trade-off.

A lot of people (I read about this a lot on slashdot...) claim Microsoft errors too far on the side of buggyness when we make this trade off. When I read things like this I typically get a littel upset. I know when I sign my name saying software can ship I mean it. I really, truly believe the software is ready to go out the door. I understand that some people will have some problems, but I know I've worked very hard to find all the problems I could. Most of the testers I know feel the same way, we get personally upset when a customer finds a bug in our area - we feel bad for not catching it first.

Anyway, what reminded me of all this is the fixed list for the recently released Windows Service Pack 2. It's sort of humbling, the number of obscure and random problems something like Windows can cause. If anyone doubts the complexity of testing something like Windows, go take a look at it. I know I'm going to go back and read it when we get close to shipping - this is the kind of thing that provides motivation to keep looking for problems, even when you think you've looked everywhere there is to look.

Chris

Comments