This Beta Bug's Life

There was an interesting question on Encarta's Beta newsgroup recently. In a nutshell, it was asked:

How does a bug submitted during a beta get fixed?

Beta testers have many drastic differences in their backgrounds. They range from professional software testers, to non-technical users who are writing their very first bug. This range of backgrounds is very important during a beta- some of the best bugs/suggestions come from our least technical users (see here for more on that topic). On the flip side of this coin, the bugs vary greatly in quality. Some are gems, complete with concise repro steps and screenshots when appropriate. Some border on incomprehensible- if there seems to be a significant issue reported then we need to follow up with that Beta user for more detail.

Encarta's Beta Bug processing technique is probably *not* how it happens in most of Microsoft. Most teams have their own subtleties, but I'll explain what we do in Encarta because that's what I know best.

Step 1: Our friendly Beta Tester enters a bug through the Bug Reporting site. This bug is then imported to our temporary Beta bug tracking database.

Step 2: Encarta PMs go through these bugs in the “Active“ state, and do our first round of elimination. If they've already been fixed, or are guaranteed to be fixed before shipping, then they'll be resolved as “Fixed“. If the bug is potentially a new issue, or more information is needed, then the PM will assign the bug to an STE.

Step 3: The STE now does their own evaluation of the bug. If it's a well written bug for a new issue, it will be ported to our core bug tracking database as is. If necessarily, we'll clean it up by eliminating un-necessary repro steps and taking appropriate screenshots. If necessary, we grab more information from the Beta tester until we can repro the same issue on our local test machines (a vital step for developer debugging).

Step 4. The bug now exists in our `main' bug tracking database, and will live and die on its own merits. If the bug is high visibility and is easily fixed, then it probably will be. There will inevitably be more bugs than can possibly be fixed, and this leads to each bug essentially being stack ranked. All bugs over a certain threshold will be fixed for the current version, or “Postponed“ for future versions. Some issues will be “Won't Fixed“ unless they reappear as more severe issues than originally thought.

The ultimate goal of a Beta tester is to get their bugs into our Primary bug tracking database. If they've done this, then they've done a great job.

Greg