Betas and Bedmates

(Yawn)

It's Saturday morning and I'm sitting in the TV room watching my 2-year-old play with her puzzles while a cartoon show drones in the background. It seems like a perfect time to blog a bit about a recent thread on AvSim that starts off speculating whether third-party developers are 'in bed' with Microsoft by virtue of them participating in our beta program. I was wondering about that myself so after waking up this morning I rolled over and asked PMDG's Lefteris Kalamaras what he thought...

Okay. Sorry for that imagery. (Remember, you can destroy all the pictures except for those in your head. <g>) Without directly addressing any of the conspiracy theorists' concerns I thought you might like to know how the beta program works and what we (the Flight Simulator team) get out of it. (Much of this information was also prevented at the AvSim conference a few years back.)

Let's start with the intent and dispel one seemingly persistent myth, that the only testing done on the product is by the beta testers. In fact, the vast majority of testing, regardless of how you measure it (except perhaps for the number of testers), is done by teams internal to Microsoft. External beta testers may be the choice of some third-party developers simply because of resource issues but for Microsoft the testing starts here. Internal testing ensures a certain level of comprehensiveness that beta testing can't guarantee. Our dedicated test team has the time to plan a series of tests that cover all new and existing functionality. Plus we have shared resources for specialized testing such as configuration compatibility.

A way to restate this is that internal testing is structured. What beta testing is really useful for is unstructured testing. A good example of this is compatibility testing for video drivers. Our internal configuration testing lab usually tests video hardware using two varieties of drivers: reference (typically the driver version that shipped with the hardware) and latest (the version available at the time of the test). Often, though, interim versions have bugs that only show up when some beta tester in the field has it on his rig.

It should be obvious then that the beta group needs to be quite diverse. After all, if it were only made up of people who keep their drivers up to date we'd never find the types of bugs I just mentioned. Speaking specifically about the FS betas, we pull from two distinct populations. The first is a pool of testers maintained by Microsoft Game Studios and used on other games. This population is representative of the broad user base that buys games. The second population is a database we maintain of FS users and developers. This population tends to delve deeper into the product and given its depth is useful for finding issues that a general gaming population would miss. Looping back to the original thread, it's important to include third-party developers because they use and stress the product in a way unique among both general gamers and FS users. In the end we're looking for a population of beta testers that represent the greater population of FS consumers.

I'll close this post by saying how proud we are of our beta testers. When we compare the effectiveness of FS betas versus other game betas the FS betas consistent come out on top. How do we measure this? One way is to look at the percentage of issues reported by beta testers that get translated into actual bugs we track and (try to) fix. FS betas typically have a rate double that of other games. So, keep up the good work!

I hope that was interesting. Feel free to post a comment and tell me what else you'd like to know.