usability testing is not beta testing

Every time that I post that I'm doing usability testing, I always get a flood of email asking about beta testing. Beta testing and usability testing are all but completely unrelated.

Usability testing is about just that: testing the usability of an application. Can you figure out how to do something specific? How long does it take you to do it? What do you do try along the way? What do you do when you can't do it? Beta testing, on the other hand, is about finding issues. Can Word open this .docx that I created a few months ago? Does Outlook play voicemails correctly?

Usability testing is done in my lab, using my equipment (and, in the case of client/server software like Outlook or Messenger, using my server too). You might not even be looking at real honest-to-goodness code, but rather a prototype of some sort. I do usability testing on prototypes running the gamut from sketches on paper (and using stickies to simulate the interaction) to awesome-looking Flash prototypes. Beta testing, of course, is done in your home or business, and it's definitely real code. It's not finished code, but it's definitely the real thing.

Usability testing is very focused on specific tasks. When you come in for a usability study, I have very specific questions to ask. For example, in my last Outlook usability study, one of the questions that I needed to answer was, "can users move a single instance of a recurring meeting to another time that is available for all attendees?" As such, I had a test account set up and pre-populated with a few users, and each of those users had a few appointments and meetings. The user had to open up a single instance of an event and find a new time for the meeting. Beta testing, on the other hand, is not focused at all. You're using it in real-world scenarios to try to find bugs, so you might (or might not!) do something similar to this. We don't give you specific directions like this during the beta.

Usability testing is conducted over a small period of time, usually no more than a couple of hours. Beta testing is conducted over a much larger period of time, usually at least a few days but more often weeks in duration.  Beta testing gives us insight into plenty of other aspects of the application, such as security, load testing, performance, and cross-platform compatibility. All of these are aspects that I can't address effectively with usability testing.

If you want to get Outlook:Mac to take home and use against your own Exchange server, signing up for my usability studies won't get you towards that goal. But participating in one of my studies will give you the chance to get the undivided attention of several MacBU employees for the two hours that you're in my lab, and your feedback is an important part of our decision-making process for deciding what Outlook will be. You can sign up to participate in our usability studies. Our beta programme for Office:Mac is a private beta; more details about how to get into a private beta can be found in this older blog post of mine: Q&A: How do I get into a MacBU private beta?

Comments

• Anonymous
  March 18, 2010
  as I make to place a site ISS of my server with IP 192.168.0.1 inside