Usability: Art and Science
Yesterday morning we were sitting in the office of one of our usability researchers watching some DVCAM tapes from tests conducted a few weeks ago.
We had a discussion that got me thinking about a set of tests we ran several years ago to determine the discoverability characteristics of contextual tabs.
At the time, contextual tabs were struggling in the usability lab. The visuals and triggers were not obvious enough, and even when people noticed them, the tabs looked so different from normal tabs in the UI that participants thought they were decorative or unactionable.
We kept iterating and iterating on the design, and one of the desperate ideas we had was to pop up a little yellow balloon the first time a contextual tab set appeared saying something like "Hey you, contextual tabs have appeared, you better click here get to the tools for working with your table."
(I'm sure the real wording was a lot more Microsoft-esque.)
Anyway, we wrote a little app to enable us to pop up the balloon at the right time--but it was a totally manual process. We had two keyboards hooked up to the usability computer, and when the contextual tabs appeared, one of us in the back room would press F10 on our keyboard to make the balloon appear. So the timing was a little weird, but it was cheaper than building the feature directly into the product itself.
The balloon wasn't the only change to the design in the new build being tested, however--we also tested improvements to the visuals and the triggers that activated the contextual tabs as well.
The result of the tests? The usability characteristics of contextual tabs improved dramatically.
But now we had a quandary: which improvements precisely had caused the uptick in usability? The balloon? The substantive changes to the interaction model? The clearer visual design?
One could imagine a world in which we ran controlled, double-blind studies to test the impact of each element of the design separately to assess the best possible combination.
In reality, though, we tend to use an iterative process in which we bring an entire design to a next level and then (if the design is successful) figure out which are the non-critical parts of the improvements. The advantage to this process is that it lets us move faster and abandon bad ideas sooner.
In this particular case, we felt kind of icky about the balloon, so we decided to run another set of tests to see how much not showing it changed the results from the previous successful test. It turned out that the test results didn't change at all mathematically; the usability of the feature was being impacted much more by the substantive changes to the design than by the notification balloon.
Developing a contextual tab design that worked well took well over six months of concentrated iterations, followed by tweaks over the last two years or so as we continued to make progress on the design surrounding them.
The biggest reset recently was when we introduced new visuals for Beta 1 Technical Refresh last winter and we had to reevaluate usability of the entire UI based on the new look.
Some of the most interesting studies we did were eye tracker comparison tests which enabled us to see how and where the new visuals affected the scanning pattern of the UI. It turns out that moving group labels to the bottom of each group in the Ribbon, for instance, helps people target the control they're looking for a bit faster than in the Beta 1 visuals.
So, could we apply an even more incremental method of usability confirmation to more fully test each element of a design change in isolation?
Perhaps, but a design is much more than the sum of its parts, and the usability of one piece always has to be weighed against the usability of the overall product. This is where art meets science.
There's a talk I give to program managers internally at Microsoft in which I present a 100% guaranteed way to improve the discoverability of a fictitious "Send via Telegraph" feature in Word:
You can't evaluate the usability of just one feature or component of an overall design without understanding its impact on the entire product.
Good design is the art of balance.
It's an art that can be infused and informed by scientific rigor, but in the end it's still an art.
Comments
Anonymous
June 21, 2006
Good to see you back writing regularily again, Jensen! As usual, very interesting stuff. Keep it up!Anonymous
June 21, 2006
Yeah, thanks for coming back!Anonymous
June 21, 2006
Ribbons? ...what useless bloatware ideas will M$ come up with next, and try to put a positive propaganda spin on? Nevermind... I don't want to know :PAnonymous
June 21, 2006
Just being curious:
The user interface usability testing must have been both a fascinating and an enormous project. How many experimental subjects did you have to bring to the labs? I suppose one test subject could only be used once?Anonymous
June 22, 2006
Fascinating! I love this blog.
I am surprised by the F10-pressing deus ex machina. Doesn't MS have a general UI testing tool? Perhaps a debugger-like tool with timers, watches, and event triggers?Anonymous
June 23, 2006
I have a problem with the Outlook's message editor's ribbon: If you author a new massage or reply and you use the ribbon to access advanced (ex. formatting) features, than when you finish working and try to send the message, you try to click to the leftmost icon by instinct. Unfortunately there well not be the send button, you have to click to the Home ribbon, to reveal the send button.Anonymous
June 27, 2006
The comment has been removedAnonymous
July 03, 2006
Same thing that happens to Bálint Kátay happens to me in outlook...a fix is necessary. Gñad to see you writing again jensen.Anonymous
April 19, 2007
PingBack from http://rickosborne.org/blog/index.php/2007/04/19/usability-vs-discoverability/Anonymous
October 27, 2008
PingBack from http://mstechnews.info/2008/10/the-office-2007-ui-bible/Anonymous
June 12, 2009
PingBack from http://cellulitecreamsite.info/story.php?id=7819Anonymous
June 13, 2009
PingBack from http://outdoordecoration.info/story.php?id=192Anonymous
June 13, 2009
PingBack from http://gardenstatuesgalore.info/story.php?id=250