the usage fallacy

  • Article

I noticed that an old post of mine, the "Word 5.1 Plus Ten" phenomenon, was referenced in Lukas Mathis's manifesto about removing features.

He starts off the article by quoting Kathy Sierra saying that her books have fewer topics than competing books but still outsell them. His solution for how to choose what features to remove is to gather usage data, and to remove features based on that data. Apparently he didn't read all of that post from Kathy, because she says later:

Give users what they actually want, not what they say they want.

Using usage data to decide which features to cut is fundamentally flawed. Usage data doesn't tell you why a feature isn't used. You're listening to people tell you what they do, but you're not listening to them tell you why they're doing it. Cutting features based on usage is giving users what you think they want based on what you think they're telling you. There are plenty of reasons that a feature might not get used, and removing the feature is not the only solution to the low-usage problem. There are many reasons that features might not get used, including the following:

  • The feature is hidden in the interface; users don't know it's there.
  • The user expects the feature to do something else.
  • The feature is difficult to use once you've figured out what it is.
  • The feature is useful, but only in limited circumstances.
  • The feature isn't actually terribly useful. It might have been useful once but isn't any longer, or has been superseded by something that's more useful, or it might never have been useful.

In the last case, I agree: cut the feature. But in the first four cases, perhaps you need a scalpel instead of a sledgehammer.

Usage data is directional. It doesn't tell you what action to take, it tells you that there might be an action to take. If you assume that the feature isn't useful based on usage data, your decision-making process is fundamentally flawed, and you're not actually listening to your users.

If your usage numbers are low, you need to figure out why they're low. If your feature is a great one but you've hidden it in the interface, or if it would fit better with users' workflow elsewhere, then you have one problem to solve. If your feature doesn't quite do what users think it should do, then you have another problem to solve. If your feature is too hard to figure out, then you have a different problem to solve. One solution to each of these might be to remove the feature, but it might also be to improve the feature such that it meets the needs of your users.

If your feature is useful but in limited circumstances, then you either need to make those circumstances less limited or accept that this feature is narrow in scope. For example, if I were to go look at Entourage usage data today, I bet you that the usage of the Out of Office feature would be relatively low. Is that a problem? Not at all. I know that Out of Office is needed only infrequently. After all, I get three weeks of vacation per year, and I rarely manage to take more than one (this is, of course, my fault), so I don't use the Out of Office feature that often. Does that mean we should remove it? Of course not. It's immensely useful when it's needed. There might be some improvements to be made to it -- for example, maybe "out of office" isn't the best name for it, maybe it should be "vacation" or something else. That might improve its usage numbers, but it's certainly not going to make it one of the top-used features in the application. It's still going to be a feature that's limited in scope, and that's perfectly okay.

I thought Mathis's piece was especially telling when he brought up the iMovie example. iMovie '09 removed features that had been available in iMovie '08 and earlier. Apple continued to make iMovie '08 available to users, which Mathis says "allowed Apple to add features users needed" to iMovie '09. I'm not sure that I see the point of taking the engineering expense of removing features (after all, removing features involves both development and test work) only to again incur the engineering expense of adding them back to the application. There are times it has to happen (to wit, our decision to remove VBA from Office 2008 and then bring it back in the next version of Office), but this is an expensive decision that has a lot of painful ramifications both for you and for your users. It's certainly not something to do based on a flawed decision-making process.

Office:Mac does collect usage data. I wrote a post about it for Mac Mojo called we are watching you soon after the release of Office:Mac 2008. Usage data is one input into our decision-making process, but it's only one of them. Usage data tells us where we might want to wield our scalpels, but it doesn't tell us to amputate the whole leg.