Windows Confidential: Retire the rubber stamp

An important aspect of intelligent product development is realizing a step in the process might not be necessary.

Raymond Chen

As a product gets closer to shipping, whether it’s a prerelease version like a consumer preview or the final release to manufacturing (RTM), every change undergoes greater scrutiny to ensure the remaining limited engineering effort is focused on the important issues. This scrutiny is also intended to verify that each approved change won’t have unintended consequences that make the cure worse than the disease.

Software product development isn’t only nonlinear, it’s discontinuous. A minor perturbation in one part of the code can magnify into a large change somewhere else. That in turn can cascade into a catastrophic failure.

In the early stages of the release process, scrutiny comes at the local level. Feature owners look over every change request and decide which ones to accept, which to postpone and which to reject. For example, if the problem only affects a small number of people or has an easily discovered workaround, it’s more likely to be postponed or rejected.

Assuming the proposed change passes the hurdles of user impact and relevance, the code to implement the change is written and tested. Then the feature team will review the proposed change and accept or reject it based on various criteria, such as the level of risk associated with that change. This isn’t a discrete process. Evaluation occurs continuously, but these two phases (initial acceptance and final review) are usually the most contentious.

As the deadline draws near, there’s often an additional layer of review added at the team level. Every change that meets the feature owners’ approval must then be presented to the overall team managers. The team managers may choose to reject a change that the feature owners accepted.

It’s usually at this level that people start asking embarrassing questions. How long has this bug been there? Does this same bug exist anywhere else? How was this bug introduced? Who signed off on the change that introduced the bug? Why didn’t testing find this bug sooner?

These questions aren’t meant to embarrass the feature owners. (That’s not their sole purpose, at least.) Team managers need to know these answers because fixing a bug that shipped in the previous version of Windows may introduce a compatibility issue. A bug with seniority often turns into a compatibility constraint. A bug present in multiple preview releases that went undiscovered until recently might not even be worth the risk of fixing.

Even closer to shipping, reviews extend further up the chain. First, they’re bumped up to the group level, then finally to the entire Windows product team. In these final stages, every proposed change to the product is taken to the Windows Ship Room. Here you have to represent the change and explain why you believe it should be accepted into the release.

Windows 8 followed this process, just like its father and grandfather. Then something interesting happened.

After one of the prerelease versions of Windows 8 shipped, the Windows Ship Room folks went through their records of all proposed fixes brought for Windows Ship Room approval during that prerelease cycle. Each bug was presented, embarrassing questions were asked, risks were balanced against the benefit, and it turns out that every single one was ultimately accepted.

The people who ran the Windows Ship Room concluded this meant the group-level ship rooms were doing a good job of deciding which bugs to fix and which to postpone or reject. The additional scrutiny of the Windows Ship Room wasn’t adding any value, because it never overturned a lower-court decision.

As a result of this post-mortem analysis, the release management team simply postponed the Windows Ship Room process for a while. They abided by the decisions of the group-level ship rooms. The Windows Ship Room wasn’t indefinitely dissolved, but after recognizing it wasn’t an essential part of the process, shutting it down for a while let everybody get a few hours of life back.

Raymond Chen

Raymond Chen's Web site, The Old New Thing, and identically titled book (Addison-Wesley, 2007) deal with Windows history and Win32 programming. Not to be taken internally.