Product Feedback Systems

Several people on the Windows Internet Explorer team have written blog posts about our feedback mechanisms (our use of automated telemetry in Windows, how to write a great bug, how to submit bugs, etc) for IE9.  After looking at the many similarities and obvious differences between manual feedback systems and projects, we decided to use Microsoft Connect for IE9 and eliminate the invitation requirement for filing bugs.  In this post, I want to take a step back and talk about how we made that decision.

Every Internet Explorer user is a Windows customer.  Listening to customer feedback is vital to the success of any business.  As communication methods have evolved, how a business listens to its customers has changed as well.  Our customers have always been at the center of how we envision and design IE but the way we listen has evolved both with technology improvements and with the way our customers communicate with each other.

Our Internet Explorer 8 beta customers asked us to look at different options for how we listen.  We took their feedback to heart and looked deeply into all aspects of our feedback systems as well as those used by other software companies and in other industries.  We also looked into research regarding the effectiveness and efficiency of various ways of listening and responding to customers.  Most importantly, we listened to real customers, real enterprises, and real developers.

Each time we start a new project we begin by stepping back and looking across the feedback from the previous project and the engineering process by which we collected it.  We challenge all assumptions, especially those based on speed, size, or connectivity.  We also talk with people to see what they like and dislike about our competitors’ product and engineering process choices to see if Microsoft’s customers might benefit from something similar or more advanced.

One of the assumptions most software organizations appear to take for granted is the need for beta releases.  In fact, some companies have even taken this assumption well beyond its commonly understood meaning. So we asked ourselves, “Should we do betas and, if so, how many?  Should we do nightly builds?  Should we continue to use Microsoft Connect or move to something else?”

A group of us sat down and thought through the customer benefits and drawbacks of various models.  Ultimately, there are two main benefits to public releases:

1. Feedback – customers reporting bugs on what is not working correctly
2. Readiness – testing sites with the new browser, adding newly possible features to pages, preparing product support, writing documentation and books, adjusting tooling, etc.

There is a continuum of feedback models that ranges from completely closed to completely open.  Examples of completely closed betas include new PC models, cars, toasters, etc.  Examples of completely open models include some academic open source projects, school PTA projects, etc.  We’ve chosen something near the middle so we can get broad feedback on quality events but not completely anonymous or unstructured either because we believe in responsible engineering.

From our point of view, the most important thing about working on a consumer technology like a browser is respecting people’s scarce time and energy.  You can do this while still achieving your goals of getting everybody ready (from web developers to support professionals to corporations) for product launch and getting feedback on quality issues that only surface in unique configurations.  I want to call out a few important aspects about feedback systems:

• Having a distinct start and end to the pre-release period is important.  Without this, people would gamble their business or personal browsing on the product always behaving in a predictable way.  Having beta customers running betas forever also neither respects their valuable time nor focuses that time on finding and reporting real defects. 
• How often you send out builds depends heavily on your own ability to find defects.   We looked at the pros and cons of releasing “nightly builds” to beta testers.  Because we have a professional testing organization and a broad range of hardware on which we test, we find defects daily (many prior to checkin), fix those, and look for more.  When the IE team starts running low on new types of bugs to find, we get thousands of other professional Microsoft testers to start using IE9 to report any defects they find with IE9 as their daily default browser.  We call these folks “Self-host Testers” and they find an additional set of interesting bugs.  Again, when it appears that this group is running low on new bugs, we expand the scope even further and publish things like IE9 betas.  The risk of putting out daily builds across all audiences is that multiple beta testers waste their valuable time finding the same bug because they’re simply testing incomplete code. 
• There is a decreasing effectiveness in the feedback as you expand the group of testers.   We looked at IE8 bug reports that were actionable versus those without enough data on which to make an effective code change to determine a signal vs. noise ratio.  For IE8, the signal to noise ratio in our bug database went from 3.1:1 to 2.6:1 when the code moved from Development to Test.  It dropped to 1.5:1 when other Microsoft testers started using it and reporting bugs.  It dropped to 0.73:1 when the rest of pre-release Windows users inside of Microsoft used it.  Finally, it dropped to just 0.64:1 across all the IE8 beta testers.  Closely related to the actual effectiveness of the bug reports is each group’s ability to produce a volume of effective bug reports.  The IE engineering teams produced more than five times the number of bug reports at six times the effectiveness of the beta program.  In other words, the IE engineering team was 30 times more efficient at finding actionable bugs than the IE beta testers in-aggregate over the same period of time.  Make no mistake though, the IE beta testers filed some great bug reports and we and Microsoft’s customers are glad they did!  For instance, bugs found on websites which require account login, special credentials such as smart cards, regionally-locked IP addresses, etc. are very important to us.
• We really want to know if something doesn’t work as expected.   If you think there’s something wrong with a feature, please file a bug so we can make adjustments before IE is done.  We will manage that in a way that respects everyone’s time by helping people target their time and bug reports on reporting actual defects we haven’t yet found at Microsoft.  Bug reports come in all flavors and sizes.  They range from “I want feature x” to “Please add a whole new mode so I can use feature y this new way” to “I tried feature x in the way you said it should work, but it doesn’t work that way. ”   Reports similar to the last example are the most useful.  When features don’t work like they should, we definitely want to know it. 

We also looked at various feedback tools when we started the project.  These included things like Bugzilla, Mantis, Launchpad, etc.  We compared them to Microsoft Connect, which is what nearly every Microsoft product uses to get beta feedback from customers.  It turns out that these tools are all amazingly similar.

We looked more deeply at Bugzilla and projects using it because it’s a tool that at least two other browser vendors use today.  We looked at the tool itself and how it handles bug workflow both from a beta site perspective and a product engineering perspective.  We found some really interesting similarities and differences:

1. All of them require some kind of login.  Mozilla’s Bugzilla needs an email address to log-in. WebKit’s needs an email to log-in.  Both of them warn you of being spammed if you actually use your primary email account so you should get a new email account if you want to file bugs.  By contrast, Microsoft Connect uses a Windows Live account with either your current email account or a free new one.  In either case, your email address isn’t visible to people or web-bots mining the site to get email addresses for spamming campaigns.  We actually went one step further and require LiveID login to query the bugs, further reducing the risk to our beta users.  Connect also works for multiple Microsoft beta projects with a single account.
2. They all handle entering issues by area, feature, or symptom.  All have some bug entry form where you fill out the details, including repro steps, title, etc.  They provide a way for you to give it a priority.  They let you provide great feedback (Bugzilla, Connect) or poor feedback (Bugzilla, Connect).
3. Bug management differs considerably across projects though.  With IE8, we took action on every issue that was reported and we staffed our beta with Microsoft professional product support engineers so they could learn how to support the product while we built it.  By contrast, if you filed an issue with Firefox 3.5, it would sit at Unconfirmed until someone with “Can Confirm” privileges saw it, which may take years.  If a privileged person does happen to see it, it may eventually be seen by the engineers. 
4. Bug closure also differs across Bugzilla and Connect.  For IE8, we took action on every bug and drove it to closure.  By contrast, as of 4/26/10, Mozilla’s Bugzilla had 12,779 unconfirmed bugs.  Webkit’s had 2,616 unconfirmed bugs across all versions but very few bugs against older versions.  Just like unemployment will never be 0%, it’s important to note that in-development projects will always have a non-zero number of bugs just as a normal part of bug management.  Each project team needs to understand and manage their inventory of bug reports to ensure a reasonable response time for their project testers.  For a project as complex as IE, that can be as large as a couple thousand issues active at any point during the project.

We made some engineering decisions on how to get the best possible feedback from our beta customers after doing all the research into different release cadences, feedback models, tools, and bug management processes. We want to respect our customers’ time and energy so we’re going to distribute more focused Platform Preview builds when there are new platform features ready for people to test drive. 

Thanks for all the great feedback in IE8. We’re looking forward to building a great IE9 release!

Jason Upton
Test Manager – Internet Explorer

Comments

• Anonymous
  January 01, 2003
  Your post doesn't actually seem to provide a rationale for why Connect-the-tool was chosen; the model you seem to be criticizing on Bugzilla is a people process that you could have fixed, using your professional support professional professionals.

• Anonymous
  January 01, 2003
  You'll notice that none of the 5 files are preview-able/downloadable.

• Anonymous
  April 28, 2010
  This is a very interesting post. You do explain why you chose not to opt for nighly builds which is understanable. But what about something in between nightly and two months like maybe two (2) releases a month or maybe a weekly build? It seems like instead of choosing nighltys wich is fine, you just went a little to far to the other side of the spectrum.

• Anonymous
  April 28, 2010
  i filed a bug on rss feeds in ie8 during the windows 7 beta, over a year ago, and it's still is not fixed. i was told by an engineer that they thought a fix was checked in back in september. it is still not released. every time i wake my pc up from sleep mode, i have to run: msfeedssync disable msfeedssync enable to get my feeds to update. if i don't run this, my feeds will not update for over 24 hours. and no, it's not my settings. i have installed windows 7 multiple times on multiple pc's and they all have the same problem. since the issue was acknowledged by an engineer, i know it's not me.

• Anonymous
  April 28, 2010
  I find it interesting that Mozilla and WebKit each have more unconfirmed reports than IE 8 has of overall reports (2437+65).  Also, all of the first 100 bug reports were closed as "Not Reproducible", "By Design", "Won't Fix", or "Postponed".  Are there any stats on how may of the 2437 reports were actually closed as fixed?  There doesn't seem to be a way to filter by closure reason in Connect.   Even more interesting is that all of the Suggestions were closed as "Postponed".  (There is one Suggestion reported fixed, but it was really a bug reported in a newsgroup, and had already been fixed.)  Is this a result of the quality of the suggestions, or from not having the feedback area open early enough to actually be able to do anything with the suggestions?

• Anonymous
  April 28, 2010
  The comment has been removed

• Anonymous
  April 28, 2010
  Your post doesn't actually seem to provide a rationale for why Connect-the-tool was chosen; the model you seem to be criticizing on Bugzilla is a people process that you could have fixed, using your professional support professional professionals. Still, it's nice to see someone from IE throwing stones at the competition. Bravo.

• Anonymous
  April 28, 2010
  Other peoples points are quite valid; the number of unconfirmed vs fixed vs won't fix etc. is up to the methodology of the team and has nothing to do with Connect, Bugzilla, Mantis, Fog Bugz, SpiraTeam or whatever other bug tracking system you might use. Microsoft seems to treat each IE release as distinct with distinct bugs and closes any that won't be fixed in the current in progress version.  While this keeps the bug list small it doesn't please your testers who would be happier with a "We can't get your changes into this release but they will remain under consideration for the future" than the cold slap in the face of "WON'T FIX" which sounds like "We don't care". I don't think, as Mitch suggests, that Bugzilla allows for any better communication, but I do think that some projects are better at providing feedback than IE.  The reason for this is more likely that the bug tracker IS the forum used to discuss the viability of fixing the bug; for IE these decisions are made internally and the results are posted to the bug tracker afterwords so we miss out on the private discussion and any chance to contribute to it.

• Anonymous
  April 28, 2010
  The comment has been removed

• Anonymous
  April 28, 2010
  Quote: "We actually went one step further and require LiveID login to query the bugs, further reducing the risk to our beta users." Was THAT the reason it required a login to view a bug????? Here is a much better solution: Don't show me(The person trying to view the bug database) other users email address, unless i am logged in. Problem solved, and no need to hit all the bugs

• Anonymous
  April 28, 2010
  Gerard-- there are many things to complain about and examine critically, but you've picked the wrong ones yet again. The current IE team is in no way responsible for the fact that they didn't exist as an ie team in 2003 and thus didn't deliver the mythical 6.5 browser version you wanted then. You can whine about it, but it's a waste of breath unless you have a time machine around somewhere. Move on. As for the tooltip: the IE team doesn't actually own Connect either. I think it's a steaming pile myself (and they are to be criticized for choosing it again and then glossing over its massive deficiencies) but again, whining about the tooltip is a waste of breath. Whatever you "personally remember" (as opposed to what, exactly?) about the state of the web in 2000, you seem to forget that IE was the best browser (by a landslide) back then. Sure, MS can be criticized for abandoning it after that, but we're back to point #1-- the current IE team had nothing to do with that. They're all likely glad that competition sprung up so that they could join the new ie team made for IE7. It's neither illogical or incoherent to agree that "yes, there is a bug, and no, it wasn't by-design", and still resolve it "Won't fix." Bugs that aren't important enough to fix (ever) have this happen to them, at EVERY software shop in the world.

• Anonymous
  April 28, 2010
  It is posts like these that make feel that one you do not listen to developers, two are trying to spin the past as happening in a way it did not. Nothing is worse to read here than PR talk whitewashing why issues that people have to spend real energy to make applications work correctly in your product that are not an issue elsewhere remain to this day and will for years.

• Anonymous
  April 29, 2010
  So far I use IE to latest IE released by the microsoft, I never experienced bugs nor problem that may cause your internet hindrance.

• Anonymous
  April 29, 2010
  @matt - instead of pointing out the issues with connect on this blog, post a bug/suggestion on Connect.microsoft.com/connect. This is the connection for the team that acutally owns and builds connect. They may be able to help out with improvements. They FINALLY added public attachments in the most recent site update.

• Anonymous
  April 29, 2010
  The comment has been removed

• Anonymous
  April 29, 2010
  A great "PR" post on bug tracking. Does bugzilla for mozilla have a lot of unconfirmed bugs? yes! but with public access to a great browser, lots of people will dig deep in the application and submit bugs. Are there "poor" bug reports? yes! sometimes you have to deal with the reality that every submitter might not be the best reporter or even a technical user.  This is why the ability to filter down the report to a simple, repeatable test case is critical (reduce the noise, increase the signal) Is Connect the best system for MSFT to use? Of course it is, and they have no intention of moving off it (as noted, this is a PR post) What is needed though is improvements to Connect - even if just for the IE connect portion. 1.) All users ******** M U S T ******** be able to see all attachments (unless a bug is reported as private) - You can't enlist a volunteer army to do the bug triage, refinement, and verification IF they can't see the test case code. 2.) Last time I checked, the search in Connect for IE was horrible. If this has been fixed, great - if Not this is needed.

• Anonymous
  April 29, 2010
  Who is responsible for the lack of a TABBED WINDOWS EXPLORER in Windows 7? Why have a tabbed browser but no tabbed file manager?

• Anonymous
  April 29, 2010
  The IE team would use Bugzilla or Mantis in a heartbeat if they could lock down either platform to be as restricted and code-hostile as possible. I'm also pretty sure that Microsoft finds the idea of using an open-source bugtracker or open-sourcing their own to be absolutely abhorrent. The "WE MUST DEVELOP EVERYTHING HERE!" mentality is astounding and stupid.

• Anonymous
  April 29, 2010
  The comment has been removed

• Anonymous
  April 30, 2010
  @gkeramidas Would you mind providing a link to the bug that you filed?  It would be helpful to be able to check on where it currently is in our system. @Mitch 74 & @jim Thanks a lot for your feedback!  As @Tom Stack points out, many of the challenges you describe are the results of the current design of the Connect system.  The Connect team is dedicated to providing the best feedback system possible, and I'm certain they'd very much appreciate hearing your input on how to make the process better.  This includes the challenges of re-activating a bug from release to release (which isn't possible if a new bug form is used for the new release), having to use ZIP attachments for code instead of the source files themselves (which is due to appropriate security concerns), linking bugs, and the search experience.  I encourage you to add your voice to their feedback program at https://connect.microsoft.com/connect. @jim Currently users can in fact see attachments, unless the submitter filed it as private.

• Anonymous
  April 30, 2010
  @justin, the original bug was 437631, but is not accessible now. it was filed on 5-4-2009. worked through email with reza on the issue. refiled the bug in february of this year. here is the link. http://connect.microsoft.com/IE/feedback/ViewFeedback.aspx?FeedbackID=482124

• Anonymous
  April 30, 2010
  Hey Jason! :-) I'm one of the primary developers of Bugzilla, and I really appreciate and found interesting the analysis you did of Bugzilla vs. Connect! :-) It's true, a lot of bug-tracking systems have a lot of things in common. That could at least partly be because Bugzilla is pretty much the oldest bug-tracking system in common use today, and so to some degree was a model (or at least something for comparison) when building other systems. There are a few points I wanted to respond to:

1. As of Bugzilla 3.4, Bugzilla no longer displays email addresses to logged-out users, so the note about being spammed isn't really true anymore, unless spammers start creating Bugzilla accounts (which I suppose is possible, but we don't have any reports of that yet). With some minor modifications, Bugzilla could actually even protect email addresses when the user is logged in, if that was important. 3 & 4. As far as there being a lot of Unconfirmed bugs in Firefox, I'd say that that's more of a function of the Mozilla Corporation and the way that resources are allocated, as opposed to an aspect of the tracker itself. It sounds like Microsoft has some people being paid to actively look at incoming bug reports, while at Mozilla it's a bit more hit-or-miss as far as whether or not any paid person will actually be there to see the bug reports. However, do you have any particular differences between Connect and Bugzilla that you think makes it easier to address incoming bugs in Connect? We'd love to improve Bugzilla if you have any thoughts on it. You know, one thing I'd love to see in Connect that is available in Bugzilla is the ability to read bug reports without having to log in. Bugzilla was built from the ground up to be an open bug tracker, so that is probably an advantage it has over Connect, in some ways, although I don't know much about the history of Connect. Any how, good luck with IE9 and the feedback process! I'd love to hear any responses you have to my comment, if you get it. My email address is mkanat@bugzilla.org. -Max

• Anonymous
  April 30, 2010
  Oh, also, I forgot to mention, but getting Bugzilla to authenticate against Windows Live would be really easy--Bugzilla has a pluggable authentication system. -Max

• Anonymous
  April 30, 2010
  @JustinSC [MSFT] re: "@jim - Currently users can in fact see attachments, unless the submitter filed it as private." This should have been announced!  Attachments in Connect have NEVER been publicly viewable, a point that developers and testers have complained about continuously since IE7 first entered beta development in Connect. Here's a screenshot of a bug I submitted (marked as public), with in this case, some screenshots attached. http://img571.imageshack.us/img571/5746/attachmentshaveneverbee.png You'll notice that none of the 5 files are preview-able/downloadable. however, I tested a brand new bug submission and yes, you can attach files that are downloadable! However, there's a hidden link tooltip that indicates you can't upload HTML, JS, or SWF files. (extremely annoying) You'll need to zip those files first I guess. There is a definite delay in seeing the files show up though, so don't be surprised if it takes 10minutes for the files to show up! Great news on this ability being added - I look forward to reviewing other attachments to reproduce bugs - and others being able to view my attachments (at least on the new bugs filed - as the old ones are still locked)

• Anonymous
  May 01, 2010
  @Max Thanks for the update on the Bugzilla 3.4 behavior. I'll pass your suggestions and contact info to the Microsoft Connect team.  Thank you.

• Anonymous
  May 07, 2010
  The comment has been removed

• Anonymous
  May 08, 2010
  Lonnie, you should read this: http://blogs.msdn.com/ieinternals/archive/2009/06/22/HTTPS-Mixed-Content-in-IE8.aspx