Product Feedback Center after VS2005 ships

In a suggestion I responded to (FDBK17182: “Hire more people, so you can fix more of customers’ suggestions”), Phylum asked me to clarify a comment I made about fixing bugs and suggestions in a future release – he asked whether I met the next official version or in a service pack.

In most cases I mean in the next official version, but it does beg the question about how Product Feedback Center (PFC) will affect our whole servicing strategy for released products. The internal focus up until now within the Developer Division has been just making it work for VS2005 – getting the teams used to handling large volumes of customer feedback, understanding what “good job/quality responses” means beyond simple turnaround times (we have more to do here – e.g. we sometimes see great, lengthy repro scenarios from customers that get a terse “no repro” back from us), figuring out which few suggestions we can address given that we’re late in the ship cycle, etc.. However, we’re also starting to think about the transition from pre-RTM to post-RTM for feedback from PFC.

Here’s what we don’t want to happen: We get lots of great feedback prior to shipping, and address as much of it as we can. On the day we ship there’s a set of suggestions and probably even low severity bugs that we’ve postponed. And then we shut down feedback on VS2005 and/or never bring up the postponed feedback again and/or stop responding on VS2005 feedback that comes in after RTM.

What we want is something more like:

- Postponed suggestions and bugs for VS2005 are the FIRST thing that gets considered for the next release.

- Customers can continue to submit feedback on VS2005 even after we ship. If you have the shipped product and you find a bug, let us know about it via PFC. Customers shouldn’t have to use some separate feedback tool just because we ship.

- We continue to acknowledge feedback as it comes in, even if we can’t provide a clear resolution/fix quickly. Then customers at least know it’s on the list.

- We use customer feedback to do a better job with servicing on existing products. This is probably the most interesting part. One of the challenges we have today is that we have no good way of knowing whether there are common bugs in shipped products that customers frequently hit. I’m sure there are a few counterexamples, but this is true in general – what we know is more anecdotal than scientific. With PFC, we potentially have the ability to get the list of bugs people most want to see serviced, and to have a goal that any service packs fix a high % of the top bugs.

- I focused that last paragraph on bugs. I think Phylum’s original question was about suggestions. It’s unlikely we’ll fix a significant number of suggestions via a service pack. I wouldn’t say we’ll never do it, but I know for sure we won’t do a lot of it because the test costs on service packs are fairly high and still not 100% coverage vs. RTM, and therefore we try to reduce all unnecessary churn. However, PFC again creates the possibility that we can set a small goal around the suggestions, e.g. look at the top 20 suggestions and see which are possible to address through low-cost/low-risk implementations.

- Even if we don’t release official fixes through service packs, PFC gives us much better way to consider alternative solutions, e.g. releasing a sample, whitepapers, etc. that get customers part of what they ask for.

I said this in my response to the bug report, but we rolled out PFC late in the VS2005 dev cycle not by design (as in “let’s not ask for feedback until it’s too late to act on many of the suggestions”) but because PFC wasn’t ready until then. PFC is a long-term commitment – we’ll turn it on earlier in future cycles, other products will start using it, etc.. And we want to make sure that we’re benefiting from all current and historical feedback at all points of our dev cycles – late in the cycle when we’re trying to make sure the quality is right, early in the planning phase when we’re figuring out what to do next, mid-way into the cycle when we’re trying to make sure the features/scenarios work well, and in-between cycles when servicing is an issue.

Let us know your thoughts here.

 

Mark Cliggett

Group Program Manager - DevDiv Customer Connection Team

Comments

• Anonymous
  November 09, 2004
  The comment has been removed
• Anonymous
  November 09, 2004
  We're changing the feedback ranking algorithm to address the ample feedback we've gotten, and to solve the problem you mention above - where just by commenting you increase the ranking. The new algorithm is essentially just the average vote/ranking, with number of votes used only to break ties among bugs with the same average. I believe this change has already been implemented, according to this FAQ:http://lab.msdn.microsoft.com/productfeedback/faq.aspx#howrank
  
  I'm surprised this change isn't already in the blog and wasn't communicated more openly. I asked about this a few days ago and was told it had been. I suspect that it was explained but only in the resolution of the top bug/suggestion on this, which is no longer easy to discover :-( I apologize for any confusion here and will make sure someone comments on this change in the blog in the next day or two.
• Anonymous
  November 09, 2004
  Ahhh that explains why everything changed ;-)
  
  What's stopping three or four friends from ranking 'their' bug with 5's and pushing other bugs to the bottom though? It's great to see that the PFC team is working on new ways to improve how the system works.
  
  I don't think the new algorithm has been fully implemented yet, closed items are still showing in the list. Probably a multi-phase rollout or something.
  
  Probably not the right place to mention this (sorry! :-) but it sure would be nice to see a way to select which .net class(es) you are submitting a bug for. With the class information being stored in the suggestion/bug database, you could build some pages to list the bugs by namespace or class. Might make it easier to find duplicates and allow the developer community to post feedback/workarounds/votes/etc.
• Anonymous
  November 10, 2004
  The new system doesn't seem great either. When I click the "More..." link underneath the suggestion and bug lists on the product feedback home page, every single entry listed has a rating of 5.0. So either these numbers aren't being reported correctly, or everyone is just always marking all their votes as "5" and it ends up being a simple "rank by number of votes".
  
  I'm actually suspecting a reporting problem - surely some lone individual out there somewhere posted some lower rankings?