Share via

Fixing Ranking

  • Article

Ranking bugs/suggestions by average score*number of votes is wrong and we want to fix it. The solution I propose would allow for tracking of several buckets of bugs and suggestions. Ideally the PFC should have a robust statistics area showing various interesting metrics and the home page should be dynamic and rotate the buckets displayed.

First to define what the rating scale should mean for bug vs. suggestions:

Rating

Bugs

Rating

Suggestions

1

Fixing this issue is not important.

1

I don’t like this suggestion. Don’t waste time implementing it.

2

This is a minor bug.

2

I’m neutral, I don’t care if Microsoft implements this suggestion.

3

This bug should be fixed but I can live with it.

3

This is a good idea.

4

This bug is very annoying and I want Microsoft to fix it.

4

I really like this suggestion, I think Micrsoft should implement it.

5

This bug is very serious, embarassing or it’s a blocking issue. It should absolutely be fixed.

5

This is a must-have feature and Microsoft should absolutely implement it soon.

Most important bugs

Most important bugs should be determined by the average vote value and ties broken by highest number of votes. To appear on the top ten list, a bug must have at least N votes (where N is at least greater then 2).

Best suggestions

Similarly to important bugs, best suggestions should be determined by the average vote value and ties broken by highest number of votes. To appear on the top ten list, a suggestion must have at least N votes (where N is at least greater then 2).

Most popular bugs

Determining the most popular bugs should take into account number of votes and validations.

Most popular suggestions

Determining the most popular suggestions should take into account number of votes.

Low traffic bugs and suggestions

Bugs and suggestions that have fewer then 3 votes or validations should appear on a list where they can receive higher visibility

 

Marie Hagman

Visual Studio Core Team

Comments

  • Anonymous
    August 27, 2004
    I opened the initial bug you linked to. And I like the idea put forware here :) Go for it!
  • Anonymous
    August 27, 2004
    NOTE: This posting has a NC-17 rating ;-)

    You started your rating scale with ignorance.

    I dislike this. I can not ignore this posting.
    I'm willing you definely not fixing this in this way.

    Did you get this idea ;-) ?

    In addition to:
    "I don’t like this suggestion. Don’t waste time implementing it."
    there must be
    "(Speak loudly) I hate this suggestion !!! I'm willing you never implement this !!!"

    As well in addition to:
    "Fixing this issue is not important"
    there must be
    "This in an issue with bug submitter himself. Fix him ;-)"

    Something like this.
    As for rating calculation - you must not assign always positive scores. You must have neutral and negative scores.
    Something like:
    -2,5 -1,5 -0,5 +0,5 +1,5 +2,5

    This way realy important bug will clearly outperform regular one. If you will count Sum(Votes)/(Number_Of_Votes * Duration_In_Days) you will have most important (and recent) issues on top.

    Homework task: How about checking all proposed scoring system using already recieved votes ?

    Create a table:
    Scoring system X
    * Top 10 reports as they will appear one month ago:
    ______
    * Top 10 reports today:
    ______

    Scoring system Y
    * Top 10 reports as they will appear one month ago:
    ______
    * Top 10 reports today:
    ______

    Post results here and only after this decide that system you are willing to adopt.

    David: My suggestion were first FDBK10304 (2004-03-11 01:01:21 ;-)
  • Anonymous
    August 28, 2004
    You are right, you were first ;) Weird that I didn't find your's when I entered mine... Well, I got more votes, though ;)
  • Anonymous
    August 28, 2004
    This scheme doesn't discuss how you deal with bugs and suggestions that are resolved. Once a bug is resolved it should be removed from the top-ten lists.

    If you do this, though, it would be good practice to not resolve the bug too quickly, so folks have a chance to provide feedback that would be useful in evaluating that resolution.
  • Anonymous
    August 28, 2004
    Why not just use a -1 to 4 scale instead of 1 to 5 and leave the ranking algorithm the same?
  • Anonymous
    September 16, 2004
    fix the spelling of "created b4" to "created by" on the search page.

    http://lab.msdn.microsoft.com/productfeedback/SearchResults.aspx?text=&stype=1&fields=1&type=1&category=0&os=0&oslang=0&status=0&msstatus=0&resolution=0&chgdays=&validation=0&votes=&voterating=0&workarounds=False&attachments=False&sortBy=Rating&order=DESC