Share via

Using Watson with Dogfood

  • Article

One of the benefits we get from Watson and our internal usage of OneNote is that we get a different set of feedback from internal users when we have a new build of OneNote for everyone to use. It works like this:

  1. We build a new version of OneNote each day and in some cases several times per day.
  2. We evaluate each build to see if it is "good enough for dogfood" - in other words, can folks around Office use this build of OneNote to get their work done? If it has some rough edges, that is OK, but we don't want anything to block people from using OneNote.
  3. If it meets this standard, we roll it out.
  4. Ultimately, the goal is that each build we get each day gets rolled out, by the way.
  5. Once we deploy it, our work on the test team is not done.

Here's where we look at the Watson data for any new spikes or hits. This can be a little tricky since not everyone will update at the same time, some folks will delay updating for weeks and so on. The end result is that we may only get a small number of users on any given build, so we have to take that into account for any Watson reports.

For example, suppose we look at the data for the first 50 people that upgrade. If anyone hits a crash, that represents a 2% failure rate, so we have to take that very seriously and start an investigation. As more people upgrade, we start to get more reports and then we can start prioritizing according to how many people are hitting each error.

Having fewer users makes this much more critical to get right since any one user can represent a large percentage of the reports. But we also have to take each report and examine it thoroughly so that we don't introduce a bug that a whole lot of people will see as well - we can afford to let these initial reports slip by.

It's a challenge we face each day on the test team and I thought it would be interesting for you to think about. There's more I could say here from a statistical viewpoint if you like, Just let me know.

Questions, comments, concerns and criticisms always welcome,
John

Comments

  • Anonymous
    April 03, 2014
    Looks like there is a problem with the page links. I was looking for the tagged tables tip but ended up staring at dogfood! ;-)

  • Anonymous
    April 03, 2014
    Which page has the brken URL?

  • Anonymous
    April 03, 2014
    This page. It is supposed to be for the tagged tables, but instead it talks about Watson. blogs.msdn.com/.../a-tip-for-getting-tagged-tables-working-better-in-onenote.aspx

  • Anonymous
    April 03, 2014
    In other words, I'm trying to find the page where the tagged tables article can be found. It seems to have been converted into the Using Watson with Dogfood article.

  • Anonymous
    April 03, 2014
    Got it.  This is not something I did so I will contact the platform owners here to see what happened.  Thanks for the report!

  • Anonymous
    April 15, 2014
    No idea what happened to the previous article so I republished it here: blogs.msdn.com/.../onenote-tip-getting-todo-checkboxes-to-work-a-little-better-with-tables.aspx