I was thinking about metrics, and it occurred to me that most of the metrics we commonly use in the industry are really good as negative indicators of quality, efficiency, testing, etc. but lousy positive indicators.  That is, most software metrics are really good at telling you when something is wrong with your project, but they don’t give you much assurance that the project is actually on the right track.  (I’m sure someone else has already thought of this, but I figured I’d pass on my random though nonetheless.)

Code coverage is a classic example.  Code coverage can be viewed as a measure of what you’re not testing.  I.e. if I have 60% code coverage then I know that 40% of my code is not being tested at all.  However, the 60% code coverage gives me no assurance that the testing of that 60% is actually any good, because there can of course be a vast combination of different code paths through any piece of complex code.

Bug numbers are another example.  A high rate of reported bugs is clearly a sign that something is amiss.  However, a low rate of reported bugs doesn’t necessarily mean you’re on track.  What if your QA team is off writing documents and not testing the product?  In that case a low rate of reported bugs is simply reflecting a lack of testing activity.  Similarly, a very low rate of bug fixing is often (though not always) a sign that something is amiss.  However, a high rate of bug fixing is not comforting – it may simply reflect that devs are rushing work in order to make the #’s look good.

Code complexity is another example.  High code complexity is generally a bad sign for your code’s maintainability (with the exception of some specific code patterns like a parsing or message handling function).  However, low code complexity doesn’t mean that your code is necessarily “good” in any other respect.  It could be utter trash, broken up into small functions.

Conformance of work estimates to actual time spent is another one.  If the actual time spent on a task is way longer (or shorter) than the estimate, that tells you that your estimation process is not very accurate.  However, if someone’s actual time spent on a task always closely jives with their estimates, that’s really not very comforting unless you’re absolutely, positively sure that no other aspect of their work has been compromised as a result (i.e. there’s no “distortion” as Robert D. Austin calls it in his excellent book Measuring and Managing Performance in Organizations, which I highly recommend to anyone interested in metrics).

By these comments I don’t mean to bash these metrics – they are very useful as a way of identifying potential problems and fixing them.  But they have to be viewed as specialized indicators, not numbers to be mindlessly met.  Most software metrics make great gauges but lousy controls.

Over & out!

Chris

We are currently using both Scrum and more traditional project management on several efforts going on within our product unit, and I thought I’d share some learnings.

Something we’re running into is that getting people dedicated to one effort can be hard, depending on the management style of the relevant managers.  For some parts of the team, the relevant leads & managers dedicated people to one effort.  For other parts of the team, the leads & managers insisted on partial assignments, e.g. saying Boffo is 50% on team A and 50% on team B.

This is a basic point of difference between task-based project management (where people are assumed to be fungible resources) and people-based project management (where the team of people is first determined, and their work items second).

As you might guess, so far it is apparent that partial assignment of people does not work very well (both for efforts using Scrum and for traditional projects).  It seems to me that a time split will seldom be 50/50 – instead it will be 80/20 or 90/10, depending on which task is more urgent or more interesting.

On several projects where we aren’t using Scrum but where people are being loaned across teams in the division to do something, we heard again & again that partial assignments are causing problems because the people aren’t fully dedicated (timewise) to the tasks, and it’s very hard to schedule.

Likewise, on the efforts where we have used Scrum, having someone partially available doesn’t jive well – hard to build cohesion and have smooth coordination with people who aren’t always around & may have the tasks from your sprint on the back burner.

Moreover, it’s pretty obvious to most people that if you are working on 2 or more project simultaneously, there is a high price to be paid for context switches.

So, based on what we’ve seen so far, my advice to my readers is that you should avoid assigning people fractionally – instead, pick one task and throw them fully at it.  As has been said for a long time, people are not fungible resources.

Over & out!

Chris

My apologies for being behind on posting.  Things have been really crazy around here with VS2005 shutting down and work on the next major release ramping up.  I’ve been tasked to help out with some engineering improvement initiatives going on in the division and this has soaked up much of my available time.

During the last sprint (sprint 8), we put all the efforts of the MSBuild team on a Scrum team.  This had one downside, which was that the team got too large (10+ people).  I knew this would be an issue, but it turned out to be really big.  The meetings got too big and the goal was too unfocused, and people definitely didn’t like it (the sprint retrospective had a very negative cast to it).

During the current sprint (sprint 9), we finally split into two separate teams, one doing more internal-focused work and the other doing more forward-looking work (although the entire team still comes together for design decisions, etc.).  This has so far worked quite well (or so it seems).  It will be interesting to see what the team thinks at the sprint retrospective.

One of the more interesting phenomena is that during the last sprint a few team members were complaining about how they thought Scrum had too much overhead and they wanted to try something different.  One of the really ironic things about this is that they were using Scrum data to argue this, and they were using the Scrum retrospective as the window into their issues!  Whenever I start to drill into why people think we should use another process, they usually come back with an answer like “we have too many other distractions”, “so-and-so is too hard to work with”, etc. – generally things that have nothing to do with Scrum, but that Scrum has showed them.  Very bizarre.  Usually I’m pretty patient with this, but sometimes I want to pull my hair out and point out that if we weren’t using Scrum, the same issues would exist but there would simply be less visibility into them and less opportunity to fix them.

Another good one is that one or two of the people on the team will occasionally complain that they don’t like having a meeting every day, but these are often the same people who would otherwise complain that they didn’t have enough insight into what the rest of the team is doing.

Anyways, I’ve found that Scrum has been useful for us in “bridging the gap” between the end of one product cycle and the beginning of another.  When other teams were trying to figure out their plans, we simply continued to execute against our product backlog and kept doing sprints, and the team stayed engaged, we have avoided analysis paralysis, and things are rolling pretty well.  Right now I really feel like the avoidance of analysis paralysis is an enormous benefit, but a hidden one – you don’t notice the benefit unless you’ve lived through the problematic situation from past experience and realize the joy in not going through it.

That’s all for now!

Chris

  Several people from MSBuild, the Developer Division, and Microsoft went to Agile 2005 last week in Denver.  It was very worthwhile.  I really enjoyed the keynotes and getting to meet a lot of others at MSFT doing agile methods as well as people from industry & academia.  I can now put a face with a lot of the agile gurus whose books I've read and that I've heard about.  The next Agile conference is July 2006 in Minneapolis.

  I would say that about 50% of the talks I went to were really good, 25% were interesting but not necessarily useful or applicable, and 25% weren't very good.

  For me, the top takeaway was the rise in prominence of automated acceptance testing (e.g. with Fit).  This is something that would be most useful inside MSFT, as we often have complex integration scenarios with other teams, and having a clear set of acceptance tests would do wonders to help clarify up front scenarios that are & aren't expected to work.  After all, if you can't provide a dozen testable examples of things that should work, you don't really understand what you're trying to do.

  Another obvious takeaway was the utility of computing velocity, which I’ve done retroactively for our past sprints.  I’ll probably blog separately about that.

  Another obvious takeaway is the trend toward automating everything that you find useful: dev & QA tests, metrics reporting, project status, etc.  If it’s automated, it will happen; if it’s not, it may not – this is just human nature.

  I’m somewhat annoyed that they have the Agile conference in the summer, as this is when I (and many others) already have travel plans.  Plus, we’re inside all day so it’s not like you really get to enjoy the weather anyway.  I would rather have it in the winter or fall.  But, not a huge deal.

Over & out!

Chris

During this sprint the team has spent a lot of time talking about improving estimates, and tracking the time spent.  Some people suggest that we track the time spent on each item, but of course some (myself included) think that's a bad and non-agile road to go down.  My personal view is that the things that slow us down tend not to be massive underestimation of work items (although we're far from perfect in that regard), but rather getting hit by other things.  We've tried to account for that in sprint 6 using a "risk reserve" as I've alluded to in prior postings.  That has worked fairly well in terms of setting aside sufficient time to deal with randomizations, although it's hard to get people to account for overhead and materialized risks.

Another issue that keeps coming up is that we keep saying we'll deploy our stuff every other week or so, but invariably it gets pushed to the end, which of course increases pressure in the last week.  Hopefully we can find a way out of this in future sprints.

I continue to see positive communication and team cohesion results from Scrum, which alone make me want to continue using it.  In addition I feel it's a great way of organizing work and getting shared focus on a set of goals.

The really interesting stuff will happen with sprint 8 when we try to do "mainline" product work using Scrum...

Over & out!

Chris

  With sprint 5 done, we are now in a quiet period, as we have balanced most of the resources on our project toward helping Whidbey get finished.  We will probably pull some resources back soon, however.

  I have been thinking about how to get the part of our team that isn't using Scrum onto the Scrum model.  It's hard, because this other part of the team is working in a reactive mode, fixing newly found bugs and other last-minute issues that arise.  This kind of thing is hard to model in a sprint backlog in any way other than sets of buffers.  So, I'm not sure what to do.  Maybe we will just wait until there's some "ordinary" work (that is, work that means forward progress on new capabilities) to be done, and include some of these reactive tasks via a buffer.

  Our entire feature team is too big to fit into one Scrum team, but would split up into two nicely-sized ones.  So I'm also thinking about how best to deal with this.  As far as scheduling, maybe we will stagger the teams off by two weeks so we don't have sprint endings and the associated planning meetings happening during the same time periods (yikes!).  Another issue is ensuring cross-team communication; we still want to operate as one feature team even though there is a split (but related) focus.  Maybe we'll have to follow the advice I read somewhere (Schwaber's book?) to have 1-2 pigs always visit the other team meetings as chickens and report back.

  If anyone has suggestions on dealing with two Scrum teams within a single conceptual "feature team", I'd be interested in hearing them.

Signing out!

Chris

For our Sprint Backlog we use an (ever more elaborate) Excel spreadsheet.

Take care,

Chris

A reader named Steve Henke writes with some questions:

First, you mentioned your concern about waterfall being used in the next major division release but with even more centralized control than before. Why do you expect more centralized control? Because of results from your team's Scrum efforts or other external factors?
 
Second, how do code reviews fit into your practices, if at all?

[…]

P.S. Have you blogged before about the most useful books on Scrum?

Let me first address the question about waterfall & centralized processes.  Most MSFT products are developed in a quasi-waterfall model where a bunch of planning is done (driven by PM’s), then a bunch of implementation milestones (driven by devs), and then a bunch of stabilization (driven by QA).  In reality we take a slightly more iterative approach by breaking the product up into milestones (but it’s certainly not “iterative” to the extent people use the term in the Agile community).  Of course, different groups can take a different approach (e.g. some internal MSFT teams exclusively use Scrum, some use XP, etc. – however most use a variation of the quasi-waterfall I described earlier).

I don’t know if there will be more centralized control in the next release, but it’s certainly a risk.  You never really know in a big organization.  However, I try to emphasize to some of the movers-n-shakers the desirability of letting each team make progress in its way (whether they will listen to me or not, I dare not guess).  To be fair, shipping a product that has thousands of people working on it introduces many challenges of scope, so there has to be a balance between centralized assessment of overall progress and local flexibility to make progress in the most locally efficient way.

Regarding code reviews, on my team they are mandatory for all checkins.  However, we do not prescribe the code review process very precisely; it is left up to the code reviewer’s discretion to decide how much time to spend analyzing someone else’s changes before signing off on them (or sending them back for changes).

As for useful books about Scrum, the one I recommend is Agile Project Management with Scrum by Ken Schwaber.

Over & out!

Chris