Software Metrics: Guidelines for establishing effective measurement programs

One of the most important artifacts we produce as testers are measures of the software product or processes. Many people scoff at metrics, but there is little doubt management team wants to see numbers. But, the true value of an effective measurement program is not simply the numbers by themselves, it is the ability of the test team to analyze the data to look for trends and to provide valuable information regarding risk.

In order to effectively analyze data testers should collect metrics with clear goals that directly impact the product or team processes the management team identifies as needing improvement. Once the test team knows the specific area they need to assess, they need to define the appropriate set of measures that will best indicate whether or not the team is tracking towards the goals. The ultimate purpose of any successful metrics program is to produce information that will help the management team make strategic decisions; it is not simply to collect numbers and build fancy charts and graphs to hang on a wall.

Below are some guidelines for effective metrics programs.

· Define the issue – ask the management team what keeps them up at night and why it is a primary area of concern

 

· Establish clear goals – ask the management team how they would define success for each issue defined

 

· Identify the proper metrics – make sure the measurements directly relate to the specific problem or issue being addressed, and the metrics can illustrate immediate benefit to the team

 

· Use more than one measurement – it is not a good idea to make a decision based upon one metric by itself; proper analysis of data requires multiple perspectives (or ways of measuring) the area being evaluated

 

· Select simple metrics – the measurements should be clearly understood by those collecting the data and the management team

 

· Make data collection easy – the best measurements are built into the production process so data collection does not distract from productivity

 

· Create a baseline – set a baseline to track direction towards (or away) from targeted goals

 

· Communicate metric program goals – preferably management should communicate the problem space and discuss what success looks like to get buy-in from the team (Remember…the testing team is collecting and analyzing the data; it’s the management team that will ultimately effect change!)

 

· Periodically review the measures – review the measurements periodically to validate the metrics are evaluating the issue effectively

 

· Avoid changing metrics during a project – changing what is being measured during a project cycle can disrupt the team and denigrate the overall use of metrics

 

· Prevent abusing the metrics – avoid using metrics to analyze things which are tangential to the specific issue the metrics are established to evaluate, and avoid changing or with-holding data to “paint a prettier picture” for management or the team

• Publicize the data and analysis – make the metric data and an analysis of the data publically available to the whole team; transparency of the information is critical to the success of change resulting from a metrics program