\"Are We There Yet?\" Estimation and Tracking on the Road to Software Development and Implementation

 

Jeremy Oosthuizen

May 2007

Summary: Cost estimation and tracking are two very important elements in software development and implementation, but the relationship between them is even more important. (9 printed pages)

Contents

Introduction
A Case in Point
Estimating Considerations
Building the Tool
So, What About That Disaster?
Critical-Thinking Questions
Further Study
Glossary

Introduction

How long do you estimate it is going to take you to read this article? You have to have a fair idea, because you had to make a decision on whether or not to start reading it. If you only have 5 minutes and it will take you 10 minutes to read, why bother starting until you know you have 10 minutes, right?

Chances are (assuming that you have an interest in the topic in the first place) that you had a quick look at the number of pages and the density of the text, and, knowing your normal reading speed, came up with a reasonable estimate. You then compared that with the time that you have available. Of course, no one does this explicitly, but it is basically the process that is followed internally. For reading an article, that seems to be accurate enough, and if you are a bit out—even quite a bit—the consequences are not that grave. And you achieved that with only about three input factors. Is that enough to estimate the cost and duration for a software project? Do not even bother to answer; of course, it is not. So, why do so many people seem to think so?

A Case in Point

A few years ago, I was working for a software company that builds and installs a customer-care and billing product. Martin, the salesperson responsible for Europe, often came to us in Professional Services to chat about which customers were currently interested in our solution. As a junior architect, it was good for me to be involved in these discussions.

One day, on my way to an unrelated meeting, I ran into Martin and someone from City Power and Light (CP&L). He introduced me to Craig, from their Purchasing department. I had assumed that something was going on, and I had guessed right, as he told me that CP&L was interested in implementing our product. This was big! Not only are they amongst the largest utilities worldwide, but, at the time, we also had not sold anything in Europe for quite a while. Nobody else in our space had, either; so, the first one to succeed, especially with a customer like CP&L, would make a big splash. If we were the ones, it would be the final nail in the coffin of a few competitors in the region and establish us as the leader in customer care and billing; if we were not, it might well be the final nail in ours.

"How much will it cost?" Martin asked me, right in front of Craig.

In my heart, I knew that I should not answer. He should not have asked me, either; he was putting me on the spot. I could not think of a non-answer quickly enough.

I turned to Craig: "You have about 12 million customers, and serve only electricity and gas—don't you?"

He confirmed.

I knew that put them in the same league as one of our U.S.-based customers. So, I did a quick mental calculation—factoring in the previous project's duration, our average charge rate over all roles in Europe, and throwing in a couple of hundred-thousand euros for travel.

"I guess around five million," I said.

Did you count the factors that went into it? I counted three: previous experience on a very high level, number of customers, and number of services. Comparing these to the reading time for the article, they would be reading speed, number of pages, and possibly typing density.

Martin frowned, looked at Craig, and then looked back at me. He was obviously currying favor with the customer. "Wow, that sounds high! Don't you think we can get away for less—let's say, around four and a half?"

"I guess we can." I replied with a shrug. "If we work very carefully."

And there it was. He had his number, and, as far as he was concerned, the conversation was over and the question answered. If only...

I knew I had put my foot in my mouth almost as soon as I had said it, but I returned to the office after my meeting to work through my calculations. I did not even really need to. I knew that I had ignored some of the best working knowledge in my possession in favor of appearing smart to the customer, helping Martin, and keeping the customer interested.

Estimating Considerations

At the time, our organization was like most, with few standards for estimation. This sounds like a gross generalization, but it was true for us at the time, and still is for most. That is why there are so many companies that offer estimation tools. My mistake made me realize that it was up to me to establish the standards. However, I knew that I had neither the experience nor the authority to do it by myself. So, I decided to call my friend Katie Jordan. Katie had been around for a while, and I always called her when I needed some advice—or, at least, a sounding board.

The first thing that she told me was to always take the request for an estimate seriously. Simply grabbing a figure from the air could have serious consequences. Estimate too low and it could cost us a lot of money, and could cost me my job. Estimate too high and we will probably lose business.

Katie told me that, even if I wanted to give the customer a quick estimate, it should have included documented factors that are explicitly missing in the estimate. It goes without saying that I should not have blurted out a number. It should have been in writing—describing the risks that are associated with the missing factors and the impact that they might have.

"The landscape of software development and implementation is changing," she told me. "Far more companies are choosing to buy packages over developing from scratch. The ratios of the different phases of development or implementation methodologies are changing. Hardware is becoming faster and cheaper. Programming languages are exploiting that, with functions taking over most of the menial programming tasks previously done and constantly redone by people. Therefore, coding is becoming a smaller percentage of the whole effort. It is also cheaper per time-unit than it was before, due to offshore software shops—like we use. You need to consider these factors when estimating a project."

Katie also told me that the type of methodology that we follow determined much about how I should build the estimate. "I'm sure you will agree that estimating an XP [extreme programming] project will require a completely different set of factors from our more traditional waterfall project, with few iterations. It is also completely different to estimate for a transaction processing system than for a business intelligence application. In the latter case, you would expect change and would not try as much to limit it, as you would for an online transaction-processing ** [OLTP] program."

Estimate Early, Estimate Often

I knew that more factors would be available as the project moved along, improving the potential for a good estimate. What Katie taught me was that it is completely acceptable to give a rough estimate early, and refine it later.

"Early in the project, estimation is more an art than the science it is later. The number of factors included in an estimate is inversely proportional to the risk of it being wrong. No estimate is ever 100 percent accurate, unless the estimate is made after the project is done!

"I'm not suggesting that it is impossible to do a reasonable estimate before the project starts," she continued. "However, keep in mind that effort during the early parts of the project is often driven by what was estimated."

Using the reading-speed analogy again, you won't have much more than the number of pages, the density of the text, and your reading speed at the pre-sales stage. As you progress with your analysis of the requirements, you will start to know a lot more about what you are estimating for: how many function points there will be, how many widgets you will have to develop, and so on.

So, How Long?

I came to realize that the people who demand estimates from us are sometimes like children. They demand, "Are we there yet?" just as children would on a long journey. They want to know right now, and they will not understand that it will take longer if they want to stop for candy, or to first install a DVD player to alleviate boredom.

Katie told me that they will always want a number (like Martin demanded from me) —anything—just to give them an idea of how long it will take or what it will cost. She advised me not to be tempted to give a ballpark figure, as it would inevitably morph into fact as the words exited from my mouth.

"This is especially true if you're later trying to increase the number, after initially guessing too low," she warned. "Never commit to giving an estimate, before you are ready and have considered enough factors to make you comfortable. And document all assumptions!"

Another thing that Katie drilled into my memory was that only those people who are authorized to give estimates to a client should actually give them; this is irrespective of who contributes to the estimation process. Normally, the authorized person would be a project director, project manager, or account executive (or whatever the title du jour is for the salesperson). She said that I should be very happy if my little transgression did not make us run into contractual or relationship issues (or both) with CP&L.

Seriously, Now, How Long?

Katie continued my lesson.

"The first thing you should have done when Martin asked you for that estimate was to check whether there was a process in place for that in our organization."

By that time, I had already found that estimates were quite ad hoc. Tracking was in place (we had to bill our customers), but there was no relationship back to estimates. I had to do some analysis to extract the process from the hive of activities that often take place when someone wants an estimate. After all, a tool (whether bought or built) is not a substitute for having a known and repeatable process. Implementing a tool on top of an ad-hoc process would create bigger problems.

Because extracting process descriptions for our customers is pretty much what we do all of the time, extracting our process was not very difficult. I learned during this process that building a tool for one's own organization is very much like building one for a customer: We do for (or possibly "to") ourselves what we are always doing for our customers! I also found out the truth of the adage "the cobbler's children are always the last ones in the village to get shoes": Software companies are often the ones with the worst software tools for their own organization.

I took Katie's advice to heart, and vowed to myself to continue our regular consultations as I went through this process.

I had already been directed by our management that we would develop the tool in-house. This was against my futile arguments; I am a big fan of buying commercial off-the-shelf (COTS), instead of developing in-house. I told them that we were going to spend a serious amount of time and money to build a tool, which we could probably buy for far less money. But, to no avail; I had to build it.

However, I decided to still look at tools that were available for our type of organization. By then, I had a good idea of what we were looking for; so, I quickly eliminated the weaker candidates. I was basically kicking tires amongst the stronger ones, still hoping to get a "Wow!" tool that I could persuade our company to buy. I figured that, just like with any other tool—such as enterprise resource management (ERP) or professional services automation (PSA)—we would then only need to do some serious evaluation, implement, and start reaping the benefits of having the best practices of many companies at our fingertips.

In reading the available literature on estimation, I found that there are (as always) two sides to the story. Vendors of estimation products all say that their product can give more accurate estimates than anyone else's, and earlier in the process. However, theory held that it is nearly impossible to accurately estimate a project very early in the life cycle, because the information that is available after analysis is just more accurate than what was available before.

After I knew what the better tools did (and unsuccessfully tried to convince management to buy one), I went back to Katie for some more advice.

She looked at the research that I had done, and then told me: "Firstly, be very careful that building this tool does not take over your life. Remember that the purpose of the tool is to build something that supports our process, not to build something you want to cater for each and every eventuality." I guess she knew me all too well. After all, doesn't everyone with a programming background make that mistake?

Building the Tool

I had documented our core process for estimating and had gotten buy-in from everyone that the process would be followed. That made it known and repeatable; so, I just had to build a tool that could support that process.

The next step that I took was to extract the model of how our organization does projects, based upon our knowledge of the process. A basic work-breakdown structure (WBS) was built at the start of every project, anyway; so, I just had to extract a generic project and include some of the more common exceptions.

The constraints that work upon every project are effort, scope, resources (which often includes effort), and budget. Quality is another, referring to the degree of generalization, as opposed to addressing specific requirements. Amongst others, I included the following major factors in the model:

• Team size.
• Project size—Percentage overhead per unit of measure are larger for very small and very large projects.
• Team makeup—Architects, managers, coders, and so on.
• Team distribution—Geographical, which influences travel costs and productive time.
• Tools—Development language and so on.
• Team experience—With the tools.
• Deliverables—Software, documentation, and so forth. These have to be named, specifying the responsible parties.
• Desired margin.
• Contingency—Some amount of time, money, and resources to cover unforeseen problems, delays, and such.

All of these factors were well-understood in our organization, which was essential to make them useful.

Estimates can be of various types, based on: a model, expert opinion, historical, ballpark figures, or formulas. I discovered that a common recommendation for an estimate is to divide the sum of: the longest that it could take + four times what it was expected to take + the shortest that it could take, by six to get the weighted average. However, even in that case, I had to have an idea of what the expected time was. I realized that there is no getting away from the need for a history of projects in our organization; using industry standards would be second prize. However, we had to be able to compare our metrics to industry standards, to see how much we had to improve. This implied that we needed the ability to map our metrics to industry standards.

That covered the generic part of our estimation process; so, it was time to fine-tune. I had to determine a process to consider the specific requirements that the customer wanted satisfied.

Top-Down Meets Bottom-Up

By this time, I had beaten a path to Katie's door, but I still needed a sounding board. Her response to my question about getting specific was surprising.

"Estimation is basically a combination of a top-down and a bottom-up approach."

At first, that was a strange-sounding concept to me, but I realized the truth of it when I thought about it for a while. What she meant by top-down was that I had to start with the methodology, consider the phases, and build the model (including a generic work-breakdown structure) according to that. I then had to assign specific roles—which have different cost levels, after all—to these generic tasks. As soon as the model was built, I would get to the bottom-up part. I had to figure out how long each generic task or building block would take, based on some kind of historical evidence (again, the importance of tracking).

Building blocks can be any measurement. One of the more popular is function points. Another is lines of code (LOC). This has mostly fallen out of favor, because it is very difficult to determine early in the life cycle, it is difficult to define, and it is highly dependent on programming language and even the overall development environment. A third is widgets—the generic name for any piece of development, enhancement, or maintenance task. Because our methodology lends itself to widgets (we implement a product, with plug-in points in specific places), that is what I decided to use. These widgets had specific roles assigned to them. I got those—including estimates on the effort for each—from our development organization.

Katie made it sound simple. "Put the WBS and the building blocks together, do some math, and there is your estimate. Does it sound simple? It is, really, if you keep all your ducks in a row."

"That's easy for you to say!" I retorted.

Even if you assumed that there would be no stops or detours on your road trip, your customer is going to ask for a stop for candy or a detour to see the Grand Canyon. It is very important to document assumptions—as well as the impact on the effort, if they were wrong or are ignored. And did I mention that you should re-estimate often?

Some Coarse Words on Granularity

Although I considered previous metrics in my disastrous estimate, it was on far too high a level of granularity: My count was three factors. So, I was going in the right direction, but I had stopped far too soon.

I knew that the metrics from previous projects could form the basis for our estimates. However, we needed our estimation categories to match our previous-metrics categories, to know exactly how to keep metrics. We built our estimation categories on the same level of granularity at which we kept metrics—rolling them up into groups, where necessary.

Tracking

Estimation and tracking are not "core business" for an architect, as they are for a manager. However, architects have to provide input. This is true both for providing information on what needs to be tracked and executing the tracking mechanism. Katie advised me to become familiar with the Project Management Body of Knowledge (PMBOK) of the Project Management Institute, where I found a lot of information on the principles of tracking.

There is no substitute for experience when improving the accuracy of estimation. However, without properly documented actual numbers to be compared with estimates, that experience means nothing. I implemented that principle to make our task-tracking traceable back to the estimate (sometimes, only the category, on a coarser level of granularity) for it. This feedback loop improved later estimates.

I made it a rule that the category "Other" would be all but banned, because it does not contribute anything to improving accuracy.

Putting It All Together

All that remained for the tool to be complete was to allocate costs per role, factor-in the expenses, and do some math. I ran through a few desk-checks and found the tool to be holding up quite well—considering the different factors. Katie wanted to see my results.

"Remember this when you're trying to make an estimated schedule shorter. There is certainly a downward spiral when you add people: It adds communication errors, complexity, management overhead, and other things. Basically, the addition of more people does not necessarily equal less time, and probably equals more time."

So, What About That Disaster?

When I think back to that number that I thoughtlessly dropped upon Martin's question, I still cringe. When I started doing the calculations I should have done in the first place, I realized how bad the situation was. There was no way that we could do the project for the price that I had tossed out. I was between a rock and a hard place. I had to go to either my manager and announce that I had just blown all our bonuses or to the customer and hope that I did not lose their business or, even worse, their trust. Katie threw the old "honesty is the best policy" at me, and I'm glad that she did. I chose to go to the customer and hoped that they also considered honesty to be the best policy. They did, and I was lucky to survive it.

The most important thing to remember is that an estimate can only be as good as the information that went into it. Less value should be attached to an estimate that is close to a ballpark figure than to an estimate that comes from a combination of thought, standard approaches, and common sense. If you have to change an estimate (and you will) when more factors become available, it will be a lot easier if the lack of available factors had been mentioned earlier.

Critical-Thinking Questions

• Why are you doing an estimate?
• Do you know the relative levels of effort of project phases and tasks in your organization?
• What factors do you have available?
• Do your organization's time- and expense-tracking numbers feed back into your estimation mechanisms?
• After you have built an estimate, can you explain where your numbers come from?

Further Study

• McConnell, Steve. Software Estimation: Demystifying the Black Art. Redmond, WA: Microsoft Press, 2006.
• Project Management Institute (PMI) Web site (search for "estimation" or "tracking")

Glossary

COTS—Commercial off-the-shelf: A product that is purchased, to be installed with no customization beyond configuration.

ERP—Enterprise resource planning: An enterprise-wide system, integrating all (or most) business data in a single application or a set of seamlessly integrated applications.

OLTP—Online transaction processing: An application that is designed for processing transactions, as opposed to analytics.

PSA—Professional services automation: Can be described as ERP for professional organizations, integrating all of the business data for a professional organization.

WBS—Work-breakdown structure: Task-by-task layout of a project.

XP—Extreme programming: Development methodology that emphasizes close cooperation between programmers and all other stakeholders, with very short delivery cycles.

 

About the author

Jeremy Oosthuizen has been working in the IT industry since 1991. He started as a Programmer in South Africa, before settling in the USA. Subsequently he progressed through various designer roles in the utilities-industry to his current role as architect. He has worked in South Africa, North America, South America and Europe.

This article was published in Skyscrapr, an online resource provided by Microsoft. To learn more about architecture and the architectural perspective, please visit skyscrapr.net.