Design & Manufacturing

Just a recent observation. Agile vs. Formal processes in the context of factories - any confusion there?

It's pretty much a given these days in software engineering we accept that we have been misled over the last couple of decades into believing that software development should follow a manufacturing process (see from your own experience the evidence in past failed development projects). Manufacturing implies 'assembly' of predictable software solutions from well designed plans. (I am sure there are more formal definitions you can insert here). For new software development projects this approach obviously fails in practice because its not so easy to plan what you don't know about what will change in the near term future, and new developments require ample amounts of discovery, learning, design, research and prototyping - all of which you are suppose to squeeze-in for free, without impact, within fixed time and resource budgets of formal processes.

Formal processes work only in practice for a small number of organisations where the outcomes of the project are highly predictable at the start, the inputs tightly controlled and the resources highly tuned and available. You can find these in very few established software businesses where the products they build don't afford large unknown variance in scope/technology etc., and can actually be predicted pretty well.

Because software development businesses need to minimize risk, meet customer expectations and a changing requirements etc., you simply can't get away with saying to your customers  "it will be ready when its ready" - and herein lies the rub, because in order to have any level of predictability, you need solid planning.

Formal processes and practices were based upon the fundamental assumption of manufacturing, that you could predict how long and how much cost, and what resources any development project would take. Which is part of the reason why its taken years (and much failure) to realise the basic assumption was incorrect for the majority of new development projects today. I don't know about you but I felt robbed! What were they smoking?

We, as software engineers, all knew this smelled to be broken somewhere, but no-one really had a practical alternative to push back to meet business goals until we decided to face the music of failure and agile was allowed to emerge - sometimes out of desperation. The beauty of agile of course is that it simply pays respect to the way most of us were actually trying to develop new software within the constraints of formal processes in the first place, and that puts the natural, empirical progress foremost.

We now, as an industry, accept superior agile processes and practices deal far better to the reality of creative new software development projects (although many organisations are sadly still to make this realization). But is agile the right process to use in all cases?

Isn't the predictable assembly of products, like in manufacturing, exactly the mantra of Software Factories? On the outside, there appears to be a disparity here - or is there?

It appears that we are saying "use factories to create your solutions" . Which sounds like factories are implying to take us a step backwards again to formal practices. Is it any wonder the message is being confused?

If you or someone else came to a similar conclusion, you/they are not listening close enough, and must be confusing the case when to use software factories, and when not to use software factories to build software.

You never build a software factory for building a one-off solution from scratch, where you don't already know the solution domain from past experience.

The clarity here is this:

We definitely should be moving software development (as much that is already known) to a manufactured process - which is what factories are all about helping to do, and what the industry needs to move forward. Yes, so, this also means that you should use formal processes, with the factories as part of the solution, to build software in the future! But when we say this we mean that the formal process should be built around the use of factories to assemble the solution.

We are not saying, use formal processes to build your new, one-off solutions. No, for those you should be using agile processes, simply because these solutions are likely to be unique to you and unknown to you at the start (i.e. requires: discovery, learning, design, research prototyping etc).

So, no factories in the solution -> no predictability to the development of the solution -> no formal process -> use agile process.

But, if factories in the solutions -> high predictability in the development of the solution -> use formal process.

This will hold true only for the parts of the whole solution which the factories themselves create. For all other parts of the solution, i.e. the glue, that require discovery, learning, design, research etc, you should employ agile processes to manage that. (Which for most software factory created solutions should be the last 20-40% of the development considering the expectation that factories today can't create the entire solution, and that the solution will always require some level of manual customization).

Now, all this supports the fact that: you should be using agile processes to create your Software Factories themselves. Since you also won't know what they look like at the start and they also require much discovery, design, research and prototyping etc. You can treat a software factory building project as a one-off solution development project in this respect - since it will almost always be a one-off development project.

So, I find it kind of ironic, that whilst the industry, as a whole, is still moving us to agile development, software factories appear to be countering that by moving us back to formal development! Of course, you know better than to confuse the two different types of software development now - right?

In actual fact, if you understand this clearly now you will know that agile development (process) is used to help design and implement new, unknown solutions to meet changing requirements. Whereas, Software Factories (tool) are used to help manufacture existing solutions, each with similar requirements.

Comments

• Anonymous
  September 02, 2007
  PingBack from http://msdnrss.thecoderblogs.com/2007/09/03/design-manufacturing/

• Anonymous
  September 03, 2007
  Spot on as ever Jezz. An aspect of what you write of course is that those involved with the production of factories are domain experts, and creative types. Whereas the users of factories are the factory drones who have to assemble the variant. Much as in the manufacturing industry This leads to the situation where we widen the gap further between the 10x creative devs and the 9-5ers. Personally i think this is a good thing, certainly for project stakeholders. The challenge is ensuring that there are enough folk in the future to move up the ladder without disenfranchising those who just can't. Otoh, we have been in the situation for a very long time where the difference between the top 5% of coders and the rest has not been reflected by reward. Perhaps some of the points you make in this discussion are also indicative of that, and perhaps it will help those at the top to earn what they are truly worth! Tim

• Anonymous
  September 03, 2007
  Thanks Tim, You said it. Just like every other industry that requires design and manufacturing, our software industry will require a broad spectrum of expertise. I think the initial fear that most have about the industrialisation implications of software factories are founded on the perception that they (the people) will be useless if a tool can 'do their job', and render them redundant without a creative purpose. We explored some of these views in an earlier post: http://blogs.msdn.com/jezzsa/archive/2007/01/04/faq-where-are-my-skills-going-to-be-required-with-factories.aspx I would like to challenge them to ask what is their job and what isn't their job. As developers today we tend take on the whole story of a solution as ours to own and control (this is part of the reason why we don't trust other peoples' work), and as part of that we take on every fine grained detail of it. With factories we are simply saying, only take on the interesting parts that require your design skills and push the mundane up a level or abstraction. Let tools do the rest. Software development is not going to get any less exciting as a result, it’s going to get more exciting as you gain higher levels of control of bigger parts of solutions. You are going to wield much more creative power than you are able to in a given timeframe than you are today, in exactly the same way as a C# developer you wield more creative power than an assembly programmer building SOA solutions (for example). Can you really say that power tools make building a house any less exciting? Certainly not if you are paying for the work and materials out of your own pocket! The way I see it now, (and I don't hold all the answers by any means), is that the very best developers (by ability) will become the elite tool makers, the researchers and prototype engineers (as most are today). This will strengthen the upper ranks. The next tier of ability will be using higher level abstractions/tools to create larger more complex solutions - 'the designers' - this will be highly creative. Certainly, the 'coders' (distinctive from ‘professional developers’) of today will have to skill-up or find other vocations or other niche markets, and this is not such a bad thing in the grand scheme of things. cheers