Factories 201 – What are the challenges?

  • Article

Previous posts in the Factory 201 series:

Now that we have looked at what a factory is, when it is applicable to build one, 'what', 'why' and 'how' you do that, it's time to step back and look at what the major challenges to building software factories are and why we haven't seen a proliferation of software factories available yet today.

There are certainly challenges building factories today, even though these are not insurmountable and very doable - so, "What are these challenges? "

We are going to break this post into 3 sections, the first discussing the practical challenges to building factories today, the second discussing the organisational challenges present today controlling adoption of building factories, and finish with market place forces in place today.

Practical Challenges

Let's start off with looking at the practical issues with building software factories today.

If you missed it, we already looked at what you need to build a factory in "When would you build one?", then the actual iterative process for building factories in "How would you build it?", so we are not going to talk about that here either, nor are we looking at the technologies you use that you use to do that, we already discussed those in "With what would you build it?".

Here, we are more focused on the broader practical challenges of developing the automated parts of software factories and what issues we face today in doing that both in the platforms and tools available, and the other drivers and affecting wide-spread adoption of factories in the market place.

Toolsets & Platform

The primary challenges we face today with building automation assets and factories are that the tools, we have today, are still at the level that requires us to fully master these toolsets (and all their technical details and intricacies) to wield any of their capability. That's no easy feat for the toolsets we outlined in "With what would you build it?" (such as: GAT, the DSL toolkit).

That may have sounded strange to you, "of course you need to know how to use a tool, to use it – well duh!"

Well the fact is, with the toolsets we are talking about here, and in the context of using them in building software factories, you shouldn't have to know all those fine details about them to leverage their capability in defining a factory. You should be able to design the things for your factory from a higher level logical description, and have someone/something else figure out how to map that and drive a specific technology or toolkit to create the implementation. Sure, if I want to squeeze out every little feature and customization of these tools, then I need to know them in more depth, but not if I just want to use them for basic factory design purposes - as would be the general case.

We want to be able to use the principals of higher level abstraction and guided automation in the very process of building our factory too! (Wouldn't you?)

Well the reality today, is that it's not yet the case, and to wield and orchestrate these toolsets to bring a factory together requires the factory builder to jump over a very high bar to leverage the capabilities provided by these individual tools.

The second challenge is that these tools were not originally designed to work with each other, and are not currently aware of each other, since they were designed and built independently for different goals, before we had a concrete vision of software factories. Which means that even though you can build factories of today happily with them, they are not ideally integrated together yet for software factory design today. This particular issue is evidenced by the fact that some groups have had to create stop-gap solutions and supplementary toolkits to help bridge the gaps between the toolsets to specifically address software factory building scenarios. The experiences are improving though.

Finally, the current toolset is pretty minimal at present, simply because we have not yet had much experience in exploring this space of building software factories and fleshing out each aspect of that. It's pretty green fields out there right now, and we expect there to be many future tooling innovations in the sorts of factories that will emerge over time.

To evolve the capabilities of software factories in the future, we need to identify what these missing tools are, most likely by discovery. 'The Next Missing Technology' explores this requirement a little further.

Factory Authoring Platform

Clearly, if we are having trouble actually practically creating software factories, because it's hard to do, then not many folks are going to, or able to, built them. Only those with the right organisational structure, long term strategy, high domain skill sets, time and money will be able to afford to build them right now. In order for factories to be successful in helping industrialise software development however, building them must be accessible to a wider developer community, so they can be used as parts of ordinary development projects.

[That is not to say that building a factory is a necessary part of any development project!]

This last point is, of course, exactly one of the challenges factories themselves are trying to solve. (i.e. packaging domain expertise of one domain into automation and guidance assets that can be re-used by domain experts of another domain!)

Everyone trying to build factories today will have a different way of doing it, (different skills sets, different approaches, different interpretation of a factory, etc.) and the factories they create will come in many different shapes and forms as a result. This is great for developing new innovations to solving the problems that factories address, but in practice, as we all know, much of the work done by all those building factories today would share similar infrastructure and authoring assets. And that part is being re-done, and re-experienced by everyone over, and over again. (Sound Familiar?)

But that's not really the area we, as factory builders, should be wasting our time on when building individual factories – this should just be out-of-some-box for us to leverage as we see fit. This is another major challenge factories are trying to solve themselves. (i.e. packaging of domain specific automation assets and tools to create a specific solution).

Therefore, it's easy to see that those building factories themselves need to swallow their own medicine – as it were, and adopt the practices that they preach in the very act of design and building of them.

We can see clearly that to increase the proliferation of factories among the software development community there is a dire need for a consistent set of authoring tools to create factories in a consistent way, on a consistent infrastructure. So that those building them can focus on higher level tasks such as logical design, domain modelling, automation asset definition, factory orchestration and solution implementation of their chosen domain.

In short, what we need here is an authoring tool, very much like a factory, to help us build our own software factory – yes, you guessed it - a 'Factory-Factory' as it is informally known. Now, although this is an amusing term, this is precisely what is needed to solve the issues we face today with re-use of factory building tools. A tool, built by experts of building factories, that gives us (factory builders) authoring tools for building our own factories. These authoring tools should be a level above the tools we have today, using abstractions and automation, so that factory builders, like you, can simply 'visualise' a solution domain and attach certain automation assets to various parts of its' logical design, and define artefacts or configuration for creating the physical solution to that design. You, the factory builders, should be then able to package up your custom factory and deploy it to a factory user who can then install it, open it up and start creating a product using the automation assets defined by your factory.

Once we have this platform of runtime and design tools in place, as factory builders, we should be able to focus on the higher level design tasks of our domain, and simply configure the factory design tools to provide the right user experience and assets to provide a solution to our domain.

Authoring Approaches

The final point here is about the approach we take to building factories today. In "How would you build it?" we explored a common-sense, practical and logical process for building a factory from the starting point of product-based assets for a known domain we already know very well. Which requires from us that we know both the problem domain we are solving and the solution domain of the thing we are using to solve the problem. This is a very appropriate process for the types of factories we have now, and what can be achieved with the tools we have today. But you may have noticed that we don't even touch on some of the concepts prescribed by the practical vision of software factories, such as: ViewPoints , Activities and StakeHolders. Realistically, today we can only really achieve logical architectural design, with: Views, Work Products, Artefacts, and Assets.

Why is that? Simply, because to properly implement the other things like ViewPoints, Activities and Roles, as they were intended, concretely in factories of today requires the support of complex runtime tools and infrastructure that is simply out-of-reach for most ordinary factory builders today.

Once that infrastructure is available to us and we can leverage these new capabilities of the factory vision, and we can start to leverage their power over what we have now so we move towards the full intended picture of a software factory with many more benefits.

So how would these additional authoring tools change things from what we have now?

The Current Approach

Well consider that today; you first need to define your solution domain from what you know about in the solutions you already know that address the problem domain. You have to possess some solution assets as the starting point to build upon (an RI or frameworks perhaps). Then, building a factory is simply the process of automating these solution assets and adding abstractions and logical architecture around them to help to factory user configure and manipulate them. Perhaps you also add some runtime components to help maintain the state of the product the factory is building, and finally somehow orchestrate those automation assets together under one roof – your factory.

If you don't have the starting solution assets, then you can't begin to automate those assets because you have nothing to describe in your factory! Sure you can envision a logical architecture of the product you want to build, but you have no way of formalising that or persisting that in your factory as a starting point to build upon. That is exactly why you need starting solution assets - to scope and define what you are doing.

A New Approach

However, what if I gave you a tool that allowed you to define and progressively refine the logical architecture of your product. Then, define its' various concerns (ViewPoints), then let you slowly define the various components of it you need to build (Work Products), what you use to build them (Automation Assets). Then, you can define the steps to build each one, what the outcomes are, and which steps rely on outcome from other steps (Activities). Then, finally how those Work Products are transformed into physical solution pieces (Solution Artefacts). Finally, what about I let you declare which concerns and activities are applicable to which roles in your project team (StakeHolders), and define which Views these particular people want to see when using the factory?

Now this is sounding very much like the factory building design experience we want to envision in the future. Indeed, this tool I described here is what a software factory of the future will look like to a factory builder, and these higher level capabilities, like: ViewPoints, Activities and Roles etc, can only realistically be achieved with an infrastructure and design tools to support them.

Unification

The point is that this tool would open some opportunities for design that are not really available to most today, and further this tool would prescribe a formal definition and process of what a factory is and how to build one. Something we don't really have today.

Today's factory development process describes the very act of rendering a factory from solution-based assets, because that makes sense in this context given the pre-requisites and objectives you would have for building a factory today – it's just guidance, not formal, rigid process. The definition of a factory today is also pretty open and broad, and really a bit broader than just the automation piece, it also includes other forms of guidance including written guidance, patterns, philosophy and samples.

As tools and platforms emerge and evolve the very process to create your factory and the definition of what a factory is will slowly change, and eventually solidify and mature for the high percentage the factory building cases. Further we will have a concrete definition of what are the pieces and how they make up a factory by then.

Until the time we have the authoring parts, the question about how to build, what to build and what is a factory *exactly* will remain debatable and subjective.

Organisational Challenges

The other major area of challenges for factories today, is simply a result of the fact that they are an emerging technology today.

Assuming we step forward a few years, and assuming that we already possess the appropriate authoring tools for creating future factories. The next challenge will be getting adoption of those authoring tools in broader use in software development programs.

[Of course, that's not to say they should be used for general software development. What we mean here, is to get them out and available to a wider developer audience, so that they are used where there is a good case for a software factory.]

A Spread of Adoption

We firmly believe that it will be the skilled professional developers who govern the adoption of the tools initially, but we do hope that it is end-customers and market forces that will drive the long and medium term demand for these tools. Why? Let's look at that quickly:

  • From a customer's/market's perspective, a solution provider that can create high-quality solutions to their specific problems at a fraction of the cost that they can today - is likely always to win the deal.
    • Of course this assumes that the company creating the solutions significantly reduces the sale price of the solution, because it should cost them considerably less to make the solution now!
  • From a solution provider's perspective, a tool that can expedite the creation of customer solutions whilst leveraging their in-house investment in resources and skills - offers a chance to increase profit and margins, and offers competitive advantages over competitors who don't have this capability.

As you can probably easily figure out, there is a dependency building here. As more customers benefit from more cost effective solutions, the market will start to demand that software development costs decrease, since they have witnessed the possibility of that already – that is the promise. This will put pressure on solution providers to make solutions cheaper than they are today. As we have seen, that's just very difficult to impossible today – something has to give, and solution development has to get cheaper. Factories offer a way out here, as you can see in more detail in the Vicious Software Development Cycle where the drivers and dependencies being built are described.

Now, to the practicalities of all that.

Mindset Changes

The major challenge solution providers' will face is a core change in the way (mind-set) they develop software and a move from one-off development approach to product line development approach, for all but the custom 'boutique' software solutions. This affects more than just how they do it. It changes the whole value network inside the organisation, their processes, their skills and their target markets.

Today, too few solution provider businesses are geared to this one-off development. Building re-use into their solutions is often a random afterthought, and even then, rarely receives appropriate investment and correct resources to execute properly. We are not just talking about physical asset reuse, but also domain expertise re-use. Since traditionally, the only place to store domain expertise was in the artefacts and documentation of one-off solutions (and individuals heads) - which then walks out the door with the solutions or individuals.

The game to play here is one of re-using value, and maximise the investment in solution domain expert resources by capturing their knowledge and experience in physical assets, and make those assets re-usable and automatable from a higher semantic level by those who don't possess those solution domain skills.

In short, the re-usable assets become a measure of the value of the organisation.

OK, so that being said, why wouldn't you want to do this?

Well good question!

Ground Up Adoption

We know that professional developers themselves have a very large part to play in the adoption of the tools they use to create solutions, regardless of what the business demands. Fair enough of course, they are the one who have to use those tools!

We explored the issue of software factory adoption from the perspective of several professional developer profiles in "Why do I need Software Factories?" What we found there is that there is a perception that software factories are somewhat of a threat (for several reasons) to some who don't fully buy that software factories can deliver its promise, over what the individuals can. This is largely based upon their own fears that they have of these kinds of automation tools and the impact on their current roles. This kind of fear will initially stifle bottom-up adoption of factories in an organisation.

The basic problem here is in comparing a tool's capabilities to a human being's. In this case, fearing that a factory will replace the role, skills and knowledge of a professional developer. The real issue here is of course recognising that some general tasks a developer performs at a low-level can in fact be captured, automated and largely replaced with higher level tools that do the job quicker and better. Whilst lower level tasks that should not be automated, require a professional's input to suit the specific objective at hand. In short it is about providing the right kind of tool for the job at hand, and directing and applying a professional developer's enthusiasm and design skills to more productive tasks.

Once this is better understood by those who are going to build and use the tools, adoption should be widespread.

Market Demand

The final point here is that today the awareness and markets for software factories are new and still emerging, and with that, the vision and execution of that vision are developing, being guided by that market demand.

The market will certainly shape what factories deliver in the future, and what tools are used to create and operate them. More importantly, the market will certainly apply forces to move to software industrialisation as solutions are proven, by initiatives like software factories, to be more financially viable to construct to satisfy growing customer demands.

It is simply not possible to guess precisely what new markets will demand or exactly how they will develop. So this will need to be discovered and addressed appropriately in a timely manner. This is a challenge in itself.

The good news though is that the whole philosophy and approaches around building software factories is iterative and progressive and should therefore be able to address this kind of change comfortably.

Summary

In this post we have had a look at the practical challenges around building software factories today. The biggest challenge today being the tools we have to get the job done. We hinted at a future where the tools supplemented by a runtime infrastructure step-up a notch and offer us the sort of logical design experience equivalent to what our factories are providing our factory users, but to us as factory builders.

We then had a look at some of the 'molasses' we see today with adoption of factories, and the attitudes many organisations and individuals have to this practice of domain knowledge re-use and automation – but this is transient.

Of course factories are new to the software development market, and although our customers have been wanting software development to advance at more than a snail's pace, to get the software they want quicker, the markets for software factories and customers demanding they be used, is still emerging.