Project and File References Revisited

Most .NET developers and architects are familiar with various approaches to Visual Studio solution design:

Single Solution

Partitioned Solution

Multiple Solutions 

Multiple Partitioned Solutions 

"What? Haven't heard of "Multiple Partitioned Solutions" before. Did you just make it up?"

Yes, you might be using it already but you didn't know what it's called. That's fine as I have come up with that name and haven't written about it before :) So please read on!

Most of the development projects contain more than one Visual Studio project. As soon as a second project is added to the solution, as a developer/architect, you start thinking about how the dependencies should look like.

Meet "Camp P"
Members of camp P believe file references are evil as they cause all sorts of versioning and build issues when the dependencies go out-of-date. The members of this camp try to put everything in the same solution and life is great as long as you can count the number of projects with your fingers. There has been a debate out there as to whether Visual Studio should become quicker when it comes to handling large solutions. This issue was partly addressed by release of Visual Studio 2005 SP1 where we introduced some performance improvements in the IDE. These improvements are also rolled forward to Visual Studio 2008. However, you need to take into account all the work the IDE and the compiler need to do while you are typing in the IDE to provide you with Intellisense and compiler warnings/errors. So at some point, you will hit a practical limit on the number of projects you can have open in Visual Studio.

Meet "Camp F"
This is where the members of "Camp F" are coming from. They want to avoid the project references in order to speed up the IDE but they usually end up spending the time they saved on resolving the versioning issues.

Now Meet Both!
None of these solutions are perfect but as long as the development team is consistent in following one of these approaches, then the team can live with it. This becomes a major problem when you have members from both camps in the same development team. This has been the case in most of the development teams I have worked with and I bet most of you have had similar experiences. So what is a typical day on such project? Somebody from "Camp F" is not happy with performance of the IDE while he is working on the "Order Processing" feature of the application. As a result, he decides to replace all of the references to the "UI Framework" projects with file references. But guess that? The dependency type is defined in the project file, so this change results in modification of the project file and the changes are checked back into the source control at some point. Few hours later, somebody from "Camp P" who is working on one of those "UI Framework" projects is trying to perform an integration test to validate a change she made to the UI Framework and finds out that the dependencies are out of date. She checks the source control as it was working fine until that morning and notices the change. She then reverts all of those changes as file references will make her job harder. And the team will soon end up in changing the references types back and forth between project and file references and and this changes cause further problems in the build process. Add to this all of the discussions/arguments as to which approach is better and you will see how much time is wasted as developers and build engineers keep changing the dependencies to suit their needs.

What is the solution then? I am going to start with re-iterating what you probably know already but bear with me - I'll try to keep it short!

The Team Development with Visual Studio Team Foundation Server guide does a pretty good job at providing high level guidelines for choosing the right approach. It provides three options for managing the solutions: Single Solution, Partitioned Solution and Multiple Solutions. I am going to skip over the Single Solution. If you can manage everything in the same solution, then you are not worried much about the solution design, right?

Partitioned Solution
If you work on a large system, consider using multiple solutions, each representing a sub-system in your application. These solutions can be used by developers in order to work on smaller parts of the system without having to load all code across all projects. Design your solution structure so any projects that have dependencies are grouped together. This enables you to use project references rather than file references. Also consider creating a master solution file that contains all of the projects. You can use this to build your entire application.

Multiple Solutions
If you work on a very large solution requiring many dozens of projects, you may encounter solution scalability limits. In this scenario, break your application into multiple solutions but do not create a master solution for the entire application because all references inside each solution are project references. References to projects outside of each solution (for example to third-party libraries or projects in another sub-solution) are file references. This means that there can be no “master” solution.
Instead, you must use a script that understands the order in which the solutions need to be built. One of the maintenance tasks associated with a multiple-solution structure is ensuring that developers do not inadvertently create circular references between solutions. This structure requires complex build scripts and explicit mapping of dependency relationships. In this structure it is not possible to build the application in its entirety within Visual Studio. Instead, you can use TFS Team Build or MSBuild directly.

The Partitioned Solution approach doesn't scale well for very large solutions since it will make the IDE very slow and this sends you down the Multiple Solutions approach. So you group related projects and create a solution for each group. You use project references for dependencies inside the solution and file references for those dependencies that cross the solution boundaries. This is where the problems start:

1- You cannot create a master solution that contains all of the projects. This is because you have created file references for cross-solution dependencies, which will cause problems when you try to build the master solution.

2- Depending on which area of the system you are working on, you may want to have a different slice of the system in your solution. For example, if you are a member of a team working on a specific feature such as Order Processing, you want to have a vertical slice that goes across application layers (presentation, business and data). But if you are working on the UI Framework for the application, you want to have a horizontal slice that includes the presentation layer from all functional areas as well as the UI Framework.

So even if all of the team members agree on the concept of Multiple Solutions, there are decision to be made on how the projects are going to be grouped into multiple solutions and as you can see in the above diagram, these categorisations don't necessarily play well with each other.

You always need to consider various factors such as the solution size and simplicity of the build process and choose one of these approaches (none of which are perfect):

• Single Solution
• Partitioned Solution
• Multiple Solutions 

Multiple Partitioned Solutions 
Some of you may know about this already but there is another approach too. One can argue that it is the same as Partitioned Solution but in order to avoid confusion(!), I usually call it "Multiple Partitioned Solutions" or "MPS" (as you know we love TLAs). So what is it and how does it work?

MSBuild is a great build engine. One of its features is the ability to follow and find the project references in a project even if those referenced projects are not inside the solution you are trying to build. This means you no longer need to have all of the project references inside the solution. Believe it or not, but you can successfully build the solution and the projects inside it even if the referenced projects are not in the solution. This will work both from the IDE and the command line as they both rely on the underlying MSBuild functionality. The only condition here is that the referenced projects need to have been built already. MSBuild is able to follow and find those referenced projects but it won't build them for you. Is this a major problem? Probably not because when you get the latest version from the source control, you usually do a full build and then start working on your specific area.

Let's see how the system will be structured then. The dependencies between the projects are created as project references wherever possible. Depending on your requirements, you can create a solution that contains only a subset of projects you need to work on. Because the dependencies are project references, you can now have cross-cutting solutions that go across functional areas and architectural layers. You can also have a master solution because the dependencies are project references. Clearly, you don't want to open the master solution in the IDE often as it will take a long time to load, it will be slow and you don't usually work on all projects at the same time. The master solution will primarily be used for build purposes.

**

Note 1: I haven't tried this but as far as I know, you won't be able to build a solution on the build server if a project reference is missing. So on your build server, make sure you always build the master solution and not the partitioned solutions. You always want to build everything on the build server anyway.

Note 2: I am not saying that you should never use file references. There are cases where you don't want the dependencies to be built in the same build step so you can use file references if needed. The advantage of the Multiple Partitioned Solutions is that it gives you maximum flexibility.

Why do I call it "Multiple Partitioned Solutions"? Because the master solution is partitioned multiple times based on various perspectives. For example, the master solution can be partitioned into three solutions, one for each logical layer (such as presentation, business and data). At the same time, it can also be partitioned into another three solutions, one for each functional area (such as Order Processing, User Management and Call Centre).

Like the other approaches, The MPS approach is not perfect and it has its own challenges. I personally prefer this approach though. It addresses the concerns around solution size and versioning and is very flexible as you can have cross-cutting solutions. But from a tooling perspective, we can provide you with better support so this is an area for improvement. As an example, if you try to remove a project from a solution in Visual Studio, it removes all of the dependencies on that project too. If you are following the MPS approach, this is not necessarily what we want. This is not as bad as it looks as in most projects I have seen, the solutions are created and maintained by dev leads, architects and/or build engineers so as long as they know what they are doing, it will be fine but it would be nice to have a proper solution here.

I have found a workaround for this, which is to unload the projects that depend on the project we are trying to remove before removing the project. This will prevent the dependencies from being dropped automatically. Another approach is to manually edit the solution file and remove the project(s).

As I said, these are workarounds not solutions. When we remove a project, we ideally want to be able to specify whether we want the project to be removed from the dependencies or we are just trying to exclude that project from the solution. So maybe we could add an "Exclude" option to the project context menu in the Solution Explorer? 

So... have you used the MPS approach in your development projects? If yes, can you share your experiences? What are the challenges you have faced and how have you addressed them? What kind of support do you expect from the IDE in this area?

Originally posted my Mehran Nikoo on February 10, 2009 here.