• Over the last 12 months we’ve done an average of 11,000 builds per month (7,500 in our RTM environment and 3,500 in our dogfood environment).
• A typical official build runs for 15-23 hours and a typical gated check-in runs 1-2 hours.
• A typical official build consumes 350 Gb of source code and produces 450 Gb of outputs (binaries, setups, etc.).
• Our controller with the most agents has 258 agents and is a dual quad core (8 logical processors) with 4 Gb RAM.

Yesterday I was asked if it was possible to control this behavior and while it isn’t by default you can enable this scenario with minor modifications to DefaultTemplate.xaml (which is in $/TeamProject/BuildProcessTemplates). By making these changes we modify the behavior of the “Update Associated Changesets and Work Items” parameter to just enable/disable the updating of work items so that the changesets and work items are always associated with the build.

1. Move the "Associate Changesets and Work Items for non-Shelveset Builds" activity outside of the "If AssociateChangesetsAndWorkItems" activity (either before or after it).
2. Delete the now empty "If AssociateChangesetsAndWorkItems" activity.
3. Select the "AssociateChangesetsAndWorkItems" activity inside the "Associate Changesets and Work Items for non-Shelveset Builds" activity and in the properties change the "UpdateWorkItems" property from "True" to "AssociateChangesetsAndWorkItems" (without the quotes).

The end result should look like this:

There’s a number of solutions to this problem but there’s a very simple one that I’ve used numerous times and have recommended to several people so I wanted to blog it so I can just point people at this post. It won’t work in all scenarios but if it does work in your scenario it’s undoubtedly the easiest solution while still offering some flexibility.

Team Build will create a directory for each platform and configuration pair so we can leverage this to split our outputs by defining solution-level configurations for each split of the outputs we want. Since most people find Configuration useful (to allow switching between Debug and Release) we’ll use Platform to define the different sets of outputs we want.

In the following example I’ll be building a solution containing four projects, “Client”, “Common”, “WebSite”, and “WindowsService”. We’re going to create three folders in the drop, “Client” (containing the outputs from the “Client” and “Common” projects), “WebSite” (containing the outputs from the “WebSite” and “Common” projects), and “WindowsService” (containing the outputs from the “WindowsService” and “Common” projects). I usually also create a platform called “All” containing all projects which is the one developers would select in Visual Studio when working with the solution.

The first thing we need to do is define each of the “platforms” we want to be able to build. To do this:

1. Open the solution in Visual Studio.
2. From the Configuration dropdown on the toolbar select Configuration Manager. You can also access this by selecting Configuration Manager from the Build Menu.
3. For each “platform” you want to create in the drop (in our example, “Client”, “WindowsService”, and “WebSite”):
   1. From the Active Solution Platform dropdown select New.
   2. Enter the name that you want the directory in the drop to be called, for example, “Client”.
   3. Select <Empty> from the Copy Settings From dropdown to start with a blank list.
   4. Make sure that Create New Project Platforms is not checked. Otherwise, the platform will be added to each of your projects. We want to create solution-level platforms and map these to existing project-level platforms.
   5. Click OK.
   6. For each project whose outputs you want dropped in this folder tick the corresponding checkbox in the Build column. You must tick any dependencies as well, these won’t automatically be built.
   7. Repeat step 3.6 for each configuration in the Active Solution Configuration dropdown (e.g. “Debug” and “Release”). This ensures that the correct projects will be built regardless of the Configuration selected.
4. You may also want to remove the built-in platforms (such as Mixed Platforms, Any CPU, x86). To do this select Edit from the Active Solution Platform dropdown, select the platform, and click Remove.
5. Save the solution and check it into version control.

These screenshots show the Debug and Release configurations for the WindowsService “platform”:

Finally, we need to configure the build definition to build each of our new platforms:

1. Edit the build definition in Visual Studio.
2. Click the Process tab.
3. Select the Items To Build process parameter and click the ellipsis button on the right-hand side.
4. Click the Configurations tab.
5. For each “platform” you want to create in the drop:

1. Click Add.
2. Select or enter the configuration you want to build in the Configuration column.
3. Enter the name of the “platform” you want to build (e.g. “Client”) in the Platform column.

• Click Close.
• Save the build definition.

This screenshot shows the Items To Build dialog configured to build the “Client”, “WebSite”, and “WindowsService” “platforms” using Debug and Release configurations:

These screenshots show the resulting drop structure for the “Client” “platform”:

Introduction

While Team Build will serialize workflow variables into an AgentScope, if they are updated within the AgentScope the updated values will not be serialized back when the AgentScope completes. To achieve this you need to attach the data you want to return to the IBuildDetail instance using information nodes.

There are three steps to this:

1. Create an information node class that will store the data we want to return to the controller from the agent.
2. In the AgentScope attach the information node to the IBuildDetail instance.
3. After the AgentScope read the information node from the IBuildDetail instance.

In my (somewhat contrived) example, I’ll return some information (machine name, processor count, and operating system) from the AgentScope to the controller and write them out as a build message. You can download the custom activities and sample process template for this blog post from: https://cid-d0a4451bfb353deb.skydrive.live.com/self.aspx/.Public/InformationNodes.zip.

Creating an Information Node Class

Information nodes are simple classes that follow a specific pattern allowing them to be stored in the TFS database against the build detail. They have a name (allowing you to retrieve just the information nodes you’re interested in) as well as a series of fields (which are stored as name value pairs).

using System;
using System.Globalization;
using Microsoft.TeamFoundation.Build.Client;
 
namespace InformationNodesActivities
{
    public class AgentInformationNode
    {
        public static readonly string Name = "AgentInformation";
 
        private IBuildInformationNode _Node;
        public AgentInformationNode(IBuildInformationNode node)
        {
            _Node = node;
        }
 
        private const string MachineNameFieldKey = "MachineName";
        public string MachineName
        {
            get
            {
                return _Node.Fields[MachineNameFieldKey];
            }
            set
            {
                _Node.Fields[MachineNameFieldKey] = value;
            }
        }
 
        private const string OSVersionFieldKey = "OSVersion";
        public string OSVersion
        {
            get 
            { 
                return _Node.Fields[OSVersionFieldKey]; 
            }
            set 
            { 
                _Node.Fields[OSVersionFieldKey] = value; 
            }
        }
 
        private const string ProcessorCountFieldKey = "ProcessorCount";
        public int ProcessorCount
        {
            get
            {
                return int.Parse(_Node.Fields[ProcessorCountFieldKey], CultureInfo.InvariantCulture);
            }
            set
            {
                _Node.Fields[ProcessorCountFieldKey] = value.ToString(CultureInfo.InvariantCulture);
            }
        }
 
        public void Save()
        {
            _Node.Save();
        }
    }
}

Storing the Information Node

Now that we have somewhere to store the values we want to pass from the AgentScope back to the controller we need to create an activity that creates one of these nodes, populates it, and then adds it to the build. We use the CreateNode method on IBuildDetail.Information to create an empty node, we then set its name (so we can retrieve it later), and create an instance of our custom information node class passing in the node we want it to store its values in.

using System;
using System.Activities;
using InformationNodesActivities;
using Microsoft.TeamFoundation.Build.Client;
 
namespace InformationNodesActivities
{
    [BuildActivity(HostEnvironmentOption.Agent)]
    public sealed class CreateAgentInformationNode : CodeActivity
    {
        protected override void Execute(CodeActivityContext context)
        {
            var buildDetail = context.GetExtension<IBuildDetail>();
            
            var node = buildDetail.Information.CreateNode();
            node.Type = AgentInformationNode.Name;
 
            var agentInformationNode = new AgentInformationNode(node);
            agentInformationNode.MachineName = Environment.MachineName;
            agentInformationNode.OSVersion = Environment.OSVersion.ToString();
            agentInformationNode.ProcessorCount = Environment.ProcessorCount;
            agentInformationNode.Save();
        }
    }
}

Retrieving the Information Node

We can retrieve information nodes using one of the methods on IBuildDetail.Information, in our case we know the name of the node we want to retrieve, and know that there’s one and only one of them so we will use IBuildDetail.Information.GetNodesByType(string name).First().

Since we created the information node in the AgentScope and want to retrieve it from the controller once we’re back on the controller we’re going to call IBuildDetail.RefreshAllDetails to re-query all of the build’s details (including information nodes) from the database.

In the following build process template snippet we:

1. Store the IBuildDetail instance in a variable.
2. Enter the AgentScope.

1. Call the custom CreateAgentInformationNode activity we created above.

• Exit the AgentScope.
• Refresh the IBuildDetail instance (using the InvokeMethod activity).
• Retrieve the AgentInformationNode from the IBuildDetail instance (using the Assign activity).
• Output values from the AgentInformationNode to the Team Build log (using the WriteBuildMessage activity).

<Sequence>
  <Sequence.Variables>
    <Variable x:TypeArguments="mtbc1:IBuildDetail" Name="BuildDetail" />
    <Variable x:TypeArguments="i:AgentInformationNode" Name="RetrievedAgentInformationNode" />
  </Sequence.Variables>
  <mtbwa1:GetBuildDetail DisplayName="Get Build" Result="[BuildDetail]" />
  <mtbwa1:AgentScope>
    <i:CreateAgentInformationNode DisplayName="Create Node" />
  </mtbwa1:AgentScope>
  <InvokeMethod MethodName="RefreshAllDetails">
    <InvokeMethod.TargetObject>
      <InArgument x:TypeArguments="mtbc1:IBuildDetail">[BuildDetail]</InArgument>
    </InvokeMethod.TargetObject>
  </InvokeMethod>
  <Assign DisplayName="Read Node">
    <Assign.To>
      <OutArgument x:TypeArguments="i:AgentInformationNode">[RetrievedAgentInformationNode]</OutArgument>
    </Assign.To>
    <Assign.Value>
      <InArgument x:TypeArguments="i:AgentInformationNode">
        [New AgentInformationNode(BuildDetail.Information.GetNodesByType(AgentInformationNode.Name).First())]
      </InArgument>
    </Assign.Value>
  </Assign>
  <mtbwa1:WriteBuildMessage 
      DisplayName="Output Node Values" 
      Importance="[Microsoft.TeamFoundation.Build.Client.BuildMessageImportance.High]" 
      Message="[String.Format(&quot;Ran on agent {0} which has {1} processors and runs {2}.&quot;, 
        RetrievedAgentInformationNode.MachineName,
        RetrievedAgentInformationNode.ProcessorCount,
        RetrievedAgentInformationNode.OSVersion)]" />
</Sequence>

https://msdn.microsoft.com/en-us/library/gg265783.aspx

On this page you’ll find both a goal-oriented listing (i.e. Get the values of environment variables) as well as an alphabetical reference by activity name. You can provide feedback through the Community Content section at the bottom of each page or general feedback about the format can be provided on Andy Lewis’ blog.

Build Automation

• Cannot delete build drops if a controller goes down
• Summary View: The first changeset in the list of 'Associated Changesets' show up twice (sometimes)
• Ancient changesets being associated with build...
• Memory leaks and performance drop on TFS 2010 Build Service under heavy load
• Watson: Application buildnotification.exe; Exception system.argumentnullexception
• Watson: Application tfsbuildservicehost.exe from Dev10 RTM; Exception 0xC000001D
• Change the default in "Build Retention"->"What to Delete" from "All" to "All - Test Result"
• Manually queueing a gated check-in build intermittedly disables the dropdown "what do you want to build?"
• Build Definition Cache on the AT can become corrupt when multiple definitions map/cloak the same paths in the workspace templates
• Undo in gated check-in does not remove unshelved adds from the disk leaving a cluttered workspace and allowing breaking check-ins to potentially make it through
• Customer can inadvertently change build quality due to odd focus and scroll wheel interaction

All of our build infrastructure runs Windows Server 2008 R2 (x64).

When a build is queued using Team Build it’s queued against a build controller. Each of our build controllers falls into one of these categories:

• Official – used for running nightly official builds for branches.
• Drop – used for dropping outputs from official builds.
• Unofficial – used for running gated check-in and unofficial nightly builds.
• Staging – used for integrating and testing changes to our build process.
• Development – used for developing and testing changes to our build process.
• Backup – pre-configured controllers that can be swapped in to replace a failed controller.
• Unassigned – contains agents that haven’t yet been assigned to another controller. This allows us to run processes against the agents (such as onboarding, patching, etc.).

Our build controllers run on modest hardware (dual processor, 4 Gb RAM) and we typically find that build controllers are RAM bound not IO or disk bound (because their primary role is co-ordination of activities across the build agents).

Once a build has been queued on a build controller we allocate the resources needed for the build, including:

• Storage Area Network (SAN) – we select a SAN based on available space in a round-robin fashion (to evenly spread load)
• Drop Server – drop servers have a VDisk attached to them that the build’s outputs are dropped to, we use DFS to make all of the drops available from a single location
• Build Machines – one per architecture/flavor

These resources are then prepared for the build, which means:

• Carving a VDisk of the appropriate size and attaching it to the drop server.
• Reimaging each of the build machines.

As mentioned above each build consumes one machine per architecture/flavor (e.g. x86 Retail) as well as a drop server. Build machines vary greatly in hardware spec with some being physical machines (up to about 8 processors with 8 Gb of RAM) and others being virtual machines. We group machines into categories based on their hardware specifications and this is used to allocate the most appropriate machines for each branch and architecture/flavor (some architecture/flavors require much fewer resources than others).

Because of the size of the source code we don’t do a full sync of it for each build (it lives on a separate drive which is not wiped by the reimaging process), instead, we scorch this drive and do an incremental sync. In situations where a build machine with a workspace for our branch isn’t available we’ll actually remap an existing workspace rather than deleting the workspace and doing a full sync.

We currently have 38 controllers, 14 of these are owned by the build lab and are used for producing official nightly builds of Main, PU branches, and a couple of feature branches. Another 23 controllers are used by PU branches and feature branches for producing unofficial builds, including gated check-in queues. The remaining controller is a drop controller which manages a number of drop machines that have SAN disks attached to them where completed builds are dropped.

Each official build typically consumes 4 build machines (for different architecture/flavor pairs) and 1 drop machine, unofficial builds typically consume 1 or 2 build machines (again for different architecture/flavor pairs).

These build resources are consumed by 93 definitions, each branch typically has 2 or 3 build definitions associated with it (English, localized, code coverage, etc.). Since introducing Team Foundation Build we’ve launched nearly 5,000 builds producing over 12 million information nodes and over 42 million fields associated with them. These information nodes contain information such as log entries but we also leverage them to store strongly-typed information against builds which I’ll discuss in more detail in a future post.

The largest Team Foundation Build tables in our database are:

The query I used to produce these statistics is:

SELECT SUM(CASE WHEN BSH.DisplayName LIKE 'DDBLD%' THEN 1 ELSE 0 END) AS 'Official Build Controllers',

SUM(CASE WHEN BSH.DisplayName LIKE 'DDROP%' THEN 1 ELSE 0 END) AS 'Official Drop Controllers',

SUM(CASE WHEN BSH.DisplayName NOT LIKE 'DDBLD%' AND BSH.DisplayName NOT LIKE 'DDROP%' THEN 1 ELSE 0 END) AS 'Unofficial Controllers',

COUNT(*) AS 'Total Controllers'

FROM tbl_BuildController BC (NOLOCK)

INNER JOIN tbl_BuildServiceHost BSH (NOLOCK) ON BC.ServiceHostId = BSH.ServiceHostId

SELECT SUM(CASE WHEN BSH.DisplayName LIKE 'DDBLD%' THEN 1 ELSE 0 END) AS 'Official Build Machines',

SUM(CASE WHEN BSH.DisplayName LIKE 'DDROP%' THEN 1 ELSE 0 END) AS 'Official Drop Machines',

SUM(CASE WHEN BSH.DisplayName NOT LIKE 'DDBLD%' AND BSH.DisplayName NOT LIKE 'DDROP%' THEN 1 ELSE 0 END) AS 'Unofficial Build Machines',

COUNT(*) AS 'Total Build Agents'

FROM tbl_BuildAgent BA (NOLOCK)

INNER JOIN tbl_BuildServiceHost BSH (NOLOCK) ON BA.ServiceHostId = BSH.ServiceHostId

SELECT COUNT(*) AS 'Total Build Definitions'

FROM tbl_BuildDefinition BD (NOLOCK)

SELECT COUNT(*) AS 'Total Builds'

FROM tbl_Build B (NOLOCK)

SELECT COUNT(*) AS 'Total Build Information Nodes'

FROM tbl_BuildInformation BI (NOLOCK)

SELECT COUNT(*) AS 'Total Build Information Node Fields'

FROM tbl_BuildInformationField BIF (NOLOCK)

EXEC sp_spaceused 'tbl_Build'

EXEC sp_spaceused 'tbl_BuildInformation'

EXEC sp_spaceused 'tbl_BuildInformationField'

Firstly, I’ve been working on the second edition of Inside the Microsoft Build Engine: Using MSBuild and Team Foundation Build which has been updated for .NET Framework 4.0 and Visual Studio 2010.

Secondly, I’ve been heads down working with a number of my colleagues in DevDiv Engineering Systems and Visual Studio ALM to implement Team Foundation Build for official nightly builds as well as gated check-ins across Developer Division. Having been involved with Team Foundation Build for quite some time, it’s exciting that we’re in a position that we can build Visual Studio using it. You can read more about this exciting milestone on Brian Harry’s blog.

During this process I learnt a great deal about architecting and implementing large build processes based on Team Foundation Build and over the next few months I’ll be publishing a number of blog posts sharing what I learnt, code snippets, etc.. If you have any specific questions you’d like answered leave a comment on this post and I’ll try to answer it in an upcoming post.

Here is an example of how to use this:

var tpcUrl = new Uri(args[0]);

var teamProjectName = args[1];

var buildDefinitionName = args[2];

var tpc = TfsTeamProjectCollectionFactory.GetTeamProjectCollection(tpcUrl, new UICredentialsProvider());

var buildServer = tpc.GetService<IBuildServer>();

var buildDefinition = buildServer.GetBuildDefinition(teamProjectName, buildDefinitionName);

var processParameters = WorkflowHelpers.DeserializeProcessParameters(buildDefinition.ProcessParameters);

foreach (var parameter in processParameters)

{

Console.WriteLine("Parameter: {0}", parameter.Key);

       Console.WriteLine("\t{0}", parameter.Value);

       Console.WriteLine();

}