Testing the build

  • Article

This is a topic I haven’t frequently read about, so I will describe my personal findings and techniques.

Testing

My technique to testing the build itself is based on those assumptions:

  • you need to test incremental change to your build, even when the project is in an advanced phase — that is when you think your script are done;
  • you need to test true code, not just some empty console application — build bugs lurks in the truly complex software your team is writing;
  • you need to keep disruption to a minimum — your team should be able to building with previous version of your scripts.

Before we start I have to remind you of my Where the build script reside post: I have a strong opinion that build scripts have to be inside the codeline and my approach here reflects this.

Given these premises, I try to follow this process:

First, I create a branch — BuildTest — of a successful build: this branch is periodically forward integrated (i.e. merged from the main trunk) or even re-created; this latter is often simpler than fighting through the merge tool.

Second, I create a ‘clone’ build definition that point to the same script folder but below the previously created branch. That definition has the same name as the “true” build prefixed by BuildTest, so if I have a Daily build, I create a BuildTest.Daily; if Daily points to $/TP/Main/Build-Scripts/Daily, BuildTest.Daily will refer to $/TP/BuildTest/Build-Scripts/Daily.

Third, the branched build script should have some conditional code to keep the environment clean and a slightly different definition:

  • it does not create workitems (i.e. bug) for a broken build (SkipWorkItemCreation=true)
  • don’t consume official build identifiers
  • keep retention to a minimum (if you set no retention, you’ll not have a log to study
  • use a different drop folder, so outputs are restricted to the build master and invisible to the team

Note that, except for build identifiers, those settings may be given on the command line or through the RSP file.

Fourth, filter out the build executions so they shows up as less as possible — hide them from build status and so on.

Troubleshooting

Let’s face it, troubleshooting a TeamBuild is hard: you need to wade through thousands of log messages. A real needle in the haystack. It cannot be avoided.

What you can do is to reduce at a minimum what you build in order to shorten your test cycle; for a nice overview see the Limit What you Build post. At least use the IncrementalGet=true and IncrementalBuild=true and leave the other branches out of the build’s workspace.

image

Launching a build from the command line with TfsBuild, makes easier to pass all those parameters.

As there is no debug for TeamBuild, you most important debug tools are the trace you add to your targets and custom task: use the Message task, and … yes, printf will never die.

Along the line of reducing what is build, comment out unneeded solution. Sometime is also possible to use the Desktop build.

Be careful when you merge your changes back to the main trunk to leave out any change made for testing purposes, like:

  • commented solution
  • commented targets
  • changes to RSP file
  • changed property values

And be sure to clean and run a full build after using the incremental ones.

Happy Build!