Create Workitems for Test Failures in TeamBuild

  • Article

Currently, TeamBuild does not have out of the box support to create workitems for test failures. I’ve attempted put some details below since this is a common request for several of our customers. It might look complicated if you are going to skim through this blog. But, I promise it is fairly straight forward :) Read on...

 

The targets that are executed in a full stack TeamBuild are defined at %ProgramFiles%\MSBuild\Microsoft\VisualStudio\v8.0\TeamBuild\Microsoft.TeamFoundation.Build.targets. The target RunTestWithConfiguration is responsible for running tests.

 

The definition of this target is as follows:

 

<Target Name="RunTestWithConfiguration" >

 

    <TeamBuildMessage

          Tag="Configuration"

          Condition=" '$(IsDesktopBuild)'!='true' "

          Value="$(Flavor)" />

 

…

…

…

 

    <!-- Test task for the end-to-end build -->

    <TestToolsTask

          Condition=" '$(IsDesktopBuild)'!='true' and '%(MetaDataFile.Identity)'!=''"

          BuildFlavor="$(Flavor)"

          Platform="$(Platform)"

          PublishServer="$(TeamFoundationServerUrl)"

          PublishBuild="$(BuildNumber)"

          SearchPathRoot="$(SearchPathRoot)"

          PathToResultsFilesRoot="$(TestResultsRoot)"

          MetaDataFile="%(MetaDataFile.Identity)"

          RunConfigFile="$(RunConfigFile)"

          TestLists="%(MetaDataFile.TestList)"

          TeamProject="$(TeamProject)"

  ContinueOnError="true" />

 

    <!-- Test task for the desktop build -->

    <TestToolsTask

          Condition=" '$(IsDesktopBuild)'!='false' and '%(MetaDataFile.Identity)'!=''"

          SearchPathRoot="$(SearchPathRoot)"

          PathToResultsFilesRoot="$(TestResultsRoot)"

          MetaDataFile="%(MetaDataFile.Identity)"

          RunConfigFile="$(RunConfigFile)"

          TestLists="%(MetaDataFile.TestList)"

          ContinueOnError="true" />

 

  </Target>

 

If you notice the highlighted part, ContinueOnError is set to true for the task TestToolsTask to ignore the test failures and continue the build process. We need to replace this with a new target and call it where there is an error in the target. MSBuild supports an element called OnError to achieve this behavior. Here is the new code after ContinueOnError is removed and OnError is added:

 

<Target Name="RunTestWithConfiguration" >

 

    <TeamBuildMessage

          Tag="Configuration"

          Condition=" '$(IsDesktopBuild)'!='true' "

          Value="$(Flavor)" />

 

…

…

…

 

    <!-- Test task for the end-to-end build -->

    <TestToolsTask

          Condition=" '$(IsDesktopBuild)'!='true' and '%(MetaDataFile.Identity)'!=''"

          BuildFlavor="$(Flavor)"

          Platform="$(Platform)"

          PublishServer="$(TeamFoundationServerUrl)"

          PublishBuild="$(BuildNumber)"

          SearchPathRoot="$(SearchPathRoot)"

          PathToResultsFilesRoot="$(TestResultsRoot)"

          MetaDataFile="%(MetaDataFile.Identity)"

          RunConfigFile="$(RunConfigFile)"

          TestLists="%(MetaDataFile.TestList)"

          TeamProject="$(TeamProject)"

/>

 

    <!-- Test task for the desktop build -->

    <TestToolsTask

          Condition=" '$(IsDesktopBuild)'!='false' and '%(MetaDataFile.Identity)'!=''"

          SearchPathRoot="$(SearchPathRoot)"

          PathToResultsFilesRoot="$(TestResultsRoot)"

          MetaDataFile="%(MetaDataFile.Identity)"

          RunConfigFile="$(RunConfigFile)"

          TestLists="%(MetaDataFile.TestList)"

          ContinueOnError="true" />

 

  <OnErrorExecuteTargets="OnTestFailure;" />

  </Target>

 

Note that I haven’t removed ContinueOnError for test task that runs for desktop build, because I don’t want to create bugs for test failures that happen in desktop build.

 

Now, we need to define OnTestFailure target. There is a task called CreateNewWorkitem available in TeamBuild to create workitems. I’ve used that here. Here is the target definition:

 

< Target Name="OnTestFailure">

   

    <CreateProperty Value="Work Item created by Team Build on BVT failure. " >

      <Output TaskParameter="Value" PropertyName="DescriptionText" />

    </CreateProperty>

 

    <CreateProperty Value="Build log is at

        &lt;a

        href='file:///$(DropLocation)\$(BuildNumber)\BuildLog.txt'

        &gt;

        $(DropLocation)\$(BuildNumber)\BuildLog.txt

        &lt;/a &gt;.

      ">

      <Output TaskParameter="Value" PropertyName="BuildlogFile" />

    </CreateProperty>

 

    <CreateProperty Value="Test Results are at

        &lt;a

        href='file:///$(DropLocation)\$(BuildNumber)\TestResults'

        &gt;

        $(DropLocation)\$(BuildNumber)\TestResults

        &lt;/a &gt;.

      ">

      <Output TaskParameter="Value"

              PropertyName="TestResultsFolder"

              Condition="Exists('$(DropLocation)\$(BuildNumber)\TestResults')" />

    </CreateProperty>

   

    <CreateNewWorkItem

          BuildId="$(BuildNumber)"

          Description="Tests are failed. $(BuildlogFile) $(TestResultsFolder)"

          TeamProject="$(TeamProject)"

          TeamFoundationServerUrl="$(TeamFoundationServerUrl)"

          Title="BVT failures in build $(BuildNumber)"

          WorkItemFieldValues="$(WorkItemFieldValues)"

          WorkItemType="$(WorkItemType)"

          ContinueOnError="true" />

   

  </Target>

 

I’m creating a new workitem with title as “BVT failures in <build number>” and adding information about the build log file and test results folder in the description. I’m using the same workitem fields specified for build failures. You can find this property in TfsBuild.proj.

 

Now, I need to bring all these things together. To achieve this, I copied the target RunTestWithConfiguration from Microsoft.TeamFoundation.Build.targets and added to TfsBuild.proj, so that this version of the target is called in the build process. Now, I made the changes described above to this newly added target. After this, I added the target OnTestFailure as defined above. This is how it looked with all these changes:

 

<!--

I’ve pulled the target RunTestWithConfiguration (this is responsible for running tests) from

Microsoft.TeamFoundation.Build.targets into TfsBuild.proj and hooked in a new target called

OnTestFailure to create a workitem when there is a test failure. The changes I made are highlighted in bold.

  -->

 

  <Target Name="RunTestWithConfiguration" >

 

    <TeamBuildMessage

          Tag="Configuration"

          Condition=" '$(IsDesktopBuild)'!='true' "

          Value="$(Flavor)" />

 

    <TeamBuildMessage

          Tag="Platform"

          Condition=" '$(IsDesktopBuild)'!='true' "

          Value="$(Platform)" />

 

    <!-- SearchPathRoot for not Any CPU -->

    <CreateProperty

          Condition=" '$(Platform)'!='Any CPU' "

          Value="$(BinariesRoot)\$(Platform)\$(Flavor)\" >

      <Output TaskParameter="Value" PropertyName="SearchPathRoot" />

    </CreateProperty>

 

    <!-- SearchPathRoot for Any CPU -->

    <CreateProperty

          Condition=" '$(Platform)'=='Any CPU' "

          Value="$(BinariesRoot)\$(Flavor)\" >

      <Output TaskParameter="Value" PropertyName="SearchPathRoot" />

    </CreateProperty>

 

    <!-- Test task for the end-to-end build -->

    <!-- New: Remove ContinueOnError=true so that OnError target defined below can be called when there is a failure -->

    <TestToolsTask

          Condition=" '$(IsDesktopBuild)'!='true' and '%(MetaDataFile.Identity)'!=''"

          BuildFlavor="$(Flavor)"

          Platform="$(Platform)"

          PublishServer="$(TeamFoundationServerUrl)"

          PublishBuild="$(BuildNumber)"

          SearchPathRoot="$(SearchPathRoot)"

          PathToResultsFilesRoot="$(TestResultsRoot)"

          MetaDataFile="%(MetaDataFile.Identity)"

          RunConfigFile="$(RunConfigFile)"

          TestLists="%(MetaDataFile.TestList)"

          TeamProject="$(TeamProject)"

          />

 

    <!-- Test task for the desktop build -->

    <TestToolsTask

          Condition=" '$(IsDesktopBuild)'!='false' and '%(MetaDataFile.Identity)'!=''"

          SearchPathRoot="$(SearchPathRoot)"

          PathToResultsFilesRoot="$(TestResultsRoot)"

          MetaDataFile="%(MetaDataFile.Identity)"

          RunConfigFile="$(RunConfigFile)"

          TestLists="%(MetaDataFile.TestList)"

          ContinueOnError="true" />

 

    <!--Execute OnTestFailure target when there is a test failure from end-to-end build -->

   

    <OnError ExecuteTargets="OnTestFailure;" />

  </Target>

 

  <!--

    New: OnTestFauilre target definition

  -->

  <Target Name="OnTestFailure">

   

    <CreateProperty Value="Work Item created by Team Build on BVT failure. " >

      <Output TaskParameter="Value" PropertyName="DescriptionText" />

    </CreateProperty>

 

    <CreateProperty Value="Build log is at

        &lt;a

        href='file:///$(DropLocation)\$(BuildNumber)\BuildLog.txt'

        &gt;

        $(DropLocation)\$(BuildNumber)\BuildLog.txt

        &lt;/a &gt;.

      ">

      <Output TaskParameter="Value" PropertyName="BuildlogFile" />

    </CreateProperty>

 

    <CreateProperty Value="Test Results are at

        &lt;a

        href='file:///$(DropLocation)\$(BuildNumber)\TestResults'

        &gt;

        $(DropLocation)\$(BuildNumber)\TestResults

        &lt;/a &gt;.

      ">

      <Output TaskParameter="Value"

              PropertyName="TestResultsFolder"

              Condition="Exists('$(DropLocation)\$(BuildNumber)\TestResults')" />

    </CreateProperty>

   

    <CreateNewWorkItem

          BuildId="$(BuildNumber)"

          Description="Tests are failed. $(BuildlogFile) $(TestResultsFolder)"

          TeamProject="$(TeamProject)"

          TeamFoundationServerUrl="$(TeamFoundationServerUrl)"

          Title="BVT failure in build $(BuildNumber)"

          WorkItemFieldValues="$(WorkItemFieldValues)"

          WorkItemType="$(WorkItemType)"

          ContinueOnError="true" />

   

  </Target>