Determining Whether Tests Passed in Team Build

Article
09/21/2006

In a forum post a while back, I laid out a method for determining whether tests had passed during a build. More recently, I have linked to this forum post in advising others on similar problems. Unfortunately, as a sharp user pointed out in this same thread, my solution doesn't actually work, since it relies on a property that is not accessible in Team Build v1!

So - to remedy the situation I have written a custom task which can be used to determine whether tests have succeeded or not. This task takes advantage of the GetTestResultsForBuild method of the Microsoft.TeamFoundation.Build.Proxy.BuildStore class.

As always, this task is provided as a sample only; its awesomeness cannot be guaranteed, etc. The task inherits from the TeamBuildTask class I presented in an earlier post.

 using System;
using System.Web.Services;
using Microsoft.Build.Framework;
using Microsoft.Build.Utilities;
using Microsoft.TeamFoundation.Client;
using Microsoft.TeamFoundation.Build.Common;
using Microsoft.TeamFoundation.Build.Proxy;

namespace MyNamespace
{
    public class CheckForTestSuccess : TeamBuildTask
    {
        /// <summary>
        /// The ConfigurationToBuild Item Group for the Build.
        /// </summary>
        [Required]
        public ITaskItem[] ConfigurationToBuild
        {
            get
            {
                return m_configurationToBuild;
            }
            set
            {
                m_configurationToBuild = value;
            }
        }

        /// <summary>
        /// An output property which will be true if all test runs succeeded and false otherwise.
        /// </summary>
        [Output]
        public bool TestSuccess
        {
            get
            {
                return m_testSuccess;
            }
            set
            {
                m_testSuccess = value;
            }
        }

        /// <summary>
        /// Returns the name of the build step to be added for this task.
        /// </summary>
        /// <returns>Name of the build step to be added.</returns>
        protected override string GetBuildStepName()
        {
            return "CheckForTestSuccess";
        }

        /// <summary>
        /// Returns the message of the build step to be added for this task - this is the
        /// string displayed in the Team Build GUI.
        /// </summary>
        /// <returns>Message of the build step to be added.</returns>
        protected override string GetBuildStepMessage()
        {
            return "Checking for test success.";
        }

        /// <summary>
        /// Put real task logic in this method.
        /// </summary>
        /// <returns>True if task is successful, otherwise false.</returns>
        protected override bool ExecuteInternal()
        {
            string platform;
            string flavor;
            TestResultData[] testResults;

            m_testSuccess = true;

            foreach (ITaskItem configuration in ConfigurationToBuild)
            {
                platform = configuration.GetMetadata("PlatformToBuild");
                flavor   = configuration.GetMetadata("FlavorToBuild");

                testResults = BuildStore.GetTestResultsForBuild(BuildUri, platform, flavor);

                foreach (TestResultData testResult in testResults)
                {
                    if (!testResult.RunPassed)
                    {
                        m_testSuccess = false;
                        break;
                    }
                }
                if (!m_testSuccess)
                {
                    break;
                }
            }

            return true;
        }

        private ITaskItem[] m_configurationToBuild;
        private bool m_testSuccess;
    }
}

To use this task, you'll need to do something like the following in TfsBuild.proj:

 <UsingTask TaskName="CheckForTestSuccess" AssemblyFile="CustomTasks.dll" />

<Target Name="AfterTest">
 <CheckForTestSuccess TeamFoundationServerUrl="$(TeamFoundationServerUrl)" 
       BuildUri="$(BuildUri)" 
     ConfigurationToBuild="@(ConfigurationToBuild)">
      <Output TaskParameter="TestSuccess" PropertyName="TestSuccess" />
 </CheckForTestSuccess>
</Target>

The property TestSuccess would then be 'true' if and only if all test runs succeeded for the build.

Comments

Anonymous
September 22, 2006
Aaron Hallberg has written a couple of posts about Team Build that started with questions from users.&nbsp;...
Anonymous
January 28, 2007
I can't seem to get TFS to run this. I get this error in the build log:Target AfterTest: C:TFSBuildAlchemyCI Test BuildBuildTypeTFSBuild.proj(153,3): error MSB4062: The "CheckForTestSuccess" task could not be loaded from the assembly C:TFSBuildTasksCheckBuildSuccess.dll. Could not load file or assembly 'file:///C:TFSBuildTasksCheckBuildSuccess.dll' or one of its dependencies. The system cannot find the file specified. Confirm that the <UsingTask> declaration is correct, and that the assembly and all its dependencies are available.Done building target "AfterTest" in project "TFSBuild.proj" -- FAILED.I have all the related TFS DLLs in the same folder. I did get a problem with dependencies, which I managed to resolve but may have done incorrectly. Which DLLs are you referencing in your project?
Anonymous
January 31, 2007
Does the service account being used by the Team Build Windows service have access to the c:tfsbuildtasks directory?Buck
Anonymous
April 25, 2007
I'm speaking at TechEd 2007 in Orlando this coming June on the whys and hows of customizing TFS. As part
Anonymous
May 01, 2007
Sharing this information from Jeff Beehler's Blog Aaron’s posts on how to extend team build through custom
Anonymous
July 13, 2007
In a blog post back in September of last year I described how to use a custom task to determine whether
Anonymous
July 13, 2007
In a blog post back in September of last year I described how to use a custom task to determine whether
Anonymous
January 02, 2008
I followed the steps and created a Task in that I am sending email with Test results.But I am not getting any results.here is mycodeprotected override bool ExecuteInternal() { StringBuilder builder = new StringBuilder(); TeamFoundationServer tfs = TeamFoundationServerFactory.GetServer(TeamFoundationServerUrl.ToString()); BuildStore buildStore = (BuildStore)tfs.GetService(typeof(BuildStore)); string buildUri = buildStore.GetBuildUri(m_teamProject, m_buildNumber); BuildData buildData = buildStore.GetBuildDetails(buildUri); string platform; string flavor; TestResultData[] testResults; m_testSuccess = true; //int testsPassed; //int testsFailed; int totalTests=0; foreach (ITaskItem configuration in ConfigurationToBuild) { platform = configuration.GetMetadata("PlatformToBuild"); flavor = configuration.GetMetadata("FlavorToBuild"); builder.Append("No Of Test Cases: " + BuildStore.GetTestResultsForBuild(BuildUri, platform, flavor).Length); builder.Append(Environment.NewLine); testResults = BuildStore.GetTestResultsForBuild(BuildUri, platform, flavor); //TestResultData test = testResults; //testsFailed=test.TestsFailed; //testsPassed = test.TestsPassed; totalTests = testResults.Length; foreach (TestResultData testResult in testResults) { builder.Append("Test Result: " + testResult.TestsPassed); builder.Append(Environment.NewLine); builder.Append("Test Result: " + testResult.TestsFailed); builder.Append(Environment.NewLine); if (!testResult.RunPassed) { FailedTests = FailedTests + 1; //m_testSuccess = false; //break; } else { SuccessTests = SuccessTests + 1; } } //if (!m_testSuccess) //{ // break; //} } MailMessage message = new MailMessage("Someone@example.com", "Someone@example.com"); message.Subject = "CI-Build Result"; //message.IsBodyHtml = true; // Construct the alternate body as HTML. string body = "<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">"; body += "<HTML><HEAD><META http-equiv=Content-Type content="text/html; charset=iso-8859-1">"; body += "</HEAD><BODY><DIV>this is some HTML text"; body += "</DIV></BODY></HTML>"; // Add the alternate body to the message. AlternateView alternate = AlternateView.CreateAlternateViewFromString(body); message.AlternateViews.Add(alternate); //"This is the HTML text for the body of the message and this is red."; string strHeader = "" + buildData.TeamProject + " Build " + buildData.BuildNumber + " " + buildData.BuildStatus + ""; builder.Append(strHeader.ToUpper(CultureInfo.CurrentCulture)); builder.Append(Environment.NewLine); builder.Append(Environment.NewLine); builder.Append("BuildMachine : " + buildData.BuildMachine); builder.Append(Environment.NewLine); builder.Append("BuildNumber : " + buildData.BuildNumber); builder.Append(Environment.NewLine); builder.Append("BuildType : " + buildData.BuildType); builder.Append(Environment.NewLine); builder.Append("DropLocation : " + buildData.DropLocation); builder.Append(Environment.NewLine); builder.Append("RequestedBy : " + UserFullName.GetCurrentUserFullName(buildData.RequestedBy)); builder.Append(Environment.NewLine); builder.Append("StartTime : " + buildData.StartTime); builder.Append(Environment.NewLine); builder.Append("TeamProject : " + buildData.TeamProject); builder.Append(Environment.NewLine); builder.Append("BuildUri : " + buildData.BuildUri); builder.Append(Environment.NewLine); builder.Append("BuildStatus : " + buildData.BuildStatus); builder.Append(Environment.NewLine); builder.Append("BuildQuality : " + buildData.BuildQuality); builder.Append(Environment.NewLine); builder.Append("FinishTime : " + buildData.FinishTime); builder.Append(Environment.NewLine); //if (SuccessTests + FailedTests != 0) //{ builder.Append("TestsPassed : " + SuccessTests); builder.Append(Environment.NewLine); builder.Append("TestsFailed : " + FailedTests); builder.Append(Environment.NewLine); //} builder.Append("Build Log Location : " + buildData.LogLocation); builder.Append(Environment.NewLine); message.Body = builder.ToString(); //builder.Append("No Of Test Passed: " + testResult.TestsPassed); //message.Body = builder.ToString(); //builder.Append("No Of Test Failed: " + testResult.TestsFailed); //message.Body = builder.ToString(); message.IsBodyHtml = true; SmtpClient client = new SmtpClient("MySMTPSERVER"); client.Send(message); return true; }I am getting the mail with 0 Test cases.but in the build log I have 900 tests Passed and 600 Failed.What may be the problem
Anonymous
January 29, 2008
One issue is that you seem to be misinterpreting the TestResultData class - this class encapsulates a test run, not an individual test. See the docs for this class here: http://msdn2.microsoft.com/en-us/library/microsoft.teamfoundation.build.proxy.testresultdata_members(VS.80).aspx. The TestsTotal property should give you the number of tests executed for the test run.It would appear, however, that you are getting 0 TestResultData objects back from the server. Are you using the task as in the example (i.e. passing in @(ConfigurationToBuild) as the ConfigurationToBuild parameter value)? If so, what platforms and flavors are specified in that item group? What target are you overriding (AfterTest, or what)?
Anonymous
May 19, 2008
Hi Aaron,Thank you for the post. This was exactly what I was looking for. One simple question but it appears your ExecuteInternal() method will always return true.I think you meant the return statement to bereturn m_testSuccess;instead of:return true;Thanks again for the help.
Anonymous
May 19, 2008
My apology. There's nothing wrong with your code. My mistake.

Partager via

Determining Whether Tests Passed in Team Build

Comments

Ressources supplémentaires