However, it turns out there is a simple work around for this – run MTM as a different user. All versions of windows support this from Windows XP to Windows 7. Vista makes you work a little harder, pushing you to the command line.

Windows XP/2003:

a) Find the shortcut for MTM on the start menu
b) Right click
c) Select ‘run as different user’

Windows 7/2008 R2:

a) Find the short cut for MTM on the start menu, or the task bar
b) shift-right click (e.g. hold down the shift key, and right click), on the shortcut
c) Select ‘run as different user’

Windows Vista/2008:

a) run a command prompt
b) type “runas /u:<domain and user> <path to MTM.exe> (e.g. runas /u:woodgrove\otherUser “c:\program files\microsoft visual studio 10.0\common7\ide\mtm.exe), and hit enter
c) enter the password, and the application will start

You now have two independent versions of MTM, which can connect to different servers.

Note, if you try to do data collection, or record actions at the same time in both, you will hit issues such as incorrect data, and potentially crashes. I don’t recommend this.

IMPORTANT: This is NOT supported. Your mileage may vary.

However, it is possible to create a solution with some coding to merge the TRX files (something we don't have in the product).

Lets work through DmytroL's questions:

1. Is it possible to publish several TRX files under the same test run (of course provided all the TRX files in question specify the same correct test run GUID in the TestRun tag)?

No. You can't. If you try to publish a run that was already published (irrespective of the actual contents), we will block you from publishing -- you'll receive an error that the run has already been published.

2. Is it permitted to overwrite the outcome of a certain test within the same test run?

No. There needs to be one outcome. While (With some simple testing i just did), it is possible to have the same test in the results file, with different executionId's, this is completely unsupported. I would expect things to become pretty sticky on the server if you published a file like this (if you have a test server, you could try... YMMV, everything may break, no I won't help you if it takes down your production server...)

3. Are there any potential problems if these TRX files are published by different people?

See (1) & (2) :)

The scenario behind all this is: a test run is a collection of manual tests put together to verify certain functionality (I know there's a notion of test list for this, but one can't slice the TFS warehouse cube by test lists). Now, subsets of the test run are assigned to different QA engineers for execution. So, each QA engineer goes through her "share" of tests and publishes the results - but the results from all QA Engineers involved should go under the same test run.

Write a tool that loads the results from each run, and then sorts them by time, and pick the post recent result (or whatever criteria you decide on), and output a single TRX, and publish

Also, it's important to note that you can slice the warehouse the way you want to -- when you publish multiple runs against one build (which is allowed, supported and expected), the roll up data goes into the warehouse with the most recent result being the "roll up" result for that test.

Notice how on the item on the right is faded out towards the end of the text box? That was super simple with the following piece of XAML:

<TextBox.OpacityMask>
    <LinearGradientBrush>
        <GradientStop Color="Black" Offset="0.8"/>
        <GradientStop Color="Transparent"
                      Offset="1.0" />
    </LinearGradientBrush>
</TextBox.OpacityMask>

You can apply this to any UIElement derived class, which basically means almost any WPF control. In our application, the text box is styled to be transparent and have no border, so it looks better. But one can see the basic end effect here. On area I'm thinking of applying it to next is items in ScrollViewers -- when the items at the bottom are cut off, it'd be nice to see them with a slight fade rather than just been unceremoniously cut off.

Note, that the XSD schemas are available with all visual studio installs in the:

%VSINSTALLDIR%\xml\Schemas\vstst.xsd file

directory, along with many other schemas.

TRX Files

Lets take a look at TRX files first. I'm not going to go through the whole file line by line -- we put a lot of info into the file, and some of it can be easily guessed. Things in the TRX file other than the obvious "results":

• Runconfig - we just dump in the whole run config that was use as part of that run
• Test List info - The test list info etc that was associated with the tests as they were run. If you just run the test from test view, or use the keyboard shortcuts (Ctrl-R, T), you won't see the test list info.

Neither of these pieces of information are required per-se, we just dump them in there for reference reasons (mostly for re-running without having all the original information).

Interesting lines:

<TestRun id="774eb9dc-b311-4cdd-9987-48dd12f1e5f4" />

The GUID here is just a randomly generated GUID -- Guid.New().ToString() is good enough here, it's just used to allows us to identify the run compared to any other arbitrary run.

<TestRunConfiguration name="Local Test Run" id="3b7e71cb-2506-4724-9b77-b3f5d5c4dd70"/>

This GUID is ID of the Test Run Config that was copied from the one specified at the time the run was created. This should match, but there's no specific need -- we don't go and look up that GUID later.

<UnitTest id="59863143-3238-122e-4d8a-637b491cc755"/>

This is where it gets interesting. This element, which is found under the TestDefinitions element enumerates/declares all the tests that are part of this run. Tests have a name, which is used to provide a friendly way for users to identify test cases from each other. However, the tests themselves need to be uniquely identified within the test system. Most test types (all but unit, for the ones that are shipped in-box) are identified using a random GUID, which is stored in the file the test is stored in. Example: Open up a word based manual test, and look on the properties of it -- you'll see an item which has a GUID in it, and is called "ID". The ID here on these child elements (children of TestDefinitions) is that same GUID as found the test. However, this is not the case for unit tests. Unit tests are generated using a hash fully qualified class name, and the method name [1] (i.e. Company.Product.TestClass.TestMethod).

<Execution id="958d14e2-2f82-4f2c-8e91-84543ca85428" />

This is another "random" GUID, however it is referred to later in the file, specifically as part of the results and tracking which tests were executed, so make sure that if you're generating here, your not just throwing the GUID out of the window after you generate it.

<TestLists>
<TestList name="All Loaded Results" id="19431567-8539-422a-85d7-44ee4e166bda" />
</TestLists>

This is similar to the test run configuration section, where it's a dupe of the actual test list file. This contains all the test lists that are in this run, with the name (as we can see), and the GUID -- again, this really is just a random GUID. But as per <Execution />, it's used later in the file.

<TestEntry testId="59863143-3238-122e-4d8a-637b491cc755" executionId="958d14e2-2f82-4f2c-8e91-84543ca85428" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" />

Test Entry provides a match between test lists, the test case, and the test result. Note that these GUIDs here refer to the IDs previously seen throughout this post. Make sure these GUIDs match up, and your results will all be peachy.

<UnitTestResult executionId="958d14e2-2f82-4f2c-8e91-84543ca85428" testId="59863143-3238-122e-4d8a-637b491cc755" />

These are the results (whose elements are determined by the test type that was executed, hence the 'UnitTestResult'), and as one can see, there are some guids here, which tally back to elements & ID's that were declared earlier in the file -- specifically the execution ID, and the test ID (since this is a unit test, the computed ID).

VSMDI Files

The VSMDI file format is much simpler, although equally filled with GUIDs :). The basic format here is the runconfig (don't ask why it's in there, it's got long historical reasons, and does allow for an... interesting scenario [2]) attached to the of the test lists (which is the same GUID no matter what), and then all the other test lists in the file, with their own GUID, a guid linking to a parent test list, and the contents of that list itself. Lets look at an example.

<TestList name="Lists of Tests" id="8c43106b-9dc1-4907-a29f-aa66a61bf5b6">

This is the "Root" test list, the one you see at the root of test list editor window - an immovable, unchangeable test list. This GUID is constant and the same in all VSMDI files, and all machines. It's been the same GUID since we shipped the first beta of VS 2005 (amusing to think that was over 4 years ago). If you don't have this, then things are going to go a bit pete tong.

Other test lists take the same form, but with there own GUID, which is merely a randomly generated GUID:

<TestList name="SampleList" id="e3d327f8-49dd-4ee0-b8c0-57e9b47d6cff" parentListId="8c43106b-9dc1-4907-a29f-aa66a61bf5b6">

Here, we can see that this node is a child of the parent node (Lists of Tests) by noting the parentListId GUID is the same as the ID guid of the "Lists of Tests" test list. This allows you to establish many levels of nesting of test lists to create the right grouping and categorization you need. However, for tests to appear in these test lists, you need to add child elements of this TestList element:

<TestLink id="59863143-3238-122e-4d8a-637b491cc755" />

Here, the ID refers to the test id of the test being linked in. If you look back through this article you can see the ID of this test showing in the TRX file section -- these IDs really do identify the test across multiple stores (even into the TFS warehouse if you push your results there).

Conclusion

There you have it, a whirl wind tour of the TRX and VSMDI file formats -- which should provide some insight for understanding the rest of our file formats. Note that while some of the information here is still relevant to VS 2005, the file format in that version is radically different, and requires some what unnatural acts to be able to generate it from outside of VS -- I would recommend upgrading to 2008 if you wish to do that :)

[1] We hash the fully qualified test method with code like this:

using System;
using System.Diagnostics;
using System.Security.Cryptography; 

namespace TeamTestSample
{
    internal static class UnitTestIdHash
    {
        private static HashAlgorithm s_provider = new SHA1CryptoServiceProvider(); 

        internal static HashAlgorithm Provider
        {
            get { return s_provider; }
        } 

        /// 

        /// Calculates a hash of the string and copies the first 128 bits of the hash
        /// to a new Guid.
        /// 
        internal static Guid GuidFromString(string data)
        {
            Debug.Assert(!String.IsNullOrEmpty(data);
            byte[] hash = Provider.ComputeHash(System.Text.Encoding.Unicode.GetBytes(data)); 

            // Guid is always 16 bytes
            Debug.Assert(Guid.Empty.ToByteArray().Length == 16, "Expected Guid to be 16 bytes"); 

            byte[] toGuid = new byte[16];
            Array.Copy(hash, toGuid, 16); 

            return new Guid(toGuid);
        }
    }
}

[2] One little know feature of the testing tools inside visual studio, is that you can re-run a test run without sources, just by picking up the test results directory, and opening the TRX file, and clicking the run button. What is even less known is that if you have the Test storages (the dlls, webtest, etc), and the VSMDI, you can open the VSMDI directly (no project, or solution), and select the tests to run. The test run configuration that will be applied will the last test run configuration that was active when the vsmdi was last edited.

Mark Seemann went and posted a short guide on how to enable this in both Visual Studio 2005 (whidbey) and Visual Studio 2008 (Orcas).

Enjoy!

Reflection, when you give it the params array to Invoke, it actually automatically unpacks (although, in reality, I don't think there is any unpacking going on) the out/ref's into the same array, so you can get back at them. This means that since Private[Type|Object] are just simple wrappers around reflection, their Invoke methods do the work for you, and in fact everything works for free.

In the case of accessors, the experience is (now) the same. But it wasn't a complete no-brainer to get it to work. See, when we call the private method from the accessor, we have to unpack all the accessor types into their raw type, to pass to the actual method. This means that when the invoke returns, the 'raw' array has those ref/out values. But before we can return to the test code (the real caller), we have to 'repack' the instances into accessor types using the attach functionality of PrivateObject. It's a bit hairy when it comes to value types rather than ref types, but ultimately it's all good.

When I started down this path, I went along my merry way creating [TestMethod]'s, and running them one-at a time. In conjunction with [TestInitialize], they were all passing wonderfully. "Great!" I thought, "lets run them all together and see lots of green....". Uh oh. All but the first failed. Had I really written that bad a custom control? (Being me, that's perfectly possible).

When I saw the error message, I knew what exactly what was going on:
Test method Foo.RendererIsCleared threw exception:  System.InvalidOperationException: The calling thread cannot access this object because a different thread owns it..

With a call stack that looks like:
System.Windows.Threading.Dispatcher.VerifyAccess()
System.Windows.DependencyObject.GetValue(DependencyProperty dp)
FooBar.get_Thingy()
FooBar.ThingyChanged()
FooBar.Doohick_PropertyChanged(Object sender, PropertyChangedEventArgs e)
System.ComponentModel.PropertyChangedEventHandler.Invoke(Object sender, PropertyChangedEventArgs e)

This is clear -- dependency properties don't support MTA scenarios (duh), which is a bit of a problem when each and every [TestMethod] is executed on a different thread. TestInitialize & first test executed are run on the same thread, and the last thread, and [ClassCleanup] are run on the last thread. This, you can see causes somewhat of a problem. How can we work around this?

My first attempt was 'ah-ha. Dispatcher allows you to invoke methods on the dispatcher thread, and thus I can get them to execute on the same thread they were created on'. Sadly this doesn't work. The thread that the DP's are created on (on the [TestInitialize]'s thread), has actually gone away. So, since you need to invoke on the thread the DP was created on... oops. It's gone away. This is a bit of a problem.

My 'quick fix' solution is to code up all the [TestMethod]s in the test class as normal, but mark them with [Ignore], so they can't be run. Then, create one [TestMethod] that is Called 'AllTests'. This method will then call all the other [TestMethod]s, and ensure they are all called on the same thread. To make this really work you should call initialize & cleanup before and after. I did the following pattern:

public delegate void TestDelegate();

public void DoTest(TestDelegate test)
{
    TestInitialize();
    test();
    TestCleanup();
}

With my AllTests method looking like:

[TestMethod]
public void RunAllWatermarkingTextboxTests()
{
    DoTest(WatermarkProperty);
    DoTest(ResetOnFocusLossPropertySettingGetting);
    DoTest(HasInputtedTextProperty);
    DoTest(PreviousTextPropertyDefaultValue);
    DoTest(LostFocusEventing);
    DoTest(ResetOnFocusLossBehaviour);
}

But this doesn't completely solve it...
You loose out on [ExpectedException], data driven, etc. This is not particularly good, but this is a good start. Additionally, the 'investigation' can be tiresome since you no-longer get the actual test that failed, just the call stack, so you have to do the leg work (Made a little bit easier thanks to the clickable call stack in Orcas).

The solution here is to build a more complex base class to do all the init on a thread that isn't going to die. When I get some time to do that, I'll make sure I post it here. :)

[1] We have a chunk of code that is 'old school' and while we've brought it forward with INotifyPropertyChanged/INotifyCollectionChanged, it's not quite there yet, and we really want to do some rich binding with dependency objects, so I ended up creating a class which aggregates some events into some house keeping code & dependency properties. It's been very powerful so far.

Is there any interest in this? If there is leave a comment on the blog, and if theres enough response we'll sort a time and place to meet up.

We can talk about the product, any problems you might have had, and if we're lucky even demo some of the team foundation features.

Let us know!

Dominic

[X-Posted from the VSTS Quality Tools Blog]

So, what’s been happening with the Testing tools in Team System? Well, I've still got Test View, Manual Testing and Test Project but I now also own Unit Test Debugging, Generic Testing (very useful) and Check-in policy support for Hatteras Source Code Control. But, I've lost Solution Integration - Michael (who owns Code Gen testing) now owns that.

In that time we've released a couple of CTPs that have included our bits - the most recent being The December CTP.

As you'd expect, all of the issues that I talked about in previous blogsicles have been fixed - we're looking in great shape right now. Some of the new functionality we have now in my feature areas are really compelling. Taking debugging as an example - enabling this has made my job of writing tests a lot easier. Just being able to set a break point in the test code and its hit - It's fantastic.

Generic testing enables you to integrate pre-existing test harness & result reporting into our infrastructure. For the full effect, you need to add some code to your test result reporting to output xml (we supply an XSD) with greater detail. Very powerful if you want to take advantage out our integrated solution but don’t want to port all your tests.

Check-in policy is quite possibly my favourite from a what-it-enables aspect - the ability to have a set of tests that must pass before a developer can check-in his code is fantastic. It's such a great way to help ensure that quality of code in the source repository is of a high standard.

I hope to blog more - it's one of my new years resolutions! I look forward to hearing comments from you, and also make sure you check on MSDN Chats for EDT chats.

There is a very cool feature, which isn’t fully surfaced yet in the UI. This allows you to define new properties for your tests, which can be shown in the property window. Under test properties you’ll see “Test Specific” group, which shows your new property.

Right now, the bit that is missing is the ability to group-by & filter on these properties rather than just the core set. I just talked to the Dev who owns Test View (since our PM is at TechEd), and I was remembering right – our current plan is to allow you to filter, group by and add columns for those properties to the test view. Also, the property window is going to allow you to edit these properties and round trip them to code – so even if you don’t know the language your tests are in you can manipulate the custom properties.

Cool, isn’t it? :)