Test Run

The Microsoft UI Automation Library

Dr. James McCaffrey

Code download available at:TestRun2008_02.exe(178 KB)

Contents

Inside StatCalc
The UI Test Automation Code
Extending the Test Harness

There are several techniques you can use to test the user interface of a Windows® application. For example, you can use System.Reflection classes to test Microsoft® .NET Framework applications or you can test both .NET and native applications by calling Win32® API functions such as FindWindow—either with unmanaged C++ or with C# or Visual Basic® by using the P/Invoke mechanism.

In this month's column, I will show you how to get started with UI test automation using the new Microsoft UI Automation library, which is included in the .NET Framework 3.0 as part of Windows Presentation Foundation (WPF). You can think of it as a successor to the Microsoft Active Accessibility (MSAA) library, which was originally designed only for accessibility features but was somewhat coincidentally found to be useful for and adapted to UI automation. On the other hand, the UI Automation library was specifically designed from the beginning for both accessibility and UI test automation tasks. You can use the UI Automation library to test Win32 applications, .NET Windows Forms applications, and WPF applications on host machines running operating systems that support the .NET Framework 3.0 (such as Windows XP, Windows Vista®, Windows Server® 2003, and Windows Server 2008).

Let me cut to the chase and state that I believe the development of the UI Automation library is one of the most important advances in test automation to date, and it will quickly become the most commonly used technique for UI test automation for Windows applications. Compared with alternative approaches to UI automation, the UI Automation library is more powerful, often easier to use, and more consistent. In much the same way that the .NET Framework transformed software application development, I think it's very likely that the UI Automation library will revolutionize UI test automation.

A good way to show you where I'm headed is with a screenshot. Figure 1 shows a simple Windows application I'm going to test. The application is called StatCalc, and it computes an arithmetic, geometric, or harmonic mean of a set of integers. An arithmetic mean is just a simple average. For example, the arithmetic mean of 30 inches and 60 inches is just 45 inches. A geometric mean is used for ratios. For example, the mean of 30:1 and 60:1 is 42.4264:1. A harmonic mean is used for rates. For example, the average of 30 miles per hour and 60 miles per hour over a fixed distance is 40 miles per hour.

Figure 1** Sample App under UI Test Automation **(Click the image for a larger view)

The UI test automation shown in Figure 1 is a console application that launches the application under test, uses the UI Automation library to obtain references to the application and user controls on the app, simulates a user entering 30 and 60, and simulates clicking on the Calculate button control. The test automation then checks the resulting state of the application by examining the result TextBox control for an expected value, and it then prints a pass or fail result. I captured the screen shot just before the test automation closed the application under test.

In the rest of this column, I will briefly describe the StatCalc Windows application I am testing and will explain how to launch the application under test; how to use the UI Automation library to get references to the application and user controls; and how to simulate user actions and check application state. I'll also describe how you can extend and modify the test system presented here to meet your own needs. I think you'll find the ability to use the new UI Automation library a great addition to your testing tool set.

Inside StatCalc

Let's briefly look at the application under test so you'll understand what the goal of the test automation is. You'll also see why it is useful, but not absolutely necessary, to have access to the source code of the application when writing UI test automation.

The StatCalc application is a simple Windows-based form. I used C# to code the application, but the UI Automation library works with Win32 applications and WPF applications, too. For simplicity, I accepted the Visual Studio® default control names of Form1, label1, textBox1, groupBox1, radioButton1, radioButton2, radioButton3, button1, and textBox2. I added top-level menu items, File and Help, using a MenuStrip control (available starting with the .NET Framework 2.0) rather than the older MainMenu control. The functionality of the StatCalc application is contained in the button1_Click method and is shown in Figure 2.

Figure 2 StatCalc Application Code

private void button1_Click(object sender, EventArgs e) {
    string[] sVals = textBox1.Text.Split(' ');
    int[] iVals = new int[sVals.Length];
    for (int i = 0; i < iVals.Length; ++i) iVals[i] = int.Parse(sVals[i]);
    if (radioButton1.Checked) {
        double sum = 0.0;
        foreach(int v in iVals) sum += v;
        double result = (double)(sum / iVals.Length);
        textBox2.Text = result.ToString("F4");
    } else if (radioButton2.Checked) {
        double product = 1.0;
        foreach(int v in iVals) product *= (double) v;
        double result = NthRoot(product, iVals.Length);
        textBox2.Text = result.ToString("F4");
    } else if (radioButton3.Checked) {
        double sum = 0.0;
        foreach(int v in iVals) sum += (1 / (double) v);
        double result = (double)(iVals.Length / sum);
        textBox2.Text = result.ToString("F4");
    }
}

When a user clicks on the Calculate button, control is transferred to the button1_Click method, which fetches the value in textBox1 as a single string. It then uses the String.Split method to parse each value into a string array. Next, the click handler converts the string array into an int array. The button1_Click logic branches based on which radioButton control has been selected, computes the appropriate mean as type double, and then displays the result in the textbox2 control as a string formatted to four decimal places.

When computing a geometric mean, the application calls a local helper method named NthRoot, which is defined as:

private static double NthRoot(double x, int n) { 
  return Math.Exp( Math.Log(x) / (double)n ); 
}

I am not doing any normal error checking here so I can keep the size of my application code small and easy to understand.

As a general rule of thumb, when writing test automation using the UI Automation library, you do not need access to the source code of the system under test. In most cases, you access controls by their captions (such as Calculate) rather than by their internal Name property (such as button1). However, some controls, such as textbox controls, do not have captions.

To access non-captioned controls with UI automation it is necessary to know their implied index order. This is the order in which controls are loaded onto the main Form control (or parent control) when the Form is initialized. The implied index order is not necessarily the same as the order in which you add controls to the Form object at design time, and it is also not the same as the tab index order.

You can most easily determine the implied index order of non-captioned controls by viewing the application source code. For example, if I examine the Form1.Designer.cs file for the StatCalc application and expand the Windows Form Designer generated code region, I see this code:

this.Controls.Add(this.button1);
this.Controls.Add(this.groupBox1);
this.Controls.Add(this.label1);
this.Controls.Add(this.textBox2);
this.Controls.Add(this.textBox1);
this.Controls.Add(this.menuStrip1);

This means that button1 has implied index 0, groupBox1 has index 1, and so on. Notice that the textBox2 control implied index is less than the textBox1 control (even though when I designed the StatCalc application, I placed textBox1 onto the Form object before I placed textBox2 onto the Form). Also, controls radioButton1, radioButton2, and radioButton3 are child controls of the groupBox1 control rather than Form1. As you'll see shortly, the level of depth in the control tree is important when writing test automation using the UI Automation library.

So, if you have access to the source for the application under test, you can easily determine the implied index order for controls. But what if you do not have access to the source code? One way to determine control implied index order is to use the Spy++ tool that ships with Visual Studio. After launching Spy++ you can press Ctrl+F5 and then drag the GUI Finder Tool onto any control/window to get detailed information about the control/window. Figure 3 shows the result of targeting the textBox1 control with Spy++. In the background you can see that textBox1 has handle 00080820. In the foreground the next control after textBox1 has caption menuStrip1, and the control previous to textBox1 does not have a caption—this is the textBox2 control.

Figure 3** Finding Implied Index Order with Spy++ **(Click the image for a larger view)

So even if you do not have source code access for the application under test, you will have the ability to eventually infer the implied index order for all the application's controls by examining each control using Spy++.

Now, manually testing even this tiny StatCalc application through its user interface would be boring, error-prone, time-consuming, and inefficient. You would have to enter some inputs, click the Calculate button control, visually verify the result value, and manually record the pass/fail result into a spreadsheet or database. A much better approach is to use the UI Automation library to simulate a user exercising the application and then determine if the application has responded correctly. By automating tedious test cases, you can free up time for more interesting and useful manual test cases where your experience and intuition play a big role.

The UI Test Automation Code

The overall structure of my test harness is shown in Figure 4. After launching Visual Studio, I create a new Console Application program. I decided to use C#, but you should be able to easily convert my test automation code to Visual Basic or any other .NET-compliant language if you wish. Next I add project references to the UIAutomationClient.dll and UIAutomationTypes.dll libraries. These libraries are part of the .NET Framework 3.0 and are typically located in the %PROGRAMFILES%\Reference Assemblies\Microsoft\Framework\v3.0 directory.

Figure 4 UI Test Automation Code Structure

using System;
using System.Windows.Automation;
using System.Diagnostics;
using System.Threading;
namespace TestScenario {
    class Program {
        static void Main(string[] args) {
            try {
                Console.WriteLine("\nBegin WPF UIAutomation test run\n"); 
                // launch StatCalc application 
                // get refernce to main Form control 
                // get references to user controls 
                // manipulate application 
                // check resulting state and determine pass/fail 
                Console.WriteLine("\nEnd test run\n");
            } catch (Exception ex) {
                Console.WriteLine("Fatal error: " + ex.Message);
            }
        }
    }
}

The UI Automation library architecture uses a client-server point of view and naming conventions. From a UI test automation standpoint this means that the application under test is called the server and the test harness is considered the client—the test harness client requests UI information from the application under test (the server). The UIAutomationClient.dll library is essentially the test automation library used by UI Automation clients. Additionally, the UIAutomationTypes.dll library contains various type definitions used by both UIAutomationClient.dll and other UI Automation server libraries.

In addition to the UIAutomationClient.dll and UIAutomationTypes.dll libraries, you will also see the UIAutomationClientSideProvider.dll and UIAutomationProvider.dll libraries. The UIAutomationClientSideProvider.dll library contains a set of code for working with controls that were not built with support for automation—this could include legacy controls and custom .NET controls. Because my application is using all standard controls (which were built to support UI automation), I don't need this library. The UIAutomationProvider.dll library is a set of interface definitions that can be used by developers who are creating custom UI controls and who want the controls to be accessible by the UI Automation library.

After I have added my Project References, the System.Windows.Automation namespace is now visible to my test harness program, and I can add a using statement to the namespace. For the sake of convenience, I add using statements that point to the System.Diagnostics namespace (so I can easily use the Process class) and to the System.Threading namespace (so I can easily use the Thread.Sleep method). As usual with any test automation, I wrap my harness with a top-level try/catch block to handle any fatal errors. Test automation is inherently tricky, and Exceptions are the rule rather than the exception so to speak.

My test automation code begins by launching the application under test:

Console.WriteLine("\nBegin WPF UIAutomation test run\n");
Console.WriteLine("Launching StatCalc application");
Process p = Process.Start("..\\..\\..\\StatCalc\\bin\\Debug\\StatCalc.exe");

Notice that I capture the process information of the application under test. Next I fetch a reference to the host machine's Desktop window as an AutomationElement:

AutomationElement aeDesktop = AutomationElement.RootElement;

Most objects of interest for test automation using the UI Automation library are of type AutomationElement. Here I am able to get a reference to the top-level visual element (the Desktop) by using the AutomationElement's static RootElement property. When performing UI test automation, it is useful to think of every visual entity (controls, windows, and so on) on the host machine as part of a hierarchical tree structure with the desktop window as the root element. This is a common principle for most kinds of UI automation including automation with the UI Automation library.

Next I need to get a reference to the main Form object, which is essentially the StatCalc application. The following illustrates a simple way to do this:

// naive approach 
Thread.Sleep(5000);
AutomationElement aeForm = AutomationElement.FromHandle(p.MainWindowHandle);
if (aeForm == null) Console.WriteLine("Could not find the StatCalc Form");

I pause for a few seconds in order to give my application under test time to launch, and then I use the FromHandle method to get a reference to the application. Recall that I saved the process information of the application under test when I launched the application.

Although this technique works, there are two minor problems with the approach. First, there is no good way to guess how long to wait before attempting to access the Form object. So you must be conservative and put in a very long delay time, which slows your test automation significantly, especially if you are running hundreds or thousands of test cases. Second, this approach assumes that the test harness launches the application under test (allowing you to save the process information about the application). But, in some testing situations, you want the application to already be running, so you will not have immediate access to the application under test process information (although you can programmatically get this information).

In most situations, a better approach to getting a reference to Form object of the application under test as an AutomationElement is code like that shown in Figure 5. Instead of pausing an arbitrary amount of time before blindly requesting a reference to the application, I put an attempt to get the reference inside a loop with a slight delay (100 milliseconds in this case). I track the number of times my harness enters the waiting loop, and I exit if I get a non-null reference (meaning I found the Form object) or if I exceed a maximum number of attempts (50 in this case).

Figure 5 Getting the Test App AutomationElement

// a better approach 
AutomationElement aeForm = null;
int numWaits = 0;
do {
    Console.WriteLine("Looking for StatCalc . . . ");
    aeForm = aeDesktop.FindFirst(TreeScope.Children, 
      new PropertyCondition(AutomationElement.NameProperty, "StatCalc"));
    ++numWaits;
    Thread.Sleep(100);
} while (aeForm == null && numWaits < 50);
if (aeForm == null) throw new Exception("Failed to find StatCalc");
else Console.WriteLine("Found it!");

Additionally, instead of using FromHandle, I use the FindFirst method, which proceeds to search for the first available reference to an AutomationElement that meets the specified condition passed in as a pair of arguments. My first argument, TreeScope.Children, instructs FindFirst to look only at immediate child controls of its context (aeDesktop in this case) rather than look at all descendants of its context.

The second argument to FindFirst is a condition:

new PropertyCondition(AutomationElement.NameProperty, "StatCalc")

You can interpret this code to mean, "The automation element with caption (Name property) equal to StatCalc." Instead of using a traditional string approach to conditions to identify controls, such as Property == StatCalc, the designers of the UI Automation library decided to create a PropertyCondition class. This approach may seem a bit awkward and wordy at first, but it becomes familiar quite quickly.

Basically, the use of a fully object-oriented PropertyCondition class to identify controls, instead of using simple strings, does require more code when you write test automation, but it also has several advantages. A PropertyCondition tends to makes the intent of your test automation code very clear (at least after you've become familiar with the paradigm). Because a PropertyCondition object is more strongly typed than a string, using a PropertyCondition allows an integrated development environment such as Visual Studio to provide design-time IntelliSense® auto-completion help. It also enables the compiler to perform better error checking.

After getting a reference to the main Form object of the application under test, I fetch references to all the user controls. For example, to get a reference to the Calculate button control, I can write this:

Console.WriteLine("Finding all user controls"); 
AutomationElement aeButton = aeForm.FindFirst(TreeScope.Children,
   new PropertyCondition(AutomationElement.NameProperty, "Calculate"));

The pattern to get a user control is exactly the same as that for getting the Form control. Consistency like this is an advantage of the UI Automation library compared to many other automation libraries. As is the case with most types of coding, with UI automation there are usually several different ways to accomplish a task. For example, instead of the previous code to get a reference to the Calculate button control, I can also write this:

AutomationElement aeButton = aeForm.FindFirst(TreeScope.Children, 
  new PropertyCondition(AutomationElement.ControlTypeProperty, ControlType.Button));

Instead of using the NameProperty, I can use the ControlTypeProperty with a value of ControlType.Button. You can interpret this to mean, "Fetch the first button control found on the Form." This approach works here only because I have only one button control on my Form object.

The ControlType class is a key Microsoft UI Automation idea. The class is very simple and can be used to identify the type of any control. Getting references to my textbox controls, which do not have captions or names, requires a different technique:

AutomationElementCollection aeAllTextBoxes = aeForm.FindAll(TreeScope.Children, 
  new PropertyCondition(AutomationElement.ControlTypeProperty, ControlType.Edit));
AutomationElement aeTextBox1 = aeAllTextBoxes[1];
AutomationElement aeTextBox2 = aeAllTextBoxes[0];

Here I use the FindAll method to retrieve all textbox controls that are direct children of my Form control. Notice I use ControlType.Edit to specify textbox controls rather than something like ControlType.TextBox, as you might have guessed. I store all my textbox controls in an AutomationElementCollection, which I can then access by index.

Recall from the previous section that the textBox2 control has a lower implied index than the textBox1 control, so textBox2 is at location [0] in the collection and textBox1 is at location [1] in the collection. There are several other approaches you can use to get references to non-captioned controls, but the FindAll method is often a good choice.

Here is how I obtain a reference to the radioButton1 control:

AutomationElement aeRadioButton1 = aeForm.FindFirst(TreeScope.Descendants, 
  new PropertyCondition(AutomationElement.NameProperty, "Arithmetic Mean"));

Notice that I use TreeScope.Descendants rather than TreeScope.Children because my current context is the Form object. The radioButton controls are children of the groupBox1 control, which is a child of the Form1 control, so the radioButton controls are descendants of Form1, but not children. Alternatively, I could first obtain a reference to the groupBox1 control (as a child of Form1) and then a reference to radioButton1 (as a child of the groupBox1 control).

After obtaining references to the radioButton2 (Geometric Mean) and radioButton3 (Harmonic Mean) in exactly the same way as with radioButton1, I can begin to exercise my application under test. Here is how I simulate a user typing some input into the textBox1 control:

Console.WriteLine("\nSetting input to '30 60'"); 
ValuePattern vpTextBox1 = (ValuePattern)aeTextBox1.GetCurrentPattern(ValuePattern.Pattern); 
vpTextBox1.SetValue("30 60");

The SetValue method probably does not come as a surprise, but notice that I do not access SetVaue directly through the aeTextBox1 object. Instead I use an intermediary ValuePattern object. The concept of AutomationPattern objects such as ValuePattern is probably the biggest conceptual stumbling block for engineers new to the UI Automation library. You can think of pattern objects as an abstract way to expose a control's functionality that is independent from the control's type or appearance. Put another way, you can use specific AutomationPattern instances such as ValuePattern to enable specific control functionality.

I simplify things further by thinking that a control's ControlType exposes what kind of control the control is, and a control's Pattern exposes what the control can do. It's important to note here that a control can support multiple patterns. Here is how I use an AutomationPattern to select the radioButton2 control:

Console.WriteLine("Selecting 'Geometric Mean' "); 
SelectionItemPattern ipSelectRadioButton2 = (SelectionItemPattern)aeRadioButton2.GetCurrentPattern( SelectionItemPattern.Pattern ); 
ipSelectRadioButton2.Select();

This time I use the SelectionItemPattern to enable a selection. The name of the GetCurrentPattern method often confuses UI Automation library beginners. From a test automation point of view, the method is setting, not getting, a specified AutomationPattern. But from a client-server perspective, the automation client code is fetching a particular property from the application under test server code. The code I use to simulate a click on the Calculate button control should help clarify:

Console.WriteLine("Clicking on Calculate button"); 
InvokePattern ipClickButton1 = (InvokePattern)aeButton.GetCurrentPattern( InvokePattern.Pattern); 
ipClickButton1.Invoke(); Thread.Sleep(1500);

Here, I use the InvokePattern to enable a button click and then execute the click using the Invoke method. Notice I pause 1.5 seconds to give my application time to respond. I could also go into a delay loop, checking periodically to see if the result textBox2 field is empty or not, using the technique I will demonstrate shortly. There are 18 AutomationPattern types, and I have listed them in Figure 6. Understanding AutomationPattern classes is the key to knowing how to exercise your application under test using the Microsoft UI Automation library.

Figure 6 Patterns

At this point in my test automation code, I have launched the application under test, entered "30 60" into the input (textBox1) control, selected the Geometric Mean (radioButton2) control, and clicked on the Calculate (button1) control. Now I examine the textBox2 control to see if I have an expected value:

Console.WriteLine("\nChecking textBox2 for '42.4264'"); 
TextPattern tpTextBox2 = (TextPattern)aeTextBox2.GetCurrentPattern(TextPattern.Pattern); 
string result = tpTextBox2.DocumentRange.GetText(-1);

Again I use an AutomationPattern, this time TextPattern, to prepare a call to the GetText method. Notice that I call GetText indirectly through a DocumentRange property, which returns a text range that encloses the main text of a "document"—which is a textbox in this case. The argument to GetText is meant to indicate that I am not placing a maximum limit on the length of the return string.

An alternative way to read the contents of the textBox2 control is to use the GetCurrentPropertyValue method, like this:

string result = (string)aeTextBox2.GetCurrentPropertyValue( ValuePattern.ValueProperty);

The GetCurrentPropertyValue method returns the current property for an AutomationElement control if a property has been set, or returns the value of the default property for the control if the property of the context control has not been set. In this case, I haven't set a property for the textBox2 control, so GetCurrentPropertyValue with an argument of ValueProperty returns the value of the default property for the tetxtBox2 control, which is the Text property. Using GetCurrentProperty saves time when coding but can be risky because current property of a control can change at run time.

With the result of the calculation in hand, I check against an expected value to determine my test scenario pass/fail result:

if (result == "42.4264") { 
  Console.WriteLine("Found it"); 
  Console.WriteLine("\nTest scenario: Pass");
} else { 
  Console.WriteLine("Did not find it"); 
  Console.WriteLine("\nTest scenario: *FAIL*"); 
}

And now I can close the application under test:

Console.WriteLine("\nClosing application in 5 seconds"); 
Thread.Sleep(5000); 
WindowPattern wpCloseForm = (WindowPattern)aeForm.GetCurrentPattern(WindowPattern.Pattern); 
wpCloseForm.Close();

I use the Close method of the WindowPattern class, which is a thematic approach for the UI Automation library.

As an alternative to close the application under test, you can manipulate the File | Exit menu item starting with:

AutomationElement aeMenu = aeForm.FindFirst(TreeScope.Children, 
  new PropertyCondition(AutomationElement.ControlTypeProperty, ControlType.Menu));

However the ControlType.Menu type is the.NET Framework 1.1 MainMenu control and does not work with the newer MenuStrip control. Because my application under test uses MenuStrip, I cannot use the ControlType.Menu type. This is a rare exception; the UI Automation library supports the vast majority of user controls and also provides automation support for custom controls through functionality contained the UIAutomationProvider.dll library. At this point my test automation is complete and I can record my test case result and launch another test scenario.

Extending the Test Harness

The code presented in this month's column gives you a solid foundation for getting started with creating your own test automation using the UI Automation library. As you have seen, the library is very extensive, and its pure object-oriented programming model is likely a bit different than other libraries you have used. However, I'm sure you will quickly learn how to use it to write powerful UI test automation. Three key concepts worth focusing on are the use of the ControlType class to identify the type of a UI control, the AutomationPattern class to identify the functionality of a UI control, and the PropertyCondition class to specify a particular UI control.

There are many, many ways you can extend the ideas in this introductory test automation harness that I've presented. One obvious extension is to parameterize the harness. Instead of hardcoding inputs and expected values, you can pass this information to your test harness on the command line. Better yet, create a test case data store as a text file, database table, or XML file and iterate through each test case, parsing out test case data and passing the data into the test harness. I discuss such design patterns in detail in my August 2005 column at msdn.microsoft.com/msdnmag/issues/05/08/TestRun.

The code featured in this column does not perform result logging—and in most cases you will want to record test case results to a data store instead of merely printing a pass/fail message to the command shell.

I strongly suspect that the UI Automation library will quickly become the most widely used framework for UI test automation. More than anything else, there is now a single, standard, comprehensive, consistent, and well-tested UI automation library that is freely available to software engineers. This is already giving rise to a community with public forums, blogs, sample code, and so on.

This is not to say I think that the UI Automation library will completely replace alternatives such as commercial UI test automation frameworks and custom UI automation frameworks. However, the UI Automation library has significant advantages over these alternatives. Compared to commercial UI test automation frameworks, the UI Automation library is less expensive, and, because it ships with the .NET Framework 3.0, it is a de facto standard.

Compared to custom UI test automation libraries written by one or two software engineers, the UI Automation library contains much more functionality, and you only have to learn one API set instead of a new API for every custom automation library. However, commercial UI test automation frameworks have an advantage in that they are often tightly integrated with tools such as test case managers and bug-reporting software tools. And lightweight custom UI test automation libraries have an advantage in that they are generally easier to use for quick-and-dirty test automation, and they can be adapted for special testing situations. Yet, as the presence of the .NET Framework 3.0 becomes more common through the adoption of Windows Vista, you can expect to see a rapid increase in the use of the UI Automation library.

Send your questions and comments for James to testrun@microsoft.com.

Dr. James McCaffrey works for Volt Information Sciences, Inc., where he manages technical training for software engineers working at Microsoft's Redmond, Washington, campus. He has worked on several Microsoft products including Internet Explorer and MSN Search. James is the author of .NET Test Automation Recipes (Apress, 2006) and can be reached at jmccaffrey@volt.com or v-jammc@microsoft.com.