In his application there was a TabItem who's Visibility property was set to Collapsed. Due to this AutomationElement.FromPoint (UIA api) was not able to return correct element(inside the hidden tabitem) from point (and hence codeduitest test was not able to recognize element from point). This looks like some optimization which WPF UIA implementation has when drilling down for element from point.

The workaround is to override the default TabItemAutomationPeer implementation and override few properties so that the drilling works fine. Attached is the custom TabItem class which I wrote to make this possible. You just need to use the TabItemCustom class instead of default TabItem class to achieve this. TabItemCustomPeer is the automationpeer implementation of TabItemCustom. TabItemCustomPeer is the important class which made this possible.

TabItemCustom (2).cs

There was an instance I came across where in the user actions where not getting recorded. It turned out that the user was interacting with a excel sheet. If you notice the below snap shot (from action log config UI), there are some list of process for which recording is disabled.

So if at any point of time you notice that your actions are not getting recorded using MTM you might want to check if that particular process is part of ignored process list.

<Grid.ColumnDefinitions>
            <ColumnDefinition MaxWidth="400" MinWidth="250" Width="250" />
            <!--Column 0-->
            <ColumnDefinition Width="2" />
            <!--Column 1-->
            <ColumnDefinition Width="*" />
            <!--Column 2-->
        </Grid.ColumnDefinitions>
        <Grid.RowDefinitions>
            <RowDefinition Height="67" />
            <!--Row 0-->
            <RowDefinition Height="30" />
            <!--Row 1-->
            <RowDefinition Height="*" />
            <!--Row 2-->
            <RowDefinition Height="57" />
            <!--Row 3-->
        </Grid.RowDefinitions>
        <!-- Resize Border-->
        <Frame x:Name="ResizeBorders" Grid.RowSpan="4" Grid.ColumnSpan="3" Panel.ZIndex="100"/>
        <Button Name="button1" Grid.Column="2" Grid.Row="1"></Button>

A user tries to automate his/her button in the application. If you record action on the button using Codeduitest builder you will notice that it actually records action on a Pane control instead of Button.

Let’s try to figure out what can be the problem here. If you notice the above code Frame has a Zindex = 100 which is greater than any other controls in the application. Internal the UIA implementation does a hit test based on Zindex. Hence if the user tries to get element from point (by clicking or doing Win + I) he/she gets pane (which is a Frame inside the XAML) instead of button. To fix this the user needs to make the Zindex of the button control greater than Zindex of frame.

Below code should work fine

<Grid.ColumnDefinitions>
            <ColumnDefinition MaxWidth="400" MinWidth="250" Width="250" />
            <!--Column 0-->
            <ColumnDefinition Width="2" />
            <!--Column 1-->
            <ColumnDefinition Width="*" />
            <!--Column 2-->
        </Grid.ColumnDefinitions>
        <Grid.RowDefinitions>
            <RowDefinition Height="67" />
            <!--Row 0-->
            <RowDefinition Height="30" />
            <!--Row 1-->
            <RowDefinition Height="*" />
            <!--Row 2-->
            <RowDefinition Height="57" />
            <!--Row 3-->
        </Grid.RowDefinitions>
        <!-- Resize Border-->
        <Frame x:Name="ResizeBorders" Grid.RowSpan="4" Grid.ColumnSpan="3" Panel.ZIndex="100"/>
        <Button Name="button1" Grid.Column="2" Grid.Row="1" Panel.ZIndex="101"></Button>

The NotSupportedException/NotImplementedException here in innocuous and is expected. The reason why we see debugger breaking at this exception is because the debugger option has of “Breaking at any CLR exceptions is switched on”. So any time an exception is thrown the VS Debugger will break at that point even if it is handled else where.

To disable this follow the below steps.

1) Click Debug->Exception menu item.

2) It will open an exception dialog. Uncheck Common language runtime exception check box (under thrown column header). See below

Now you are good to go with debugging your codeduitest.

There are two ways (exclusive) in which search can happen:-

a)    Plugin search :- By plugin search I mean that the respective plugin for that Technology (as specified by the user) searches for the control.

b)    Playback Engine search :- In this case playback navigates through the UI tree of the Application in BFS manner to search for the control.

Today in case of MSAA (Winforms/Win32 controls) and UIA (WPF controls) plugins, playback searches for the control, where as in case of Web (IE controls) the search happens using plugin search.

Let us try to understand a simple search that happens using Playback engine. Consider the following UI tree.

Suppose user specifies following Search Criteria for Control “F”. (I will name the variable same as what is specified in the blocks to make things clear)

If you see in above case A is the container element for C and C is container element for F. Container elements are those element which act as a pivot for for searching another element. In above case playback engine will search C only inside C and will search F only inside C (and not B). As mentioned before playback search happens using BFS traversal of the UI tree. Lets try to mimic what happens when engine tries to search for Control F.

In the above case A is the Toplevel element. Top Level search happens using Window search. Lets assume that playback was able to find A using Window based search. Lets put A in the array. This A will now act as the Pivot.

Now engine gets all the children of A (it queries the plugin for all the child) and puts them in the array. Here is what the updated array looks like

Now engine sees that the C’s search properties matches the Search properties that user has specified in the code. Bingo! element is found!!

So till now engine is able to search for A and then inside A, C. Now it goes and tries to search for F. It puts C in the array. Updated array looks something like (just cleared the array for clarity)

As it did before it gets the child of C and tries to match the properties. In this case the array looks like

In this case it matches the properties specified by the user and properties of F, which eventually matches and hence engine is able to search F.

Above example is a very basic scenario in which search happens, just to get things setup.

Now let us analyze the UI tree drawn above and try to get relationship among the controls. As you can see D and E are siblings which has common parent B.

F does not have a real sibling. But it has siblings D and E with common ancestor being A.

Question: What if we are able to define a relation of controls (like what we do for ancestor elements what we say container elements) which are siblings of each other for in case of search.

Answer: Yes it is possible. This is why NextSibling SearchConfiguration is there.

Moving ahead with NextSibling SearchConfiguration. Today this is used in case of MSAA and UIA plugin which also means that this configuration is understood by the playback engine.

EXAMPLE 1

Lets take for example the below code

In short I have defined a relationship between E as a next sibling of D with common ancestor B.

Below diagram shows how search for E happens in this case. Let us assume that B is already searched by Playback engine. Hence the array will look something like :

Now the motive of Engine is to search D (because that is the next element it needs to search before searching E). So it gets the children of B one by one and puts it into the array. So it puts D (The first child of B) in the array and eventually marks it as element D because search properties matches. Here is the updated array

Now it goes and try to search E. Engine on seeing that NextSibling is the SearchConfiguration for E, it doesn’t gets the children of D (as it did in the original search example) rather it continues getting children of B, because E will be the next sibling of D. Doing so it finally hits E which matches the search properties specified by the user.

This was a simple scenario of a NextSibling match.

EXAMPLE 2

Let us discuss one complex example. Below code shows one of the complex scenarios :

In this above scenario I am trying to search F with NextSilbling configuration. If you notice A is the common ancestor of both D and F and hence I have written the code in this way (have not used B as the container element of D).

Let us see how does this work. Assume A is already searched and part of the array. So the array looks like

As the search progresses for element D engine pushes children of A inside the array, but none of them matches. The state of array will be like:

The pivot element (meaning the element of whose children was getting queued in the array) was initially A. Now when all the children of A is in the array the pivot element becomes B (next element in the array). Hence children of B is now queried for and queued into the array. In this case D is the element which matches the search properties specified for control D. Hence the state of array looks like

Now the engine tries to search for F. It sees a NextSibling Search configuration of F. Hence rather getting children of D it continues getting children of B which has only one element left that is E. Array looks something like :

The pivote element moves from B to C. The only child of C ie F gets queued into the array and also matches the SearchProperties specified by the user. Hence engine returns element F.

Uses of NextSibling SearchConfiguration

The codeduitest recorder (in case of UIA and MSAA plugin cases) generates NextSibling SearchConfiguration when the element does not have a good Searchable properties (ie they have just have ControlType property), but it’s previous sibling has good identification strings.

In example 1 E didn’t had any searchable property but D had. Hence recorder in this case will generate NextSibling for element E.

In example 2 F didn’t had searchable property and so did E. Hence it generated NextSibling for F with D being the previous sibling with good Searchable property.

User can always hand code and use this configuration as and when required.

Note: As mentioned before NextSibling today is applicable for MSAA/UIA plugin and not web.

If you just want to Click the control with both middle and left button  then you can do this by following steps

1) Record the scenario of click using only one Mouse button. Say Left

2) The code that gets generated will be something like

Mouse.Click (uIForm1Client, new Point(63, 226));

3) Now modify the code as follows

Mouse.Hover(uIForm1Client, new Point(63, 226), 3);

// Press down of two buttons

DoMouseAction(MOUSEEVENTF_LEFTDOWN | MOUSEEVENTF_MIDDLEDOWN);

// Now do up

DoMouseAction(MOUSEEVENTF_LEFTUP | MOUSEEVENTF_MIDDLEUP);

If you just want to Drag on the control with both middle and left button  then you can do this by following steps

1) Record your drag scenario with only one Button

2) The code that gets generated will be something like

// Move 'Form1' client from (63, 226) to (194, 230)

Mouse.StartDragging(uIForm1Client, new Point(63, 226));

Mouse.StopDragging(uIForm1Client, 131, 4);

3) Now modify the code as follows.

Mouse.Hover(uIForm1Client, new Point(63, 226), 3);

// Press down of two buttons

DoMouseAction(MOUSEEVENTF_LEFTDOWN | MOUSEEVENTF_MIDDLEDOWN);

// The points should be used as much as needed. These points resembles the Screen coordinate

Mouse.Move(new Point(Cursor.Position.X + 40, Cursor.Position.Y));

// Now do up

DoMouseAction(MOUSEEVENTF_LEFTUP | MOUSEEVENTF_MIDDLEUP);

private const int MOUSEEVENTF_ABSOLUTE = 0x8000;
private const int  MOUSEEVENTF_LEFTDOWN = 0x2;
private const int  MOUSEEVENTF_LEFTUP = 0x4;
private const int  MOUSEEVENTF_MIDDLEDOWN = 0x20;
private const int  MOUSEEVENTF_MIDDLEUP = 0x40;
private const int  MOUSEEVENTF_MOVE = 0x1;
private const int MOUSEEVENTF_RIGHTDOWN = 0x8;
private const int  MOUSEEVENTF_RIGHTUP = 0x10;

[DllImport("user32.dll", CharSet = CharSet.Auto, CallingConvention = CallingConvention.StdCall)]
public static extern void mouse_event(long dwFlags, long dx, long dy, long cButtons, long dwExtraInfo);

public void DoMouseAction(int buttons)
{
    //Call the imported function with the cursor's current position
    int X = Cursor.Position.X;
    int Y = Cursor.Position.Y;
    mouse_event(buttons, X, Y, 0, 0);
}

a) Which plugin gets picked up for which controls

b) Introducing MSAA plugin

c)  Introducing what does it mean that a control is windowed.

d) Properties specific to Windowed control.

e) How to do Windowed based search and how to search a particular non windowed control using a windowed properties.

Plugins currently picked up (Beta2)

Below is the info in which plugins gets picked up for which controls

1) IEPlugin (web) :- This is picked up for web controls inside IE 7/8.

2) UIA:- This is picked up for WPF controls.

3) MSAA (Default plugin):- This is picked for Winforms Controls, Win32 controls, MFC applications. In short which ever controls that are not specified in above two plugins gets picked up by this plugin which is a default plugin. Thing to note here is that if for some platform we are sure that default plugin wont work (for eg. FireFox, SIlverlight etc) this plugin also doesn’t get picked up and we throw exception.

Introducing MSAA plugin

MSAA plugin uses “Micorsoft Active Accessibility” to identify controls. So if a control has a good Accessibility support then it could be identified by the plugin (if there is no other plugin that support that control this plugin is hit. To see how to write your own plugin for the platform that are not supported please go through this.)

Internally the plugin generates search properties/exposes properties given by interface IAccessible. For richer properties for Winforms/Win32 controls it uses Windows messages to get those properties. Also some of the SearchProperties are windowed specific properties. In the next couple of topic we will be seeing what does a windowed control means and what are the specific properties that this plugin generate.

What are windowed controls?

By Windowed controls we mean all the controls who have their own window handle. You can identify if a particular control has a window handle of it’s own by using SPY++. Below are the steps to check if a control has a window handle of it’s own or not.

1) Open SPY++.

2) Press Ctrl+F.

3) Use the Cross hair and drag it to task bar button of any application that is opened. You will notice that the whole task bar is getting highlighted instead of the buttons. This is because it uses a window handle from point and highlights it. In this case the buttons don’t have their own window handle instead the whole task bar’s window. This means all the these buttons inherits the window handle of the parent task bar window control.

Note: Not all controls with ControlType = Window are windowed and also even a control which is of ControlType other than Window can also be a Windowed control (eg the hosting control inside the Internet Explorer who’s controltype is a client)

Properties of Windowed Control

Some the properties that a windowed control can have are :-

1) WinProperties.Common.ControlName (The ControlName which is got by Sending WM_GETCONTROLNAME message to the window handle of the control)

2) WinProperties.Common.ControlId (Got using GWL_ID message)

3) WinProperties.Common.ClassName (Got using GetClassName Pinvoke)

4) WindowText (Got using GetWindowText PInvoke)

5) AccessibleName (Got using IAccessible pointer for the control)

All of these properties are got using window handle of the control and hence these are specific to windowed controls. The last property though (as the name suggest is not a property got using window handle but using IAccessible pointer.

How is Name and AccessibleName different?

With Respect to the MSAA plugin Name is IAccessible name for a non windowed control. In case of Windowed controls the name Property returns WindowText and AccessibleName returns IAccessible Name. For Non Windowed controls AccessibleName property should not return anything and you will get IAccessible name using Name property as mentioned before.

Searching of a control

You can do two type of Search using MSAA plugin :-

1) Window based search :- This is done by navigating through all the window either inside Desktop or inside a specified Window handle.

2) MSAA search :- This uses IAccessible based navigation to search a particular control.

Search a Window Control using Window Search

If you want to search a window control you can do that by specifying atleast one Window Property (as mentioned above)

So you can do this to search a control who’s ControlName is “foo” and who’s controltype is window

WinWindow window = new WinWindow (containerelement);
window.SearchProperties.Add(WinProperties.Common.ControlName, "foo");

Internally the plugin checks if there is even a single window property mentioned in SearchProperty. Seeing that it does a Window based search.

Note: Following are the properties you can mention as part of SearchProperties so that a window based search kicks in.

WinProperties.Common.ControlName

WinProperties.Common.ControlId

WinProperties.Common.ClassName

“AccessibleName”

Searching a Non Windowed control using Windowed property

Let’s take an example of a Simple Windows Form Application which has a button inside it. I want to search the button if I know it’s parent window’s ControlName (let’s say “foo”) and don’t want to search using button’s name property. Below code will show how to search a button using it’s window’s controlname property

WinWindow window = new WinWindow (topLevelElement);

// Assign a ControlName property to the window control so that a window search kicks in.
window.SearchProperties.Add(WinProperties.Common.ControlName, "button1");

// Now you search a button control without specifying any property here

WinButton button = new WinButton (window);
Mouse.Click(button);

Let’s understand the code. The window instance is having a Windowed SearchProperty assigned to it. Hence it will hit a window search which matches the control name property before returning the element.

Now we have assigned the Container element as window of the button control (see the ctor of the button control). So internally a search for the window will happen first using window search  and then button will be searched under the window control using MSAA based search.

Event Overview

Coded UI Test is a new feature in Microsoft Visual Studio 2010 for functional UI automation testing. Coded UI Test is targeted towards automation engineers and developers who can use the feature to ensure there are no regressions in the UI. Coded UI Test includes a rich API library to code against and a resilient record and playback tool. In this webcast, we discuss various extensibility features of Coded UI Test like extending the support to custom controls or using the record and playback tool for completely different scenarios.

Here comes Microsoft Indic Language Input tool. It currently supports Bengali, Hindi, Kannada, Malayalam, Tamil and Telugu.

This is what I get when I type “Hindi”.

I suggest giving it a try.