However, it is not possible to associate the test-case with the test-method from Microsoft Test and Lab Manager (MTLM). When viewing the “Associated Automation“ tab in any test-case, we can notice all the entries are read-only. This linking can be done from Visual Studio IDE.

Snapshot of Test-case opened in Microsoft Test and Lab Manager

Steps to associate a Test-method with a Test-case:

1. Open the Visual Studio IDE.

2. Open the test-project or solution in which the needed test-method to be associated.

3. Connect to the required TFS Project Collection and then the Team Project.

4. Query for the needed test-case using Id / Owner / Created data and so on.

5. Open the required test-case and click on “Associated Automation” tab

6. Click on the ellipses near the “Automated Test Name”

7. In the “Choose Test” dialog, Choose the required test method and Click on “Ok”. This dialog only shows the list of tests that are part of currently loaded test project or solution.

Snapshot of Test-case opened in Visual Studio IDE

In order to find the control in test execution phase, Coded UI test captures some set of properties on the control and name them as “Search Properties” and “Filter Properties”. Each instance of these is a  Name-value pair, with Name being the Property-Name and Value being the actual value of the property during the time of the capture.

Search for a control always starts from desktop. Lets take an example of a control whose search properties and filter properties are captured

Code

                    HtmlEdit  mUIEmailEdit = new HtmlEdit(someAncestorControl);
                    mUIEmailEdit.SearchProperties[HtmlEdit.PropertyNames.Id] = "email";
                    mUIEmailEdit.SearchProperties[HtmlEdit.PropertyNames.Name] = "email";
                    mUIEmailEdit.FilterProperties[HtmlEdit.PropertyNames.LabeledBy] = null;
                    mUIEmailEdit.FilterProperties[HtmlEdit.PropertyNames.Type] = "SINGLELINE";
                    mUIEmailEdit.FilterProperties[HtmlEdit.PropertyNames.Title] = null;
                    mUIEmailEdit.FilterProperties[HtmlEdit.PropertyNames.Class] = null;
                    mUIEmailEdit.FilterProperties[HtmlEdit.PropertyNames.ControlDefinition] = "id=email size=25 name=email";
                    mUIEmailEdit.FilterProperties[HtmlEdit.PropertyNames.TagInstance] = "7";

          mUIEmailEdit.Find();

Description

From the code, we will be able to get all the properties that are associated with the control under test from the control object ( mUIEmailEdit ). When the search for the control starts ( explicit by Find() or implicit by any usage of the control in actions or property validations ), it starts finding the controls under the Container element of the control ( someAncestorControl ).

Search in Coded UI test is a Breadth-First which compares each of the controls in the traversal for the match of the properties given in the code.

First pass of the Search is done for the match of All properties given in SearchProperties list. Basically this can be referred as AND Condition of search properties. In cases, where the results of the search using SearchProperties finds exactly one control or doesn’t find any control, Coded UI test skips using the Filter Properties (even if is defined in the code) because of the 2 cases,

    1. There is exactly one control, so there is no need to apply filters.

    2. There are no controls to apply filters.

But, in cases where the list of SearchProperties is not good enough to find the exact control, Coded UI tests uses the filter properties (one by one) till it finds One exact match. Basically this can be referred as Ordered Match.

In remote cases where Search using FilterProperties also results in more than one control, the First match is returned as the resultant control.

Example

Lets take the instance of search for the code written above

1. Search starts at the “someAncestorControl” control as it is specified as the ancestor for the control to be found.

2. At every traversal, we match the properties of the current control “X” with the search properties mentioned in the code. Let say we have a list of controls (X1, X2 and X3) under the container’s hierarchy that matches the properties.

3. Now we need to nail down the search to one control. From the list (X1, X2 and X3) , we check for the match of 1st Filter property ( HtmlEdit.PropertyNames.LabeledBy ). Let say the list now trims to (X2 and X3). In case of no match , we skip this filter property and keep using the next filter property for search.

4. We continue #Step3 with the following filter properties (one-by-one) till the list of resultant controls trims to 1 or till all the filter properties compared.

Workflow of Search

Technology specific data

1. Search for Web controls                      : Both SearchProperties and Filter Properties are supported.

2. Search for Winforms, Wpf controls      : Only SearchProperties are supported.

Choice of SearchProperties

Each technology has a limited set of properties that can be used for search via Coded UI tests. So, Coded UI tests chooses only the properties exposed by the technology as SearchProperties. These are chosen by targeting a good trade-off between robustness and performance.

For instance, the Web technology exposes many properties including all the attributes of the controls rendered in the page, whereas .Net controls’ accessibility exposes only a sub-set of the properties via accessibility.

Coded UI test also honors the parameters used in manual test-case in Team Foundation Server 2010 while code-generating the action recording associated with it. Let see how to author a manual test-case with parameters.

In TCM, use ‘@’ symbol to prefix any parameter name while authoring the test-case. With addition of any parameter in the test case, parameter value table will show the parameter name in new column. Enter all the values that will be used in consecutive iterations of the manual test case. In this example of a Login test-case, notice that 2 values are added for each of the parameters (Username and Domain)

Lets assume that the test-case was tested manually with Action recording ON for one iteration of values (Balagans and Fareast). Now that the action recording is ready for the test-case, we can create a Coded UI test from the automation strip for all iterations of the test.

1. Create a test project and add a new Coded UI test to it

2. In the options dialog box, select 2nd option to generate automation code from the action recording that is stored with the test-case in Team Foundation server

3. Connect to the appropriate server and team project and Query for the test case for which the code needs to be generated.

4. When pressed “Ok”, Visual Studio uses the Automation strip, parameter names and parameter values to create a Data-driven Coded UI test

Test Code

[DataSource("Microsoft.VisualStudio.TestTools.DataSource.TestCase",
            "https://vsttidcserver2:8080/tfs/defaultcollection;MTR",
            "4571", DataAccessMethod.Sequential),
        TestMethod]
        public void CodedUITestMethod1()
        {
            this.UIMap.EnterUserNameinuserNameeditBoxParams.UINameEditText = TestContext.DataRow["UserName"].ToString();
            this.UIMap.EnterUserNameinuserNameeditBox();
            this.UIMap.EnterDomainindomaineditBoxParams.UIEmailEditText = TestContext.DataRow["Domain"].ToString();
            this.UIMap.EnterDomainindomaineditBox();
        }

Note that for every Recorded method (EnterUserNameinuserNameeditBox ), Coded UI test generates a parameter class ( EnterUserNameinuserNameeditBoxParams )that groups all the values of actions used in the automation. Same is the case for the property validation methods. For every validation method, Coded UI test generates a parameter class (<MethodName>”ExpectedValues”), whose properties can be assigned to any value for changing the behavior based on the requirements.

So, whenever the tests needs to be data-driven, the only thing that needs to be done is defining a data-source and assign values to the properties of this parameter class before calling the corresponding recorded method or validation method.

Contents of the Data-driven Coded UI test

1. Connection String – Contains the Team Foundation Server name , Test-case Id and the Data source type name based on the data source chosen (can be Database, XML or CSV). In this case, the parameter values are pulled from the Team Foundation server test-case (Microsoft.VisualStudio.TestTools.DataSource.TestCase)

2. Data-Access – All the values from the parameter values can be accessed via TestContext variable that is part of any unit tests. DataRow property gives the data in one row from which the values of the columns / fields can be accessed.

3. Value-Assignment and Method calls – Finally to data-drive the test-case, Coded UI test uses the data-source to assign values to the parameter class and then calls the corresponding recorded method or validation method.

For more details on Data-driven test, visit Link

It was pretty eclectic group with people from different countries involved in different levels of testing : From Test Consultants like Michael Holden to Manual testers, who were keen is getting the knowledge share about the new testing solutions in the market

Day-1 started with the keynote from Michel Holden about Exploratory Testing and the importance in sapience in testing. His insights about How testing cant be automated completely without involving sapience and analogy of How Testing and leadership requires same set of skills was an eye-opener. The keynote was followed by a interesting talk on Model-based testing by Bruno Legeard, CTO and Co-Founder, Smartesting.

Many testers were surprised to see Microsoft among the players in Testing solutions. One of the attendees had an interesting quote ”Why Microsoft is doing testing tools ? Go and develop software!“. Tanuj Vohra, Partner Director PM in Visual Studio gave a quick round of the presentation about the rich feature set in VS 2010, which made many think about the value props and the ease with which testing can be done.

We had a pretty busy booth with attendees in the following buckets.

1. Test Consultants

2. Test Managers

3. Senior Executives

4. Testers

One of the main value props that impressed many people was the Application Lifecycle Integration that breaks the walls prevailing in today’s software business. Testers seldom knew the requirement changes, Business analysts fail to understand the quality of their user stories, and of course developers are in their own world fixing bugs that is not linked to any requirements or test plan. Visual Studio 2010 in integration with Team Foundation Server 2010 addresses all these issues by providing integration between Developers and Testers, Business analysts and Developers and so on.

Testers who attended the conference were pretty much impressed with the feature set we are shipping as part of V1. There were many questions related to Migration of assets from existing tools to TFS 2010. We also received quite a bunch of requests from different organisations to have a POC / Demo in their premises for assessing the value wrt their projects.

CodedUITest:

Visual Studio 2010 introduces a new test type – Coded UI Test [CUIT], which enables you to create automated UI tests which can then be added to a regression test suite. Coded UI Tests may be run as part of a Build Verification Test suite, thus giving the developer immediate feedback on any UI regressions. The developer can run these UI Tests before his check-in and ensure that there are No more UI regressions.

It supports the following tools

1. Coded UI Test recorder allows users to record test actions and generate code for them. Generated code is re-factored into methods based on user gestures

2. UI Control Locator allows users to add UI controls to a repository and generate assertion code on their properties

3. Coded UI Test has excellent integration with Team Foundation Server

The following sub-sections provide details on the features supported by the engine.

· Intent-aware, non-intrusive and resilient action recording: Instead of just capturing raw user actions, the engine has been designed to be intent aware. Set of actions are aggregated into equivalent action to provide an intent aware playback.

Additionally, the engine uses smart algorithms to accommodate changes in configurations like window resize/Minimize/Maximize, screen resolution changes etc to provide a resilient playback support.

· Strong UI technology and platform support: The engine supports a wide range of platforms out-of-the box covering thick client applications built with .Net technologies to latest web applications

• Internet Explorer 7.0 and Internet Explorer 8.0
  • Support for applications written in ASP.Net, PHP, JSP, ASP, Ajax, SharePoint and other Web 2.0 applications.
• Win-forms Controls for Microsoft .Net framework 2.0, 3.5 SP1 and 4.0. Win32 / MFC controls with good accessibility.
• Windows Presentation Framework (WPF) for .Net framework 3.5 SP1 and 4.0

Internally the engine uses features of IEDOM for recording actions on the web applications being tested via Internet explorer. It uses accessibility technologies in Windows to capture actions on AUTs. Specifically it uses Microsoft Active Accessibility (MSAA) for Win-form support and UI automation (UIA) for WPF support. Custom controls that are developed for Win-form and WPF that follow accessibility guideline are also supported.

· In-built sophisticated mechanisms for handling complex UI action recording scenarios

• Fast Search: In order to be able to playback user actions, the engine must be able to search for the Control that was acted upon. It uses fast search algorithms to quickly identify the controls and take appropriate actions on them.
• Wait For Ready Mechanism: When playing back an action the engine needs to know that the UI is ready to act on the next action. For example, if the application was navigating to a web page, the engine needs to wait for the navigation to complete before taking an action. The WaitForReady mechanism provides this support by intelligently detecting the application’s readiness to act on the next action.
• UI Synchronization: If the user acted on a control (e.g. a mouse click), the engine needs to ensure that the control actually did receive the input. UI synchronization feature provides this support to the engine allowing it to perform a more resilient playback.
• Ensuring UI Control is Visible: In a fast-forward scenario, the UI element (e.g. a list box) that needs to be acted upon might not be visible (scrolled our) or hidden behind another window. This feature ensures that the element is visible before taking an action on it.

How is CUIT different?

CodedUITest has an edge over the existing automation Frameworks in terms on the following

1. Robustness

2. Scalability in different dimensions like number of actions, number of controls and the length of the recording and playback

3. Performance of the Automation

4. Breadth in Platform Support

5. Integration with Visual studio Team System for Planning, Authoring, Executing test cases and Reporting.

To get more insight on the comparison with other automation frameworks, visit the UI Automation Wiki page here [Microsoft internal link]

Creating Automated Functional Tests using CUIT:

A typical functional test encompasses a series of Actions to be performed on the application under test followed by set of Validations that will verify the functionality of the test application at that stage of execution.

The following are the steps to be followed to create a CodedUITest. The underlying sample application is a Form-filling application.

1. Start Visual Studio 2010 IDE

2. Click on “Test” menu and select “New Test”

3. Selected “CodedUITest” from the list of available test types and create a Project

Note: It is preferable to add the Test project to the Application’s solution itself so that the Test code will be built as part of the product code build process.

Screenshot of the Application under Test:

Recording the actions and Generating code for Automated Test:

1. From the Dialog that opens on creating a new instance of CUIT, Select “Record Actions, Edit UIMap or Add Assertions”

2. CodedUITest Builder Window opens just above the task bar with 4 buttons [ Record/Pause, View Actions, Add Assertions, Generate Code ]

3. Click on “Record” and Start performing actions on the Application.

In this example of Form-Filling Application, we will be keying in the details for Name, DOB, Sex and Profession fields and Click on Submit to generate some results

4. Now that we have recorded the application, we need to generate code for the recorded actions to create an automated test for the underlying scenario. To do so, Click on “Generate Code”

5. Key in a Name for the Method and Click on “Add and Generate”

Inserting Assertions for validations:

1. Any functional testing scenario is complete only if the validation is done for the actions done on the application. In our example, we will be validating the Output of the Results Textbox at the bottom.

2. To Insert Assertion, Click on “Add Assertion”. This opens up CodedUITest Builder in Assertion mode.

3. The Window has 2 panes.

a. Left pane shows all the controls interacted till now. This is called UIMap Viewer.

b. Right pane shows all the Properties of the selected control. This is called Property Explorer.

4. To add validation for a control that is already added to UIMap

a. Click on the control in Left pane. Select the validation property. Right Click on it and Click “Add Assertion”

b. In order to undo adding Assertion on the same session, Right Click on the property to select “Delete Assertion”

5. To add validation for a new control

a. Point to the control on the Application. Use the Global Hot-key for Add Control to UIMap [ Win + I ]

b. Same as above, select the validation property to Assert on and Key in the Assert Value in the dialog that opens up.

6. Click on “Generate code” to generate the code for the Assertion added.

Code Snippet of the generated assertion:

/// <summary>

/// CheckResult - Use 'CheckResultExpectedValues' to pass parameters into this method.

/// </summary>

public void CheckResult()

{

WinEdit rtbDetailsEdit = this.FormFillingApplicatiWindow.RtbDetailsWindow.RtbDetailsEdit;

// Verify that 'rtbDetails' text box's property 'Text' equals 'You are Hari born on 19 July 1988 working as Engineer

Assert.AreEqual(this.CheckResultExpectedValues.RtbDetailsEditText, rtbDetailsEdit.Text);

}

Contents of CodedUITest Project:

1. UIMap.uitest – This is a xml file that is a repository for the UIObjects interacted in the test and the list of Recorded and Assertion methods added to the test

2. UIMap.Designer.cs – Auto-generated file [ Not to be edited by the users ] , which contains the Class definitions of the UIObjects in the UITest

3. UIMap.cs – Partial class, which is a place-holder for overriding the properties of the UIObjects in UIMap

4. CodedUITest.cs – Test Class that instantiates the UIMap in the current project and calls the Recorded and Assertion methods

5. UserControls.cs – Control Type classes for each of the types of the UIObjects added to the UITest file

a. In case of a Winforms button, the generated UserControls class name will be WinButton

Finally, run the test to verify the scenarios / validations and hence the functionality of the application will be tested.

Maintainability of CodedUITest:

Any Application’s User interface is subject to change over the product cycle. This adds up to the maintainability cost of the automated functional tests.

In the context of CodedUITest, the recorded actions and assertions are code-generated on the controls in the application, which are found through a set of properties called “Search and Filter Properties”. In many instances, the functional tests are not agnostic to the changes in the application. So, the changes in the functional tests are highly vulnerable. The following segment will explain in detail about the series of the recommended steps that needs to be followed for fixing the CodedUITest for a corresponding change in the application.

What are the possible changes in the Application that will affect CodedUITest?

1. Change in the Expected Value of the Validation Property

2. Change in Value of the Search Properties.

3. Change in the UI Hierarchy of the control [ like moved to a different container ]

How to deal with changes in Validation properties:

1. The obvious solution would be to make the CodedUITest fail because of incorrect expected value. On the Error Stack Trace in Test results window, Click on the Assertion line link and fix the Expected value.

2. CodedUITest generated code has a Partial class from which the tester will have access to Recorded methods, Assertion methods and the ExpectedValues classes. So the other way to fix the expected value is to write an Initialization function that will change the expected value of the assertion. See sample code below

//Partial Class Code

public partial class UIMap

{

/// <summary>

/// Method to set the Expected value to String.Empty

/// </summary>

public void Init()

{

this.CheckResultExpectedValues.RtbDetailsEditText = String.Empty;

}

}

//CodedUITest Class Code

[TestMethod]

public void CodedUITestMethod1()

{

UIMap map = new UIMap();

//Call Init() to reset the Expected value

map.Init();

//Call Assertion method

map.CheckResult();

}

How to deal with changes in Search properties:

Any change in Search Properties will result in failure of Recorded methods as well as the assertions added, as both are dependent on the Search properties of the underlying control.

1. One of the recommended solutions is to re-fetch the control to the UIMap. Add the new control to the UIMap and generate code to reflect the changes of the UIMap in the code. To update the code replace all instances of the old object with the new object added.

a. This can be done by copying the “Reference of the new and old Object to clipboard” and do a Find-replace in the Test project

b. You may also delete the old object from the UIMap by clicking on the “Delete” button on CodedUITest builder toolbar. This can be done only when no references to the object is present in the test classes.

c. Note that, we need to generate code from updated UIMap to remove the obsolete UIObjects in UIMap.Designer.cs file.

2. Search properties can be overridden inside the Partial class using some Initialization methods. See sample code below

/// <summary>

/// Method to override the Search properties in UITest

/// </summary>

public void ChangeProperties()

{

this.FormFillingApplicatiWindow.SubmitWindow.SearchProperties[WinProperties.Window.Name] = "SubmitDetails";

}

//CodedUITest Class Code

[TestMethod]

public void CodedUITestMethod1()

{

UIMap map = new UIMap();

//Call ChangeProperties () to change the search properties

map.ChangeProperties ();

//Call Recorded method

map.Bvtmethod();

}

3. The last resort would be edit the UITest [xml formatted] file directly and “Generate Code” for the changes to reflect in UIMap.Designer.cs file. This is not the recommended way as editing the UITest incorrectly may result in incorrect functionality.

Ready to try out CodedUITest?

Visual Studio 2010 Beta2 is available in this link.

Visual Studio Beta2 Forum : https://social.msdn.microsoft.com/Forums/en-US/vsautotest/threads

Visual Studio Team Test [VSTT] comes to the rescue in these situations !

VSTT encompasses a Test Controller-Test Agent setup for facilitating Remote Execution of automated tests. The following will briefly give an idea of the High level architecture .

Prerequisites :

• Development machine has Visual Studio Team System installed with product and test sources
• A Test machine that is a mimic of the Developer machine [ where the tests will be run ]

Setting up the Controller - Agent :

• Install VSTT Controller on the Developer machine using the below steps and configure it appropriately

1. Enter the credentials, in case the tests needs to be run on a different credential from Network Service
2. Apply Settings

• Install VSTT Agent on the test machine, which is already setup with all the needed configuration and settings for the unit tests to run
  1. Enter the credentials, in case the tests needs to be run on a different credential from Network Service
  2. Connect the Agent to the configured Controller which is setup above
  3. Apply Settings

Note : In case of running Automated UI tests [ or CodedUITest ], the Agent should be running as an Interactive process instead on Service to act on the UI Components under desktop.

Running Test using Controller - Agent :

• Open the product sources in Visual Studio along with the corresponding test entities [ including *.testSettings ]
• Open Test-Settings file and Click on Role, and Select "Remote Execution" and Enter the Test Controller machine name .

• In order to control the load on each of the agents, we can Filter the Agents in which the tests needs to run by giving a Attribute Name and Value condition. These Attributes are assigned via "Manage Test Controllers" options under "Test" menu in Visual-Studio IDE

• Start running tests from Test View and see the results in "Test Results" pane.

Advanced Options :

Controller - Agent setup allows the test to collect more information from the test machine using "Data Collectors"

1. Some of the Data Collectors are Event Log, System Information and Code-coverage

Deployment :

1. Test Binaries and dependant binaries needs to be given as Deployment Files and Folders in Test settings
2. So the best practice would be to build the solution with Outdir path set in Build Command and use the corresponding Folder in Test settings

Note : The Symbol “*” beside the name of the Agent indicates that it is running as an Interactive process and is ready top run CodedUITest or any Automated tests that interacts with desktop ]]>