I was in a discussion with the customers and they had this strong opinion that they are paying a lot of money for testing teams to just review requirements and write test cases. They recommended us to write automation scripts only without spending any time on designing test scenarios and test cases. I highlighted our role and contribution in requirement elicitation process to identify the gaps and save the effort upfront rather than discovering them in the build or stabilization phase. To accomplish that goal we need to spend effort in coming up with a detailed test plan and strategy that best fits the unique needs and requirements of the project in hand. We methodologically identify the areas that are best candidates for automation based on the ROI (Return on Investment) to save effort and time during the test execution phase. The quality of the automation scripts depends on the coverage of the underlying test cases / test scenarios and hence it is important to have them. At the end of the discussion I could sense that the customers were still skeptical and I didn’t expect them to get totally convinced either. I bought time and promised them to give a small demonstration of our capabilities by doing some quick POCs (Proof of Concepts).

From the requirements, I picked the most important ones and studied the pattern, the behavior and looked at the testability aspect of them and prepared a list of test techniques. I wanted to reduce my test cases to a number that is manageable and maintainable. I figured out that they had lot of complex process workflows where there are multiple jumps, branches and loops. I couldn’t cover them even if I wrote a whole bunch of test cases and even after doing that it will not guarantee that I didn’t miss an important path. We had the Visio's of those workflows so why not use a “State Transition Table”? I came up with a quick template to represent their most complex workflow and ensured I covered at least every state and every transition once. I had added a flag to be able to filter the main flows/alternate flows.

Then there were a few complex business rules that had lot of conditional clauses and the result depended on the outcome of all those clauses. Again I realized that there can be lot of permutation here and I didn’t want to take chances. “Decision Table” just sounded right to me. I quickly prepared one covering positive and negative flows and here I had a table that was covering it all.

For lot of data entry forms, there were many fields and especially drop downs, radio buttons and checkboxes. I didn’t want my team to randomly pick values and tomorrow miss double-mode types of defects. “Orthogonal array of testing” can be helpful and I thought of using its Pairwise implementation to solve that problem. I used a free tool from Microsoft called PICT and voila from 72 permutations I had now arrived at just 9 permutations scientifically that ensures every pair is covered at least once.

After a couple of days I had put together a few running POCs of these techniques in addition to test cases that used standard techniques like EP & BVA (Equivalence Partitioning and Boundary Value Analysis). I quickly came up with the best way to structure and organize my test artifacts using Microsoft Test Manager. I created a sample Requirement Traceability Matrix using Test Manager. I decided on my test suites and also defined different test configurations as the application is being targeted for multiple platforms.

In 3 days, we were in a conference room with all the customers and stakeholders including our project team. I ran the demos of my test techniques on the real requirements. I showed them we would ensure test coverage and provide them the requirement traceability reports using Test Manager. PICT was like magic to them. State transition tables taught them that testing is much more than randomly clicking buttons and following plain steps. Automation can be done only after you perform these key activities and test cases are not just plain word documents.

The meeting went well and ended with encouraging remarks like “Can you send me the link of that tool? “, “Can you share these with us for Unit testing?”, “We want you to demo this to our test team", “This sounds great”. It was evident from their response that we had their confidence.

Stay tuned for more post. I can be reached at rajkamal@microsoft.com

Disclaimer: This blog is my personal opinion and I apologize if it hurt your sentiments. In the words of my friend Arun (aruncha@microsoft.com) , “When a developer starts in a project he is already 100 miles high whereas a tester is 100 miles buried under the ground and it takes the tester lot of work to reach up there and prove his value before his work is acknowledged “

TechNet Webcast: Automated Data Quality Checks Using SQL Server and Visual Studio (Level 300) – Raj Kamal, Microsoft

Microsoft IT's business intelligence engineering team has reduced its development and testing phases by automating the data quality audits and finding data quality issues earlier in the development cycle than in the past, using a metadata-driven, user-configurable tool. Attend this webcast to learn about automating data quality checks using Microsoft SQL Server and Microsoft Visual Studio and to hear directly from the development team how this can apply to you.
Presenter: Raj Kamal, Microsoft IT Test Lead, Microsoft Corporation
Raj Kamal is a senior test engineer who specializes in different types of testing techniques, test automation, and testability in different domains like manufacturing, healthcare, and higher education. Raj holds several certifications, including Certified Software Test Engineer (CSTE), Microsoft IT Professional for Microsoft SQL Server Business Intelligence (BI) 2008, International Software Testing Qualifications Board (ISTQB), and APICS certification in supply chain management. Raj also has expertise with Rational and Mercury testing tools, and he has helped teams develop test automation strategies and architectures for other major software companies.

Event: https://msevents.microsoft.com/CUI/EventDetail.aspx?culture=en-US&EventID=1032452088&CountryCode=US

View Online: https://www106.livemeeting.com/cc/MSEventsBMO/view?cn=guest&id=1032452087&pw=3CE735D1

Thanks,

Raj

Author:

Raj Kamal (rajkamal@microsoft.com)

New !!! On special demand, sharing the below content in PDF format as well as sample reusable Coded UI Framework for your user

http://blogs.msdn.com/cfs-file.ashx/__key/communityserver-blogs-components-weblogfiles/00-00-01-42-73-CodedUI/2086.Automation-Guidance-_2800_VS-2010-Coded-UI_2900_-ExternalCommunity.pdf

http://blogs.msdn.com/cfs-file.ashx/__key/communityserver-blogs-components-weblogfiles/00-00-01-42-73-CodedUI/0167.CodedUISampleFramework.rar

Objective

The objective of this artifact is to provide a L200 test automation guidance leveraging VS 2010 Coded UI features to foster a continuous, consistent and standard automation approach. The document can also be used as a quick jump start guide for beginners.

However the bigger objective is Build, Deploy and Test leveraging Visual Studio Lab Management features, which will allow Automation Testers to reap the benefits of the following

· Integrating with the build process for automatic build deployment

· Execution on VMs

· Capabilities of getting the test automation results reported as part of Test Manager (MTM) for better reporting.

Pre-requisites: Ensure you have ‘CodedUISampleFramework’ sample Coded UI project For VS 2010 installation & Configuration, go to Appendix: VS 2010 Installation Reference section If you are new to Visual Studio Coded UI, go to Appendix: Getting started with VS2010 Ensure your application is supported by Coded UI, go to MSDN link

Jump to Case Study, if your objective is to get started with Coded UI using our recommended framework and practices.

Test Automation Process using Visual Studio 2010

It is not enough just to automate your tests but there is also an important need to automate the build process as well as automatic installation and deployment. This will eliminate manual intervention required to trigger the automation due to build and deployment dependencies.

Automated Build, Deploy and Test framework will also enable us to find out the build quality at the earliest by publishing the test results for every run against a new build.

You can run the test suites that are most important for your continuous builds because too many tests can delay the completion of build. Make sure that you run these tests for the current iteration.

Figure 1: The topology required for setting up Build, Deploy and Test environment using various VS components like Build Controller and Agents for Build, Test Controller and Agent for distributing test and collecting results, Virtual environment for Deployment and Execution (Reference taken from MSDN)

To run unit or functional automation tests, we can create a set of environments that are managed Hyper-V images in Microsoft Test Manager. Tests that are created at the start can be used as part of continuous integration.

1. For Daily build, we want to run very minimal set of test which are like Build Verification test or Smoke test

2. For Weekly builds, we want to run “Regression test” which will be more in number.

Figure 2: Steps for Build, Deploy and Test workflow

If you plan to run your automated tests by using Microsoft Test Manager, you must use a physical or virtual environment that contains a set of roles to run your tests from your test plan.

**Build, Deploy and Test using Visual Studio Lab Management will be demonstrated using the below case study in future editions of this document.

We will now learn Coded UI automation features as well as our recommended automation framework and best practices in rest of the document.

Case Study: Coded UI Automation

We are going provide guidance on Coded UI automation using a real world case study. We assume you are familiar with Microsoft Test Manager for writing test cases (otherwise please jump to Get started section in Appendix)

Here is a sample snapshot of your test cases in MTM:

In subsequent sections, we will present the various approaches and steps involved in Automation Development Lifecycle that you should take to achieve your objective while following the automation standards and guidelines that are recommended.

Step 1: Automation Planning

Like Project Planning or Test Planning, Automation planning is equally important to ensure that the teams are not directly jumping into automation scripting without taking all the important factors into consideration which are extremely critical for providing accurate estimates, deciding the right framework, choosing the right approach, the long term maintenance of automation and getting early ROI (Return on Investment).

This activity will be primarily focused on identifying the feasibility and the scope of automation.

Figure 3: Steps involved in Automation Scope and Planning

Taking our case study, we will walk you through each of these steps below:

a) Identify Automatable Test Cases, Scenarios & plan Order of Execution

We must clearly know beforehand the test cases that are good automation candidates, the scenarios that can be converted into library methods for reuse and the sequence of execution if the tests are dependent on each other.

SL No.	Checklist	Description	Y/N
1.	Have you identified and marked the automatable tests as PLANNED in Test Manager?	Open your test cases in Test Manager that you want to automate and mark Automation Status = “Planned” for those For the case study, we have marked the automation candidate as “Planned” as shown below. Whereas the test which are not good candidates for automation are marked as “Not Automated” to show that they are out-of-scope for automation.	☐
2.	Have you identified your data-driven test cases?	Identify your test cases that are data-driven (this will be required during automation design) For our case study, look at the example of one data-driven test related to Bing where you want to perform search multiple times for the list of values provided as Test data	☐
3.	Have you identified the common functionalities/scenarios OR SHARED STEPS which can be automated for Reusability?	Ensure that you identify the Shared Steps or Common functionalities that can be later be converted into methods inside your application specific libraries to reduce automation effort and redundancy For the case study, here is one of the examples of Shared step or common functionality which can be added to the automation library. Note: This will be used during automation design	☐
4.	Have you identified the precedence / sequence of execution for your automated tests?	Ensure that you mark the sequence / precedence for the automated tests that have to be run As shown below, define the order of execution for your automaton scripts using OrderID field	☐

b) POC (Proof of Concept)

Now that we know the scope, we must also ensure that the application or technology in hand is actually supported by VS Coded UI Test by doing a POC on key scenarios. Many a times there might be custom controls and other components that are not automatable / supported by Coded UI out-of-the-box or need customization or are not worth automating due to low ROI. The outcome of this activity should help you decide your approach and next steps. You may mark the non-automatable test cases as out-of-scope for automation.

SL No.	Checklist	Yes/No
1.	Have you done a feasibility study to ensure VS 2010 Coded UI supports your application? Identify key scenarios of your application and ensure that they are supported by VS 2010 Coded UI by coding few scripts and checking the results.	☐
2.	Does your application have 3rd party controls or custom controls? Ensure that you call out the tests which are not supported by VS 2010 as “out-of-scope” for automation.	☐
3.	Testability: Have you worked with the development team to ensure that the properties of the objects are exposed and set appropriately for smooth automation? It is extremely important to work with the development team to work on the testability right from the design phase to make automation easier later in the cycle.	☐

c) Automation Estimation

The estimation will decide the success or failure of your automation project as it has been observed that automation can take much more time than anticipated if all the key tasks involved in automation lifecycle development are not clearly thought of while estimating.

The following major activities in each phase of automation must be estimated.

Automation Initiation

· Tool/Framework Feasibility Study

· POC

Framework Design/Implement

· Design and Implement Various Framework Components

· Testing the Framework Components and End to End Testing

· Training Resources on the Framework/Tool

· Environment Setup

Script Development Phase

· Script Creation

· Script Debugging

· Script Review

· Review Comments Incorporation

Script Execution Phase

· Environment Setup

· Result Analysis

Follow the below checklist while coming up with you initial estimation apart from effort for automation design and execution.

SL No.	Checklist	Y/N
1.	Check if you have included automation planning effort in your estimation including POC (Proof of Concept)- if any	☐
2.	Check that you include environment preparation (installation, configuration) effort into estimations.	☐
3.	Have you identified the Shared Steps or Common functionalities that can be later on converted into methods inside your application specific libraries to reduce automation effort and redundancy	☐
4.	Ensure that you include effort for building framework reusable components (Libraries)	☐
5.	Ensure that you include extra effort required if third party or custom controls are there in the application.	☐
6.	Ensure that you include effort for code review, walkthroughs and compliance with the standards defined. Ensure that you call out the tests which are not supported by VS 2010 as “out-of-scope” for automation.	☐
7.	Ensure that you include effort for code review, walkthroughs and compliance with the standards defined	☐
8.	Ensure that you include effort for debugging and lead time to resolve bugs discovered in the product (troubleshooting and workarounds)	☐
9.	Ensure that you include effort for reporting and result analysis.	☐

Now that we have the clarity on the scope, feasibility and estimates, we can move the next logical step which is creating your automation solution

Step 2: Creating a new Coded UI Project & Adding project to Source control

You can create a project from scratch or by using an existing template but it’s important to add your project to source control for version management and change tracking.

SL No.	Topics	Description	Y/N
1.	Have you created new Coded UI project & Added it to Source Control (TFS)?	Every automation project must be source controlled. Add project to Source Control by clicking the check box as shown Project Check-in Path: All application / project specific automation code tobe maintained under respective ‘project node’ of the TFS being used for the projects. Recommended Source Control Path: <Project-Node>/Main/Test/Test Automation Refer to Appendix: TFS Check-In Policies and Best Practices for details.	☐
2.	Open existing test project and using it as Template	Launch Visual Studio, Browse and open existing ‘CodedUISampleFramework’ template and add it to Source control. You can change the properties as per your needs. Please find below a short description on the folder structure of the template project which will be used for case study: 1. Config: All your application configuration details and playback settings. etc will go here Example: Also refer to: http://msdn.microsoft.com/en-us/library/a65txexh(v=VS.100).aspx 2. Library: All your libraries- application level, data access, exceptions, utilities will go here. Please refer to Step 3 3. Resources: All your keys / strings that’s need to globally defined can be added here rather than hardcoding their values in code. Example: The resource file key “SearchTimeOut” can be used to set playback settings as per your project needs without hardcoding it in your code.   4. Scripts: Your actual test are stored in these .cs files along with components like Test Initialize, Test Cleanup etc 5. Test Data: These will contain your .CSV or .XML test data files for your data-driven automation test 6. UIMaps: These will contain your test object repository along with any test methods or assertions (if any) 7. Main Class file: One class that acts as a single point interface to all the test methods and contains instantiations of the UI maps	☐

With next step, we are now entering into the most important phase of Automation Lifecycle Development which involves designing your automation framework. It is worth spending time on coming up with a good design rather than doing lot of rework later due to shortsightedness and eagerness to start scripting. I would strongly recommend you spend a good effort in coming with design/framework that suits your project requirement before writing your automation test.

Step 3: Automation Design & Development

Automation design and framework is one of the very important pillars of your automation success and it serves as a guiding light for your team members to follow a standard and consistent automation approach across the team.

In this section we will be walking through the automation framework design for Coded UI projects using our Case Study’s template project.

Figure 5: Proposed 3-Tier Architecture for maintaining automated test that promotes high modularity and maximum reuse with minimum maintenance effort in longer run

Follow the below steps for Automation design guidance:

SL No.	Steps	Description
1.	If you have created project from scratch, add Multiple UI Map files (In the template project, we already have 2 UIMap files for Case Study)	Managing Object Repositories: Single UI Map File: By default only one UIMap File is created per project and it is appropriate for a project of small size Multiple UI Map File: (We recommend using Multi UIMap files for modularization) For large application, we should use multiple UI Map. Using multiple UI Map files can provide the following benefits: - Each map can be associated with a logical subset of the application. This makes changes easier to manage. - Each tester can work on a section of the application and check in their code without interfering with other testers working on other sections of the application. - Additions to the application UI can be scaled incrementally with minimal effect on tests for other parts of the UI You can add new UIMap files by selecting Add new Items option and then selecting CodedUI Test Map type and renaming it as per your needs. The system will launch Coded UI Test Builder which can be used to add object properties to your UIMap file. Press Generate Code button (Alt+G). For Case Study, you will find the UIMap files as shown below: Open UIMap .CS files and add User friendly names to constructor: Go to your Main Class file (Bing.cs) in our Case Study and instantiate each Multi UIMap file   Please refer to Appendix: Managing objects using UITestEditor for more details
2.	Add/Create Libraries to your project (For Case Study, We have already created libraries for Bing)	Managing Libraries: Application Functional Libraries using Multiple UI Map files Ensure common functionalities identified above in planning phase are automated as reusable methods which are then invokes from your Coded UI Test methods Create functional libraries as per functional modules for your application. Common Library / Helper Classes: a) Ensure common utilities like SQL connection, Excel/ XML, String/Date handling etc. are referenced from common library repository in your automated solution (instead of creating them from scratch) All common libraries need to be browsed and added to automation solution as “References” b) Ensure that common helper classes which are technology specific like WPF, WCR, SharePoint, Silverlight, .Net etc. are referenced in your solution (if any). Refer to Appendix: Managing Common Libraries in TFS For our case study, we have created/referenced following libaries based on the Shared Steps and scenarios identified from ‘automatable test cases’ selected in Step 1 (Automation Planning) Example of Application specific library methods Example of Common/Utilies based methods:
3	Add your CodedUI scripts / Test Methods	Automate your ‘automatable test cases’ as CodedUI Test Methods using application libraries, utilities and UI Maps defined above a) Set the test context which provides information about and functionality for the current test run. b) Test Initialize: To perform steps before running each test like instantiating UIMap files, setting TestContext for logging etc. c) Add your Test Methods As you can notice above that we have highlighted certain things which are not recommended like using static playback interval or hardcoding test data. We will see in next section how to handle these. d) Test Cleanup: To run code after running each steps
4.	Adding Checkpoints / Assertions	For validation in Coded UI Testing, you need to use “Assertions” - Creating assertions using CUIT (Coded UI Test Recorder): You can create assertions without coding by using out-of-the-box feature provided with Coded UI Choose the property you want to validate and Click “Add Assertion”. - Creating assertions by hand coding: You can create assertions by writing you own code and capturing properties against expected values to validate at the run time. Verify.WaitForControlPropertyEqual(BING.SearchResultsUI.searchText, "Text", "Bill Gates", 15000); e.g. We have a method in our Utility “Verify” that waits for control property to be equal to some text value for certain period and returns PASS if found otherwise FAIL
5.	Make your scripts data-driven	Data-Driven Test Ensure every variable is parameterized by passing the test data to each test using data-driven capabilities in VS 2010. Right-click on the Coded UI test and select Properties, to view the properties of the test To start making this a data-driven test, we have to first create a data source for the test to draw from. On the Data Connection String property, click the ellipse button to start the New Test Data Source Wizard There are three options for creating a data source: Database -- which allows you to connect to a database or Microsoft Excel spreadsheet CSV File -- which allows you to use a comma-delimited file XML File -- which allows you to use an XML file When you add a data source A Data Source attribute is added to the Coded UI Test. [DataSource("Microsoft.VisualStudio.TestTools.DataSource.CSV", "|DataDirectory|\\data.csv", "data#csv", DataAccessMethod.Sequential), DeploymentItem("data.csv"), TestMethod] For our Case study, let’s look at one of our data-driven test: a) We have a method in our data access library (Utilies) to read the data from data sources using the test context b) Another method as part of Application library returns the data from data source and inputs the same to application using the above data access library method c) Test data source which is stored in .CSV format d) Our Data-driven test method calls the above method as part of our functional

Now I suppose you have your automation tests candidates designed and developed. After you write and debug your scripts its important to run them in a batch or as part of a suite. There are various way to do that and factors to keep in mind as we will see in next section.

Step 4: Automation Execution & Deployment

Ensure you have configured test settings to capture logs, code coverage, test impact etc. and you also need to keep in mind that your script should be able to successfully run in multiple environment (for ex. Test to UAT)

After automation design and debugging is completed, you can now run and deploy your automation scripts using the below steps:

SL No.	Areas	Description
1.	Configuring Test Settings	Test Settings a) Data and Diagnostic: Enable appropriate diagnostic are selected as shown below (Code Coverage, Event Log, Test Impact etc.) b) Define other settings like Roles, Test Timeouts, Setup and Clean scripts
2.	Deploy your test to multiple environments	Deployment to different environments: We want to run the same test across multiple environments to ensure everything is still working correctly. Development-> Test -> UAT -> STAGE Example: We want to run these tests after each deploy to each environment. Each environment has as a different URL, username and password. public UIMap() { // Bind the parameters using Environment Variables. string testServer = Environment.GetEnvironmentVariable("TestServer"); string testUserName = Environment.GetEnvironmentVariable("TestUserName"); string testUserPassword = Environment.GetEnvironmentVariable("TestUserPassword"); } Also refer: http://blogs.msdn.com/b/gautamg/archive/2010/02/23/how-to-get-same-test-running-under-different-environments.aspx
3.	Command Line Execution (MSTest.Exe)	Grouping & Classifying Test Methods using various parameters It lets you run groups of tests based on their assigned categories without the requirement to maintain test lists. Go to Test View and you can filter and view your tests by setting these parameters MSTest.EXE Options: MSTest.exe is the command-line command that is used to run tests. This command has several options that you can use to customize your test run. You can specify these options in any order on the MSTest.exe command line. Example: mstest /testcontainer:"C:\TestProject2\generictest1.generic" /test:testproject32\generic Refer to http://msdn.microsoft.com/en-us/library/ms182489.aspx

With this, we come to the end of our case study where we walked you through the complete automation development life cycle that involved Automation Planning, Automation Design and Development, and finally Automation execution and Deployment.

Automation scripts are just ‘Code’ at the end of the day and we can’t deny the importance of having high quality of code and best practices adopted for making our automation tests not only readable and easy to troubleshoot but also for long term maintainability.

Coding standards and best practices

We recommend following the below checklist and recommendations for your automation code. It is the responsibility of the entire team to follow these and imbibing a culture of peer and lead review of the automation code on a regular basis.

Checklist

We are also providing a compilation of best practices collected from various forums which can be used as a reminder when writing coded UI test.

Do’s and Don’ts of UI Automation

Do(s)

o Each recorded method should act on a single page, form, or dialog box. Create a new test method for each new page, form or dialog box.

o Use the Coded UI Test Builder whenever possible.

o Explicitly set focus to the window on which the test Case is expected to input data.

o When possible, limit each recorded method to fewer than 10 actions. This modular approach makes it easier to replace a method if the UI changes.

o Create a separate UIMap file for each module in your application under test. For more information, see Testing a Large Application with Multiple UI Maps.

o If you are creating assertions by coding with the API, create a method for each assertion in the part of the UIMap class that is in the UIMap.cs file. Call this method from your test method to execute the assertion.

o Capture screen shots for failures. Helps in debugging.

o Leave the UI in a known state after a test Case is done executing. E.g.: Close the wizard even if the test Case is to test the first screen of a wizard by cancelling out of the wizard after the first screen.

o Validate that the object being created using the UI with the one being stored in the backend system

o Log before you do something, not after.

o In code reviews for test cases, or new framework automation support review the log messages, is it clear what the test did from just the log?

o Log the origin of messages. If the framework logs, say that it logged, if the test case logged, say that.

o Have two assertion modes that are configurable: 1. Fail instantly. 2. Delay assertion of failure until the test has completed. Steps might be mutually exclusive, you’re losing coverage.

o Log non-fatal exceptions so as not to abort a test, bubble fatal exceptions up to the test and let the exception assert failure.

o Log every valid UI operation being performed in the test Case.

o Use resource files for all data required to be entered in the UI.

o Use Refactoring: Refactoring implies removing a block of code that is being used repeatedly throughout the program into a single function. This helps in reuse of code and makes the code cleaner, and improves readability. 

o Use Constructor of classes for meaningful things: In any class avoid unnecessary overloading of constructors. Minimize the amount of work done in a constructor. Constructors should mainly be used for initialization of members. It is always good practice to have accessory methods (get/set) to expose member variables and also initialize them.

o Using string. Compare () instead of "==" or "Equals ()": It is advisable use string.compare() as in that case we don't have to worry about NullPointerException, unlike using Equals() -- you have to make sure that the object that calls Equals() is not null. 

o Using List<class name> instead of Arrays: Arrays are not type safe, so it is advisable to use the inherent .Net 2.0 collection objects like IList<T>, ICollections<T>. They provide additional type-checking during compile time.

o Use “this.”: Whenever you use a member variable of class, it is a good practice to put "this" in front of the variable. This improves readability and avoids conflicts when the parameter name and the member variable names are same.

o Using .NET attributes for test cases: Attributes provide a mechanism to enable additional data to the Test Code during runtime and it also helps in keeping information associated with the Test Case in the code itself.

Don’t(s):

o Do not modify the UIMap.designer.cs file directly. If you do this, the changes to the file will be overwritten.

o Do not log multiple times.

o Do not sleep after every UI operation.

o Do not give unusually long timeouts for every major operation.

o Fail immediately after an unexpected screen.

o Do not re-launch the UI application for every test Case as it is a time consuming process.

o Test cases should never have hard-coded strings. This includes control ids, menu paths, etc. Even control IDs and menu paths change over time.

o Do not use sleep

o Avoid too many parameters in a Method/Function. It is best practice to avoid passing too many parameters to a function or method. This indicates lack of abstraction. In this case encapsulating the parameters in a separate class or struct may help.

o Avoid Lengthy Functions: If a function is longer than a page, it is a good practice to break it up into smaller functions. However, if the code has a switch statement or straight line code with no loop back, then it is still OK, but in almost all other cases, the code can be simplified by splitting it up into multiple functions.

o Avoid Hard-Coded strings: Avoid the usage of hard coded strings in the code. Instead use the constants utility to store them.

Appendix

i. Getting Started with VS 2010 Coded UI

ii. Visual Studio 2010 Installation

This step is required to ensure that environment is setup and configured to get started with automation,

- Installation: Have you installed VS 2010 Ultimate?

Go to Microsoft.com Download page

- Installation: Have you installed VS 2010 Feature Pack 2

Refer to http://msdn.microsoft.com/en-us/vstudio/ff655021.aspx to find out new features in FP2

- Installation: Have you connected to you Source control (TFS node)?

Go to VS 2010 Ultimate -> Team -> Connect to Team Foundation Server

Add your TFS server name, Choose Project Collection, Select your team project and say Connect

iii. TFS Check-in Policies & Best Practices

- Ensure that policies enforced by project administrator are met

Tip: For example “comment” must be entered every time checking the code

http://msdn.microsoft.com/en-gb/library/ms181460.aspx

Apart from that, following are available by default

o Builds:   Requires that the last build was successful before a check-in.

o Code Analysis:  Requires that code analysis is run before check-in.

o Testing Policy:  Requires that certain tests are completed before check-in.

o Work Items:   Requires that one or more work items be associated with the check-in.

http://msdn.microsoft.com/en-gb/library/ms181411.aspx

http://msdn.microsoft.com/en-gb/library/ms245460(v=VS.100).aspx

- Ensure that any custom policies enforced by administrator are met.

- As a best practice, check-in the files at the end of the day

- Code Branching and Merging

One of the most common branching strategies is to align branches with product releases. A branch holds all the software development assets for a single release. Occasionally, you must merge updates from one release to another, but they usually never merge. You discontinue a branch when you discontinue its release.

The mechanics of branching in TFS are simplified to a single right-click à Branching and Merging | Branch command.

Figure 6: Courtesy: ALM Rangers Codeplex Branching guidance

If you have more than one work (DEVELOPMENT) branch, forward integration to all work branches should occur as soon as any branch integrates into the MAIN branch. Because the MAIN branch is kept stable, forward integration is safe..

Refer to http://msdn.microsoft.com/en-us/library/ee782536.aspx and http://tfsbranchingguideiii.codeplex.com/ for detailed guidance from ALM Rangers team

- Ensure Reverse Integration and Forward Integration process when multiple teams (Onsite Offshore are working together on same engagement)

FORWARD INTEGRATE (FI) – merges from parent to child branches.

REVERSE INTEGRATE (RI) – merges from child to parent branches

Refer to http://branchingguidance.codeplex.com/wikipage?title=Code%20Promotion%20Best%20Practices

- Gated Check-ins: Ensure the solution is built before the automation code check-in

By using a gated check-in for the MAIN branch, you help make the reverse integration much easier because quality gates help avoid conflicts or errors in the MAIN branch. For more information, see http://msdn.microsoft.com/en-us/library/dd794776.aspx and http://blogs.msdn.com/b/patcarna/archive/2009/06/29/an-introduction-to-gated-check-in.aspx

http://intovsts.net/2010/04/18/the-gated-check-in-build-in-tfs2010/

When this trigger is selected for a build definition, any check-in that is made to a file that is mapped in the associated workspace will trigger a verification build. If the build succeeds, the changes will be submitted to the repository. If the build fails, the changes are not allowed to be submitted and must be fixed and resubmitted.

iv. Coded UI Frameworks

v. Managing common libraries in TFS

All common code and libraries to be maintained at a centralized node which wil be accessible (read-only) to all the projects/apps to be included for reusability.

Note: Only the automation group member will have the add/edit permission on this node to control the change.

Common TFS Project & Node: < placeholder >

Library Path: Automation/Libraries

vi. Data driven testing

Use below link if you need more information about Data driven testing using Coded UI and access/store data in MS EXCEL from Coded UI

Data Driven Testing by Mathew

http://msdn.microsoft.com/en-us/library/ee624082(VS.100).aspx

http://msdn.microsoft.com/en-us/magazine/cc163536.aspx

vii. Managing object repository using UI Test Editor

This is a new feature launched in FP2 which lets you better manage your object repository.

The Coded UI Test Editor toolbar offers the following commands:

1. Find helps to locate UI actions and controls.

2. Delete removes unwanted UI actions.

3. Rename changes the names for test methods and controls.

4. Properties Opens the Properties Window for selected item.

5. Split into a new method lets you modularize the UI actions.

6. Move Code adds custom code to your test methods.

7. Insert Delay Before adds a pause prior to a UI action specified in milliseconds.

8. Locate the UI Control identifies the location of the control in the UI of application under test.

9. Locate All helps to verify control property and significant changes to the application’s controls.

Refer to: http://msdn.microsoft.com/en-us/library/gg269469.aspx

viii. Extending Coded UI

VS 2010 Coded UI support extensibility for technologies which are not supported out-of-the-box like Excel or custom controls.

Refer this example for WPF calendar support

http://blogs.msdn.com/b/tapas_sahoos_blog/archive/2010/12/10/adding-new-control-support-in-a-coded-ui-test-plugin.aspx

ix. Support & Help

- Discussion forum (External)

http://social.msdn.microsoft.com/Forums/en/vsautotest/threads

1. Abstract

2. Prospects Hot Candidates

3. Proposed Solution

4. Process: Intelligent Sampling Process Flow & Engine

5. Sampling Algorithms selected for Database Sampling

6. Cost/Benefit Analysis for the selected algorithms

7. Extended Features

8. Benefits

9. References

1. Abstract

   When the data volume is not too huge, the entire data of the population can be validated as a part of the test phase (Unit testing/ Integration Testing/ System testing). Such a total study gives an excellent confidence on the quality and coverage which can be very expensive method, because validating all the data involves huge hardware cost, resource dependency and the time required for the completion of the activity. Remember, too, that the objectives of a project do not always require an absolutely exact account of the entire population.

   Sampling is one approach which can be adopted when the data is voluminous. Note that sampling does not mean that you were not equally interested in all the items in the population. On the contrary, you would like to study all of them, but you pick the sample for practical reasons. Perhaps you have a population of millions of objects and it is impossible to reach even a major part of them. Also in those cases (with populations of, say, up to 10,000) where you might choose to study every object, the sampling study may be a prudent choice, because it saves your time and you can then use the time you save to study the sampled items more carefully.

   

   Few months back, we were in the middle of testing our application, which processes more than 1.50 TB of data and takes approx 36 hours of time when we received this great news that Microsoft was voted as the Best Employer by BT TNS Mercer’s Best Companies to work for in India & it was no doubt a moment of pride but i wasn’t interviewed and none of my friends were either but it struck us how they were still able to echo our feelings so well (for most of us i spoke to were not surveyed)

   After thinking it over for a couple of days we realized that this very same concept is very much applicable to our IT solution as well. Why do we have to process 1.5 GB of data and wait for 36 hours to run or why should i make up the data by generating it randomly and taking the risk of compromising on the coverage when i have access to customer/ production equivalent data (non PII)?
   
   The Big Question:
   “Why didn’t we do intelligent sampling yet?”

   

   The Big Question:
   “Choosing Sample Test Data from Population/ Customer data is not a challenge but how do you ensure that sample is a GOOD sample”
   I agree, “GOOD” is a quite subjective term. Let me rephrase
   “How do you ensure that the sample chosen is a GOOD representative of your entire data population?”
   

2. Prospects Hot Candidates:

   Bigger the Size of font in the image below, more is the importance & applicability.
   

3. Proposed Solution

   Best Sample selection based on your requirement is the true objective. There are multiple solutions in silo to cater to specific scenarios.

   What better place to have this solution than in Visual Studio DbPro
   

   

4. Process: Intelligent Sampling Process Flow & Engine

   

   1. Selection Criteria

      i) Population Selection

      The data set which needs to be sampled should be chosen first. This can be chosen based on the parameters of

      1. Size – the dataset for sampling should be of reasonable size (>500 MB). This is to make the process of data sampling worth the effort
      2. Complexity – If the dataset to be sampled is of lesser complexity, data generation approaches may be more suitable. It is for complex data set which involves some business rules plus data integrity to be maintained where subset data selection would help.
      3. Test cases to be run – Subset data selection is for a scenario where number of test cases to be run is high and considerable amount for data regression is required
      4. Variety in data – The dataset to be sampled contains a variety in data. For Eg. Databases of an enterprise system would be ideal. On the other hand one which contains only Taxonomy related data or domain values may not be suitable for sampling

      ii) Business Rules / Requirements or user input / Sampling Frame

      Whenever a sample data is selected from an existing database, one needs to ensure that the underlying business rules are intact. In order to cater to negative test case scenarios, some sample breaking the rules is also fine. However on the whole, business rules have to be maintained.

      Most datasets where sampling is required also have some ‘business rules’ attached to them. For Eg. Expense related calculations may need checks of exchange rates; billing related calculations may need checks of status and so on. If there are business rules which are critical to the transformation of the system to be tested, these can be input by the user, to ensure that the sampling happens accordingly.

      iii) Get Expected Cost Associated From Sample from the User.

      The user should also be able to select an expected associated cost to the sample which would be created. This cost would be based on the size/time taken for the sample generation and coverage based on data variety included in the sample. The expected associated cost entered by the user would be used to prioritize on the sample algorithms to be run, which are known to provide the expected cost. Running these algorithms with the expected cost as an input would help reach the required end result and propose different data sampling options to the user.

   2. Sampling Method

      i) Data Profiling Required - Domain/Transactional Data

      In order to go for effective sampling, we also recommend categorizing the existing dataset into tables containing ‘Master’^* data and those containing transactionaldata.This could be an outcome of the Data Integrity relationship detected, or a step within the sampling engine while applying the algorithm. This categorization helps in selecting required amount of samples from each of these tables as well as selecting an appropriate sampling method. For instance, if a table contains date/calendar information, thus making it fall under a master category, would not need to be sampled or reduced; where as one containing daily transactions of an account can be a candidate for sampling. Detection of data set impacting the coverage is critical before coming up with the sampling plan.This would be done based on

      1. Profiling the data and checking for columns which contain distinctively unique values which are sure to impact the test case/data coverage.

      2. Get input from the user based on his prior knowledge.

      Once this input is provided, save it to use during generating the sampling plans.

      ii) Generate Plans with associated cost

      Generate the different plans possible for data selection and provide the‘cost’ associated with it based on the rules applied prior to sampling This cost would be calculated based on following factors:

      • Expected size of the sample
      • Expected data coverage of the sample.

      The plans along with the indicator of cost associated with each of the plan would be provided to the user. Thereafter, the user could either choose the best recommended plan, or go for another based on his discretion.

      

      iii) User to choose plan or go for regeneration

      If the user is not satisfied with the plans and their cost, he can ask for a regeneration of plan which would involve trying out a different set of inputs or providing different columns for driving data coverage.

   3. Sampling Engine

      The sampling engine is expected to perform all of the above steps after scanning through the target population, as well as consuming specific inputs provided by the user. Next Steps are:

      i. Implementing the sampling plan

      ii. Reviewing the sampling process

      The sampling engine is the core of the data selection process. Stages involved within the sampling engine:

      i) Building the Sampling Rules

      • The sampling rules would be built based on the Data Integrity Checks run on the database as well as after profiling the data for any exceptions to business rules and categorizing data sets into master or transactional.
      • This could also include setting any delta rule condition (or in other words reducing the scope of data to be sampled
        When a Data Volume is sampled already, the next time the Sampling Engine Runs on the same dataset, the sample is generated for the new added data rows only. This would save time, effort and money and would give more ROI to the proposed feature inclusion in VSTF DB Pro.
      • These sampling rules would form the underlying conditions which need to be satisfied by the selected database.
      • This ensures that after selecting a sample from the source system, when we run our ETL process on it, the ETL should not break due to incorrect sample set.

      ii) Apply algorithm

      • The algorithms chosen for application within the sampling engine are covered in next section (Section 7). Each of the above algorithms would be applied to the source system data and cost for the same would be calculated

      iii) Calculate Sampling Cost :

      We define sampling cost based on two parameters

      • Sample Size
      • Data coverage

      The Sampling Cost would be calculated based on the Size (This could be Row Count of each table available in the sample or the actual Size in MB/GB). This percentage reduction in Size would act as one parameter in Cost. Data Coverage in the sample would be detected differently by differently algorithms. If the algorithm is probabilistic, frequency of data would be a key in determining coverage, if statistical algorithm is used, distinct values in columns would form key criteria and in T-Way, the requirements would be primary. This would be covered in detail in the subsequent section.

      iv) Check for correctness and update the Sampling Rules

      Identify if the sample satisfies the basic rules and if the user wishes he/she could opt for re-sampling, and this time fine-tune the rules as well as inputs provided.

      v) Log Errors

      Any errors encountered during the process of profiling/sampling method generation would have to be logged and reported.
      The diagram below shows an end-to-end mock of the screens that could show up during this process of sampling method generation and selection and the various inputs that need to be provided by the user and the actions that take place.
      

5. Sampling Algorithms selected for Database Sampling

   Within any of the types of frame identified above, a variety of sampling methods can be employed, individually or in combination. Factors commonly influencing the choice between these designs include:

   • Nature and quality of the frame
   • Availability of auxiliary information about units on the frame
   • Accuracy requirements, and the need to measure accuracy
   • Whether detailed analysis of the sample is expected
   • Cost/operational concerns

   a) Probabilistic Sampling Methods - Multi-Stage Sampling

   In most real applied application, we would use sampling methods that are considerably more complex than simple methods like (random sampling, cluster, stratified, systematic etc). The most important principle here is that we can combine the simple methods in a variety of useful ways that help us address our sampling needs in the most efficient and effective manner possible. When we combine different probabilistic sampling methods, we call this Multi-Stage sampling. By combining different sampling methods we are able to achieve a rich variety of probabilistic sampling methods that can be used in a wide range of social research contexts. This leads to less Sampling Error by relying on multiple randomizations by choosing random samples of preceding random samples.

   3-Stage Multi-Stage Sample Process

   1. Select the clusters from the population
   2. Select all strata from each clusters
   3. In next stages, select additional random sample from selected strata units and so on
   4. Finally all ultimate samples selected are chosen

   b) Statistical Sampling – Purposive Sampling

   The difference between non-probability and probability sampling is that non-probability sampling does not involve random selection and probability sampling does. Most sampling methods are purposive in nature because we usually approach the sampling problem with a specific plan in mind. In purposive sampling, we sample with a purpose in mind. We usually would have one or more specific predefined groups we are seeking. Useful when you need to reach a targeted sample quickly and where sampling for proportionality is not the primary concern.

   For ex. have you ever run into guys in a mall asking people if they could interview them? Most likely they are conducting a purposive sample. They might be looking for Caucasian females between 30-40 years old as that’s their need and they do that by estimating

   One caveat is that you are also likely to overweight subgroups in your population that are more readily accessible.

   c) T -Way Sampling Method

   T-way Testing is a type of interaction testing which requires that for each t-way combination of input parameters of a system, every combination of valid values of these t parameters must be covered by at least one test case. It involves selecting test samples in such a manner that it covers all the t-wise interactions between the parameters and the possible values of a given system.

   For example, Let us consider Employee Entity which contains attributes like Employee Discipline and Employee Role. Also, consider the Country Entity has attributes like CountryName and IsActive. Correspondingly, the possible values Employee Entity and Country Entity are as shown below.

   

   The total Number of Combinations possible for values between Employee and Country = 3*2*3*2 = 36 combinations. We can also say, that our code will work if I have 1 combination, but neither of them strikes an optimal balance between Quality Assurance and Delivery!

   Applying the t-way testing methodology and setting t=2, we get the Dataset which has only 9 combinations

   
   Similarly, when t=3, the number of combinations increase but the data coverage also increases and hence the sample size increases, and it strikes an optimal balance between the size of the dataset and the code coverage. A sample run would generate 14 records for the current example.
   We propose, set t=3 to find out the optimal data combination both in terms of coverage and time.

6. Cost/Benefit Analysis for the selected algorithms

   This section will be used internally by the system to determine best algorithm for a selected population by doing a cost/benefit of each algorithm statistically

   a) Cost Benefit for Statistical Algorithms

   
   

   b) Cost Benefit for Statistical Algorithms

   
   

   c) Cost Benefit for T-Way Algorithm

   

7. Extended Features:

   7.1 Avoid Re-sampling of the Sampled Data to increase data coverage

   Sampling schemes should be generally defaulted to Without Replacement (‘WOR’). No element should be re-selected more than once in the subsequent sampling attempts if it’s already sampled in one of the past attempts (unless RESAMPLI NG = 1 is explicitly configured by the user). By Default, Re-Sampling Flag is set to 0 to ensure that already sampled data doesn’t always get picked in every sample run and rest of the data should get a fair chance of getting selected.

   For example, if we catch fish, measure them, and immediately return them to the water before continuing with the sample, this is a WR design, because we might end up catching and measuring the same fish more than once. However, if we do not return the fish to the water (e.g. if we eat the fish), this becomes a WOR design.

   Marking already sampled data helps ensuring that the already sampled data is not picked in the future runs unless we run out of data. This basically follows the assumption that the sampling method chosen by the user / system picks the best available sample for the first run but say for the second run we don’t want to pick the already sampled data (selected based on run #) and so on.

   

   7.2 Weighted Sampling

   Weights can be provided by the user in the proposed user configuration settings to give more value / importance to a specific column which is more important based on the business need.

   

   More generally, data should usually be weighted if the sample design does not give each individual an equal chance of being selected. In many situations the sample fraction may be varied by stratum and data will have to be weighted to correctly represent the population.

   Real World Example, a simple random sample of individuals in the United Kingdom might include some in remote Scottish islands who would be inordinately expensive to sample. A cheaper method would be to use a stratified sample with urban and rural strata. The rural sample could be under-represented in the sample, but weighted up appropriately in the analysis to compensate.

8. Benefits

   The following are direct /indirect benefits to the various stakeholders like Dev, Test, PM’s & Business Community.

   • Overall Test Execution Time Reduction
   • Higher Requirement & Test Coverage
   • Higher Code Coverage
   • Effective Unit Testing & Functional Testing -Boundary Values, Equivalence Partitioning
   • Reduces Hardware Cost for IT
   • High confidence while delivering to Test, UAT & Production
   • Increased Domain knowledge & business understanding of the resource while choosing & profiling the sample data
   • Earlier detection of potential Data Quality issues by analyzing the data (while profiling) for sampling

9. References

   http://en.wikipedia.org/wiki/Sampling_%28statistics%29

   http://doi.acm.org/10.1145/1102351.1102456

   http://www.ai.mit.edu/projects/jmlr/papers/volume3/scheffer02a/scheffer02a.pdf

   http://www.uiah.fi/projects/metodi/152.htm

   http://en.wikipedia.org/wiki/Quantitative_marketing_research

   http://en.wikipedia.org/wiki/Expectation-maximization_algorithm

   http://www.gabormelli.com/RKB/EM_Algorithm

Authors

Raj Kamal (rajkamal)

For all deliverables & details: http://webservicestesting.codeplex.com

Authors

Raj Kamal (rajkamal@microsoft.com) ,

Vipul Mehta (vmehta@microsoft.com ) ,

Ranjit Gupta (rankumar@microsoft.com )

Gunjan Jain (gunjain@microsoft.com )

Microsoft Services - Global Delivery, India

Objective

This artifact will provide Web Services Automation guidance for Functional & Performance Testing leveraging VS 2010. The document will provide recommendation on using either Unit Test or Web Test for functional testing depending on your project and test objective as will be later covered in the document. Load Test will be used for Performance Testing in both cases.

The primary objective is to propose a viable web services test automation approach & reusable libraries to customers, partners and consultant. It will also be a step in standardizing web services automation using VS 2010 platform across the Industry and filling the gaps by extending our tools as suggested in the talk. The document can also be used as a quick jump start guide for beginners to get started on Web Services Automation Testing.

Other than the guidance included in this document, key deliverable will primarily consists of the following:

a) Unit Test & Web Test Template projects for Web services testing

b) Reusable libraries for Unit Test

c) Reusable Add-ins (Extraction & Validation Rules) for Web Test that can be reused to reduce automation development effort drastically.

d) Performance Testing using Load Test once the Web Tests/Unit Tests are in place.

e) Sample Web Services application for beginners to gets started

Pre-requisites: Ensure you have downloaded and deployed WebServiceAutomation_SampleApp that contains sample Web services projects (WCF/ASMX/WF etc.) that you can use to getting started with Web Services automation. Verify you have downloaded our sample Template projects WebServiceAutomation_SampleUnitTest and WebServiceAutomation_SampleWebTest that contain sample unit test/web test which you can reuse / customize for your needs Ensure you have downloaded our Reusable Libraries for Unit Testing (WebServicesTesting.Automation.UnitTest.Helper.dll) & Web Test (WebServicesTesting.Automation.WebTest.Helper.dll) Visual Studio 2010, SQL Server Express or higher version, IIS (Internet Information Server 6.0 or above

Importance of Web Service Automation

It’s important to first understand the importance of Web Services Automation and the ideal scenarios where it must be recommended.

Web Services (WCF/ASMX) due to their worldwide implementation and popularity but limited test tool support, often raises the question around testing them effectively. Testers are craving to find effective ways to Test Web Services. Even VS 2010 doesn’t have dedicated test types for verifying Web Services (like for UI, there is Coded UI and for DB there is Database Unit Test) which results in low test coverage and non-standardization of proven ways to test web services especially WCF 4.0 Web Services. Web Services Testing in VS 2010 so far has been not been given a first-class citizen treatment and this guidance will bridge that gap by making it much easier and effective.

· In this agile development era, we are moving to a model where UI is ever changing and consummation of web services is taking the center stage and hence this layer attention it truly deserves by shifting the onus from UI testing to Web Services based validation.

Traditionally teams have also used Fiddler and other 3^rd Party tools for the same purpose (many times for manual testing alone) but as Web services have different needs, the knowledge harvested doesn’t get reused to its potential in future engagements

Ideal Candidates

· If your application is consuming APIs exposed by other upstream systems or 3rd Parties

· If your application is exposing APIs to be consumed by the downstream applications

· When your UI is dynamic and constantly changing and cost/maintenance of UI automation is much more.

· When your business logic and rules are primarily implemented using services (ASMX, WCF & Workflow Services) and testing at this layer is much faster and efficient.

· When you need to come up with a common automation framework for web services that can be used to validate both functional and performance aspect of the web services with minimum effort.

Test Approach – Identify your goal (Functional Test or Performance Test or Both)

When we understand and realize the importance of Web Services Automation, the next challenge is to select the right Test approach and Test Type before jumping into automation design and implementation.

Test Objective

It is important to answer the basic question “Why do you want to do Web Services Testing?” and that will help you decide the right kind of Test you should choose to automate them. Let’s begin with few possible answers

To validate the functional behavior of your application / APIs

To validate the performance aspect of your application/ APIs

To first validate the functionality and then also test the performance aspect of it. (a) + (b)

Now when you know about your test objective, you should also look at the technologies being used for your Web Services as that can make a difference as well. If you use Microsoft platform, chances are your services are built on:

ASMX – Legacy aps

WCF – New apps

WF – Window Workflow Foundation

Choosing the appropriate Test Type (Unit Test or Web Test)

Please find below the recommendation on the type of test you should decide depending on your test objective defined in the section above

Unit Test – provides you with flexibility to directly access your Web services APIs programmatically using familiar c# and test their behaviour based on your requirements, business rules etc. We will be covering this in depth in the section below

Web Test – also lets you invoke your web services methods, though not through better known record and playback feature of Web Test but by directly adding services method call and parameters, properties. We will be covering this in depth in the section below

At a high level, you can say that unit test is more of glass box testing whereas Web Test gives you ‘black box’ view.

As shown in the image below, Unit Test is an excellent choice when:

You have complex web services involving lot of business rules and logic which you need to validate and primary focus is on detailed functional validation

You are using WCF based services that includes RESTFul API, services returning non-xml kind of data

Web Test is a better choice when:

You have relatively simpler web services where you objective is to test simpler / straightforward functionality and behaviour and primary focus is to test their performance down the line

You are working on legacy app that are based on ASMX kind of services which have a better support in Web Test

Test Types vs. Test Objective Recommendation

Sample Lab Application: Web Service Automation

As many of you might be new to Web Services automation, we have created and shared sample Web Services applications that will help you better understand our helper functions/ Add-ins and sample unit test/web test shown later in the document.

If you already have sample Web Services deployed in your machine, you can skip this section.

Sample Application: Student Administration This is a sample Student Administration application that comprises of simple functionalities around enrolling students and displaying their details back to the user. The UI app is built using Win forms and services are implemented using both ASMX and WCF for demonstration purpose. There are other sample projects to demonstrate features like Authentication, Caching, Workflow, Asynchronous calling etc.

Sample Student Administration Application UI

Sample Application project structure in TFS

Please refer to Appendix: Deploying Sample Web Services Application to know more about our sample application and instructions around deployment.

At this point we assume you have deployed our sample web services application. In subsequent sections, we will present the various approaches and steps involved in Web Services Automation Design, Development & Execution that you should take to achieve your objective while following the automation standards and guidelines that are recommended.

Web Services Automation – Unit Test Walkthrough

If “Unit Test” was selected as the Test approach for Web service automation based on the analysis and your test objective as covered in above section: Test Approach then proceed with the walk through.

Creating a new Unit Test Project / Reusing Sample Template Unit Test Project

You can create a new project from scratch or using our existing sample template project but it’s important to add your project to source control for version management and change tracking.

SL No.	Topics	Description
1.	Have you identified & marked the automatable tests as PLANNED in Microsoft Test Manager (MTM)	Open your test cases in Test Manager that you want to automate and mark Automation Status = “Planned” for those     Whereas the test which are not good candidates for automation are marked as “Not Automated” to show that they are out-of-scope for automation.    This is an optional step if you don’t want to run your test from Microsoft Test Manager and map your automation to manual test cases. We however recommend this.
2. a	Have you created new Test project & Added it to Source Control (TFS)?	Every automation project must be source controlled.   Add project to Source Control by clicking the check box as shown and give appropriate name.
2.b	Alternatively, Open existing Test project and using it as Template	Launch Visual Studio, Browse and open existing ‘WebServiceAutomation_SampleUnitTest’ template project and add it to Source control. You can change the properties as per your needs. For this sample template project we have SampleUnitTestCases.cs that contains all our sample reusable unit test cases. Please find below a short description on the folder structure of the template project which will be used for case study: 1. Config: All your application configuration details that you need to run your test will go to app.config Example: Also refer to: http://msdn.microsoft.com/en-us/library/a65txexh(v=VS.100).aspx 2. Common Libraries References: Add references to our reusable helper library “WebServicesTesting.Automation.UnitTest.Helper.dll” for Unit Test Web Services Testing (provided as part of the release and documented below) . 3. Service References: Add / update services reference to point to your web services and provide a logical name   4. Scripts: Your actual test are stored in these .cs files along with components like Test Initialize, Test Cleanup etc You can create a folder structure to organize your test methods by grouping them in multiple class files depending on the size of the project. It is advisable to group test belonging to same feature/functional module to be grouped together in a .CS file and given an appropriate name. 5. Solution Configurations & Compilation Symbols With our template solution you get additional solution configuration which will help you run your same test for both functional and load test seamlessly by controlling the sections of the code that you may want to execute for functional and load testing For example. When you do load test, you might not want to do certain validation which makes sense for functional testing alone and these configurations will control that by just selecting the appropriate solution configuration as shown below. a) Custom Solution configurations b) Configuration – Functional Testing As shown below it uses a compilation symbol #WEBSERVICES_FUNCTIONAL_TESTING that is used in our test methods to control the behavior. c) Inside your test method, you will have code like this to control the behavior based on the solution configuration file that is selected For more reference: http://www.codeproject.com/Articles/49448/Use-Visual-Studio-Conditional-Compilation-to-manag.aspx

Unit Test Automation Design & Development (WCF & ASMX based services)

Now that you have your test project in place and our sample reusable unit test cases for automating common web services test scenario, in this section we will be walking you through few sample reusable tests that you can customize it for your own needs or create new unit test for your project scope. The approach shown below would remain same for WCF as well as ASMX based services and we have shown examples of both.

Follow the below steps for Automation design guidance:

SL No.	Steps	Description
1.	Global Settings	a) Set the test context which provides information about and functionality for the current test run. b) Test Initialize: To perform steps before running each test like instantiating objects, defining global variables including connection strings etc. c) Test Cleanup: To run code after running each steps
2a.	Add New Test Methods / Reuse Existing Template Test Methods (Simple Lookup Verification)	Sample Test scenario: Verifying Web service call response value against expected results Below is sample data driven test method for web services testing that comes with our template project for Unit Test. You can add your new test methods by modeling the below example. Step 1. Create a new Test method / Modify our existing sample test method Step 2. Instantiate and invoke your web service method (in this example: ASMX based service) as shown in the code snippet below and get the results in a variable of appropriate data type In our sample example, the web service returns an integer datatype which we cast to String for comparison. Your services method can return any data type like int, double, decimal, bool, string etc and we can use the same helper method below to validate the response. Step 3. Get your expected results from data source of your choice as shown below. Step 4. Reuse our appropriate library / helper method for validating the web service response against expected results. In this example CompareString is a reusable method in our helper / library that can be called from your test methods to save effort related to validating the results and marking the test as pass/fail. Note: Depending on your test scenario, you need to choose different helper/library methods. The complete documentation about our reusable library method is given in sections below
2b	Add New Test Methods / Reuse Existing Template Test Methods (Simple Transactions Verification)	Sample Test scenario: Verifying Web service call results in database operation like Insert Below is sample test method when tester needs to validate if the web service call results into a Database Insert by validating the database for ensuring the transaction was committed. Instead of Insert, you might have different operations like update, delete etc. for which we have reusable helper methods in our library. Here the web service method Enroll_Student (in this case WCF based Service) is inserting a record into the database and the expected result is the query against the database to validate the operation to ensure transaction was committed. Reuse our appropriate library / helper method for validating the web service response against expected results. In this example Verify_DBInsert is a reusable method in our helper / library that can be called from your test methods to save effort related to validating the database insert and marking the test as pass/fail. Note: Depending on your test scenario, you need to choose different helper/library methods. The complete documentation about our reusable library method is given in sections below
2c	Add New Test Methods / Reuse Existing Template Test Methods (Handling Complex Object Types)	Use this approach as a generic way of comparing / validating web services response of various data types including user defined/abstract data types like Struct, Classes as well as primitive data types like Int, String, Dictionary, List etc. DTO (Data Transfer Objects) Having the results of a remote method available in an object instance makes it easy to pass this data to a test module or to compare it with the desired outcome It works by converting the service response (whether abstract or primitive data types) and expected results into standard generic objects which then are finally converted into XML for comparison with the help of our reusable library methods as described below in the example Sample Test scenario: Verifying Web service response which returns a Struct object against test data from database. Step 1: Create Data structure for your object which your service returns as shown below (In this case it returns a Struct which contain elements like First_Name, Last_Name etc. Step 2. This is an optional step for better organizing your code which invoked your web services and returns the response to the main test method for verification. Step 3: Add your test method Step 4. You can read your data source (in this case database) and convert it into generic DTO object using reusable method in our library called DTO_PopulateDBResults which takes Dataset as input parameter. DTO Type should be same as the type of your complex object (in this case Student which is struct) List<Student> ExpectedStudentDTOList = null; ExpectedStudentDTOList = helper.DTO_PopulateDBResults<Student>(dt); Step 5. In this step you call you Action code to get your web service response Actions.DTO_Action complexObj = new Actions.DTO_Action(); Object myObj = complexObj.ComplexObjectSample(); Step 6. Convert web services response into generic DTO using our generic library method called DTO_PopulateObject which accepts Object and structure of the object defined above as parameters. object ActualStudentDTOList = helper.DTO_PopulateObject<Student>(myObj, s1); Step 7. Finally the generic DTO objects returned from expected results i.e. Test data source and Web services response are converted into XML and compared using our reusable library method called DTO_XMLComparison and accordingly marks the test method as pass/fail bool result = helper.DTO_XmlComparison(ExpectedStudentDTOList, ActualStudentDTOList, typeof(List<Student>)); Assert.AreEqual(true, result, "Expected does not match actual"); Step 8. Expected and Actual XML files with the test method name are created under your folder c:\TestResults for your future reference. Please create this folder if it doesn’t exist
3.	Make your scripts data-driven Irrespective of kind of test methods you have created above	Data-Driven Test Ensure every variable is parameterized by passing the test data to each test using data-driven capabilities in VS 2010. Right-click on the test method and select Properties, to view the properties of the test To start making this a data-driven test, we have to first create a data source for the test to draw from. On the Data Connection String property, click the ellipse button to start the New Test Data Source Wizard There are three options for creating a data source: Database -- which allows you to connect to a database or Microsoft Excel spreadsheet CSV File -- which allows you to use a comma-delimited file XML File -- which allows you to use an XML file When you add a data source A Data Source attribute is added to the Test method. [DataSource("System.Data.SqlClient", "Data Source=(local);Initial Catalog=Sample_DB;Integrated Security=True", "Sample_App", DataAccessMethod.Sequential), TestMethod]
4.	Map your automation test to your test cases in MTM	This is an optional step if you don’t want to run your test from Microsoft Test Manager and map your automation unit test to manual test cases. We however recommend this.

Repeat the above steps for all your test scenarios against your web services to be tested. After you write and debug your scripts its important to run them in a batch or as part of a suite. There are various way to do that and factors to keep in mind as we will see in next section.

Web Services Automation – Web Test Walkthrough

If “Web Test” was selected as the Test approach for Web service automation based on the analysis and your test objective as covered in above section: Test Approach then proceed with the walk through.

Creating a new Web Test Project / Reusing Template Web Test Project

You can create a new project from scratch or using our existing sample template project but it’s important to add your project to source control for version management and change tracking.

SL No.	Topics	Description
1.	Have you created new Test project & Added it to Source Control (TFS)?	Every automation project must be source controlled. Add project to Source Control by clicking the check box as shown
2.	Alternatively, Open existing Test project and using it as Template	Launch Visual Studio, Browse and open existing “ WebServiceAutomation_SampleWebTest” template project and add it to Source control. You can change the properties as per your needs. Please find below a short description on the folder structure of the template project which will be used for case study: 1. Common Libraries References: a) Add references to our reusable helper library “WebServicesTesting.Automation.WebTest.Helper.dll” for WebTest based Web Services Testing (provided as part of the release and documented below) . b) Reusable Validation Rules c) Reusable Extraction Rules 2. Service References: Add / update services reference to point to your web services and provide a logical name 3. Scripts: Your actual test are stored in these .webtest files. You can create a folder structure to organize your test methods by grouping them in multiple class files depending on the size of the project. It is advisable to group test belonging to same feature/functional module to be grouped together in a .CS file and given an appropriate name. For this sample template project we have many reusable webtest that can be reused / customized for your needs. 4. Test Data: You can choose to have you test data as CSV or XML or Database as support by Visual Studio Testing framework for making your test data driven.

WebTest Automation Design & Development (WCF & ASMX based services)

Now that you have your test project in place, in this section we will be walking you through our sample reusable Web test for automating common web services test scenario that you can customize it for your own needs or create new additional Web tests for your project scope.

Follow the below steps for Automation design guidance:

S No.	Steps	Description
1a.	Add New Web Test / Reuse Existing Template Web Test (Dataset Verification for WCF/ ASMX Service)	Sample Test scenario: Verifying Web service method that returns a dataset against the expected test data from Database If you are adding a new web test from scratch then follow the steps: Step 1. Open your web service method request XML by launching WCFTestClient utility or any other way you are comfortable with. Step 2. Add a new Web Test or Reuse our existing template Web Test and customize them Step 3. Right click the node and press “Add Web Service Request” and provide your web service URL     Step 4. Right click the node and press “Add Header” and provide SOAPAction value as shown below by retrieving it from request XML of Step 1. Step 5. Add <string body> a) From your request XML as shown below, extract Envelope and Body (you shouldn’t copy Header part) <s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"> <s:Body> <Display xmlns="http://tempuri.org/" /> </s:Body> </s:Envelope> b) Add this to to you string body now Step 6. Add our reusable Validation and Extraction Rule called DatasetValidation that reads the expected data from database using SQL query and validate it against the response from web services by using XPathExpression Note: Depending on your test scenario, you may need to choose different Extraction and Validation Rule complete documentation about our reusable library method is given in sections below
1b.	Add New Web Test / Reuse Existing Template Web Test (Method Prototype Verification for WCF/ ASMX Service)	Sample Test scenario: Verifying Web service method parameters and response values against expected test data. Repeat Step 1 to 5 from 1 a) above as explained Step 6. Add our reusable Extraction Rule called ParameterExtraction and provide XPathExpression for example Enroll_Student and it will retrieve all the parameters name and their respective values and store it inside ContextParam Step 7. Step 6. Add our reusable Validation Rule called ParamValidation and provide ParamValues that contain expected test data as shown below along with contextParam name as defined in Step 6. Step 8 & 9. As shown in this web test you can also add our reusable ServiceResponseExtraction and ServiceResponseValidation rules from our helper class for validating the response values against expected test data. Step 10. After you run this web test, the result will show Pass / Fail with context tab showing the discrepancies if any. Note: Depending on your test scenario, you may need to choose different helper/library methods. The complete documentation about our reusable library method is given in sections below
2.	Make your scripts data-driven	Data-Driven Test Ensure every variable is parameterized by passing the test data to each test using data-driven capabilities in VS 2010. To start making this a data-driven test, we have to first create a data source for the test to draw from. On the Data Connection String property, click the ellipse button to start the New Test Data Source Wizard There are three options for creating a data source: Database -- which allows you to connect to a database or Microsoft Excel spreadsheet CSV File -- which allows you to use a comma-delimited file XML File -- which allows you to use an XML file

Repeat the above steps for all your test scenarios against your web services to be tested. After you write and debug your scripts its important to run them in a batch or as part of a suite. There are various way to do that and factors to keep in mind as we will see in next section.

Automation Execution & Deployment (Unit Test/Web Test)

Ensure you have configured test settings to capture logs, code coverage, test impact etc. and you also need to keep in mind that your script should be able to successfully run in multiple environment (for ex. Test to UAT)

After automation design and debugging is completed, you can now run and deploy your automation scripts using the below steps:

SL No.	Areas	Description
1.	Configuring Test Settings	Test Settings a) Data and Diagnostic: Enable appropriate diagnostic are selected as shown below (Code Coverage, Event Log, Test Impact etc.) b) Define other settings like Roles, Test Timeouts, Setup and Clean scripts
2.	Command Line Execution (MSTest.Exe)	Grouping & Classifying Test Methods using various parameters It lets you run groups of tests based on their assigned categories without the requirement to maintain test lists. Go to Test View and you can filter and view your tests by setting these parameters MSTest.EXE Options: MSTest.exe is the command-line command that is used to run tests. This command has several options that you can use to customize your test run. You can specify these options in any order on the MSTest.exe command line. Example: mstest /testcontainer:"C:\ WebServiceAutomation_SampleUnitTest" /test:testproject32\generic Refer to http://msdn.microsoft.com/en-us/library/ms182489.aspx
3.	TCM.exe or Test Execution from Microsoft Test Manager (Optional though recommended)	a) If you have mapped you automation unit test with test cases in Microsoft Test Manager then you can use TCM.exe as shown below to run your web service test and it will also update your test cases results in MTM without any custom code. You don’t need to use MSTest.exe in that case Example: tcm run /create /title:title /planid: id /collection:CollectionURL /teamproject:project (suiteid:id /configid:configid | /querytext: query) [/settingsname:name] [/owner:owner] [/builddir:directory] [/testenvironment:name] [/login:username,[password]] [/include] Refer to : http://msdn.microsoft.com/en-us/library/dd465192.aspx & http://msdn.microsoft.com/en-us/library/ff942469.aspx b)You can also Run Automated Tests from a Test Plan Using Microsoft Test Manager. To run the latest version of your automated test, you must make sure that the correct build is selected for your test plan. You must also use a physical or virtual environment to run your automated tests. This is out of scope of this guidance however we recommend this the most. Refer to: http://msdn.microsoft.com/en-us/library/dd553270.aspx

Web Service Performance Testing using Unit Test/Web Test + Load Test

Now when you have your Web Services Test automated either using Unit Test or Web Test as suggested in this guidance above, you can go ahead and create you Load Test to performance test your web services.

SL No.	Areas	Description
1.	Configuration – Load Testing	a) Ensure you have selected our custom solution configuration for Load Test i.e. Debug – Load Test b) As shown below it uses a compilation symbol #WEBSERVICES_LOAD_TESTING that is used in our test methods to control the behavior. c) Inside your test methods, you will have code like this to control the behavior based on the solution configuration file that is selected
2.	Creating Load Test using Web Services Unit Test or Web Test (Designed in above sections)	Step 1. Add a new Load Test Step 2. Give a scenario name for your Load test Step 3. Specify your load pattern (Constant Load/ Step Load) Step 4. Choose an appropriate test mix Step 5. This is the most important step where you have an option to select Unit Test or Web Test to be added to your Load test depending on the kind of approach you had selected for your Web services automation above Press Add and select your “Web Services Unit Test” OR “Web Services Web Tests” created in the sections above that you want to performance test Press OK and adjust Distribution of tests. Step 6. Refer to below section for adding custom web services counters other than standard counters that you may want to monitor Step 7. Specify Load run settings Step 8. You load test is created Our sample Unit Test and WebTest template project has this reusable SampleLoadTest.loadtest sample load test for you to model your load tests around.
3.	Running Load Test and analyzing results	For Web services Load testing, we recommend you adding following custom counters against the web services server (where the web services are hosted): ServiceModelEndPoint 3 ServiceModelEndPoint 4 ServiceModelOperation ServiceModelServices Please follow Adding custom counters set using Load Test Editor for detailed steps. Step 1. Open you load test. Right click you counter set and add “Custom Counter Sets” Step 2. Right click and now click “Add Counters” Select Performance Category to include counters related to Web Services as shown below Step 3. Repeat it for all the categories as shown below. Tip: You can enable performance counters for a WCF service through the app.config configuration file of the WCF service as follows: <configuration> <system.serviceModel> <diagnostics performanceCounters="All" /> </system.serviceModel> </configuration> Sample Load Test Summary Report

Web Services Helper Documentation for Unit Test & Web Test Add-ins

This section provides documentation of the complete list of reusable helper functions (WebServicesTesting.Automation.WebTest.Helper.dll) and web add-ins (WebServicesTesting.Automation.UnitTest.Helper.dll) for your reference and usage. We have shown few of these in sample walkthrough above to give you a glimpse of the same.

Category	Title	Description	Unit Method Signature & Example invoke call	Web Custom Extension (Extraction & Validation Rules)
Lookup Verification	Verifies the Scalar value or multiple scalar values of even different data types returned by services against expected results	This method will let you compare the scalar values of different types like <INT>, <STRING> etc. returned by the service method call converting them to <string>against the expected values	public void CompareStringArray(int condition, string query, string[] ArrgetAgeActual) Example: obj.CompareStringArray(2, query, ArrgetAgeActual);	For scalar value Extraction Rule ServiceResponseExtraction Context Parameters 1.XpathExpression 2.ContextParam Validation Rule ServiceResponseValidation Context Parameters 1.Expected 2.ContextParam For multiple values Validation Rule DataSetValidation Context Parameters 1.ConnString 2.Query 3.XPathExpression 4.ContextParam The validation rule extracts the dictionary returned by web service from xml and compares with the expected dataset based on the query
Lookup Verification	Verifies the values of <DICTIONARY> data type returned by services against the expected results	This method will let you compare the <Dictionary> <key, value> values returned by service method call against the expected values <key, value>.	public void CompareDictionaryWithExpectedArray(Dictionary<int, string> ActualDic, string[,] ArrgetAgeExpected) Example : obj.CompareDictionaryWithExpectedArray(col_Symbols, ArrgetAgeExpected);	Validation Rule DataSetValidation Context Parameters 1.ConnString 2.Query 3.XPathExpression 4.ContextParam The validation rule extracts the dictionary returned by webservice from xml and compares with the expected dataset based on the query
Lookup Verification	Verifies the values of <LIST> data type returned by services against the expected results	This method will let you compare the values of data type <LIST> returned by service method call against the expected values	public string[] CompareList(string query) Example: string[] ArrgetAgeExpected = obj.CompareList(query);	Validation Rule DataSetValidation Context Parameters 1.ConnString 2.Query 3.XPathExpression 4.ContextParam The validation rule extracts the List returned by webservice from xml and compares with the expected dataset based on the query
Lookup Verification	Verifies the result set returned by services against expected results	This method will let you compare the result set returned by service method call against the expected result set	Method Prototype public void Verify_Dataset(string ConnString, string Query, DataSet dst) Method Invocation UnitTest help = new UnitTest(); help.Verify_Dataset(conn, query, set);	Validation Rule DataSetValidation Context Parameters 1.ConnString 2.Query 3.XPathExpression 4.ContextParam The validation rule extracts the dataset returned by webservice from xml and compares with the expected dataset based on the query
Lookup Verification	Verifies the row count of the values returned by Web Services against expected result	This method will let you validate the row count of the values/list/dataset returned by service method call against the expected row count	Method Prototype public void Verify_RowCountDataSet(int expected, int actual) Method Invocation UnitTest help = new UnitTest(); help.Verify_RowCountDataSet(expected, actual);	Extraction Rule RowCountExtraction Context Parameters 1.XpathExpression 2.ContextParam This method extracts the row count and stores the value to Context Param specified by the user. Validation Rule RowValidation Context Parameters 1.ConnString 2.Query 3.ContextParam 4.ContextParam This method validates the value set to the context param and the one returned by the specified query
Metadata Verification	Verifies the called web service method has schema as expected	This method validates the XML file to make sure it fits to your application's needs to prevent any exception in your code. To check whether the XML document conforms to an XSD Schema, the document must be validated against that XSD Schema	Method Prototype public bool Verify_Schema(string targetNameSpace, string xsdPath,string requestXml) Event Handler to raise error private static void ValidationEventHandler(object sender, System.Xml.Schema.ValidationEventArgs arg) Method Invocation UnitTest help = new UnitTest(); bool validation = help.Verify_Schema(targetNameSpace ,XsdPath,requestXml);	Validation Rule SchemaValidation Context Parameters 1.targetNameSpace 2. xsdPath
Metadata Verification	Verifies that web service methods exist and their parameters definition (like input parameters, return type) against expected result	This method will let you smoke test / BVT if the expected list of Web Service methods exists and their definition like input parameters and return type matches.	Method Prototype public void Verify_methodPrototype(object myObject, string Methodname, string paramlist, string returnTypelist) Method Invocation UnitTest help = new UnitTest(); help.Verify_methodPrototype(obj, methodlist, paramlist, returnTypelist);	Extraction Rule ParameterExtraction Context Parameters 1.XpathExpression 2.ContextParam This method extracts the parameter passed to the request and set to the contextparam Validation Rule ParamValidation Context Parameters 1.ParamValues 2.ContextParam This method validates the value set to the context param and the Values passed to the ParamValues The above extraction & validation rule is used to validate the parameter passed The below extraction & validation rule is used to validate value returned by the webservice Extraction Rule ServiceResponseExtraction Context Parameters 1.XpathExpression 2.ContextParam Validation Rule ServiceResponseValidation Context Parameters 1.Expected 2.ContextParam
Transactional Verification	Verifies that called web service method creates the output file as expected	This method verifies the output file (e.g. CSV, XML, TXT) has got created by the web service method call	Method Prototype public void Verify_FileCreation(string FilePath) Method Invocation UnitTest help = new UnitTest(); help.Verify_FileCreation("D:\\employees.xml");	Validation Rule FileCreationValidation Context Parameters 1.FilePath
Transactional Verification	Verifies that web service INSERTS/CREATES a new record into the database	This method verifies the web service call results into new record insertion in a given database	Method Prototype public void Verify_DbInsert(string ConnString, string Query, string expected) Method Invocation UnitTest help = new UnitTest(); help.Verify_DbInsert(conn, query, expected);	Extraction Rule ParameterExtraction Context Parameters 1.XpathExpression 2.ContextParam This method extracts the parameter passed to the request and set to the contextparam Validation Rule DbInsertValidation Context Parameters 1.ConnString 2.ContextParam 3.Query 4.ContextParam This method validates the contextparam and the values returned by the query
Transactional Verification	Verifies that web service UPDATES existing records into the database	This method verifies the web service call results into update of existing record(s) in the given database	Method Prototype public void Verify_DBUpdate(string ConnString, string Query, string expected) Method Invocation UnitTest help = new UnitTest(); help.Verify_DBUpdate(conn, query, expected);	Extraction Rule ParameterExtraction Context Parameters 1.XpathExpression 2.ContextParam This method extracts the parameter passed to the request and set to the contextparam Validation Rule UpdateDbValidation Context Parameters 1.ConnString 2.ContextParam 3.Query 4.ContextParam This method validates the contextparam and the values returned by the query
Transactional Verification	Verifies that web service DELETES records into a database	This method verifies the web service call results into deletion of existing record(s) from a given database	Method Prototype public void Verify_DBDelete(string ConnString, string Query) Method Invocation UnitTest help = new UnitTest(); help.Verify_DBDelete(conn, query);	Validation Rule DeleteDbValidation Context Parameters 1.ConnString 2.Query This method verifies the rows returned by query is 0
Workflow service	Verify that web service executed the workflow and end result is as expection	_________For this one we just need a test method in our test file instead of a separate helper method____	Sample code to verify the result returned by the workflow IDictionary<string, object> _d = new Dictionary<string, object>() { { "userArgument", arg } }; _d = WorkflowInvoker.Invoke(new Workflow1(), _d); Int32 outPut = (Int32)_d["OutputValue"]; Assert.AreEqual(expected, outPut); output can be validated with expected for its correctness	WorkFlow cannot be invoked. We can invoke the web service to verify the result returned by service method against the expected result. Extraction Rule ServiceResponseExtraction Context Parameters 1.XpathExpression 2.ContextParam Validation Rule ServiceResponseValidation Context Parameters 1.Expected 2.ContextParam
Authentication	Verifies that user credentials & certificate are validated against the server to authenticate the user trying to access the web service	This method verifies the user credentials and certificate to ensure that he is authenticated to get the results from the web service	var ServiceendPoint = new EndpointAddress(new Uri("http://localhost:91/Service1.svc"), EndpointIdentity.CreateDnsIdentity("MyWebSite")); var binding = new WSHttpBinding(); binding.Security.Mode = SecurityMode.Message; binding.Security.Message.ClientCredentialType = MessageCredentialType.UserName; var result = new Auth_Service.Service1Client(binding, ServiceendPoint); try { result.ClientCredentials.ServiceCertificate.Authentication.CertificateValidationMode = X509CertificateValidationMode.None; result.ClientCredentials.UserName.UserName = auth_username; result.ClientCredentials.UserName.Password = auth_passwd; time = result.GetServertime(); }
Encryption	Verifies that web service is encrypting the message/transport layer when returning data	This method will verify that web service is encrypting and decrypting the data as required by business.	Configuration appConfig = ConfigurationManager.OpenExeConfiguration(ConfigurationUserLevel.None); ServiceModelSectionGroup serviceModel = ServiceModelSectionGroup.GetSectionGroup(appConfig); Console.WriteLine("Configuration Name : Address : Binding : Contract"); foreach (ChannelEndpointElement endpoint in serviceModel.Client.Endpoints) { if (endpoint.Contract == "Auth_Service.IService1") { binding_used = endpoint.Binding; break; } else { } } Auth_Service.Service1Client result = new Auth_Service.Service1Client(); result.ClientCredentials.ServiceCertificate.Authentication.CertificateValidationMode = X509CertificateValidationMode.None; result.ClientCredentials.UserName.UserName = auth_username; result.ClientCredentials.UserName.Password = auth_passwd; string time = result.GetServertime(); TestContext.WriteLine("Bindings used by this contract is " + binding_used);
Caching	Verifies that data returned by called web service is returned from cache	_________For this one we just need a test method in our test file instead of a separate helper method____ Caching is frequently used as an optimization in distributed systems. It can be used to avoid re-doing computations or complex database access when the results remain valid for an extended period of time. In this case, subsequent requests for the same information can be served with the cached version rather than repeat the processing with the associated overheads.	string url = "http://localhost/Cache/Service1.svc/xml/12"; Uri uri = new Uri(url); WebRequest request = (HttpWebRequest)WebRequest.Create(uri); request.Method = "GET"; HttpWebResponse response = (HttpWebResponse)request.GetResponse(); Stream receiveStream = response.GetResponseStream(); StreamReader readStream = new StreamReader(receiveStream, Encoding.UTF8); string firstResponse = readStream.ReadLine(); Above code snippet shows to retrieve server response. To test caching two request should be made befor cache expires and should be validated.	Extraction Rule CacheExtraction Context Parameters 1.XpathExpression 2.ContextParam 3.Delay To test caching two request should be made before caching expire and one after caching expire. Parameter Delay has to be passed for the second request And plugin DelayPlugin should be associated with the second service CacheOutcomePlugin ContextParameters 1.ContextParam1 2.ContextParam2 3.ContextParam3
Caching	Verifies that data returned by called web service is returned from database when cache is expired	_________For this one we just need a test method in our test file instead of a separate helper method____	string url = "http://localhost/Cache/Service1.svc/xml/12"; Uri uri = new Uri(url); WebRequest request = (HttpWebRequest)WebRequest.Create(uri); request.Method = "GET"; HttpWebResponse response = (HttpWebResponse)request.GetResponse(); Stream receiveStream = response.GetResponseStream(); StreamReader readStream = new StreamReader(receiveStream, Encoding.UTF8); string firstResponse = readStream.ReadLine(); Above code snippet shows to retrieve server response. To test caching two request should be made before caching expire and one after caching expire and should be validated.
Non-XML Data	Verifies that called web service method sends/returns non-xml data (like blob, image) as expected	This method will verify that web service method is storing or retrieving the non XML data like blob from file system or database	Refer our reusable [TestMethod] public void VerifyNonXMLData()
Asynchronous messaging	Verifies that called web service method if implemented as asynchronous then returns the data as expected at a later point in time	This method verifies the asynchronous behavior of web service method as expected and implemented.	Refer our reusable [TestMethod] public void VerifyAsync_Response()
Exception and Error handling	Verify that service returns appropriate error message when there is a fault exception like underlying database connectivity issue to ensure exception and error handling	This method will verify that web service method that is called handles the exceptions like timeout, connectivity/networking issue as expected	<serviceDebug includeExceptionDetailInFaults="True" /> Set the IncludeExceptionDetailInFaults property to true to instruct to return managed exception information to the client in SOAP faults to ease debugging try { WCFServiceRef.Service1Client obj = new WCFServiceRef.Service1Client(); obj.Display(); } catch (FaultException e) { string str = e.Message.ToString(); TestContext.WriteLine(str); }	Extraction Rule ExceptionExtraction ContextParameter 1.contextParam Validation Rule ExceptionValidation ContextParameter 1.contextParam 2.ErrorMsg ExceptionPlugin ContextParameter 1.contextParam Plugin is used to pass/fail the test case contextParam name provided in extraction/validation/plugin should be same
Merge	Verifies the result obtained from multiple web services calls by merging them matches against consolidated expected output	This method verifies when similar result is obtained from multiple web services and then need to be validated against single expected result set Think of a scenario where you are getting similar result from various source - each a different web service call. The tester wants to validate the consolidated output from these services by comparing it against expected values	WCFServiceRef.Service1Client obj = new WCFServiceRef.Service1Client(); string expected = "5 Test"; string exmaple1 = obj.mergeExample().ToString(); string example2 = obj.mergeExample1(); string actual = exmaple1 + " "+ example2; Assert.AreEqual(expected, actual);	Extraction Rule ServiceResponseExtraction Context Parameters 1.XpathExpression 2.ContextParam The extraction rule is used with both the request to extract the response from the server Associate the validation rule with the 2nd request Validation Rule MergeTwoServcResponseValidation ContextParameter Context Parameters 1. ContextParam1 2.ContextParam2 3.expected
DTO based Validations	DTO (Data Transfer Objects) based validation is a generic way to validate web services response of any data type against expected result by converting them into generics objects	It works by converting the service response (whether abstract or primitive data types) and expected results into standard generic objects which then are finally converted into XML for comparison with the help of our reusable library methods as described below in the example	public List<T> DTO_PopulateObject<T>(object DTOObject, object myObj) public List<T> DTO_PopulateDBResults<T>(DataTable dt) where T : class public bool DTO_XmlComparison(Object ExpectedDTOList, Object ActualDTOList, Type ObjectType) Examples: ExpectedStudentDTOList = helper.DTO_PopulateDBResults<Student>(dt); object ActualStudentDTOList = helper.DTO_PopulateObject<Student>(myObj, s1); bool result = helper.DTO_XmlComparison(ExpectedStudentDTOList, ActualStudentDTOList, typeof(List<Student>));

Tools to speed up Web Services Testing

WCF Load Test

This tool helps us in accelerating the generation of Skeleton unit test which can be used for load test of Application.

This tool takes the WCF or Fiddler2 message log file and a WCF client proxy, or a WCF interface contract, and generates a unit test that replays the same sequence of calls found in the message log file for WCF and ASMX services. The code generated is easily modifiable so that data variation can be introduced for the purpose of doing performance testing.

The tool will also integrate with Visual Studio 2005/2008 Team Edition for Software Testers and Visual Studio 2010 Ultimate and installs a wizard for creating the message log and processing it from within Visual Studio.

Tool can be downloaded from

http://wcfloadtest.codeplex.com/releases/view/51762

This tool is an add-in to VS and after running the .msi, open Visual studio and from the test menu, create a new test .It contains a template for the WCF Test.

1. Create a new Test Project and select the WCF test from the templates available for the test.

2. Click on Next

3. Give the path of the executable, it’s your application exe files, in this case it’s the sample App application which we are using for the generation of unit test.

4. Click on Run, will run the application and load the data as expected.

5. Test the application; change the values in the List of Students to fire the service with different set of Data. Once this is done, close the application.

And click on Next on the ‘New WCF Service Test Generator wizard’

6. There will be screen which depicts what all methods were invoked when we navigated the application with different set of inputs.

7. Click Next, Now we need to add the Dll/exe which define the contracts which are implemented for the WCF service which we have consumed.

8. Click on finish and the Unit tests are created for actions which we have performed on the Application.

Web Test Plug In for Silverlight + RIA based services

When testing Silver Light applications, we frequently run into dealing with non-Html content types. This is because most of the Silver Light applications use WCF RIA web services. Typically, this will be a binary format called MSBin1. Also, Silver Light applications may use other content types such as JSON, SOAP and REST to communicate with server. In general there is significantly greater use of Web Services and JSON in today's web applications.

The Web Test Plugins for Web Service Messages leverage these extensibility points to provide the following functionality for all these content types:

1. Select the Web Service plugin to enable Web Service message plugins before recorder shows up when we create a new web test project.

2. After Recording is completed it adds a reference to the runtime assembly to project, and adds the web service plugin to the Web Test

3. In recording result we can see a new tab named as “Web Service”

4. Right click on the selected node, and select “Add Extraction Rule” or “Add Validation Rule” from the context menu.

Reference- <http://blogs.msdn.com/b/mahipalkante/archive/2010/04/07/web-test-plugins-for-web-service-messages.aspx>

Web Services Automation Benefits

· Increase the quality of the application and find issues/bugs quicker and easily by validating web services for functionality without relying or testing the UI layer too much which is many times slower.

· Many times when UI is under development or for V1.0 projects where UI is changing a lot often, UI automation doesn’t give you’re the ROI in short term and Web Services automation can help you quickly validate the functionality and business layer.

· The effort and investment on automating Web Services for functionality will also pay off later to quickly build you performance test suite with lesser effort and reuse.

· This will also help to scale up when doing performance testing to ensure that web services behave as per SLA and as-expected when put under lot of stress and load.

· As a performance engineer, you will be able to do effective performance (load/stress) testing of web services along with validating their behaviour.

· As developer/consultant, you will be able to use web services automation as part of continuous Integration to improve the build and code quality.

· As a support engineer, you can also run automation tests quickly to find the culprit such an unintentional regression impact and that will help in debugging and troubleshooting critical failures.

Appendix

i. Getting Started with VS 2010 Unit Test & Web Test

Walkthrough: Creating and Running Unit Tests

Web Performance Test Walkthroughs

Introduction to WCF Testing

ii. Deploying Sample Web Services Application

Please find below the steps to get started with our sample web services

Step 1: Deploy sample DB

From our sample solution, run Create_Sample_DB.sql which will create the Database and Tables required for our sample app.

Step 2. Deploy our services

a) Sample WCF Service –

This project contains the WCF services will be consumed by Sample_APP_UI. It contains the contract as well as the implementation of the contracts. This project also uses the LinQ feature for integration with the database.

Below is sample step to deploy services on your machine

i) Launch solution and publish “WcfService” project to FileSystem

ii) Add an application to your IIS and provide the File system directory and App pool details

iii) Browse and test your services in browser or using WCFTestClient

Browser: From IIS, select your application and press Browse

WCFTestClient:

- Launch VS Command Prompt and type WCFTESTCLIENT

- Add URL of your web service

- Click on any test method, provide request parameter and invoke

b) Sample ASMX Web Services

This project contains the services which are built on ASMX types. These services are also consumed by the Sample_App_UI.

For deployment follow the steps given in above step (a).

Note: You can skip this as well if your scope doesn’t include testing ASMX based services

c) Sample Windows Workflow Foundation Services

This project contains the sample WCF service which is consumed by the workflow application.

For deployment follow the steps given in above step (a).

Note: You can skip this as well if your scope doesn’t include testing Workflow based services

d) Sample Authentication Project

This project contains the WCF service which entails the authenticate service. We have used the “UserNamePasswordValidator” validate method for the verification of the client credentials to authenticate the client.

The main intent is to check the authentication of the client, so this service has 1 contract implemented “GetServertime” which returns the current time at the server.

Note: You can skip this as well if your scope doesn’t include testing Authentication of your services

e) Sample Non-XML Data Project

This project contains the WCF service which is required for returning the non xml data from the service. For this case we are using the service to return the image file using the stream.

There are two response from the service implemented under this project.

- Buffered response

- Streamed response

Note: You can skip this as well if your scope doesn’t include testing of Non-XML based response from web services

f) Sample Web Services Caching Project

This project contains a WCF service which contains the service representing the cache behaviour of the WCF service. For this project we have leveraged the Asp.Net cache in the WCF. The response is cached for 60 seconds if the request URL doesn’t changes. On changing the parameter to the request URL or elapse of 60 seconds the call again to this service returns the fresh response from the service.

Note: You can skip this as well if your web services don’t use caching

g) Sample Asynchronous Web Services Project

This project shows another feature of the WCF asynchronous call where in client sends a request to the server and continues with the regular task and on completion of the service i.e. response comes from server the required action is performed using the response received.

For this example we have used client making the Asynchronous call using the call-back mechanism. This project contains the contract as well as the Windows client which uses the service return.

Note: You can skip this as well if your web services don’t use Asynchronous calls

Step 3. Run Sample Client apps

a) Sample UI application

This is the main project in the solution which consumes different services WCF and ASMX. This project is a sample app to demonstrate the services when consumed from UI and their return

Note: Please update the service reference to point to the URL of the our sample web services (WCF and ASMX) as shown below by using “Configure service reference”

You can now run sample UI app that consumes WCF and ASMX based services.

b) Sample Authentication Console Application

This project in the solution contains the client which will test service, and provide the valid credentials so that the service returns the valid return.

This is a console based application which hosts the Auth_Service and send the valid credentials to test it.

c) Sample Workflow Console Application

This is a Workflow project which uses the WCF services with in the workflow, and executes the rest of action in the workflow.

This project is just a sample to be demonstrated on how the services are integrated with the WF app.

iii. Sample Restful Services

iv. Request/Request XML for services not supported by Fiddler/ WCFTestClient

Working on a project where we create WCF Services there may be a need to capture the raw xml passed in to the service operation and also capture the reply xml sent back by the service. Our reusable code captures raw xml even for the scenarios (e.g. Non Xml data being returned by service) where WCF Test Client or fiddler are not able to trace the request\response xml.

Reusable Code snippet:

ImageServc.StreamingServiceClient ClientHandle = new ImageServc.StreamingServiceClient();

ResponseXmlInterceptingBehavior responseXmlInterceptingBehaviour = new ResponseXmlInterceptingBehavior();

ClientHandle.Endpoint.Behaviors.Add(responseXmlInterceptingBehaviour);

OutputFile = ClientHandle.GetData("Dryden");

string requestXml = responseXmlInterceptingBehaviour.ResponseXmlInterceptor.RequestXml;

v. Binding & Hosting configuration & consideration

This project one of the features of WCF to host the WCF service on the console. In this project we have used the basic test for the call of contract/method of WCF which is hosted on the console app.

Please refer the snapshot which will give you an idea on hosting the service on the console.

Another sample unit test

vi. Data driven testing

Use below link if you need more information about Data driven testing using Coded UI and access/store data in MS EXCEL from Coded UI

Data Driven Testing by Mathew

http://msdn.microsoft.com/en-us/library/ee624082(VS.100).aspx

http://msdn.microsoft.com/en-us/magazine/cc163536.aspx

<<Please refer to Web Services Automation Guidance (VS 2010 Unit Test
& Web Test).PDF http://webservicestesting.codeplex.com/releases/view/78473#DownloadId=312126
for detailed documentation>>

The artifact will provide Web Services Automation guidance for Functional
& Performance Testing leveraging VS 2010. The document will provide
recommendation on using either Unit Test or Web Test for functional testing
depending on your project and test objective as will be later covered in the
document. Load Test will be used for Performance Testing in both cases.

The primary objective is to propose a viable web services test automation
approach & reusable libraries to customers, partners and consultant. It will
also be a step in standardizing web services automation using VS 2010 platform
across the Industry and filling the gaps by extending our tools as suggested in
the talk. The document can also be used as a quick jump start guide for
beginners to get started on Web Services Automation Testing.

Have fun guys. We want to hear from you.

Thanks,

Raj, Vipul, Ranjit &  Gunjan