Before describing the new feature, let's dive into the problem a bit further. A book that I highly recommend, XUnit Test Patterns: Refactoring Test Code, uses the term fixture to describe the setup for a test. There are several different fixture patterns, but I will focus on only a couple.

The fresh fixture pattern creates a new environment for each test method. This is the most common approach for tests at the bottom of the test pyramid, primarily unit tests (see this post on the test pyramid). This typically produces the most reliable tests as there is no possible carryover from previous tests.

The shared fixture pattern creates an environment that is reused by many tests. As tests get broader in scope as in integration or business cycle tests, more and more data needs to exist for the test to execute in a reasonable amount of time. This is where a shared fixture is frequently used, but it has its downsides. A significant challenge can be inadvertent state carryover that impacts downstream tests. This situation impacts test reliability, can be very frustrating to debug, and results in lost confidence in the test suite.

An ERP system, with its mass of back end, interconnected, data, lends itself to a shared fixture for tests broader than unit tests. Internally, we've come up with different approaches to address the data sharing and fixture tear down challenges over past releases - things like test suites and various isolation approaches. Most of these were "bolt-ons" to the SysTest framework, and all ultimately had some drawbacks.

For AX'7', we focused on reliably rolling back data at different layers directly within the SysTest framework. To accomplish this, the transaction savepoint feature in SQL Server was leveraged. Savepoints are created at the start of each class and each method, enabling nested rollback to these known states as part of the teardown process. Because it leverages native SQL capabilities, it is both reliable and fast.

Data rollback using SQL transaction savepoints is the default behavior for all SysTest classes. You can choose to disable it at the class level using the SysTestTransaction attribute with the 'NoOp' value. This might be helpful while debugging tests, but you should almost always take the default behavior.

Now that we have reliable and fast data rollback, we've discouraged the use of legacy test isolation mechanisms. This has resulted in big reduction of 'flaky' tests… and that has been a huge benefit to our engineering efforts!

My last post introduced a testability feature in the new AX called FormAdaptors.  That post described changes in the product architecture that enabled FormAdaptors and why they have been so valuable for our internal automated regression testing strategy.  In summary, FormAdaptors provide an API to interact with forms from X++ and run server side tests that are fast and reliable.

This post provldes a simple example of FormAdaptors so you can get a feel for the API.  The form that we’re going to be targeting is the VendParameters form.  We’re specifically going to interact with the two controls in the red and green boxes in the screenshot of the form shown below. 

The scenario that we want to test is to verify that the green box control gets enabled and defaulted to false whenever the red box control is set to ‘Yes’. The red box control is the InvoiceMatching_UseInvoiceMatching control.  The green box control is the InvoiceMatching_UseTotalPriceMatching control. 

The X++ test code shown below implements the test scenario.

/// <summary>
/// Tests Invoice totals matching field settings
/// </summary>
[SysTestMethod]
public void InvoiceMatching_TotalsFieldSettings()
{
       using (VendParametersFormAdaptor vendParameters = VendParametersFormAdaptor::open())
       {
           vendParameters.InvoiceMatching_UseInvoiceMatching().setValue(true);
           vendParameters.InvoiceMatching_UseTotalPriceMatching().validateEnabled(true);
           vendParameters.InvoiceMatching_UseTotalPriceMatching().validate(false);
       }
}

The goal for this API is that the test is self explanatory when read… hopefully that is true for you!  A few notes on this sample follow:

• [SysTestMethod] identifies the X++ method as a test method.
• Each form has a FormAdaptor class associated with it named <FormName>FormAdaptor.  The class has multiple static constructors on it, one of which is the open() method shown.
• The FormAdaptor is wrapped in a using statement so the class is properly disposed after use.
• Chaining calls together is how you initiate actions on the form.  For example, setValue() is a method on a CheckBoxControl.
• Convenience calls such as validateEnabled() interact with SysTest methods to assert failures and log additional information to aid in debugging failures. 

Because this test is running on the server, there is no actual visualization of the test running.  This is one disadvantage of executing form tests in this manner and was an initial concern for engineers on the team.  This quickly went away when they saw how easy it was to debug the tests in addition to the benefits outlined in the previous post.

In order to describe FormAdaptors, we first need to take a step back to discuss the architecture changes made to enable a web based client that executes against an Azure service in the new AX. The pictures below comparing AX 2012 with the new AX illustrate how the logical control tier has moved to the server. The physical controls, which are now web friendly HTML and JavaScript based, still reside on the client.

       >>>>         

Since the form logic is now running on the server, we have the opportunity to execute form based tests on the server.

But why would we want to do this? Don't we want tests to replicate exactly what the customer does with the product? To do that, you would want to exercise the full product stack starting with the physical control on the client… right?

Well… I can't put it any other way… that's wrong. And if you don't understand why, you should go back and read the posts that are linked at the end of this post. But let me summarize why we want to do this:

• Server side form based tests are 10X to 100X faster than client side tests. Our form based test times are now measured in seconds (at most) whereas AX 2012 client side tests were measured in minutes.
• Server side form based tests are much more reliable as the number of moving parts in the test interface is drastically reduced.
• Because of the previous two bullets, it is easier to write more granular tests that are more precise, simpler, and more maintainable.
• Replicating what the customer does with the product is frequently not the best way to write a regression test that supports the engineering team.
• Writing test code in the same language and environment as the product code is a win for engineering productivity.

Now that I've justified why we want form based server side tests, we still need to author these tests. This is where FormAdaptors come into play. The adaptor pattern "is a software design pattern that allows the interface of an existing class to be used from another interface." (https://en.wikipedia.org/wiki/Adapter_pattern). FormAdaptors provide an API for test classes to interact with the form and its controls in a natural, strongly typed way. Under the covers, the API interacts with the form layer on the server ("Logical control" in the picture above) to replicate what a user working with the product in a browser would do.

In summary… I guess I wasn't very precise in my opening paragraph. FormAdaptors are really just a wrapper around the big change in testability - the capability to run fast and reliable server side tests written in X++. FormAdaptors are the 'face' of this capability and make it easier to use, which is an important facet of test development as well! More about the FormAdaptor APIs in future posts..

Related:

https://blogs.msdn.microsoft.com/dave_froslie/2016/02/03/achieving-balance-in-testing-software/

https://blogs.msdn.microsoft.com/dave_froslie/2016/02/18/automated-regression-test-qualities/

https://blogs.msdn.microsoft.com/dave_froslie/2016/03/09/test-automation-pyramid/

https://blogs.msdn.microsoft.com/dave_froslie/2016/03/25/testing-is-infinite/

In my last post, Testing is Infinite, I described a risk based approach to test prioritization. While the number of forms in AX'7' is not infinite, we do ship a few thousand forms and creating a regression suite for a few thousand forms is a very large task. We needed to look at the ROI for creating these tests on different forms and prioritize which ones to create tests for first.

The first step was to identify the relative importance of the form for regression testing.  For this, I used the risk calculation example in my last post: Regression Exposure = (likelihood of regression) X (impact of regression). Here's how I defined the two inputs:

• Likelihood of regression - Intuitively, the probability of a regression goes up with the complexity of the artifact being changed. So I embarked on determining a complexity metric for forms. Ultimately, I settled on three factors: number of controls, number of data sources, and number of lines of code. Through the metadata API in the product, I found these three inputs for all forms, normalized each of them, and then used an average of the three factors to arrive at a complexity score for each form.
• Impact of regression - From a customer perspective, the more often a form is used the bigger the impact of a regression. Fortunately, we have data from AX 2012 showing form opens for customers that opted into SQM data collection. This data was used for the impact… well, actually the log value of the number of open forms to help take out some of the skew in this data.

From these two inputs, I calculated the risk exposure, or importance factor, for each form in the product. I took it one more step and categorized the forms into Most Important, High Importance, Medium High Importance, Medium Low Importance, Low Importance, and Not Important.

This data validated an assumption that many on the product team had - the most complex forms are also the most used forms in the system. In fact, the top three importance categories accounted for the vast majority of form opens in AX 2012:

Our prioritization activity became quite clear and measurable - focus on providing regression coverage starting with the most important forms.  Just covering the top 1% was a huge step forward in providing regression coverage.

The last thing I'll mention is how we built some analytics to visualize the coverage of the forms. The Power BI (wicked cool, by the way) scatter chart below pulls complexity (Y-axis), AX 2012 usage (X-axis), and number of form opens in our regression tests (size of the bubble) together into one view. The most important forms for regression coverage are on the upper right of the chart. Those forms should also have the largest bubbles since they deserve the most test coverage. While this screen shot is limited to the Most Important forms, you can see that we generally followed the prioritization described in this post - build the most tests for the highest risk forms.

  ]]> https://docs.microsoft.com/archive/blogs/dave_froslie/testing-is-infinite Fri, 25 Mar 2016 18:36:58 GMT https://blogs.msdn.microsoft.com/dave_froslie/?p=1083 One of my favorite test quotes comes from a former peer of mine - "development is finite but testing is infinite". He used that statement to reinforce the challenge of the software test function and the need to constantly be looking at the ROI (return on investment) of testing activities. There's always something more to test.

Short of hiring Buzz Lightyear ("to infinity… and beyond!") as a software tester, we need to figure out how to prioritize various testing activities. As noted above, ROI needs to guide the way but that isn't an easy thing to measure.

For me, the first step in test prioritization is assessing product risk. The definition of risk that I use is the following (for a detailed example, go to https://technet.microsoft.com/en-us/library/cc535373.aspx):

• Risk exposure = (probability of the risk event happening) X (impact of the risk event)

So let's think about this in the context of a prime test activity - creating an automated regression test (see Automated Regression Test Qualities for more information). Here's how I think about prioritizing regression tests for a particular software artifact based on risk:

• Probability = the likelihood of a regression happening in this artifact. Some key inputs to assessing probability include churn, complexity, and existing regression coverage of the artifact.
• Impact = the estimated cost of the regression should it happen. Some key inputs to assessing impact include the severity of a bug resulting from the regression (security, data damaging, etc), the relative use of the artifact, and the customer perception. For example, an issue on the welcome screen of the product may not be severe, but the impact may still be high if the screen comes up often and the issue results in a perception of low quality by the customer.

So Regression Exposure = (likelihood of regression) X (impact of regression). While measuring these inputs absolutely is difficult, getting a relative comparison is something that you can work through.

Note that this will provide the return aspect of ROI. Since some tests are easier to create and maintain than others (see Test Automation Pyramid), determining the effort to create the regression test must also be considered for the investment portion of the ROI assessment.

This approach can be applied to other testing activities as well. Great testers are constantly thinking about the risk they are mitigating and the ROI of their activities. In my next post, I will walk through how we applied the regression risk approach to prioritize form based tests during AX'7' development.

Test granularity refers to the scope of the test, or how much of the system under test (SUT) that the test covers. There are a variety of terms used for the granularity, typically starting with 'unit' for tests that are narrow in scope. At the other extreme, tests that are broad in scope are typically called 'integration', 'scenario', or 'end to end (E2E)'.

The test pyramid is a simple visualization of the suite make-up using different granularities. Do an image web search for 'test pyramid' and you'll got lots of examples of them. For Dynamics AX'7', here's the test pyramid that we used:

The test suite's base needs to be a large volume of unit and component tests. This is important because these tests are better across the board with respect to the test qualities that I discussed in the last blog post: reliable, maintainable, fast, simple, and precise. Hopefully this should be self-evident, but for a much longer discussion on this topic I suggest you take a look at https://googletesting.blogspot.com/2015/04/just-say-no-to-more-end-to-end-tests.html.

The pyramid also indicates that having some broader integration or business cycle tests is important. Even though these tests are more costly, they are very good for building confidence in the product. They are less useful than granular tests when they fail, however. They are typically less reliable so they could be failing for a reason other than a product issue. If it is a product issue, the issue can be challenging to isolate.

The next question you might have may be about the ideal ratio between the different tiers of tests. That's a good question and the answer depends on the context of your product, but the starting point that I use is an order of magnitude difference between each tier. For example, having 1,000 unit tests, 100 component tests, 10 integration tests, and 1 business cycle test would be reasonable. We ended up with roughly this distribution for the test suite in AX'7'.

While this all looks good in theory, my experience with the last two major releases of AX provides great evidence that this approach works. For AX 2012, our pyramid looked more like an hourglass. We had a lot of unit tests and an almost equal number of integration tests. The integration tests mostly worked through the user interface and the tests were relatively slow, unreliable, and very costly from a maintenance standpoint. Their feedback value was significantly diminished because the failure rates for non-product reasons was high.

For AX'7', we have some new testability features (future post fodder!) and, as mentioned above, our suite follows the pyramid. The test suite runs at least an order of magnitude faster and our pass rate is consistently above 99.5%. Maintenance cost is significantly less and feedback value is significantly higher. We've also moved a large number of the tests further upstream in our development process to catch product and test issues earlier.

In my last post, I discussed the need for a balanced approach to testing efforts. The goal, informed by the context of the project, is to consider all four quadrants of the agile testing matrix.

In this post, I'm going to turn my focus to a testing activity that is typically found on the Support Engineering half of the matrix - automated regression testing. First, let me describe what I mean by 'automated regression' testing as these are terms that can be overloaded in the software test world.

Automated - an 'automated' test is one where the system under test is exercised by a computer and not by a human. It does not mean "have the computer use the system under test like a human would use it". It is this second inference that drives some software test industry leaders to never use the term ‘automated’ or ‘automation’ and prefer something more like 'computer assisted testing'.

Regression - a regression happens when something that used to work no longer works. In the software world, this typically happens when a change is introduced - either directly by an engineer or by an change in a dependency.

Putting these together, automated regression testing involves a computer exercising the system under test to identify and/or prevent changes in the software that cause it to no longer work as it did before.

When creating the set of automated regression tests, the goal is find the most effective test design that will support the engineering effort. Software test teams sometimes will try to serve too many needs with their 'automation' efforts by using it as their primary bug finding activity. While bugs will be found while developing automated tests, there are far more efficient ways to find important bugs that customers care about. I expect to cover exploratory testing in a future post.

Defining desirable qualities in an automated regression test helps us to achieve an effective test design. The following graphic is one that we're referred to frequently during the Dynamics AX'7' project:

While these qualities seem fairly obvious, they are deceptively difficult to achieve in practice. In my next post, I will address different test granularities that will lead to an automated regression suite with these qualities.  

A question that I occasionally use with college students is a scenario where they have two hours to test the Windows calculator on their first morning on the job. They have access to any information that they want, but they must provide a recommendation to their new manager at the end of the two hours.

Most candidates do a good job of taking a structured approach to functionally testing the calculator, moving from basic functions to advanced functions to error conditions. What very few recognize is the need to think about performance, usability, localization, help, being a good citizen in the OS, etc etc. Even a fairly simple application requires different approaches to do a thorough validation job.

A tool that I've found helpful to explain this to others internally throughout the development life of AX'7' is a variation of Brian Marick's testing matrix. Please see https://www.exampler.com/old-blog/2003/08/21/#agile-testing-project-1 for the original post from Brian. My version with some different types of tests layered into the matrix is below.

An effective testing program will have activities in all four quadrants. You can visualize emphasizing a particular aspect by moving the vertical middle line to the left or right. Move it to the left to make the Critique Product quadrants larger and you take on regression risk as the engineering team makes changes. Move it to the right and you don't have confidence that the product actually solves the customer needs, even if you have really good engineering support. I've been in projects that have made both of these mistakes. Moving the horizontal middle line up or down results in different project risks, but risks nonetheless.

The matrix can also help you realize that the same tool / approach can't be all things for the project. For example, the tendency for many software professionals is to believe that the same scenario oriented tests that we use to manually Critique Product are the best tests to automate to Support Engineering. While these tests can give you confidence in the product, they are expensive to automate and maintain, slow to run, and are imprecise (it's difficult to pinpoint the cause of a failure). A regression suite is far more effective when primarily made up of component and unit tests. In future posts, I will talk more about test automation qualities and regression suite make up.

If an interview candidate were to draw Marick's testing matrix and describe to me how he/she would approach testing the Windows calculator, that candidate's chances of getting hired would be very good!!

Well, a bit of a deeper dive into my blogging history shows that I wrote a bunch of posts in 2005 and 2006, between 3 and 8 posts per year from 2007 to 2012, and a big fat zero for 2013. And 2014. And 2015. And now it's 2016… over 10 years since I started blogging and about time to start doing some writing again.

Let's get the (re)introduction part of this post out of the way. There are definitely some constants from 2005. I still work for Microsoft at the Fargo campus. I still work on the Dynamics product line, specifically Dynamics AX which I've worked on for several years now. Fargo is still in the prairie, so the blog title still works.

But there are changes as well. My beard has more gray hair to start with. My elementary school daughter is in college. My, how the time flies…. but I digress. I've played several different roles in my 14 years at MSFT - dev lead, dev manager, test manager, test architect, and engineering manager. I've worked both on the foundation side of AX (the kernel, development tools, client, etc) and in the past four years on the application side (the modules like human resources and the various financial modules). While I've been personally changing roles, the software industry and Microsoft have also been making significant changes. Lots of change.

The trigger for recharging this blog is having something to write about, of course. That means that I a) am passionate about sharing the content, b) feel that it will be useful to a Dynamics focused audience, and c) able to share it.

All three of these things are coming together in my current role which I've dubbed 'test architect v2'.  How we think about quality in a "cloud first world mobile first world with unified engineering" has changed - most of it for the better. I'll be writing about how it has changed internally and provide thoughts on how it could and should impact the Dynamics AX ecosystem. Hopefully that will be useful to you. Finally, I'll be able to share more about the new version of Dynamics AX over the coming months.

If you’ve spent some time using VS 2010 Coded UI with Dynamics AX 2012 you’ve probably noticed that playback can be slow.  Watching the playback, you will see that there will be some activity, then a pause, then some more activity, more pausing, a little more playback, more pausing, <sigh>.  It makes you wonder.. what is it that the playback tool has to think so hard about that it has to pause?

It turns out the playback tool is not thinking.  Instead, it’s looking around – looking around for the window that it needs to do its next operation on.  It takes so long to find the window because of the uniqueness of Dynamics AX window titles and how Coded UI looks for windows. 

When you do a recording with Coded UI, the recorder identifies certain search properties that are used to find the UI element on playback.  For top level windows, the primary search property is the title of the window.  Let me explain using an example.

Let’s say I’m recording a script to create a new sales order.  The Sales Order form has a title that looks like the following:

After you complete the recording and generate code, go back and open the UIMap.uitest element in the Solution Explorer in VS.  In the UI Control Map, right click on the top level node that represent this form (UISalesorder1ceuSalesoWindow was the name in my example).  Select Properties in the pop-up menu.  Then go to Search Properties in the Properties pane and click on the […] button.  A screen shot of what I just described is shown below.

After you click on […], an Edit Search Properties form will come up.  In it you will see one of the search properties is:

Name EqualsTo “Sales order‬ (‎‪1‬ - ‎‪ceu‬)‎‪ - ‎‪‪‪Sales order‬: ‎‪SO-101375‬‬, ‎‪‪Forest Wholesales”

If I play this back, it takes over 30 seconds.  In fact, it takes about 25 seconds to do the first operation on the Sales Order form.  Too slow!

But here’s the deal, if I look at the Sales Order form title, I see that it has a different title than the one that I recorded and that the Coded UI playback engine is looking for.  The ID has incremented by one so that the title now is:

It’s surprising that the form is found at all given our Search Properties.  It does find it because the playback engine’s Smart Match capability kicks in.  What the engine is doing is first looking for a window with the exact match.  It looks through all windows currently displayed.  Then it waits.  And looks some more.  After all, it might take some time for the window to be displayed so the playback engine doesn’t want to be in a hurry.  So it waits for a little while longer and then looks some more.  Eventually, it gives up on finding a window that is an exact match.  That’s when the Smart Match algorithm kicks and and it looks for windows with titles that are similar.  Smart Match can pretty quickly figure out that this new window has a title that is almost the same as the other one and decides it’s the one to operate on.  And off it goes.

So it’s cool that it works, but waiting for 20+ seconds for the exact match phase to timeout is tedious.  Fortunately there’s a way around this.  If you go back to the Search Properties window described above, you can change the properties.  In this case. I’m going to change

Name EqualsTo “Sales order‬ (‎‪1‬ - ‎‪ceu‬)‎‪ - ‎‪‪‪Sales order‬: ‎‪SO-101375‬‬, ‎‪‪Forest Wholesales”

to

Name Contains “Sales order‬ (‎‪1‬ - ‎‪ceu‬)‎‪ - ‎‪‪‪Sales order‬: ‎‪SO”

Now my playback was complete in around 8 seconds instead of over 30 seconds!

To summarize, minimize the pauses in your Coded UI playback with Dynamics AX 2012 by changing the window name search property from an EqualsTo statement to a Contains statement.  This speeds up the playback engine’s search significantly, and speeds up your development and test execution process!