Microsoft Dynamics AX 2012 R2 Benchmark for Multiple Retail Workloads

In 2014, Microsoft conducted a benchmark on Microsoft Dynamics AX 2012 to measure the performance and scalability characteristics of the Microsoft Dynamics AX 2012 R2 retail solution in a simulated retail scenario. The benchmark showcases the ability of AX 2012 R2 to handle various specialized loads concurrently, without compromising performance and scalability on critical business processes. The benchmark includes many functional scenarios across different client and integration technologies, thereby providing a view of retail back-end scenarios.

The following scenarios were executed to complete the benchmark:

• Trickle feed of point of sale (POS) transactions from stores
• Posting trade agreements to stores
• Publishing product assortments to stores, including converting pre-actions to actions

End-of-day processing of POS transactions:

• Inventory reservations were made for the imported transactions.
• Imported POS transactions were consolidated and aggregated, and then transferred to statements in a peak-load situation.
• Statements were processed and transferred to sales orders, and financial updates were posted.

Please find the benchmark documents here:

https://mbs.microsoft.com/partnersource/northamerica/deployment/documentation/msdax2012r2multiretailwl
https://mbs.microsoft.com/customersource/northamerica/AX/learning/documentation/msdax2012r2multiretailwl

Microsoft Dynamics AX 2012 R3 Financial Benchmark

In August 2014, Microsoft conducted a financial benchmark of Microsoft Dynamics AX 2012 R3 to measure the application’s performance and scalability characteristics for core financial functionality.

Multiple different workloads were simulated:

• A set of financial transactions was run concurrently.
• A batch of 500 general ledger (GL) journals of 1,000 journal lines each was posted in a separate test.
• A set of management reporter reports and scenarios was executed.
• A year-end closing process was run in a separate test.
• A consolidation process was run in a separate test.

The benchmark was modeled by using two financial periods, previous period and current period. It assumes that the customer is in a closing state for the previous period while entering regular business transactions in the current period. For the closing process (for example, the summary trial balance), some transaction data was created in the previous period before the benchmark run. The defined scenarios were run with 500 concurrent financial users who generated load on the Application Object Server (AOS) instances by using the Microsoft Dynamics AX 2012 R3 Benchmark SDK.

The overall benchmark result shows the capability of Microsoft Dynamics AX and Management Reporter to handle core financial scenarios with volume data while maintaining good performance measures.

Please find the benchmark documents here:

https://mbs.microsoft.com/partnersource/northamerica/deployment/documentation/msdax2012r3financialbm
https://mbs.microsoft.com/customersource/northamerica/AX/learning/documentation/msdax2012r3financialbm

Microsoft Dynamics AX 2012 R3 Day in the Life Benchmark for Retail and Distribution

Microsoft conducted a 24-hour “day in the life” benchmark of Microsoft Dynamics AX 2012 R3 to measure the application’s performance and scalability characteristics when running retail and distribution workloads. The benchmark ran a variety of functional scenarios that use different client and integration technologies, thereby providing a broad view of enterprise resource planning (ERP) workload performance on AX 2012 R3.

• The initial benchmark run showcased the daytime and nighttime activities at a complex customer installation that has 2,000 stores, each with 10 point of sale (POS) devices. The benchmark modeled stock movement to replenish stores, and also the use of a web storefront built on Windows Communication Foundation (WCF) services, a call center doing order entry, and a mobile app that connected to Retail Server.
• The benchmark also simulated having 3,000 concurrent users perform background tasks.
• In additional benchmark runs, we increased the volume to 5,000 concurrent users and also increased the volume of POS transactions.

These scenarios generated load on an instance of Application Object Server (AOS). In this benchmark, some AOS instances are configured as multiple logical AOS instances in a cluster. Batch processes run simultaneously with online processes but on separate AOS instances.

AX 2012 R3 demonstrated the ability to scale up and scale out on the AOS tier, and on the database server.

AX 2012 R3 also showed the ability to handle a more complex business process flow smoothly. User experience measures were below 2 to 3 seconds for intensive posting operations, and line save operations averaged sub-second response times.

The benchmark results demonstrate the ability of AX 2012 R3 to concurrently handle more complex scenario flows with a very high volume, without compromising performance and scalability for critical business processes.

Please find the detailed documents here:

https://mbs.microsoft.com/partnersource/northamerica/deployment/documentation/msdax2012r3dayretaildisbm
https://mbs.microsoft.com/customersource/northamerica/AX/learning/documentation/msdax2012r3dayretaildisbm

In Ax2012, we introduced class SrsReportDataProviderPreProcess class as the solution for long running SSRS reports that cause SSRS time out. The approach is to process data in Ax session before calling SSRS. The preprocessed data stored in a regular SQL table shared by all user sessions, only to be striped by session id. That approach creates a bottleneck in case of many concurrent user sessions. In R2, we introduced this new class to allow use of tempDB to carry report data across sessions, from Ax data processing session to SSRS data retrieval session. AOS kernel will not purge tempDB table when session end leaving it to application code to clean up the tempDB table. In case of report, this base class will take care of the clean up.

You should stop use the old SrsReportDataProviderPreProcess class but use the new base class instead for long running reports. You will see performance gain in case of multiple concurrent sessions on the same report.

Currently there are many OOB Ax2012 R3 reports already take advantage of this feature. You can check out how they are leverage the feature there. One typical example is TaxListDP class.

This R3 feature is implemented in the kernel. The usage is not limited to reporting. Attached is a sample project showing how this kernel feature is used in case of multi-threading in Ax: each thread creates its own instance of a temp table, inserts rows into the table, and passes the table name back to the main thread (crossing session boundary), where contents from each temp table is then merged into one DB table using set-based insert. At the end, the code must despose the temp tables. The code is for demonstraction only. A more practical application to leverage this feature is in batch processing.

Using Dynamic Set-based insert operation

In AX2012, set based insert operation is part of X++ syntax and is precompiled. That limits its use when filters are dynamic, i.e. taking from end user at run time.

In AX2012, R3 we implemented a DCR that allows constructing a set based insert where clause based on Query range on the fly. This feature allows much wider adoption of set based operation and greatly improve performance when applied.

A good coding example can be seen in TaxListDP.insertTaxListTaxTmpData. It is done using a new static method on Query object. Here is how it is used.

                Query::insert_recordset(taxListTaxTransTmp, insertRecordsetMap, query);

 the first parameter is the tempDB table and the second one is the mapping between the tempDB table fields and query fields. See detail in the AOT class source code. Here is the highlight of the row-based old code:

Without the DCR, we can’t change this code to set-based because query range is set by end users. Now, the highlight of the new high performance code is as follow:

PrivateProject_TempDBTableDCRSample.xpo

Some of the AX customers have uneven disparity in data distribution for different Companies/Les (typically one or two companies may have most of the data volume and all other companies have very less data). Under these conditions the cached plan for a specific company/LE may result in very poor performance for another company/LE. 

It is possible to change how SQL Server is optimizing queries using trace flag 4136 and this may have a positive effect if parameter sniffing is an issue in your system. More often than not the overall performance degraded when trace flag 4136 has been used. Customers should be careful about using the trace flag 4136, as it can result in poor performance in undesired areas as it is trying to ‘Optimize for UNKNOWN’. Please evaluate using below described feature instead.

Many customers/partners brought this to Microsoft Dynamics AX team’s attention.  The latest cumulative update for R2 (CU6) addresses this by passing 2 kernel parameters PARTITION, DATAAREAID as literals.  If you have R2- CU6 installed and if you know you are facing parameter sniffing issues due to uneven data distribution, you could try enabling this feature.

Note: This feature is turned OFF by default.

When you installed AX2012 – R2 CU6, it would have created 2 new records in SYSGLOBALCONFIGURATION. With names ( 'DATAAREAIDLITERAL', 'PARTITIONLITERAL')

To turn the feature ON you need to update these records and set the value to 1.  Following update will enable the feature on all the AOS.

UPDATE SYSGLOBALCONFIGURATION SET [VALUE] = 1 WHERE NAME IN ( 'DATAAREAIDLITERAL', 'PARTITIONLITERAL')

To turn the feature OFF you need to update these records and set the value to 0. 

Note: There is no UI available for this and this needs to be turned ON/OFF from SQL.

After updating the value to 1, when you restart the AOS this feature gets enabled.

Here is a sample parameterized SQL when this feature is OFF. (Captured thru the trace/traceparser.  The field list of the select clause bellow is shown truncated)

SELECT TOP 1 T1.SALESID,.......,T1.RECID

FROM SALESTABLE T1

WHERE (((PARTITION=?)

      AND (DATAAREAID=?))

      AND (SALESID=?))

Here is a sample parameterized SQL when this feature is ON. (Captured thru the trace/traceparser.  The field list of the select clause bellow is shown truncated)

SELECT TOP 1 T1.SALESID,.......,T1.RECID

FROM SALESTABLE T1

WHERE (((PARTITION=5637144576)

      AND (DATAAREAID=N'ceu'))

      AND (SALESID=?))

The internal performance testing of different scenarios did not show any performance regression when the feature is turned on.  But before you enable this feature in production, you should test this feature in your test system thoroughly and ensure it works for you.

Note: AX32Serv binary (AX32Serv.exe) of CU6 contain the fix.  If you are just taking the binary alone, you may have to insert two records into SYSGLOBALCONFIGURATION with names 'DATAAREAIDLITERAL', 'PARTITIONLITERAL' and set the values to 1 to enable this feature.

Remove duplicated DB calls

Most of the duplicated DB calls are hidden deep in the stack. For example, Vendor report calls DirPartyTable::ElectornicAddressType2primaryFieldId() 4 times (4 db access) per line, from VendTable::email, telefax, telex, and phone methods, respectively.

The case of duplicated DB call are mostly implicit, such as calling find() method inside a table method. Some of them are cached in the global cache, some are not. You need to make the decision whether we use global cache, or fix it at the report level. (Note that cache approach won’t help in cold start case.)   

A way to detect this pattern is to run a report under the tracer and analyze the trace by comparing number of report lines with number of DB method calls in a code path. Say if a report has 100 output lines and a corresponding DB method call count is 200, then you should exam further if the duplicated method call can be reduced.

More report examples are ProdPickList report, LedgerJournal report, CustInvinoiceSpec report, etc.

Refactor Parent Level Operations

Sometimes, parent level information is retrieved or calculated at the detail level. It causes repeated data access or calculation. An example is that Customer or Company field values are retrieved through CustTrans table method. One way to fix it is to cache the values in global cache. However, adding them to cache will encure system wide impact. Secondly, caching does not help in case of cold start.

There are two ways to fix it. First, try to sort the query data source by parent level field values, Customer RecID in the above case, and get the new data row only if a change occurs. This way, we can reuse the parent level values in the subsequent lines until parent record changed. 

Another way to fix is to separate parent level values with the details by processing parent level values outside the loop of lines. You can likely use set-based operations at parent level and join the result with the line-based result. Here is an example from CustInvoice report:

In Contoso as an example, CustTran table has 10,000 records. The number of combinations of the two keys, AccountNum and PostingProfile, is just over 100, which is the total number of summaryAccount records. The current code use a Map to cache the result in memory, but it is still a line-based look up.

To optimize it, you can save the result of getSummaryAccount() in a lookup temp table, keyed by AccountNum and PostingProfile. Here are the new steps in the sequence:

1. Update the main while loop by removing call to getSummaryAccount() method and the logic after to get display names. You need to added AccountNum and PostingProfile in to the temp table and save the field values in the insert method.
2. Add a set-based insert after the main loop into the lookup temp table from custInvoiceTmp table, with group by the two key fields.
3. Add a select-while loop to the lookup temp table to call getSummaryAccount method and getDisplayValueForFocusName method; save the display name in the same table using update. Note that you don’t need the map look up anymore.
4. Add a set-based update to set the DismensionFocusDisplayValue field in CusInvoiceTmp table by joining the lookup temp table on the key fields.
5. Optional of doing step 3 and 4, you can avoid calling getDisplayValueForFocusName method line-by-line in step 3 by duplicate its logic with the set-based update in step 4 above; just add another join.

This pattern applies not only to the parent-child tables, but also to any cases that the field values of a temp table comes from a table with much fewer distinct lines, say with ratio of 1:10 or larger.

Fewer Instantiations of Helper Class Objects

A typical example of this pattern is the use of CurrencyExchangeHelper, which instantiates an object for each line. The cost can become significant. In PurchRanking report for example, it takes up to 60% of total processing time.

You can try to instantiate one instance for the entire process; and then assign different parameters in the iteration of lines for calling calculations.

Skip postLoad Kernel Call

After a data row is loaded, Ax kernel makes a call back to application postLoad() override method, which will give application the first chance to tweak the data or fire dependency logic. If the table used by your report does have the method override, you can improve the report performance by telling the Ax kernel to skip the method call. Report developers must make the decision to see if the postLoad override method can be skipped in the context of the report.

In one measurement, we see about 5% of performance improvement just on the method call overhead. The saving could be more significant considering the time spend on  processing the method logic.

Here is the code to show how to do it:

And, here is the trace comparison between a normal default trace and the trace with the skipping enabled.  The postLoad method here does not do much:

With skippostLoad set to true, the extra calls are gone:

Working with Dynamic Parameters

As it is mentioned in an earlier blog (part 1), Ax2012 set-based operations are X++ language elements compiled into SQL at compile time. They do not take, as input, the dynamic query ranges from users at run time. To leverage set-based operations, what you can do is to keep the original line-base loop but move every possible set-based process outside this loop. We improved the trial balance report In Ax2012 R2 this way.

In the trial blance report, it first detects of any dynamic parameter is added by calling a new method queryHasRangesOrFilters(). If it is, the main queryRun.next() loop is executed but only to collect those filtered line ides from GL table. Then, the rest of the process are changed to set-based.

The line-by-line loop on queryRun.next() can’t be replaced whenever dynamic parameter is enabled. But you should make this loop lean, only to process those that must be line-based and leave the rest to set-based processing. Vend report in Ax 2012 R2 is another example.

Note: Ax2012 R3 enables set-based insert with dynamic parameters. See post Part 6 on this series for detail.

All the method calls in insertIntoTempTable() method are database find() type of methods, which can be expressed as join clauses in a set-based statement. To use set-based data access when dynamic parameter is enabled, you first store all the VentdTable line recIDs, and available field valuess, in the temp table without calling any method. The second step is to do set-based update over the first results to join with other tables to get the rest of the field values.

Please note that replacing a line-base table display method duplicates the business logic in the method code in the form of set-based expressions. Such denormalization is a tradeoff for performance. Developers need to make the call based on the each scenario.

Table Methods of Database Operation

As mentioned above in Vend report, a table method of database operation, such as look up or aggregate, can be replaced with a join clause in the set-based operation. One typical example is find() method that most tables have.  Basically, the loop up logic in the table function needs to be repeated by an exist-join clause in the set-based statement, such as this code snip from VendAccruedPurchasesDP_NA.

The look up and the subsequent comparison need to be converted to table join clauses of the insert_recordset statement, such as this:

 Set-based approach in the Nested Loops

Many report RDP classes have nested loops. The outer loop iterates over the result of queryRun object line-by-line. And then for each line there is a nested inner loop, which is mostly un-related to the dynamic parameter as the outer loop does. In most cases, such a nested loop can be in set-based to reduce the X++ code complicity from O(n*n) to O(n).

Report examples of this type are CustBillofExchange, TaxPurchLedger, CustLedgerTrans, etc.

Sometimes, the nested loop is wrapped in a private method call.  Here is an example from LedgerTransStatementDP class:

This method is called per line. The equivalent set-based code as implemented in Ax2012 R2 appears quite different:

Notice the use of a temp table, LedgerTransStatementOffsetAccountTmp, which stores the intermediate result. Such a table is often needed for set-based approach.

The set-based approach is not straightforward, but it is much faster than the line-based method call.

Aggregation and Sequencing

In a line-based processing loop, it is easy to aggregate a total or a rolling total in a variable. It is done differently in the set-based approach.

To get the summary in the temp table, it is often done in two steps: inserting all the detail lines first and followed by inserting summary lines in the same table. You can add a flag field to the table, such as isFinalRecord, to separate the aggregated values with detail values. In the end, you delete all the detail lines before rendering the report. In LedgerTransStatement report (R2 code), the set based operations are in stages: retrieve and process the raw detail records, and then grouping. In the end in recorderRows() method, only aggregated lines are kept:

A rolling total, or a current amount, depends on the order of the result set. We can define an incremental id field to tag the temp table based on the order-by. Then do set-based with aggregation on those that has the id value smaller than the current row.

See Ax6.2 code example in LedgerTrialBalanceDP.updateAccumulatedAmounts():

Working with Complicated Code Logic

We already know that some line-based method calls contains only database look up logic. They can be combined into join clauses in the set-based operations. However, whenever there is line-based code logic, it is often hard or impossible to rewrite it into set-based without affecting fundamental logic of the report and data schema. Examples are rounding and exchange rate code. They will remain as line-based.

For example, display method InventTable.itemName() has code logic that can’t be in set-based.  Reports that use it must keep the call in the line-based loop. The general task for developers is to identify what code can be in set-based and what must stay as line-based.

The table display methods above look up VendInvoiceJour and VendTrans tables to get the last dates respectively.

However, there is a performance cost associated with this convenience. The call to a table display method is line-based. When a report is critical in performance, you can make it run faster by replaying the display method calls with Ax Views.

For the above report, we added two views, with aggregation, that duplicate what the display methods do.  Please See xpo attatchment for one of them. The other view is almost the same. In one test case, this particular change improved the db access time from 0.42” to 0.05”. Here is the AOT of one view.

This pattern of improvement makes sense for the display method that is database bound and the logic in the display method can be expressed in views. For example, display method InventTable.itemName() can’t be expressed in views.

Please consider using calculated fields in Views before using display method. Calculated fields will be translated into SQL expression as part of target SQL statement, instead of being executed row-by-row in AOS.

View_VendInvoiceJourLastInvoiceDate.xpo

In one case, there are over 8000 CustTrans lines fetched but only 41 lines are added to the report. In this example, more lines are retrieved less restrictively and filtered later in code instead. Most lines are discarded anyway, resulting in excessively unnecessary DB access.

The fix is to remove the if-condition inside the main loop and add perDate to the queryRun object data range.

The key thing to look for this pattern is to see if every iteration of the main loop results in one or more insert/update to the report data temp table. If not, then you need to identify skipping conditions and see if the conditions can be represented in terms of the main loop query range expression.

Ax2012 R2 improved CustDueReportDetail report this way. Here are the high light of the old code. You can test yourself to see if how you would improve it using this pattern.

Another case in this pattern is trying to do aggregation in SQL instead of in X++ code. LedgerAccountBlanceSheetByDim report can be re-written in set-based, but the line-based fix in this pattern also yields huge performance gain:

In method LedgerAccountBlanceSheetByDim.prepareTmpTableData(), a while loop iterates over GJAE table to total up the amount.

The improved code uses SQL aggregation, which improved the speed in multy folds:

Ax reports were migrated to Microsoft SSRS in Ax2012, mostly in the form of Report Data Provider classes (RDP).  We want to share with our customers and ISVs, through this series of blogs, with some best practices to improve SSRS report performance. 

Our goal is to improving report performance without introducing much disruptions or risks to the existing reports. So, the recommended patterns of changes are mostly focus on optimizing data access and reducing duplicates in the scope of your current report designs. In some cases, a report needs to be redesigned for better performance, which is outside the scope of this discussion.

Using Set-Based Operations

Set-based operations in Ax are exponentially faster than the equivalent line based operations. Most RDP reports today access data line by line.  In Ax2012 R2, we updated a few reports with set-based data access and observe better performance. 

A typical line based logic in a RDP class processReport() method looks like this:

And another example – a method in a RDP:

You can re-write it into X++ set-based operations to transform data from source tables into the target tempDB table through selects and joins. A general approach of doing this is to exam the Query object range conditions and trying to express them in terms of set-based expressions. Relations in the query object need to be expressed as join clause in the set-based statement. An example for the above line-based code would be something similar to this:

However, not all line based RDP providers can be in set-based, for mainly two reasons: first, the use of dynamic parameters in reports. Second, complicated line based logic in X++ code that can’t be expressed in set-based expression, such as exchange rate calculations. In later posts, we will show how to improve the performance using partially set-based approach, which we call it the hybrid pattern.

The two constrains are easy to identify. Here are the details.

Identifying Dynamic Parameters Constraint

When dynamic parameter is enabled for a report, end users can define and modify the query ranges in the report query object. The Report framework runtime will execute the query with user modified query ranges. Such runtime feature is not supported in today’s X++ set-based operations, where all filter conditions are precompiled. A simple way to find out if a report is enabled with dynamic parameter is to run the report and see if the query form has the “Select” button:

Or in VS report designer see if the property is checked.

Of all the 1100 RDP reports shipped in Ax2012, about 700 of them have dynamic parameters enabled.  

Ax2012 R3 added support for set-based insert with dynamic parameters. Please refer to Part 6 of this posting series for details.

Identifying Code Logic Constraint

An RDP class typically contains code logic to retrieve and/or calculate field values for each target record line, in a line-by-line way. Generally speaking, such X++ code can’t be expressed in set-based operation clauses. For example, CurrencyExchangeHelper class has the logic that can’t be written in the form of X++ set-based clause expression. When this happens, the line-based code in question has to stay. 

In a few cases, a field value that comes from complicate code logic may already exist in some other related table field. If you can identify that duplicating field, you should use that field value. A good example is in the Ax2012 VendAccruedPurchaseDP:

The table method itemName() contains the logic that can’t be expressed as set-based. A proposed option of using set-based in this case is to use PurchLine.Name.

In many cases, table methods, such as find(), are used only to get values, line-by-line, through a database select statement. Such select statement can become an exist-join clause to the set-based operation. This is discussed later in this blog.

Replacing Line-based code with Set-based

Finally, if none of the above two constrains applies to a report,  you should be able to transform the report into set-based. We identified a few reports shipped in Ax2012 to be the candidates. Here are some of them:

TaxLedgerReconciliation, VendLedgerReconciliation, BankLedgerReconciliation, CustLedgerReconciliation, LedgerAccountBalanceSheetByDim, BankAccountStatement, etc.

In Ax2012 R2, we optimized three reports by using some set-based operations: LedgerTrialBalance, VendAccruedPurch, and LedgerTransStatement. We will continue improving SSRS reports in the future releases.

(to be continued)

Normally, a report is unlikely to reach this limit to cause out-of-memory exception or a significant performance drop. However, Check report is more subject to it, where data are prepared in a temp table and then streamed to SSRS. A check often contains images of the company Logo and signature.  A batch of check printing requests often contains a large number of checks, say 500 or even 1000s. In such cases, customers will get close to or even hit the limit. The check printing either becomes unbearably slow or fails.

You can work around the issue in the current released version by reducing the image size of both company logo and signature using compressing method.  You may not see the improvement unless you batch up a large number of checks to print.

To change the company logo, you need to use an image compressing tool to compress your company logo and signature files into smaller files. Avoid using BMP file format; use compressed image format such as JPG instead. Once you have the smaller image file ready,  open bank accounts form and select Set up/Check: Company>Cash and bank management>Common>Bank accounts

Click on Company and Signature tab respectively to load the smaller files you created earlier.

Please note that smaller images in reports in general benefit the printing speed.

Please find the reports here:

PartnerSource

CustomerSource