BizTalk Patterns part 2–Sync Async

• co-Authored by Thiago Almeida (Datacom)
• reviewed by Conor Brady (Oakton)

Developing a BizTalk Server solution can be challenging, and especially complex for those who are unfamiliar with it. Developing with BizTalk Server, like any software development effort is like playing chess. There are some great opening moves in chess, and there are some great patterns out there to start a solution. Besides being an outstanding communications tool, design patterns help make the design process faster. This allows solution providers to take the time to concentrate on the business implementation. More importantly, patterns help formalize the design to make it reusable. Reusability not only applies to the components themselves, but also to the stages the design must go through to morph itself into the final solution. The ability to apply a patterned repeatable solution is worth the little time spent learning formal patterns, or to even formalize your own. This entry looks at how architectural design considerations associated to BizTalk Server regarding messaging and orchestration can be applied using patterns, across all industries. The aim is to provide a technical deep-dive using BizTalk Server anchored examples to demonstrate best practices and patterns regarding parallel processing, and correlation.

The blog entry https://blogs.msdn.com/b/appfabriccat/archive/2010/09/29/biztalk-patterns-part-1.aspx examined a classic example of parallel execution, and handling of large message structures. It consists of de-batching a large message file into smaller files, applying child business process executions on each of the smaller files, and then aggregating all the results into a single large result. Each subset (small file) represents an independent unit of work that does not rely or relate to other subsets, except that it belongs to the same batch (large message). Eventually each child process (the independent unit of work) will return a result message that will be collected with other results, into a collated response message to be leveraged for further processing.

Asynchronous processing of the message / Synchronous response to the client

This post will examine another common design pattern. It is called the Sync-Async pattern. This pattern consists of a client making a request to a server service for information, whereby the client will provide an endpoint for the server to send that information to, in an asynchronous response. However, the client also requires an immediate technical acknowledgement from the server service, within the same synchronous connection that the client has established, to be assured that the request has been well received. An example of this scenario is RosettaNet – where a client initiates a PIP3A4 Managed Purchase Order request, and the server sends back a PIP3A6 order status update (Distribute Order Status) some time later.

The problem becomes even more complex when there are several (million) client requests (each with their own endpoint) that are sent to the server to act upon. Management of all these requests can be onerous, and potentially sap a lot of hardware resources.

The pattern is relatively straight forward, including the technical acknowledgement that is returned in the same synchronous connection that the client established. There are several approaches that can be pursued, all of which require correlation. 

Let’s go into the details of this pattern by taking a look at an example. A client application seeks to obtain book information from an Internet Book Seller. The client communicates to the server by sending the International Standard Book Number (ISBN), as well as its endpoint information for the server to know where to return the response (i.e.: book information) to. The server receives the request, and immediately returns a technical acknowledgement within the same synchronous connection that was established by the client. The technical acknowledgement will  include the message size of the request. When the server has assembled the response (book details), it knows where to direct it to, by using the endpoint information that was part of the original request message.

Click on the image to view it in its original larger size.

The sequence of message flows is described as follows

1. Inbound request from client submitted a two-way receive port in BizTalk called SyncToAsync.Request.ReceivePort. The request header contains endpoint information for the server to return the book information result message back to the client, and the body contains the ISBN number. A custom pipeline on this receive location determines the request message size and creates a wrapper around the request with this information. That will be used as part of the technical acknowledgement response within the same synchronous connection that was established by the client.

2. The request is published to the Message Box database. It is important to note that all the messages going in and out of BizTalk Server go through the message box database, even in synchronous scenarios.

3. A send port group SendPortGroup1 is configured with a filter to receive all messages from port SyncToAsync.Request.ReceivePort. Send port groups are collections of send ports that allow the same message to be sent to multiple destinations while only needing to configure the filters once, at the send port group level.

4. The send port group contains two send ports: SyncToAsync.ASyncRequest.FILE.SendPort and SyncToAsync.Request.FILE.SendPort. There are now two paths in the solution – one using an orchestration and the other using send port filters:

• One follows the synchronous response back to the client request as technical acknowledgement containing the size of the request message.

• The other demonstrates two asynchronous ways of forwarding the client request to one-way services. One is an orchestration that uses the ‘ReturnURL’ value sent in by the client to determine where the message should be dynamically sent to for cases when the client wants to determine where the message should be forwarded to, as per our book request example. The other is a static send port that shows how the request messages can be forwarded to a static send port where administrators could change the pipeline, maps, URL and adapter configurations when needed.

Here is the description of the two paths as per the diagram.

Asynchronous processing of the message:

4.1. Send port SyncToAsync.AsyncRequest.FILE.SendPort initiates the asynchronous path. It contains a map from the wrapper message back to the original client request (including ReturnURL), and saves the message to an Out  folder

5.1 The original request message is picked up from the Out folder by a one-way receive port called SyncToAsync.Response.AsyncReceivePort.

6.1 The message is put on the message box.

7.1 An orchestration bound to the SyncToAsync.Response.AsyncReceivePort receive port receives the message, maps it to a final response message containing the book information, configures a dynamic send port with the value in ReturnURL, and sends out the message. This step is to illustrate that any workflows can be initiated by the message, and how it could it could be routed dynamically to any end point including a URL requested by the client.

8.1 The message sent by the orchestration is put on the message box.

9.1 The message from the orchestration is routed to a dynamic send port that uses the values set by the orchestration to the message via WCF to a service.

10.1 The message is delivered to the service.

11.1 A separate send port called SyncToAsync.AsyncResponse.WCF-NetTcp.SendPort subscribes to messages received by the SyncToAsync.Response.AsyncReceivePort as well. This send port can contain a pipeline and a map to create the response message. This is to show how static send ports can also be used to handle the asynchronous send.

12.1The message is sent by the static send port to the WCF service.

Synchronous response to the client:

4.2 Send port SyncToAsync.Request.FILE.SendPort initiates the synchronous path. It contains a map from the wrapper message to an outbound schema that contains the file size for the client technical response and five different internal context properties. These properties are internal to BizTalk and are used to route messages back to the original synchronous connection created by the client. These properties are demoted from the message context properties into XML elements by the XML Transmit pipeline and the message is saved to an Async folder. Note that maps on send ports happen before the pipeline execution, so the pipeline receives the message created by the map, the schema of which contains those five routing properties (mentioned in the previous entry).

These key properties are EpmRRCorrelationID ReqRespTransmitPipelineID IsRequestResponse Correlation Token RouteDirectToTP

5.2 The message with the demoted properties and the original message size is picked up from the Async folder by a one-way receive port called SyncToAsync.Response.SyncReceivePort. The XML Receive pipeline in the receive port promotes the five values that had been demoted onto the message.

6.2 The message is put on the message box.

8.2 Because the correct fields have been promoted when the file was picked up, the message is routed back to the synchronous two-way receive port.

8.2 The response message with the request size is sent back to the client application on the same synchronous connection it initiated.

Hence, the example illustrated how Asynchronous processing of the message / Synchronous response to the client can be achieved. Note that this solution avoids the use of a BizTalk Orchestration (and Windows WF) to send the synchronous response and control the overall solution implementation – it does that using only BizTalk Server’s messaging capabilities.

• A custom receive pipeline was used to create the technical acknowledgement, as well as to forward the request into the BizTalk Server system.
• Correlation was required to keep track of the requests and the responses that the server must return the respective clients. Leveraging correlation with BizTalk Orchestration (or even Windows Workflow Foundation) is relatively easy, but creates overhead and could require a heavy investment on hardware. However, leveraging correlation in the BizTalk messaging (i.e.: without Orchestration) is also possible, and requires a lot less footprint on hardware. This mechanism was also leveraged in the previous post of BizTalk Patterns (part 1).

Note: LOB is an acronym for Line of Business (LOB) application.

This example does not cover retries / attempts over an unreliable transport, such as HTTP/S. Perhaps that exercise is best left up to reader (yourselves).

Sample code

Sample code is provided courtesy of Ji Young Lee (Microsoft South Korea), and enhanced by Thiago Almeida (Datacom). Sync-Async_v2010 code can be found at https://code.msdn.microsoft.com/appfabriccat/Release/ProjectReleases.aspx?ReleaseId=5050

Acknowledgements to: Thiago Almeida (Datacom), Conor Brady (Oakton), Werner Van Huffel (Microsoft), Ji Young Lee (Microsoft), Paolo Salvatori (Microsoft), Shelley Shapley (HP)

Technorati Tags: BizTalk Server,Asynchronous,Synchronous,Correlation