In-order delivery of correlated messages in Logic Apps by using Service Bus sessions

In this blog, we will describe a new feature that has been added in Logic Apps to support In-order delivery of correlated messages. This pattern is known as “Sequential Convoy” in BizTalk terminology. It is implemented in Logic Apps by using Service Bus sessions.

In-order message delivery ensures that messages are delivered in the same order in which they were published. Correlated messages have a property which defines the relationship between the messages, for instance, Service Bus uses session id for many correlation scenarios.

Let’s take an example, we have 10 messages for a session “S1” and 5 messages for session “S2”. By implementing sequential convoy in Logic Apps as explained in detail below, 10 messages associated with session “S1” will be processed by one Logic App instance and the 5 messages for session “S2” will be processed by another Logic App instance.

Here is a brief overview on how a Logic App can be configured to implement the “sequential convoy” pattern. We will use peek-lock triggers for receiving messages from Service Bus queue provided by Service Bus connector.

1. In the first step, we will initialize a session based on the message read from a session-aware queue by the trigger.
2. The Logic App instance will lock the session which is established in step 1 so that no other instance can take it.
3. All the subsequent get operations on this queue using the session id will be processed by this Logic App instance.

We have added a new Logic Apps template called “Correlated in-order delivery using Service Bus sessions” to show the usage of the “sequential convoy” pattern. In this template, we use queues but topic subscriptions work as well. The queues/topic subscriptions used for implementing this pattern should be configured with sessions at the entity creation time. Below, we provide details of the steps used in the template.

1. Firstly, the template initializes a session by using the “Session id” advanced option on “When a message is received in a queue (peek-lock)” trigger. There are three options available for the “Session id” option:
   • None - This option is selected by default and no sessions are used when it is selected. Sequential convoy can’t be implemented when this option is selected.
   • Next Available - This option will create a new session for the Logic App instance based on the session id of the message received from Service Bus queue. By using this, each new instance will pick up a new session available from the queue and the rest of the workflow will process all the messages associated with that session as described in the steps below.
   • Enter custom value – This option should be used when the session id is known in advance and you always want to create a new instance whenever a message with that session id is received.

2. Now that we have initialized the session, we will perform actions described below on the initial message read.
   • Send the message to a topic named {session id}-topic.  This way all the messages associated with a specific session will go to a topic.
     
   • Complete the message by selecting the “Session id” advanced options of the “Complete the message in a queue” Service Bus action.
     

3. Now we will receive all the correlated messages for this session and process them by using the “do until” scope. The end of the sequential convoy can be decided based on the following criteria:
   • Message content – When the content of the last message has some information which decides the end of a set of correlated messages, “do until” scope should be defined as shown below. This approach has been used in the sample Logic Apps template.
     
   • Message count – This can be used when the total number of messages to be read in a correlated sequence is already known. For example, below scope defines that 10 messages should be read for this session id.
     
   • Action timeout – The loop can be terminated by setting a predefined timeout on the peek-lock trigger which is used as an action. If no message appears on the queue within the defined timeout, the workflow will end. This can be set on the peek-lock trigger which is used as an action by using the “Code View” in the Logic Apps. The sample template defines a predefined 5 minute  timeout. The “do-until” scope in the template will end when either a message with “done” content is received or no message appears on the queue within 5 minutes .  If the action timeout fires the do-until loop will exit with an error. More details about this configuration can be found here.  This can be achieved by defining limit on the action as shown below.
         “limit”: {
             “timeout”: “PT5M”
         }

Now that we have configured the “do until” condition, we receive next message and perform similar actions we performed  on the initial message. Following are the details of the actions performed in the “do until” scope.

• 1. Receive next message from the session by using “When a message is received in the queue (peek-lock)” trigger. We set the “session id” advanced option to the session id read from the initial message.
     
  2. Send the message to a topic named {session id} -topic. This way all the messages associated with a specific session will go to a topic.  You must ensure that the topics are pre-created based on the values you use for the sessionId property set on each message.
  3. Complete the message by selecting the “Session id” advanced options of the “Complete the message in a queue” Service Bus action.

Steps b and c are same what we performed on the first message read.

Please ensure that “Session id” advanced option is set for all the actions to implement the Sequential Convoy.

To run this template please ensure you have completed below steps:

1. Create a session-aware queue.
2. Send messages to the queue by setting the session id property.
3. Create topics which have names based on the values you use for the sessionId property set on each message. The topic names will look like {session id}-topic.

Once you have configured these steps save the Logic App, you will see each Logic App instance will send messages associated with a session id “S1” to a topic name “S1-topic”.  Each group of correlated messages will be routed to the same topic.

Comments

• Anonymous
  May 05, 2017
  Nice post, but could you explain a little more why you are doing this?It looks to me that we are reading Session based messages and then creating Topic based messages from them. Are the messages being read from and then posted to the same ServiceBus - or are two used?What is the point, couldn't the client simply create Topic based messages instead of Session base ones?Presumably the rest of the solution has an App Logic listening to the topic queue?
  • Anonymous
    December 13, 2017
    Thanks Jason. The intent is to show how one can use service bus sessions in Logic Apps. We are reading messages from a queue and messages associated with a session are sent to a Topic in this example, although you can process these messages any way you want.