Integration Question/Pattern

Question

Hi All,

Currently I have a project whereby an application will send out a number of records followed by a request for a report on the data loaded. The endpoint receiving these requests (event stream) is asynchronous so the records need to be processed in order. I have looked and event hub or a session based queue to try to enforce the order of operations. However I was wondering if there was a known integration pattern built to handle this.

The number of records can vary before the report request so that will only be known when the application submits the request. One way would be to provide the endpoint with a list of record ids to process and then somehow ensure that each had been processed before continuing with the report. Again I am unsure of the integration pattern (https://www.enterpriseintegrationpatterns.com/) or if there is one that accommodates

Any help would be appreciated

Answer 1

Hi
I would use the message queue protocol. An azure service bus provides these features and was built to provide the integration pattern to the use case you are high levelling.

The service bus queue has two features that controls the ordering whilst added to the queue. Sequence number and timestamp. Read more in link below.

https://learn.microsoft.com/en-us/azure/service-bus-messaging/message-sequencing

Furthermore you can consume messages from this queue in two ways simply put. Either you consume it directly which means the oldest message with lowest sequence number is removed from the queue immediately upon you retrieving it or you can retrieve the message firstly and agree with the queue to lock the queue and this message until you send a settlement message. This last feature is used when you need to make sure that the message broker has completed it services before next message can be retrieved. Read more below.

https://learn.microsoft.com/en-us/azure/service-bus-messaging/message-transfers-locks-settlement

Please let me know if the answer was helpful by accepting the solution.

Mox

Answer 2

Thankyou MOXiMOX-2301,

Will the sequence of messages work if I have multiple instances of a consumer. Or would this require the use of sessions? i.e. If I have multiple threads of the consuming process how do I ensure that another process wouldn't pick up a message regardless of the sequence

Regards

Answer 3

Yes it will work with multiple brokers. You can also have scenarios that you share queue with multiple brokers allowing all of them to retrieve same message and/or set the maximum retrieval limit to 1 forcing explicit asynchronous flow.

I haven’t come across a use case scenario as of yet where I haven’t been able to apply the above mentioned integration strategy whereas I need an explicit ordering of messages delivered.

Answer 4

Excellent,

So applying the sequence number ensures that as long as one message is processed at a time the message with the lowest number is always retrieved first? Effectively turning it into a FIFO queue. If there is an issue with message processing does this mean that no other messages will be processed?

Thanks

Answer 5

Yes very simplistically seen the sequence number and timestamp is what the service bus uses in combination of other built in logic to ensure this flow. The technology has a lot more to offer than this.