How to get only new message from IBM MQ in Logic apps

Question

How to get only new message from IBM MQ in Logic apps

Sharma, Rahul 80

Hi All,

I am trying to setup Near Real time message ingestion from IBM MQ using Logic apps.

But logic apps poll at a fixed interval and not able to fetch on only the new messages from the queue. It seems it is always picks the duplicate message.

I am using MQ trigger in standard logic app and MQ browser to read one message. I want every time when trigger the workflow it fetches only the new message which hasn't been read by previous execution of workflow.

Please let me know the correct way and what is the issue in my flow ?

TIA

Bukke SanthiSwaroop Naik 390 Reputation points

2023-07-30T08:50:35.04+00:00

The behaviour you described, where Logic Apps polls at a fixed interval and fetches potentially duplicate messages, is expected when using the MQ trigger in Logic Apps. The reason for this is that the MQ trigger is designed to be based on polling, and it doesn't inherently support tracking or handling message deduplication.

To achieve near real-time message ingestion and avoid processing duplicate messages, you need to implement a mechanism for tracking the last processed message or the message offset. This way, the next time the Logic App triggers, it can start from the last processed message and fetch only the new messages.

One common approach to achieve this is to use a combination of Logic Apps and an external storage service (e.g., Azure Blob Storage, Azure SQL Database, or Redis) to store the last processed message's unique identifier or offset.

Here's a high-level outline of the approach:

When the Logic App processes a message from IBM MQ successfully, store the message's unique identifier or offset in the external storage service.

Next time the Logic App triggers, retrieve the last processed message's identifier/offset from the storage service.

Use the retrieved identifier/offset to fetch only the new messages from IBM MQ, starting from the last processed message.

After successfully processing the new messages, update the last processed message's identifier/offset in the storage service.

By incorporating this logic, the Logic App can continue from the last processed message and avoid processing duplicates.

It's worth noting that this approach might require some custom development in the Logic App to integrate with the external storage service. Additionally, depending on the exact requirements and message volume, you might need to consider the performance and scalability aspects of the storage service and Logic Apps.

If you still have questions, please let us know in the "comments" and we would be happy to help you. Commenting is the fastest way of notifying the experts.

If the answer has been helpful, we appreciate hearing from you and would love to help others who may have the same question. Accepting answers helps increase the visibility of this question for other members of the Microsoft Q&A community.

Thank you for helping to improve Microsoft Q&A! User's image
Sharma, Rahul 80 Reputation points

2023-07-30T09:02:44.25+00:00

Hi Bukke,

Thanks for your explanation. I get what you are saying. Even in this approach the MQ Browse action doesn't have any customization to provide the offset to read from the Queue, so in this case Logic apps wouldn't be ideal for reading the data from queue.

Could you please confirm the behavior of MQ Trigger that it doesn't have functionality to check the new message and trigger even if old message are also there ?

Thanks,

Rahul
Bukke SanthiSwaroop Naik 390 Reputation points

2023-07-30T09:08:51.91+00:00

@Sharma Rahul Please refer more in the Documentation https://www.ibm.com/docs/en/ibm-mq/9.2?topic=queuing-starting-mq-applications-using-triggers
Bukke SanthiSwaroop Naik 390 Reputation points

2023-07-30T09:10:06.44+00:00

the behavior of the IBM MQ trigger in Azure Logic Apps is based on polling, and it does not have built-in functionality to trigger only for new messages while ignoring old messages.

When the MQ trigger executes, it polls the IBM MQ queue at a fixed interval and retrieves all available messages in the queue, including both new and old messages that are still present. The trigger then passes these messages to the Logic App for processing.

As a result, the MQ trigger does not have a direct mechanism to distinguish between new and old messages based on the trigger itself. If you need to process only new messages and ignore the old ones, you would need to implement custom logic within your Logic App.

One common approach to handle new messages while ignoring old ones is to keep track of the last processed message's timestamp or identifier in external storage (e.g., Azure Blob Storage, Azure SQL Database). Then, during each trigger execution, you can compare the messages' timestamps or identifiers with the last processed value and filter out messages that have already been processed.

It's essential to carefully design and implement the filtering logic to ensure that you process messages accurately while avoiding duplication or missing any important messages.

2 answers

Your answer

Sharma, Rahul 80 Reputation points

2023-07-30T09:02:44.25+00:00

Hi Bukke,

Thanks for your explanation. I get what you are saying. Even in this approach the MQ Browse action doesn't have any customization to provide the offset to read from the Queue, so in this case Logic apps wouldn't be ideal for reading the data from queue.

Could you please confirm the behavior of MQ Trigger that it doesn't have functionality to check the new message and trigger even if old message are also there ?

Thanks,

Rahul
Bukke SanthiSwaroop Naik 390 Reputation points

2023-07-30T09:08:51.91+00:00

@Sharma Rahul Please refer more in the Documentation https://www.ibm.com/docs/en/ibm-mq/9.2?topic=queuing-starting-mq-applications-using-triggers
Bukke SanthiSwaroop Naik 390 Reputation points

2023-07-30T09:10:06.44+00:00

the behavior of the IBM MQ trigger in Azure Logic Apps is based on polling, and it does not have built-in functionality to trigger only for new messages while ignoring old messages.

When the MQ trigger executes, it polls the IBM MQ queue at a fixed interval and retrieves all available messages in the queue, including both new and old messages that are still present. The trigger then passes these messages to the Logic App for processing.

As a result, the MQ trigger does not have a direct mechanism to distinguish between new and old messages based on the trigger itself. If you need to process only new messages and ignore the old ones, you would need to implement custom logic within your Logic App.

One common approach to handle new messages while ignoring old ones is to keep track of the last processed message's timestamp or identifier in external storage (e.g., Azure Blob Storage, Azure SQL Database). Then, during each trigger execution, you can compare the messages' timestamps or identifiers with the last processed value and filter out messages that have already been processed.

It's essential to carefully design and implement the filtering logic to ensure that you process messages accurately while avoiding duplication or missing any important messages.

Answer 1

the behavior of the IBM MQ trigger in Azure Logic Apps is based on polling, and it does not have built-in functionality to trigger only for new messages while ignoring old messages.

When the MQ trigger executes, it polls the IBM MQ queue at a fixed interval and retrieves all available messages in the queue, including both new and old messages that are still present. The trigger then passes these messages to the Logic App for processing.

As a result, the MQ trigger does not have a direct mechanism to distinguish between new and old messages based on the trigger itself. If you need to process only new messages and ignore the old ones, you would need to implement custom logic within your Logic App.

One common approach to handle new messages while ignoring old ones is to keep track of the last processed message's timestamp or identifier in an external storage (e.g., Azure Blob Storage, Azure SQL Database). Then, during each trigger execution, you can compare the messages' timestamps or identifiers with the last processed value and filter out messages that have already been processed.

It's essential to carefully design and implement the filtering logic to ensure that you process messages accurately while avoiding duplication or missing any important messages.

Answer 2

We had a continuous discussion with Microsoft Support about this requirement. They mentioned that Logic app is capable to get trigged whenever a new message is available in the MQ. And they have actions to mark complete in that queue so that they will be deleted and won't picked up next time. For this they have auto complete trigger and Complete message action as well.

They don't have any mechanism to maintain the offset due to this if you are the dedicated consumer for a queue this method work for you as in our case. But if there would be multiple consumers for the same queue then logic apps are not suitable for the read from MQ.

Thanks,

Rahul

Share via

How to get only new message from IBM MQ in Logic apps

2 answers

Your answer