Monitor MDM events processing delay(lag) with Azure Event Hubs

Question

We use and rely on Master Data Management (MDM) . At the moment , there are 2 'queues' involved in the MDM landscape at the moment. For these queues, an azure event hub is used. As MDM is an internal system and not mission critical in terms of operations, the processing of changes from and towards other systems are allowed to have some delay. How can we monitor the queues for delay in processing of messages. Please share the thoughts. At the moment I am following this link: https://learn.microsoft.com/en-us/azure/event-hubs/monitor-event-hubs?tabs=AzureDiagnostics%2CAzureDiagnosticsforRuntimeAudit%2CAzureDiagnosticsforAppMetrics

Answer 1

@NewDBA Welcome to Microsoft Q&A forum and thanks for reaching out here.

As per understanding your requirement is to setup a monitoring and alerting mechanism for your EventHub messages that are being delayed. Please correct if I understood incorrectly. To monitor the queues for delay in processing of messages in Azure Event Hubs, you can use Azure Monitor. Azure Monitor provides a centralized platform for monitoring and analyzing the performance and health of your Azure resources, including Event Hubs.

Below are some steps you can follow to monitor the queues for delay in processing of messages: Enable diagnostics logging for your Event Hubs namespace and Event Hubs. This will allow you to collect diagnostic logs and metrics for your Event Hubs. Create an Azure Monitor workspace and configure it to collect diagnostic logs and metrics from your Event Hubs.

1. Create a log query in Azure Monitor to retrieve the diagnostic logs and metrics for your Event Hubs. You can use the log query to filter and analyze the logs and metrics based on your specific requirements.
   

Create an Azure Monitor alert to notify you when the processing of messages in your Event Hubs is delayed beyond a certain threshold. You can configure the alert to trigger when the number of messages in the queue exceeds a certain limit or when the processing time for messages exceeds a certain duration. Configure the alert to send notifications to the appropriate stakeholders, such as email or SMS notifications.

By following these steps, you can monitor the queues for delay in processing of messages in your Event Hubs and take appropriate actions to address any issues that arise. The link you provided has detailed information on how to setup the monitoring based on your specific requirement.

To know more about Azure Monitor configuration, you may refer to this youtube video: Configuring Alert and Monitoring for Azure Services (Please note this is a generic demonstration and you will have to configure as per your specific need)

Hope this helps.

---

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.