Receive change notifications through Azure Event Hubs

Question

Hi All,

I have a little inquiry regarding ‘Receive change notifications through Azure Event Hubs’.

As you know, we load Azure Databricks Diagnostic logs (plus a few system tables via Azure EventGrid) into Azure EventHub, which we then consume into ADX and run KQL queries. This is all fine.

 

What I am trying to achieve here is to enable another internal team to access our Databricks logs directly through Azure EventHub using this ‘Receive change notification’ approach explained in the above article. For this test, I have set up another EventHub and configured it in our Azure Databricks workspace to send Diagnostic logs. Logs are flowing in this EventHub. Please note, this new EventHub has nothing to do with the ADX as it is not part of any ADX Data connection.

 

Now my objective is to enable this ‘other internal team’ to receive notifications whenever there is any new content in this EventHub. Also, they should be able to consume the content directly out of EventHub. I have followed all the steps in this article. What I am slightly confused about is ‘Create the subscription and receive notifications’.

 

Is it correct to understand that:

• I need to create a subscription so that this ‘other XYZ team’ can receive notifications whenever there is new content in the EventHub?
• If true, then I need to offer them a ‘notificationUrl’ through which they would receive a ‘change notification’ that there is a change. However, I am then confused about the ‘Receive notifications’ part. In other words, what is the difference between ‘receiving a change notification’ as part of the create subscription and ‘receive the notifications’ described under Receive notifications? If you could briefly explain by giving an example for my use case, it would be helpful.
• What information do I need to provide to this ‘other internal team’ so that their application can programmatically receive the Databricks diagnostics whenever there is a change? Do I need to provide them with both the Send and Listen EventHub Shared Access Policy ‘Connection string–primary keys’?

I need to precisely understand this article and approach explained here by Microsoft here so that I can take required steps in order to fulfill the objective.

Lastly when it comes to this other team building a client to receive/process the event, they simply need to follow this “receiving events” after we provide them with the EventHub connection string (Send/Listen both)?

 

Thanks in advance for your support.