Receive change notifications through Azure Event Hubs

Webhooks may not be suitable for receiving change notifications in high throughput scenarios or when the receiver can't expose a publicly available notification URL. As an alternative, you can use Azure Event Hubs.

Good examples of high throughput scenarios include applications subscribing to a large set of resources, applications subscribing to resources that change with a high frequency, and multitenant applications that subscribe to resources across a large set of organizations.

The article guides you through the process of managing your Microsoft Graph subscription and how to receive change notifications through Azure Event Hubs.

Using Azure Event Hubs to receive change notification

Azure Event Hubs is a popular real-time events ingestion and distribution service built for scale. You can use Azure Events Hubs instead of traditional webhooks to receive change notifications. Using Azure Event Hubs to receive change notifications differs from webhooks in a few ways, including:

  • You don't rely on publicly exposed notification URLs. The Event Hubs SDK relays the notifications to your application.
  • You don't need to reply to the notification URL validation. You can ignore the validation message that you receive.
  • You need to provision an Azure Event Hub.
  • You need to provision an Azure Key Vault.

Set up the Azure KeyVault and Azure Event Hubs

This section walks you through the setup of required Azure services.

The Azure CLI allows you to script and automate administrative tasks in Azure. The CLI can be installed on your local computer or run directly from the Azure Cloud Shell.

# --------------
# TODO: update the following values
#sets the name of the resource group
resourcegroup=rg-graphevents-dev
#sets the location of the resources
location='uk south'
#sets the name of the Azure Event Hubs namespace
evhamespacename=evh-graphevents-dev
#sets the name of the hub under the namespace
evhhubname=graphevents
#sets the name of the access policy to the hub
evhpolicyname=grapheventspolicy
#sets the name of the Azure KeyVault
keyvaultname=kv-graphevents
#sets the name of the secret in Azure KeyVault that will contain the connection string to the hub
keyvaultsecretname=grapheventsconnectionstring
# --------------
az group create --location $location --name $resourcegroup
az eventhubs namespace create --name $evhamespacename --resource-group $resourcegroup --sku Basic --location $location
az eventhubs eventhub create --name $evhhubname --namespace-name $evhamespacename --resource-group $resourcegroup --partition-count 2 --message-retention 1
az eventhubs eventhub authorization-rule create --name $evhpolicyname --eventhub-name $evhhubname --namespace-name $evhamespacename --resource-group $resourcegroup --rights Send
evhprimaryconnectionstring=`az eventhubs eventhub authorization-rule keys list --name $evhpolicyname --eventhub-name $evhhubname --namespace-name $evhamespacename --resource-group $resourcegroup --query "primaryConnectionString" --output tsv`
az keyvault create --name $keyvaultname --resource-group $resourcegroup --location $location --enable-soft-delete true --sku standard --retention-days 90
az keyvault secret set --name $keyvaultsecretname --value $evhprimaryconnectionstring --vault-name $keyvaultname --output none
graphspn=`az ad sp list --display-name 'Microsoft Graph Change Tracking' --query "[].appId" --output tsv`
az keyvault set-policy --name $keyvaultname --resource-group $resourcegroup --secret-permissions get --spn $graphspn --output none
keyvaulturi=`az keyvault show --name $keyvaultname --resource-group $resourcegroup --query "properties.vaultUri" --output tsv`
domainname=`az ad signed-in-user show --query 'userPrincipalName' | cut -d '@' -f 2 | sed 's/\"//'`
notificationUrl="EventHub:${keyvaulturi}secrets/${keyvaultsecretname}?tenantId=${domainname}"
echo "Notification Url:\n${notificationUrl}"

Note: The script provided here is compatible with Linux-based shells, Windows WSL, and Azure Cloud Shell. It will require some updates to run in Windows shells.

Creating the subscription and receiving notifications

After you create the required Azure KeyVault and Azure Event Hub services, you will be able to create your subscription and start receiving change notifications via Azure Event Hubs.

Creating the subscription

Subscriptions to change notifications with Event Hubs are almost identical to change notifications with webhooks. The key difference is that they rely on Event Hubs to deliver notifications. All other operations are similar, including subscription creation.

The main difference during subscription creation will be the notificationUrl. You must set it to EventHub:https://<azurekeyvaultname>.vault.azure.net/secrets/<secretname>?tenantId=<domainname>, with the following values:

  • azurekeyvaultname - The name you gave to the key vault when you created it. Can be found in the DNS name.
  • secretname - The name you gave to the secret when you created it. Can be found on the Azure Key Vault Secrets page.
  • domainname - The name of your tenant; for example, contoso.com or contoso.com. Because this domain will be used to access the Azure Key Vault, it is important that it matches the domain used by the Azure subscription that holds the Azure Key Vault. To get this information, you can go to the overview page of the Azure Key Vault you created and select the subscription. The domain name is displayed under the Directory field.

Note

Duplicate subscriptions are not allowed. When a subscription request contains the same values for changeType and resource that an existing subscription contains, the request fails with an HTTP error code 409 Conflict, and the error message Subscription Id <> already exists for the requested combination.

Receiving notifications

Events are now delivered to your application by Event Hubs. For details, see receiving events in the Event Hubs documentation.

Before you can receive the notifications in your application, you need to create another shared access policy with a "Listen" permission and obtain the connection string, similar to the steps listed in Configuring the Azure Event Hub.

Note: Create a separate policy for the application that listens to Event Hubs messages instead of reusing the same connection string you set in Azure KeyVault. This ensures that each component of the solution has only the permissions it needs and follows the least permissions security principle.

Note: Your application receives validation messages whenever it creates a new subscription. You should ignore these notifications. The following example represents the body of a validation message.

 {
    "value":[
        {
            "subscriptionId":"NA",
            "subscriptionExpirationDateTime":"NA",
            "clientState":"NA",
            "changeType":"Validation: Testing client application reachability for subscription Request-Id: 522a8e7e-096a-494c-aaf1-ac0dcfca45b7",
            "resource":"NA",
            "resourceData":{
                "@odata.type":"NA",
                "@odata.id":"NA",
                "id":"NA"
            }
        }
    ]
}

Subscriptions for rich notifications with large payloads

The maximum message size for Event Hubs is 1 MB. When you use rich notifications, you might expect notifications that exceed this 1-MB limit. To receive notifications larger than 1 MB through Event Hubs, you must also add a blob storage account to your subscription request.

Set up storage and create a subscription

  1. Create a storage account.
  2. Create a container in that storage account and assign it a name.
  3. Retrieve the storage account access keys or connection string.
  4. Add the connection string to the key vault and give it a name (this is the secret name).
  5. Create or recreate your subscription, now including the blobStoreUrl property in the following syntax: blobStoreUrl: "https://<azurekeyvaultname>.vault.azure.net/secrets/<secretname>?tenantId=<domainname>"

Receiving notifications

When Event Hubs receives a notification payload that is larger than 1 MB, the Event Hubs notification won't contain the resource, resourceData, and encryptedContent properties that are included in rich notifications. The Event Hubs notification will instead contain an additionalPayloadStorageId property with an ID that points to the blob in your storage account where these properties have been stored.

What if the Microsoft Graph Change Tracking application is missing?

It's possible that the Microsoft Graph Change Tracking service principal is missing from your tenant, depending on when the tenant was created and administrative operations. The service principal's globally unique appId is 0bf30f3b-4a52-48df-9a82-234910c4a086 and you can use this value to confirm whether it exists or create it if it doesn't.

To confirm whether the service principal exists in your tenant, run the following query. If the service principal exists, the request returns a 200 OK response code and the corresponding Microsoft Graph Change Tracking object in the response body. You must grant the calling app the Application.Read.All permission to run this operation.

GET https://graph.microsoft.com/v1.0/servicePrincipals(appId='0bf30f3b-4a52-48df-9a82-234910c4a086')

If the service principal doesn't exist, create it as follows. You must grant the calling app the Application.ReadWrite.All permission to run this operation.

POST https://graph.microsoft.com/v1.0/servicePrincipals

{
    "appId": "0bf30f3b-4a52-48df-9a82-234910c4a086"
}