Receive change notifications through Azure Event Hubs

Webhooks aren't suited for receiving change notifications in high throughput scenarios or when the receiver can't expose a publicly available notification URL. As an alternative, you can use Azure Event Hubs.

Examples of high throughput scenarios where you can use Azure Event Hubs include applications subscribing to a large set of resources, applications subscribing to resources that change frequently, and multitenant applications that subscribe to resources across a large set of organizations.

The article guides you through the process of managing your Microsoft Graph subscription and how to receive change notifications through Azure Event Hubs.

Using Azure Event Hubs to receive change notification

Azure Event Hubs is a popular real-time events ingestion and distribution service built for scale. Using Azure Event Hubs to receive change notifications differs from webhooks in a few ways, including:

• You don't rely on publicly exposed notification URLs. The Event Hubs SDK relays the notifications to your application.
• You don't need to reply to the notification URL validation. You can ignore the validation message that you receive.
• You need to provision an event hub.
• You need to provision an Azure Key Vault.

Set up the Azure Event Hubs authentication

Use Azure CLI

The Azure CLI allows you to script and automate administrative tasks in Azure. The CLI can be installed on your local computer or run directly from the Azure Cloud Shell.

# --------------
# TODO: update the following values
#sets the name of the resource group
resourcegroup=rg-graphevents-dev
#sets the location of the resources
location='uk south'
#sets the name of the Azure Event Hubs namespace
evhamespacename=evh-graphevents-dev
#sets the name of the hub under the namespace
evhhubname=graphevents
#sets the name of the access policy to the hub
evhpolicyname=grapheventspolicy
#sets the name of the Azure KeyVault
keyvaultname=kv-graphevents
#sets the name of the secret in Azure KeyVault that will contain the connection string to the hub
keyvaultsecretname=grapheventsconnectionstring
# --------------
az group create --location $location --name $resourcegroup
az eventhubs namespace create --name $evhamespacename --resource-group $resourcegroup --sku Basic --location $location
az eventhubs eventhub create --name $evhhubname --namespace-name $evhamespacename --resource-group $resourcegroup --partition-count 2 --message-retention 1
az eventhubs eventhub authorization-rule create --name $evhpolicyname --eventhub-name $evhhubname --namespace-name $evhamespacename --resource-group $resourcegroup --rights Send
evhprimaryconnectionstring=`az eventhubs eventhub authorization-rule keys list --name $evhpolicyname --eventhub-name $evhhubname --namespace-name $evhamespacename --resource-group $resourcegroup --query "primaryConnectionString" --output tsv`
az keyvault create --name $keyvaultname --resource-group $resourcegroup --location $location --enable-soft-delete true --sku standard --retention-days 90
az keyvault secret set --name $keyvaultsecretname --value $evhprimaryconnectionstring --vault-name $keyvaultname --output none
graphspn=`az ad sp list --display-name 'Microsoft Graph Change Tracking' --query "[].appId" --output tsv`
az keyvault set-policy --name $keyvaultname --resource-group $resourcegroup --secret-permissions get --spn $graphspn --output none
keyvaulturi=`az keyvault show --name $keyvaultname --resource-group $resourcegroup --query "properties.vaultUri" --output tsv`
domainname=`az ad signed-in-user show --query 'userPrincipalName' | cut -d '@' -f 2 | sed 's/\"//'`
notificationUrl="EventHub:${keyvaulturi}secrets/${keyvaultsecretname}?tenantId=${domainname}"
echo "Notification Url:\n${notificationUrl}"

Note: The script provided here is compatible with Linux-based shells, Windows WSL, and Azure Cloud Shell. It requires some updates to run in Windows shells.

Use the Azure portal

Configure the event hub

In this section, you:

• Create an Event Hubs namespace.
• Add a hub to that namespace to relay and deliver notifications.
• Add a shared access policy that allows you to get a connection string to the newly created hub.

Steps:

1. Sign in to the Azure portal with privileges to create resources in your Azure subscription.
2. Select Create a resource > type Event Hubs in the search bar > select the Event Hubs suggestion.
3. On the Event Hubs creation page, select Create.
4. Fill in the Event Hubs namespace creation details, and then select Create.
5. When the event hub namespace is provisioned, go to the page for the namespace.
6. Select Event Hubs and + Event Hub.
7. Give a name to the new event hub, and select Create.
8. After the event hub is created, select the name of the event hub, and then select Shared access policies and + Add to add a new policy.
9. Give a name to the policy, check Send, and select Create.
10. After the policy is created, select the name of the policy to open the details panel, and then copy the Connection string-primary key value. Record the value; you need it for the next step.

Configure the Azure Key Vault

In order to access the event hub securely and to allow for key rotations, Microsoft Graph gets the connection string to the event hub through Azure Key Vault.

In this section, you:

• Create an Azure Key Vault for storing the secret.
• Add the connection string to the event hub as a secret.
• Add an access policy for Microsoft Graph to access the secret.

Steps:

1. Sign in to the Azure portal with privileges to create resources in your Azure subscription.
2. Select Create a resource > type Key Vault in the search bar > select the Key Vault suggestion.
3. On the Key Vault creation page, select Create.
4. Fill in the Key Vault creation details, and then select Review + Create and Create.
5. Go to the newly created key vault using the Go to resource from the notification.
6. Copy the DNS name; you need it later in this article.
7. Go to Secrets and select + Generate/Import.
8. Give a name to the secret, and keep the name for later; you need it later in this article. For the value, paste in the connection string you generated at the Event Hubs step. Select Create.
9. Select Access Policies and + Add Access Policy.
10. For Secret permissions, select Get, and for Select Principal, select Microsoft Graph Change Tracking. Select Add.

Use the Azure portal (RBAC)

Configure the event hub

In this section you:

• Create an Event Hubs namespace.
• Add a hub to that namespace to relay and deliver notifications.
• Add a shared access policy that allows you to get a connection string to the newly created hub.

Steps:

1. Sign in to the Azure portal with privileges to create resources in your Azure subscription.
2. Select Create a resource > type Event Hubs in the search bar > select the Event Hubs suggestion.
3. On the Event Hubs creation page, select Create.
4. Fill in the Event Hubs namespace creation details, and then select Create.
5. When the event hub namespace is provisioned, go to the page for the namespace.
6. Select Event Hubs and + Event Hub.
7. Give a name to the new event hub, and select Create.
8. After the event hub is created, select the name of the event hub, and then select Access Control (IAM) from the sidebar.
9. Select Role Assignments.
10. Select + Add and select Add Role Assignment.
11. Under Role > Job function roles > select Azure Event Hubs Data Sender > select Next.
12. Under the Members tab, select Assign access to User, group, or service principal.
13. Select + Select members > search for and select Microsoft Graph Change Tracking.
14. Select Review + assign to complete the process.

Create the subscription and receive notifications

After you create the required Azure KeyVault and Azure Event Hubs services, you can now create your change notification subscription and start receiving change notifications via Azure Event Hubs.

Create the subscription

Subscriptions to change notifications with Event Hubs are almost identical to change notifications with webhooks. The main difference during subscription creation is the notificationUrl. You must set it to EventHub:https://<azurekeyvaultname>.vault.azure.net/secrets/<secretname>?tenantId=<domainname>, with the following values:

• azurekeyvaultname - The name you gave to the key vault when you created it. Can be found in the DNS name.
• secretname - The name you gave to the secret when you created it. Can be found on the Azure Key Vault Secrets page.
• domainname - The name of your tenant; for example, contoso.com or contoso.com. Because this domain is used to access the Azure Key Vault, it's important that it matches the domain used by the Azure subscription that holds the Azure Key Vault. To get this information, you can go to the overview page of the Azure Key Vault you created and select the subscription. The domain name is displayed under the Directory field.

Note

Duplicate subscriptions aren't allowed. When a subscription request contains the same values for changeType and resource that an existing subscription contains, the request fails with an HTTP error code 409 Conflict, and the error message Subscription Id <> already exists for the requested combination.

Receive notifications

Change notifications are now delivered to your application by Event Hubs. For details, see receiving events in the Event Hubs documentation.

Before you can receive the notifications in your application, you need to create another shared access policy with a "Listen" permission and obtain the connection string, similar to the steps listed in Configure the event hub.

Tip

Create a separate policy for the application that listens to Event Hubs messages instead of reusing the same connection string you set in Azure KeyVault. This separation follows the principle of least privilege by ensuring that each component of the solution has only the permissions it needs.

Handling validation notifications

Your application receives validation notifications whenever it creates a new subscription. You should ignore these notifications. The following example represents the body of a validation message.

 {
    "value":[
        {
            "subscriptionId":"NA",
            "subscriptionExpirationDateTime":"NA",
            "clientState":"NA",
            "changeType":"Validation: Testing client application reachability for subscription Request-Id: 522a8e7e-096a-494c-aaf1-ac0dcfca45b7",
            "resource":"NA",
            "resourceData":{
                "@odata.type":"NA",
                "@odata.id":"NA",
                "id":"NA"
            }
        }
    ]
}

Subscriptions for rich notifications with large payloads

The maximum message size for Event Hubs is 1 MB. When you use rich notifications, you might expect notifications that exceed this limit. To receive notifications larger than 1 MB through Event Hubs, you must also add a blob storage account to your subscription request.

Set up storage and create a subscription

1. Create a storage account.
2. Create a container in the storage account and assign it a name.
3. Retrieve the storage account access keys or connection string.
4. Add the connection string to the key vault and give it a name. This value is the secret name.
5. Create or recreate your subscription, now including the blobStoreUrl property in the following syntax: blobStoreUrl: "https://<azurekeyvaultname>.vault.azure.net/secrets/<secretname>?tenantId=<domainname>"

Receive rich notifications

When Event Hubs receives a notification payload that is larger than 1 MB, the notification doesn't contain the resource, resourceData, and encryptedContent properties that are included in rich notifications. The notification instead contains an additionalPayloadStorageId property with an ID that points to the blob in your storage account where these properties are stored.

What if the Microsoft Graph Change Tracking application is missing?

The Microsoft Graph Change Tracking service principal might be missing from your tenant, depending on when the tenant was created and administrative operations. The service principal's globally unique appId is 0bf30f3b-4a52-48df-9a82-234910c4a086 and you can run the following query to confirm whether it exists in the tenant.

HTTP

GET https://graph.microsoft.com/v1.0/servicePrincipals(appId='0bf30f3b-4a52-48df-9a82-234910c4a086')

C#

// Code snippets are only available for the latest version. Current version is 5.x

// To initialize your graphClient, see https://learn.microsoft.com/en-us/graph/sdks/create-client?from=snippets&tabs=csharp
var result = await graphClient.ServicePrincipalsWithAppId("{appId}").GetAsync();

Read the SDK documentation for details on how to add the SDK to your project and create an authProvider instance.

CLI

// THE CLI IS IN PREVIEW. NON-PRODUCTION USE ONLY
mgc service-principals-with-app-id get --app-id {app-id}

Read the SDK documentation for details on how to add the SDK to your project and create an authProvider instance.

Go

import (
	  "context"
	  msgraphsdk "github.com/microsoftgraph/msgraph-sdk-go"
	  //other-imports
)

graphClient := msgraphsdk.NewGraphServiceClientWithCredentials(cred, scopes)



appId := "{appId}"
servicePrincipals, err := graphClient.ServicePrincipalsWithAppId(&appId).Get(context.Background(), nil)

Read the SDK documentation for details on how to add the SDK to your project and create an authProvider instance.

Java

// Code snippets are only available for the latest version. Current version is 6.x

GraphServiceClient graphClient = new GraphServiceClient(requestAdapter);

ServicePrincipal result = graphClient.servicePrincipalsWithAppId("{appId}").get();

Read the SDK documentation for details on how to add the SDK to your project and create an authProvider instance.

JavaScript

Snippet not available

Read the SDK documentation for details on how to add the SDK to your project and create an authProvider instance.

PHP

<?php
use Microsoft\Graph\GraphServiceClient;


$graphServiceClient = new GraphServiceClient($tokenRequestContext, $scopes);


$result = $graphServiceClient->servicePrincipalsWithAppId('{appId}', )->get()->wait();

Read the SDK documentation for details on how to add the SDK to your project and create an authProvider instance.

PowerShell

Import-Module Microsoft.Graph.Applications

Get-MgServicePrincipalByAppId

Read the SDK documentation for details on how to add the SDK to your project and create an authProvider instance.

Python

from msgraph import GraphServiceClient

graph_client = GraphServiceClient(credentials, scopes)


result = await graph_client.service_principals_with_app_id("{appId}").get()

Read the SDK documentation for details on how to add the SDK to your project and create an authProvider instance.

If the service principal doesn't exist, create it as follows. You must grant the calling app the Application.ReadWrite.All permission to run this operation.

Method 1

HTTP

POST https://graph.microsoft.com/v1.0/servicePrincipals
Content-type: application/json

{
    "appId": "0bf30f3b-4a52-48df-9a82-234910c4a086"
}

C#

// Code snippets are only available for the latest version. Current version is 5.x

// Dependencies
using Microsoft.Graph.Models;

var requestBody = new ServicePrincipal
{
	AppId = "0bf30f3b-4a52-48df-9a82-234910c4a086",
};

// To initialize your graphClient, see https://learn.microsoft.com/en-us/graph/sdks/create-client?from=snippets&tabs=csharp
var result = await graphClient.ServicePrincipals.PostAsync(requestBody);

Read the SDK documentation for details on how to add the SDK to your project and create an authProvider instance.

CLI

// THE CLI IS IN PREVIEW. NON-PRODUCTION USE ONLY
mgc service-principals create --body '{\
    "appId": "0bf30f3b-4a52-48df-9a82-234910c4a086"\
}\
'

Read the SDK documentation for details on how to add the SDK to your project and create an authProvider instance.

Go

import (
	  "context"
	  msgraphsdk "github.com/microsoftgraph/msgraph-sdk-go"
	  graphmodels "github.com/microsoftgraph/msgraph-sdk-go/models"
	  //other-imports
)

graphClient := msgraphsdk.NewGraphServiceClientWithCredentials(cred, scopes)


requestBody := graphmodels.NewServicePrincipal()
appId := "0bf30f3b-4a52-48df-9a82-234910c4a086"
requestBody.SetAppId(&appId) 

servicePrincipals, err := graphClient.ServicePrincipals().Post(context.Background(), requestBody, nil)

Read the SDK documentation for details on how to add the SDK to your project and create an authProvider instance.

Java

// Code snippets are only available for the latest version. Current version is 6.x

GraphServiceClient graphClient = new GraphServiceClient(requestAdapter);

ServicePrincipal servicePrincipal = new ServicePrincipal();
servicePrincipal.setAppId("0bf30f3b-4a52-48df-9a82-234910c4a086");
ServicePrincipal result = graphClient.servicePrincipals().post(servicePrincipal);

Read the SDK documentation for details on how to add the SDK to your project and create an authProvider instance.

JavaScript

const options = {
	authProvider,
};

const client = Client.init(options);

const servicePrincipal = {
    appId: '0bf30f3b-4a52-48df-9a82-234910c4a086'
};

await client.api('/servicePrincipals')
	.post(servicePrincipal);

Read the SDK documentation for details on how to add the SDK to your project and create an authProvider instance.

PHP

<?php
use Microsoft\Graph\GraphServiceClient;
use Microsoft\Graph\Generated\Models\ServicePrincipal;


$graphServiceClient = new GraphServiceClient($tokenRequestContext, $scopes);

$requestBody = new ServicePrincipal();
$requestBody->setAppId('0bf30f3b-4a52-48df-9a82-234910c4a086');

$result = $graphServiceClient->servicePrincipals()->post($requestBody)->wait();

Read the SDK documentation for details on how to add the SDK to your project and create an authProvider instance.

PowerShell

Import-Module Microsoft.Graph.Applications

$params = @{
	appId = "0bf30f3b-4a52-48df-9a82-234910c4a086"
}

New-MgServicePrincipal -BodyParameter $params

Read the SDK documentation for details on how to add the SDK to your project and create an authProvider instance.

Python

from msgraph import GraphServiceClient
from msgraph.generated.models.service_principal import ServicePrincipal

graph_client = GraphServiceClient(credentials, scopes)

request_body = ServicePrincipal(
	app_id = "0bf30f3b-4a52-48df-9a82-234910c4a086",
)

result = await graph_client.service_principals.post(request_body)

Read the SDK documentation for details on how to add the SDK to your project and create an authProvider instance.

Method 2

POST https://graph.microsoft.com/v1.0/servicePrincipals(appId='0bf30f3b-4a52-48df-9a82-234910c4a086')
Content-type: application/json
Prefer: create-if-missing

{
    "displayName": "Microsoft Graph Change Tracking"
}

Related content

• Change notifications overview
• See the following Azure Event Hubs quick starts:
  • .NET Core
  • Java
  • Python
  • JavaScript