Edit

Share via

Manually create resources for Event Grid ingestion

  • Article

Azure Data Explorer offers continuous ingestion from Azure Storage (Azure Blob storage and Azure Data Lake Storage Gen2) using an Event Grid Ingestion pipeline. In the Event Grid ingestion pipeline, an Azure Event Grid service routes blob created or blob renamed events from a storage account to Azure Data Explorer via an Azure Event Hubs.

In this article, you learn how to manually create the resources needed for Event Grid Ingestion: an Event Grid subscription, an Event Hubs namespace, and an event hub. Event Hubs namespace and event hub creation are described in the Prerequisites. To use automatic creation of these resources while defining the Event Grid ingestion, see Create an Event Grid data connection in Azure Data Explorer.

Prerequisites

Note

For best performance, create all resources in the same region as the Azure Data Explorer cluster.

Create an Event Grid subscription

  1. In the Azure portal, go to your storage account.

  2. In the left menu, select Events > + Event Subscription.

    Screenshot of the create Event Grid page.

  3. In the Create Event Subscription window within the Basic tab, provide the following values:

    Screenshot of create Event Grid subscription page, , showing the basic tab with subscription details.

    Setting Suggested value Field description
    Name test-grid-connection The name of the Event Grid subscription that you want to create.
    Event Schema Event Grid Schema The schema that should be used for the Event Grid.
    Topic Type Storage account The type of Event Grid topic. Automatically populated.
    Source Resource gridteststorage1 The name of your storage account. Automatically populated.
    System Topic Name gridteststorage1... The system topic where Azure Storage publishes events. This system topic then forwards the event to a subscriber that receives and processes events. Automatically populated.
    Filter to Event Types Blob Created Which specific events to get notified for. When creating the subscription, select one of the supported types: Microsoft.Storage.BlobCreated or Microsoft.Storage.BlobRenamed. Blob renaming is supported only for ADLSv2 storage.

  4. Under ENDPOINT DETAILS, select Event Hubs.

    Pick an event handler to receive your events - event hub - Azure Data Explorer.

  5. Select Select an endpoint and fill in the event hub you created, for example test-hub.

  6. Under MANAGED IDENTITY FOR DELIVERY, optionally select a managed identity type using the information in the following table.

    Screenshot of the managed identity for delivery section, showing the select managed identity type option.

    Type Description
    None (default) Events aren't delivered using managed identities.
    System assigned Events are delivered using a system-assigned managed identity.
    IMPORTANT: Before using this option, make sure you enabled system-assigned identity on the Event Grid system topic and gave it the Azure Event Hubs Data Sender role on the event hub. For more information, see Enable managed identity for system topics.

    Important

    If you disable local authentication on the Event Hubs namespace that contains the event hub used for streaming notifications, you must use managed identities to deliver events when using this event subscription.

  7. Select the Filters tab if you want to filter events.

    Use Subject Filters to track specific subjects events. Set the filters for the notifications as follows:

    Screenshot of the filters tab, showing the filter event options.

    1. Select Enable subject filtering
    2. Subject Begins With field is the literal prefix of the subject. Since the pattern applied is startswith, it can span multiple containers, folders, or blobs. No wildcards are allowed.
      • To define a filter on the blob container, set the field as follows: /blobServices/default/containers/[container prefix].
      • To define a filter on a blob prefix (or a folder in Azure Data Lake Gen2), set the field as follows: /blobServices/default/containers/[container name]/blobs/[folder/blob prefix].
    3. Subject Ends With field is the literal suffix of the blob. No wildcards are allowed.
    4. Case-sensitive subject matching field indicates whether the prefix and suffix filters are case-sensitive.

    For more information about filtering events, see blob storage events.

    Use Advanced Filters to add custom filters that meet your requirements.

    For example, When using the Azure Data Lake SDK to upload a file, file creation triggers an Event Grid event with size 0. This event is discarded by Azure Data Explorer. File flushing triggers another event if the Close parameter is set to true. This event indicates that this is the final update and the file stream has been closed. To avoid unnecessary processing and empty file ingestion errors, filter out CreateFile events using the following filter:

    Screenshot showing how to filter out create file events.

    Setting Suggested value Field description
    Key data.api The field in the event schema used for filtering.
    Operator String is not in An operator to be evaluates on the selected key.
    Value CreateFile The value used to evaluate the key.

  8. Select Create

Next step

Create an Event Grid data connection