Manually create resources for Event Grid ingestion

Azure Data Explorer offers continuous ingestion from Azure Storage (Azure Blob storage and Azure Data Lake Storage Gen2) using an Event Grid Ingestion pipeline. In the Event Grid ingestion pipeline, an Azure Event Grid service routes blob created or blob renamed events from a storage account to Azure Data Explorer via an Azure Event Hubs.

In this article, you learn how to create manually the resources needed for Event Grid Ingestion: Event Grid subscription, event hub namespace, and event hub. event hub namespace and event hub creation are described in the Prerequisites. To use automatic creation of these resources while defining the Event Grid ingestion, see Create an Event Grid data connection in Azure Data Explorer.

Prerequisites

Note

For best performance, create all resources in the same region as the Azure Data Explorer cluster.

Create an Event Grid subscription

  1. In the Azure portal, go to your storage account.

  2. In the left menu, select Events > Event Subscription.

    Create event grid subscription.

  3. In the Create Event Subscription window within the Basic tab, provide the following values:

    Create event subscription values to enter.

    Setting Suggested value Field description
    Name test-grid-connection The name of the event grid subscription that you want to create.
    Event Schema Event Grid Schema The schema that should be used for the Event Grid.
    Topic Type Storage account The type of event grid topic. Automatically populated.
    Source Resource gridteststorage1 The name of your storage account. Automatically populated.
    System Topic Name gridteststorage1... The system topic where Azure Storage publishes events. This system topic then forwards the event to a subscriber that receives and processes events. Automatically populated.
    Filter to Event Types Blob Created Which specific events to get notified for. When creating the subscription, select one of the supported types: Microsoft.Storage.BlobCreated or Microsoft.Storage.BlobRenamed. Blob renaming is supported only for ADLSv2 storage.
  4. In ENDPOINT DETAILS, select Event Hubs.

    Pick an event handler to receive your events - event hub - Azure Data Explorer.

  5. Click Select an endpoint and fill in the event hub you created, for example test-hub.

  6. Select the Filters tab if you want to filter events.

    Use Subject Filters to track specific subjects events. Set the filters for the notifications as follows:

    Filters tab event grid.

    1. Select Enable subject filtering
    2. Subject Begins With field is the literal prefix of the subject. Since the pattern applied is startswith, it can span multiple containers, folders, or blobs. No wildcards are allowed.
      • To define a filter on the blob container, set the field as follows: /blobServices/default/containers/[container prefix].
      • To define a filter on a blob prefix (or a folder in Azure Data Lake Gen2), set the field as follows: /blobServices/default/containers/[container name]/blobs/[folder/blob prefix].
    3. Subject Ends With field is the literal suffix of the blob. No wildcards are allowed.
    4. Case-sensitive subject matching field indicates whether the prefix and suffix filters are case-sensitive.

    For more information about filtering events, see blob storage events.

    Use Advanced Filters to add custom filters that meet your requirements.

    For example, When using the Azure Data Lake SDK to upload a file, file creation triggers an Event Grid event with size 0. This event is ignored by Azure Data Explorer. File flushing triggers another event if the Close parameter is set to true. This event indicates that this is the final update and the file stream has been closed. To filter for FlushAndClose events and remove file creation events with size 0, use the following filter:

    Screenshot showing how to filter for flush and close events.

    Setting Suggested value Field description
    Key data.api The field in the event schema used for filtering.
    Operator String is in An operator to be evaluates on the selected key.
    Value FlushWithClose The value used to evaluate the key.
  7. Select Create

Next steps