Use the ingestion wizard to create an Azure Event Hubs data connection for Azure Data Explorer

Azure Data Explorer offers ingestion (data loading) from Event Hubs, a big data streaming platform and event ingestion service. Event hubs can process millions of events per second in near real-time. In this article, you connect an event hub to a table in Azure Data Explorer using the ingestion wizard.

Prerequisites

Note

To enable access between a cluster and a storage account without public access (restricted to private endpoint/service endpoint), see Create a Managed Private Endpoint.

Note

The cluster and event hub should be associated with the same tenants, for different tenants please use SDKs

Ingest data

  1. In the left menu of the Azure Data Explorer web UI, select the Data tab.

    Select the ingestion wizard in the Azure Data Explorer web UI.

  2. In the Ingest data from Event Hub card, select Ingest.

The Ingest data window opens with the Destination tab selected.

Destination tab

Screen shot of destination tab. Cluster, Database, and Table fields must be filled out before proceeding to Next-Source.

  1. The Cluster and Database fields are auto-populated. You may select a different cluster or database from the drop-down menus.

  2. Under Table, select New table and enter a name for the new table. Alternatively, use an existing table.

    Note

    Table names must be between 1 and 1024 characters. You can use alphanumeric, hyphens, and underscores. Special characters aren't supported.

  3. Select Next: Source.

Source tab

  1. Under Source type, select Event Hub.

  2. Under Data Connection, fill in the following fields:

    Screenshot of source tab with project details fields to be filled in - ingest new data to Azure Data Explorer with Event Hubs in the ingestion wizard.

    Setting Suggested value Field description
    Data connection name TestDataConnection The name that identifies your data connection.
    Subscription The subscription ID where the event hub resource is located.
    Event hub namespace The name that identifies your namespace.
    Event hub The event hub you wish to use.
    Consumer group The consumer group defined in your event hub.
    Event system properties Select relevant properties The event hub system properties. If there are multiple records per event message, the system properties will be added to the first one. When adding system properties, create or update table schema and mapping to include the selected properties.
  3. Select Next: Schema.

Schema tab

Data is read from the event hub in form of EventData objects. Supported formats are CSV, JSON, PSV, SCsv, SOHsv TSV, TXT, and TSVE.

For information on schema mapping with JSON-formatted data, see Edit the schema. For information on schema mapping with CSV-formatted data, see Edit the schema.

Screenshot of schema tab in ingest new data to Azure Data Explorer with Event Hubs in the ingestion wizard.

Note

  • If streaming is enabled for the cluster, the option to select Streaming ingestion is available.
  • If streaming is not enabled for the cluster, the option to select Batching time is available. For Event Hubs, the recommended default batching time is 30 seconds.
  1. If the data you see in the preview window isn't complete, you may need more data to create a table with all necessary data fields. Use the following commands to fetch new data from your event hub:

    • Discard and fetch new data: discards the data presented and searches for new events.
    • Fetch more data: Searches for more events in addition to the events already found.

    Note

    To see a preview of your data, your event hub must be sending events.

  2. Select Next: Summary.

Continuous ingestion from Event Hubs

In the Continuous ingestion from Event Hub established window, all steps will be marked with green check marks when establishment finishes successfully. The cards below these steps give you options to explore your data with Quick queries, undo changes made using Tools, or Monitor the Event Hubs connections and data.

Screenshot of final screen in ingestion to Azure Data Explorer from Event Hubs with the ingestion wizard.

Next steps