What is the ingestion wizard?

The ingestion wizard is a user interface for ingesting data in the Azure Data Explorer web UI. The ingestion wizard can be used to ingest data in various forms, as a one-time or continuous ingestion, and create database tables and mapping structures.

The ingestion wizard provides the following features:

  • Intuitive experience guided by the ingestion wizard
  • Ingest data in a matter of minutes
  • Ingest data from different kinds of sources: local file, blobs, and containers (up to 10,000 blobs)
  • Ingest data in various formats
  • Ingest data into new or existing tables
  • Table mapping and schema are suggested to you and easy to change
  • Continue ingestion easily and quickly from a container with Event Grid

Prerequisites

Note

To enable access between a cluster and a storage account without public access (restricted to private endpoint/service endpoint), see Create a Managed Private Endpoint.

Access the ingestion wizard

There are many entry points to the ingestion wizard.

  • In the web UI, select Data in the left pane. Within the Data Management page, select a type of ingestion and select Ingest.

    Screenshot of options to ingest data from the data management window of the Azure Data Explorer web UI interface - Azure Data Explorer.

  • In the web UI, select Query in the left pane. Right-click the database or table and select Ingest data.

    Screenshot of selection of the ingestion wizard in the Azure Data Explorer web UI.

  • In the web UI, select My cluster in the left pane and then select Ingest.

    Ingest new data from welcome to Azure Data Explorer.

  • To access the ingestion wizard from the Azure portal, select Query from the left menu, right-click on the database or table, and select Ingest data.

    Access the ingestion wizard from Azure portal.

Ingestion wizard

Note

This section describes the ingestion wizard in general. The options you select depend on what data format you are ingesting, what kind of data source you are ingesting from, and whether you are ingesting into a new or existing table.

For sample scenarios, see:

The wizard guides you through the following options:

Schema mapping

The service automatically generates schema and ingestion properties, which you can change. You can use an existing mapping structure or create a new one, depending on if you're ingesting to a new or existing table.

In the Schema tab, do the following actions:

  • Confirm the autogenerated compression type.
  • Choose the format of your data. Different formats allow you to make further changes.
  • Change mapping in the Editor window.

File formats

The ingestion wizard supports ingesting from source data in all data formats supported by Azure Data Explorer for ingestion.

Editor window

In the Editor window of the Schema tab, you can adjust data table columns as necessary.

The changes you can make in a table depend on the following parameters:

  • Table type is new or existing
  • Mapping type is new or existing
Table type Mapping type Available adjustments
New table New mapping Change data type, Rename column, New column, Delete column, Update column, Sort ascending, Sort descending
Existing table New mapping New column (on which you can then change data type, rename, and update),
Update column, Sort ascending, Sort descending
Existing mapping Sort ascending, Sort descending

Note

When adding a new column or updating a column, you can change mapping transformations. For more information, see Mapping transformations

Note

At any time, you can open the command editor above the Editor pane. In the command editor, you can view and copy the automatic commands generated from your inputs.

Mapping transformations

Some data format mappings (Parquet, JSON, and Avro) support simple ingest-time transformations. To apply mapping transformations, create or update a column in the Editor window.

Mapping transformations can be performed on a column of Type string or datetime, with the Source having data type int or long. Supported mapping transformations are:

  • DateTimeFromUnixSeconds
  • DateTimeFromUnixMilliseconds
  • DateTimeFromUnixMicroseconds
  • DateTimeFromUnixNanoseconds

Data ingestion

Once you have completed schema mapping and column manipulations, the ingestion wizard starts the data ingestion process.

  • When you ingest data from non-container sources, the ingestion takes immediate effect.

  • If your data source is a container:

    • Azure Data Explorer's batching policy aggregates your data.
    • After ingestion, you can download the ingestion report and review the performance of each blob that was addressed.
    • You can select Create continuous ingestion and set up continuous ingestion using Event Grid.

Initial data exploration

After ingestion, the wizard gives you options to use Quick commands for initial exploration of your data.

More data management actions

Discover the available options within each Data management tab by selecting the links in the following table.

Tab Ingestion options
Manage Create tables, external tables and update batching and retention policies.
One-time ingestion Ingest from local files, blob storage, or a container.
Continuous ingestion Configure continuous ingestion from Event Hubs or a blob container.
Backfill Ingest data from sources as a one time or continuous ingestion.
SDKs The sample app generator wizard is a tool that allows you to create a working app to ingest and query your data in your preferred programming language. Learn more about connectors to expand and explore the capabilities of Azure Data Explorer.

Next steps