Ingest from storage
.ingest into command ingests data into a table by "pulling" the data
from one or more cloud storage files.
For example, the command
can retrieve 1000 CSV-formatted blobs from Azure Blob Storage, parse
them, and ingest them together into a single target table.
Data is appended to the table
without affecting existing records, and without modifying the table's schema. Minimal required permission levels are required so that you can ingest data into all existing tables in a database or into a specific existing table. For more information, see Permissions on the Engine Service.
table TableName SourceDataLocator [
= IngestionPropertyValue [
async: If specified, the command will return immediately, and continue ingestion in the background. The results of the command will include an
OperationIdvalue that can then be used with the
.show operationcommand to retrieve the ingestion completion status and results.
TableName: The name of the table to ingest data into. The table name is always relative to the database in context, and its schema is the schema that will be assumed for the data if no schema mapping object is provided.
SourceDataLocator: A literal of type
string, or a comma-delimited list of such literals surrounded by
)characters, representing storage connection strings. Kusto uses a URI format to describe the storage files containing the data to pull.
It is strongly recommended to use obfuscated string literals for the SourceDataPointer that includes actual credentials in it. The service will be sure to scrub credentials in its internal traces, error messages, etc.
The following table lists the properties supported by Azure Data Explorer, describes them, and provides examples:
||A string value that indicates how to map data from the source file to the actual columns in the table. Define the
||A string value that indicates how to map data from the source file to the actual columns in the table using a named mapping policy object. Define the
||The datetime value (formatted as an ISO8601 string) to use at the creation time of the ingested data extents. If unspecified, the current value (
||A Boolean value that, if specified, instructs the command to extend the schema of the table (defaults to
||If the original table schema is
||For ingest-from-query commands, the folder to assign to the table. If the table already exists, this property will override the table's folder.||
||The data format (see supported data formats).||
||A string value that, if specified, prevents ingestion from succeeding if the table already has data tagged with an
||A Boolean value that, if set to
||A Boolean value that, if specified, indicates that the command should persist the detailed results (even if successful) so that the .show operation details command could retrieve them. Defaults to
||A Boolean value that, if specified, describes whether to enable the Ingestion Time Policy on a table that is created by this command. The default is
||A Boolean value that, if specified, describes whether the command may recreate the schema of the table. This property applies only to the
||A list of tags to associate with the ingested data, formatted as a JSON string||
||A JSON string that indicates which validations to run during ingestion. See Data ingestion for an explanation of the different options.||
||Use this property when ingesting data from storage that has a ZIP archive. This is a string value indicating the regular expression to use when selecting which files in the ZIP archive to ingest. All other files in the archive will be ignored.||
The result of the command is a table with as many records as there are data shards ("extents") generated by the command. If no data shards have been generated, a single record is returned with an empty (zero-valued) extent ID.
||The unique identifier for the data shard that was generated by the command.|
||One or more storage files that are related to this record.|
||How long it took to perform ingestion.|
||Whether this record represents an ingestion failure or not.|
||A unique ID representing the operation. Can be used with the
This command doesn't modify the schema of the table being ingested into. If necessary, the data is "coerced" into this schema during ingestion, not the other way around (extra columns are ignored, and missing columns are treated as null values).
The next example instructs the engine to read two blobs from Azure Blob Storage
as CSV files, and ingest their contents into table
an Azure Storage shared access signature (SAS) which gives read access to each
blob. Note also the use of obfuscated strings (the
h in front of the string
values) to ensure that the SAS is never recorded.
.ingest into table T ( h'https://contoso.blob.core.windows.net/container/file1.csv?...', h'https://contoso.blob.core.windows.net/container/file2.csv?...' )
The next example is for ingesting data from Azure Data Lake Storage Gen 2
(ADLSv2). The credentials used here (
...) are the storage account credentials
(shared key), and we use string obfuscation only for the secret part of the
.ingest into table T ( 'abfss://email@example.com/path/to/file1.csv;...' )
The next example ingests a single file from Azure Data Lake Storage (ADLS). It uses the user's credentials to access ADLS (so there's no need to treat the storage URI as containing a secret). It also shows how to specify ingestion properties.
.ingest into table T ('adl://contoso.azuredatalakestore.net/Path/To/File/file1.ext;impersonate') with (format='csv')
The next example ingests a single file from Amazon S3 using an access key ID and a secret access key.
.ingest into table T ('https://bucketname.s3.us-east-1.amazonaws.com/path/to/file.csv;AwsCredentials=AKIAIOSFODNN7EXAMPLE,wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY') with (format='csv')