Azure Data Explorer Kusto emulator
The Kusto emulator is a local environment that encapsulates the query engine. You can use the environment to facilitate local development and automated testing. Since the environment runs locally, it doesn't require provisioning Azure services or incurring any cost; it's a free offering under the Microsoft Software License Terms.
- The emulator is provided as-is, without any support or warranties.
- The emulator isn't intended for use in production environments.
- The license terms explicitly prohibit benchmark tests using the emulator as it is not intended or optimized for that purpose. The emulator has a very different performance profile compared to the Azure Data Explorer service.
The emulator supports all commands and queries within its architecture limitations. For instance, Row Level Security policy commands are supported but since there's no authentication in the emulator, they aren't useful.
The emulator doesn't include the data management service and hence doesn't support queued, managed, or streaming ingestion. However, you can ingest data using ingestion commands. Data can be ingested from local files, external tables, or external data. By default, ingested data is persisted on the disk within the container; hence, the data is as transient as the container itself. If you want to retain the data, you can persist it outside the container by mounting a volume on the container and persisting the data to the volume.
Kusto emulator scenarios
The main scenarios for the emulator are:
- Local development: You can use the emulator to develop queries without provisioning an Azure service and without a connection to the internet.
- Automated testing: You can perform automated tests using the emulator without provisioning an Azure service and without a connection to the internet. It can be an efficient way to perform automated tests using the Kusto Query Engine.
Kusto emulator vs. free cluster
There are some overlaps between the Kusto emulator and the free cluster offerings. The following table contrasts the features of the offerings.
|Item||Kusto emulator||Free cluster|
|Storage (uncompressed)||Only limited by host capacity||Approx. 100 GB|
|Databases||Up to 10000||Up to 10|
|Tables per database||10000||Up to 100|
|Columns per table||10000||Up to 200|
|External tables||To local files||No support|
|Materialized views per database||1000||Up to 5|
|Managed ingestion pipelines||None||All of them|
|Security features||None||Encrypted connection only|
|Long term data management||No extent merge capability||Full support|
|Internet requirement||No connectivity required||Cloud service|
When choosing the best option for your use case, you should consider the following:
- Local development: The best fit for local development depends on the features you require. For instance, if your scenario requires the use of managed pipelines, the free cluster offering works best. On the other hand, if your scenario requires local development in a disconnected environment, the Kusto emulator would be a better fit.
- Automated testing: In general, the Kusto emulator is more suited to automated testing than the free cluster offering. It's faster to provision and doesn't require any Azure principal to set up.
In general, this offering isn't suited for production workloads.
- Provided as-is, without any support or warranties
- Doesn't provide any security features, including:
- Access control
- Encrypted connections; connection is through an HTTP connection
- Encryption at rest
- No managed pipelines, including:
- Event Hubs
- IoT Hub
- Event Grid
- No support for ingestion endpoints, including Kusto.Ingest SDKs
- No streaming ingestion
- Although ingested data can be stored externally to the container, we don't recommend persisting data for long periods of time for the following reasons:
- The Python plugin isn't supported