Install the Azure Data Explorer Kusto emulator

You can install the Azure Data Explorer Kusto emulator in the following ways:

  • On your own device: Consider using this option if you need to provision a local development environment
  • On a CI/CD agent virtual machine (VM): Use this option if you require a CI/CD pipeline for running automated tests

In this article, you'll learn how to:

Prerequisites

  • The host operating system must be either:
    • Windows Server 2022
    • Windows Server 2019 Version 10.0.17763.2928 or newer
    • Windows 11
  • 2 gigabytes (GB) of RAM minimum; we recommend using 4 GB or more
  • Docker Client

Install the Kusto emulator

The following steps are for using PowerShell to start the emulator using the Kusto emulator container image. For other options, see Run emulator options.

  1. Switch Docker to run with Windows containers. You may need to enable the feature in the Docker settings.

    Screenshot of the Docker settings, showing the Switch to Windows containers option.

  2. Run the following command to start the emulator.

    Important

    The Kusto emulator container image is a free offering under the Microsoft Software License Terms. Since the emulator runs in a container, you must accept the license terms by passing the ACCEPT_EULA environment variable to the container with its value set to Y indicating.

    Note

    • The first time this command is run, Docker pulls the container image which is several GBs in size and may take several minutes to download. Once downloaded, the image is cached and available for subsequent runs without having to download it again.
    • The container must be run in process-isolation mode. This is the default on some versions of Docker. For other versions, you can start the container in Hyper-V isolation mode by adding --isolation=hyperv to the run command.
    • To start the emulator on a Windows Server operating system, make sure you use the latest or stable tag.

      docker run -e ACCEPT_EULA=Y -m 4G -d -p 8080:8080 -t mcr.microsoft.com/azuredataexplorer/kustainer:latest
      
    • To start the image container on a Windows 11, make sure you use the windows11 tag:

      docker run -e ACCEPT_EULA=Y -m 4G -d -p 8080:8080 -t mcr.microsoft.com/azuredataexplorer/kustainer:windows11
      
  3. Run the following command to verify that the container is running.

    docker ps
    

    The command returns a list of running container instances. Verify that the emulator image mcr.microsoft.com/azuredataexplorer/kustainer:latest appears in the list.

    CONTAINER ID   IMAGE                                                  COMMAND                  CREATED          STATUS          PORTS                    NAMES
    a8b51bce21ad   mcr.microsoft.com/azuredataexplorer/kustainer:latest   "powershell -Command¦"   11 minutes ago   Up 10 minutes   0.0.0.0:8080->8080/tcp   contoso
    
  4. Run the following command to verify that Kusto emulator is running. The command runs the .show cluster query against the management API and it should return a StatusCode with value 200.

    Invoke-WebRequest -Method post -ContentType 'application/json' -Body '{"csl":".show cluster"}' http://localhost:8080/v1/rest/mgmt
    

    The command should return something like the following:

    StatusCode        : 200
    StatusDescription : OK
    Content           : {"Tables":[{"TableName":"Table_0","Columns":[{"ColumnName":"NodeId","DataType":"String","ColumnType":"string"},{"ColumnName":"Address","DataType":"St
                        ring","ColumnType":"string"},{"ColumnName":"Name","...
    RawContent        : HTTP/1.1 200 OK
                        Transfer-Encoding: chunked
                        x-ms-client-request-id: unspecified;d239f3aa-7df0-4e46-af0a-edd7139d0511
                        x-ms-activity-id: a0ac8941-7e4c-4176-98fa-b7ebe14fae90
                        Content-Type: application...
    Forms             : {}
    Headers           : {[Transfer-Encoding, chunked], [x-ms-client-request-id, unspecified;d239f3aa-7df0-4e46-af0a-edd7139d0511], [x-ms-activity-id, 
                        a0ac8941-7e4c-4176-98fa-b7ebe14fae90], [Content-Type, application/json]...}
    Images            : {}
    InputFields       : {}
    Links             : {}
    ParsedHtml        : System.__ComObject
    RawContentLength  : 988
    

Run emulator options

You can use any of the following options when running the emulator:

  • Mount a local folder to the container: Use this option to mount a folder in the host environment into the container. Mounting a host folder enables your queries to interact with local files, which is useful for creating a persistent database and ingesting data.

    For example, to mount the folder "D:\host\local" on the host to the folder "c:\kustodatadata" in the container, use the following command on Windows Server:

    docker run -v d:\host\local:c:\kustodata -e ACCEPT_EULA=Y -m 4G -d -p 8080:8080 -t mcr.microsoft.com/azuredataexplorer/kustainer:latest
    
  • Run on a different port: The Kusto emulator exposes access to the Kusto Query Engine on port 8080; hence in other examples you mapped the host port 8080 to the emulator port 8080. You can use this option to map a different host to the engine.

    For example, to map port 9000 on the host to the engine, use the following command on Windows Server:

    docker run -e ACCEPT_EULA=Y -m 4G -d -p 9000:8080 -t mcr.microsoft.com/azuredataexplorer/kustainer:latest
    

Connect to the emulator

You can use any of the following tools to connect to and interact with the emulator:

In the following sections, you'll use Kusto.Explorer to create a database, ingest data, and query it. To learn more, see Using Kusto.Explorer.

Create a database

You'll need a database in your emulator for your data. You can create the following types of database:

  • A volatile database where the database only exists in RAM and is lost when the container is stopped
  • A persistent database where the database is stored in a folder in the container. This allows your database size to exceed the size of your container's RAM. If you want to keep the data between container runs, you'll need to persist the database to a mounted folder.

To create a volatile database

In the Kusto.Explorer Query mode, run the following command to create a volatile database:

.create database <YourDatabaseName> volatile

To create a persistent database

In the Kusto.Explorer Query mode, run the following command to create a volatile database:

.create database <YourDatabaseName> persist (
  @"c:\kustodata\dbs\<YourDatabaseName>\md",
  @"c:\kustodata\dbs\<YourDatabaseName>\data"
  )

Ingest data

To ingest data, you must first create an external table linked to a file and then ingest the data into a table in the database.

Use the steps in the following example to create an external table and ingest data into it. For the example, in the local folder c:\kustodata\sample, you'll create a file called sample.csv with the following data:

Alice, 1
Bob, 2
Carl, 3
  1. Run the following command to create a table to receive the data:

    .create table MyIngestedSample(Name:string, Id:int)
    
  2. Run the following command to ingest the file into the table:

    .ingest into table MyIngestedSample(@"c:\kustodata\sample.csv")
    

Query data

You can view the data in the table using the following query:

MyIngestedSample
| summarize sum(Id), avg(Id)

Stopping the container

  1. You can stop the container by running the following command to obtain the container ID:

    docker ps
    
  2. Run the following command with the container ID:

    docker stop <containerID>
    

Next steps