Connect to and manage Azure Data Explorer in Microsoft Purview

This article outlines how to register Azure Data Explorer, and how to authenticate and interact with Azure Data Explorer in Microsoft Purview. For more information about Microsoft Purview, read the introductory article.

Supported capabilities

Metadata Extraction Full Scan Incremental Scan Scoped Scan Classification Access Policy Lineage Data Sharing
Yes Yes Yes Yes Yes No Limited** No

** Lineage is supported if dataset is used as a source/sink in Data Factory Copy activity

Prerequisites

Register

This section describes how to register Azure Data Explorer in Microsoft Purview using the Microsoft Purview governance portal.

Authentication for registration

There are several methods available for authentication for Azure Data Explorer:

Service Principal to register

To use service principal authentication for scans, you can use an existing one or create a new one.

Note

If you have to create a new Service Principal, please follow these steps:

  1. Navigate to the Azure portal.
  2. Select Azure Active Directory from the left-hand side menu.
  3. Select App registrations.
  4. Select + New application registration.
  5. Enter a name for the application (the service principal name).
  6. Select Accounts in this organizational directory only.
  7. For Redirect URI select Web and enter any URL you want; it doesn't have to be real or work.
  8. Then select Register.

It is required to get the Service Principal's application ID and secret:

  1. Navigate to your Service Principal in the Azure portal
  2. Copy the values the Application (client) ID from Overview and Client secret from Certificates & secrets.
  3. Navigate to your key vault
  4. Select Settings > Secrets
  5. Select + Generate/Import and enter the Name of your choice and Value as the Client secret from your Service Principal
  6. Select Create to complete
  7. If your key vault is not connected to Microsoft Purview yet, you will need to create a new key vault connection
  8. Finally, create a new credential using the Service Principal to set up your scan

Granting the Service Principal access to your Azure data explorer instance

  1. Navigate to the Azure portal. Then navigate to your Azure data explorer instance.

  2. Add the service principal to the AllDatabasesViewer role in the Permissions tab.

System or user assigned managed identity to register

  • System-assigned managed identity - As soon as the Microsoft Purview Account is created, a system-assigned managed identity (SAMI) is created automatically in Azure AD tenant. It has the same name as your Microsoft Purview account.

  • User-assigned managed identity (preview) - Similar to a system-managed identity, a user-assigned managed identity (UAMI) is a credential resource that can be used to allow Microsoft Purview to authenticate against Azure Active Directory. For more information, you can see our User-assigned managed identity guide.

To register using either of these managed identities, follow these steps:

  1. If you would like to use a user-assigned managed identity and have not created one, follow the steps to create the identity in the User-assigned managed identity guide.

  2. Navigate to the Azure portal. Then navigate to your Azure data explorer instance.

  3. Select the Permissions tab on the left pane.

  4. Add the SAMI or UAMI to the AllDatabasesViewer role in the Permissions tab.

Steps to register

To register a new Azure Data Explorer (Kusto) account in your data catalog, follow these steps:

  1. Navigate to your Microsoft Purview account
  2. Select Data Map on the left navigation.
  3. Select Register
  4. On Register sources, select Azure Data Explorer
  5. Select Continue

register new data source

On the Register sources (Azure Data Explorer (Kusto)) screen, do the following:

  1. Enter a Name that the data source will be listed with in the Catalog.
  2. Choose your Azure subscription to filter down Azure Data Explorer.
  3. Select an appropriate cluster.
  4. Select a collection or create a new one (Optional).
  5. Select Register to register the data source.

register sources options

Scan

Follow the steps below to scan Azure Data Explorer to automatically identify assets and classify your data. For more information about scanning in general, see our introduction to scans and ingestion.

Create and run scan

To create and run a new scan, follow these steps:

  1. Select the Data Map tab on the left pane in the Microsoft Purview governance portal.

  2. Select the Azure Data Explorer source that you registered.

  3. Select New scan

  4. Select the credential to connect to your data source.

    Set up scan

  5. You can scope your scan to specific databases by choosing the appropriate items in the list.

    Scope your scan

  6. Then select a scan rule set. You can choose between the system default, existing custom rule sets, or create a new rule set inline.

    Scan rule set

  7. Choose your scan trigger. You can set up a schedule or run the scan once.

    trigger

  8. Review your scan and select Save and run.

View your scans and scan runs

To view existing scans:

  1. Go to the Microsoft Purview governance portal. Select the Data Map tab on the left pane.

  2. Select the desired data source. You can view a list of existing scans on that data source under Recent scans, or you can view all scans on the Scans tab.

  3. Select the scan that has results you want to view.

    The page that appears shows you all of the previous scan runs, along with the status and metrics for each scan run. It also displays:

    • Whether your scan was scheduled or manual.
    • How many assets had classifications applied.
    • How many total assets were discovered.
    • The start and end times of the scan, and the total scan duration.

Manage your scans - edit, delete, or cancel

To manage or delete a scan:

  1. Go to the Microsoft Purview governance portal. Select the Data Map tab on the left pane.

  2. Select the desired data source. You can view a list of existing scans on that data source under Recent scans, or you can view all scans on the Scans tab.

  3. Select the scan that you want to manage. You can then:

    • Edit the scan by selecting Edit scan.
    • Cancel an in-progress scan by selecting Cancel scan run.
    • Delete your scan by selecting Delete scan.

Note

  • Deleting your scan does not delete catalog assets created from previous scans.
  • The asset will no longer be updated with schema changes if your source table has changed and you re-scan the source table after editing the description on the Schema tab of Microsoft Purview.

Next steps

Now that you have registered your source, follow the below guides to learn more about Microsoft Purview and your data.