Connect to and manage Azure Databricks Unity Catalog in Microsoft Purview

  • Article

This article outlines how to register Azure Databricks, and how to authenticate and interact with Azure Databricks Unity Catalog in Microsoft Purview. For more information about Microsoft Purview, read the introductory article.

Supported capabilities

Metadata Extraction Full Scan Incremental Scan Scoped Scan Classification Labeling Access Policy Lineage Data Sharing Live view
Yes Yes No Yes Yes Yes No No No No

When scanning Azure Databricks Unity Catalog, Microsoft Purview supports:

  • Metastore
  • Catalogs
  • Schemas
  • Tables including the columns
  • Views including the columns

When setting up scan, you can choose to scan the entire Unity Catalog, or scope the scan to a subset of catalogs.

Note

This connector brings metadata from Azure Databricks Unity Catalog. To scan Azure Databricks workspace-scoped metadata, refer to Azure Databricks Hive Metastore connector.

Known limitations

  • When object is deleted from the data source, currently the subsequent scan won't automatically remove the corresponding asset in Microsoft Purview.

Prerequisites

  • You must have an Azure account with an active subscription. Create an account for free.

  • You must have an active Microsoft Purview account.

  • You need an Azure Key Vault, and to grant Microsoft Purview permissions to access secrets.

  • You need Data Source Administrator and Data Reader permissions to register a source and manage it in the Microsoft Purview governance portal. For more information about permissions, see Access control in Microsoft Purview.

  • To scan Azure Databricks Unity Catalog, Microsoft Purview connects to a SQL Warehouse in your workspace, and uses Personal Access Token for authentication. You need to have an Azure Databricks workspace that is Unity Catalog enabled and attached to the metastore you want to scan. In your Azure Databricks workspace:

    • Generate a personal access token, and store it as a secret in Azure Key Vault.

      • For all the objects that you want to bring into Microsoft Purview, the user needs to have at least SELECT privilege on tables/views, USE CATALOG on the object’s catalog, and USE SCHEMA on the object’s schema.

      • In order to scan all the objects in a Unity Catalog metastore, use a user with metastore admin role. Learn more from Manage privileges in Unity Catalog and Unity Catalog privileges and securable objects.

      • For classification, user also needs to have SELECT privilege on the tables/views to retrieve sample data.

    • Create a SQL Warehouse. You can use the autocreated Starter warehouse as well if applicable.

      • Note down the HTTP path. You can find it in Azure Databricks workspace -> SQL Warehouses -> your warehouse -> Connection details -> HTTP path.

      • Make sure the user has the Can Use permission so as to connect to the Azure Databricks SQL warehouse. Learn more from SQL warehouse access control.

  • If your Azure Databricks workspace doesn’t allow access from public network or if your Microsoft Purview account doesn’t enable access from all networks, you can use the Managed Virtual Network Integration Runtime for scan. You can set up a managed private endpoint for Azure Databricks as needed to establish private connectivity.

Register

This section describes how to register an Azure Databricks workspace in Microsoft Purview by using the Microsoft Purview governance portal.

  1. Go to your Microsoft Purview account.

  2. Select Data Map on the left pane.

  3. Select Register.

  4. In Register sources, select Azure Databricks > Continue.

  5. On the Register sources (Azure Databricks) screen, do the following:

    1. For Name, enter a name that Microsoft Purview will list as the data source.

    2. For Azure subscription and Databricks workspace name, select the subscription and workspace that you want to scan from the dropdown. The Databricks workspace URL is automatically populated.

    3. Select a collection from the list.

    Screenshot of registering Azure Databricks source.

  6. Select Finish.

Scan

Tip

To troubleshoot any issues with scanning:

  1. Confirm you have followed all prerequisites.
  2. Review our scan troubleshooting documentation.

Use the following steps to scan Azure Databricks to automatically identify assets. For more information about scanning in general, see Scans and ingestion in Microsoft Purview.

  1. Go to Sources.

  2. Select the registered Azure Databricks.

  3. Select + New scan.

  4. Provide the following details:

    1. Name: Enter a name for the scan.

    2. Extraction method: Indicate to extract metadata from Hive Metastore or Unity Catalog. Select Unity Catalog.

    3. Connect via integration runtime: Choose the default auto resolved Azure integration runtime or a Managed VNet IR you created.

    4. Credential: Select the credential to connect to your data source. Make sure to:

      • Select Access Token Authentication while creating a credential.
      • Provide secret name of the personal access token that you created in Prerequisites in the appropriate box.

      For more information, see Credentials for source authentication in Microsoft Purview.

    5. HTTP path: Specify the Databricks SQL Warehouse’s HTTP path that Microsoft Purview will connect to and perform the scan, e.g. /sql/1.0/endpoints/xxxxxxxxxxxxxxxx. You can find it in Azure Databricks workspace -> SQL Warehouses -> your warehouse -> Connection details -> HTTP path.

  5. Click Test connection to validate the settings.

    Screenshot of setting up Azure Databricks Unity Catalog scan.

  6. Select Continue.

  7. In Scope your scan page, select the catalog(s) you want to scan.

    Screenshot of setting up the scope for Azure Databricks scan.

  8. Select a scan rule set for classification. You can choose between the system default, existing custom rule sets, or create a new rule set inline. Check the Classification article to learn more.

  9. For Scan trigger, choose whether to set up a schedule or run the scan once.

  10. Review your scan and select Save and Run.

Once the scan successfully completes, see how to browse and search assets.

View your scans and scan runs

To view existing scans:

  1. Go to the Microsoft Purview portal. On the left pane, select Data map.
  2. Select the data source. You can view a list of existing scans on that data source under Recent scans, or you can view all scans on the Scans tab.
  3. Select the scan that has results you want to view. The pane shows you all the previous scan runs, along with the status and metrics for each scan run.
  4. Select the run ID to check the scan run details.

Manage your scans

To edit, cancel, or delete a scan:

  1. Go to the Microsoft Purview portal. On the left pane, select Data Map.

  2. Select the data source. You can view a list of existing scans on that data source under Recent scans, or you can view all scans on the Scans tab.

  3. Select the scan that you want to manage. You can then:

    • Edit the scan by selecting Edit scan.
    • Cancel an in-progress scan by selecting Cancel scan run.
    • Delete your scan by selecting Delete scan.

Note

  • Deleting your scan does not delete catalog assets created from previous scans.

Browse and search assets

After scanning your Azure Databricks, you can browse data catalog or search data catalog to view the asset details.

When browsing by source types, you see two entries for Azure Databricks Unity Catalog and Azure Databricks respectively. The former contains the Unity Catalog artifacts including the metastore and its catalogs/schemas/tables/views, while the latter contains the workspace.

Screenshot of browsing assets by source type.

From the Azure Databricks workspace asset, you can find the associated Unity Catalog under Properties tab, reversed applies too.

Screenshot of finding the associated Unity Catalog with Azure Databricks source.

Next steps

Now that you've registered your source, use the following guides to learn more about Microsoft Purview and your data: