Connect to and manage Azure Databricks in Microsoft Purview (Preview)
This article outlines how to register Azure Databricks, and how to authenticate and interact with Azure Databricks in Microsoft Purview. For more information about Microsoft Purview, read the introductory article.
|Metadata Extraction||Full Scan||Incremental Scan||Scoped Scan||Classification||Labeling||Access Policy||Lineage||Data Sharing|
When scanning Azure Databricks source, Microsoft Purview supports:
Extracting technical metadata including:
- Azure Databricks workspace
- Hive server
- Tables including the columns, foreign keys, unique constraints, and storage description
- Views including the columns and storage description
Fetching relationship between external tables and Azure Data Lake Storage Gen2/Azure Blob assets (external locations).
Fetching static lineage between tables and views based on the view definition.
This connector brings metadata from Databricks metastore. Comparing to scan via Hive Metastore connector in case you use it to scan Azure Databricks earlier:
- You can directly set up scan for Azure Databricks workspaces without direct HMS access. It uses Databricks personal access token for authentication and connects to a cluster to perform scan.
- The Databricks workspace info is captured.
- The relationship between tables and storage assets is captured.
When object is deleted from the data source, currently the subsequent scan won't automatically remove the corresponding asset in Microsoft Purview.
You must have an Azure account with an active subscription. Create an account for free.
You must have an active Microsoft Purview account.
You need Data Source Administrator and Data Reader permissions to register a source and manage it in the Microsoft Purview governance portal. For more information about permissions, see Access control in Microsoft Purview.
Set up the latest self-hosted integration runtime. For more information, see Create and configure a self-hosted integration runtime. The minimal supported self-hosted Integration Runtime version is 5.20.8227.2.
Ensure JDK 11 is installed on the machine where the self-hosted integration runtime is installed. Restart the machine after you newly install the JDK for it to take effect.
Ensure that Visual C++ Redistributable for Visual Studio 2012 Update 4 is installed on the machine where the self-hosted integration runtime is running. If you don't have this update installed, download it now.
In your Azure Databricks workspace:
Generate a personal access token, and store it as a secret in Azure Key Vault.
Create a cluster. Note down the cluster ID - you can find it in Azure Databricks workspace -> Compute -> your cluster -> Tags -> Automatically added tags ->
Make sure the user has the following permissions so as to connect to the Azure Databricks cluster:
- Can Attach To permission to connect to the running cluster.
- Can Restart permission to automatically trigger the cluster to start if its state is terminated when connecting.
This section describes how to register an Azure Databricks workspace in Microsoft Purview by using the Microsoft Purview governance portal.
Go to your Microsoft Purview account.
Select Data Map on the left pane.
In Register sources, select Azure Databricks > Continue.
On the Register sources (Azure Databricks) screen, do the following:
For Name, enter a name that Microsoft Purview will list as the data source.
For Azure subscription and Databricks workspace name, select the subscription and workspace that you want to scan from the dropdown. The Databricks workspace URL will be automatically populated.
For Select a collection, choose a collection from the list or create a new one. This step is optional.
To troubleshoot any issues with scanning:
- Confirm you have followed all prerequisites.
- Review our scan troubleshooting documentation.
Use the following steps to scan Azure Databricks to automatically identify assets. For more information about scanning in general, see Scans and ingestion in Microsoft Purview.
In the Management Center, select integration runtimes. Make sure that a self-hosted integration runtime is set up. If it isn't set up, use the steps in Create and manage a self-hosted integration runtime.
Go to Sources.
Select the registered Azure Databricks.
Select + New scan.
Provide the following details:
Name: Enter a name for the scan.
Connect via integration runtime: Select the configured self-hosted integration runtime.
Credential: Select the credential to connect to your data source. Make sure to:
- Select Access Token Authentication while creating a credential.
- Provide secret name of the personal access token that you created in Prerequisites in the appropriate box.
For more information, see Credentials for source authentication in Microsoft Purview.
Cluster ID: Specify the cluster ID that Microsoft Purview will connect to and perform the scan. You can find it in Azure Databricks workspace -> Compute -> your cluster -> Tags -> Automatically added tags ->
Mount points: Provide the mount point and Azure Storage source location string when you have external storage manually mounted to Databricks. Use the format
/mnt/<path>=abfss://<container>@<adls_gen2_storage_account>.dfs.core.windows.net/;/mnt/<path>=wasbs://<container>@<blob_storage_account>.blob.core.windows.netIt will be used to capture the relationship between tables and the corresponding storage assets in Microsoft Purview. This setting is optional, if it's not specified, such relationship won't be retrieved.
You can get the list of mount points in your Databricks workspace by running the following Python command in a notebook:
It will print all the mount points like below:
[MountInfo(mountPoint='/databricks-datasets', source='databricks-datasets', encryptionType=''), MountInfo(mountPoint='/mnt/ADLS2', source='abfss://firstname.lastname@example.org/', encryptionType=''), MountInfo(mountPoint='/databricks/mlflow-tracking', source='databricks/mlflow-tracking', encryptionType=''), MountInfo(mountPoint='/mnt/Blob', source='wasbs://email@example.com', encryptionType=''), MountInfo(mountPoint='/databricks-results', source='databricks-results', encryptionType=''), MountInfo(mountPoint='/databricks/mlflow-registry', source='databricks/mlflow-registry', encryptionType=''), MountInfo(mountPoint='/', source='DatabricksRoot', encryptionType='')]
In this example, specify the following as mount points:
Maximum memory available: Maximum memory (in gigabytes) available on the customer's machine for the scanning processes to use. This value is dependent on the size of Azure Databricks to be scanned.
As a thumb rule, please provide 1GB memory for every 1000 tables.
For Scan trigger, choose whether to set up a schedule or run the scan once.
Review your scan and select Save and Run.
Once the scan successfully completes, see how to browse and search Azure Databricks assets.
View your scans and scan runs
To view existing scans:
- Go to the Microsoft Purview governance portal. Select the Data map tab on the left pane.
- Select the desired data source. You can view a list of existing scans on that data source under Recent scans, or you can view all scans on the Scans tab.
- Select the scan that has results you want to view. The page shows you all of the previous scan runs, along with the status and metrics for each scan run.
- Click the run ID to check more about the scan run details.
Manage your scans - edit, delete, or cancel
To manage or delete a scan:
Go to the Microsoft Purview governance portal. Select the Data Map tab on the left pane.
Select the desired data source. You can view a list of existing scans on that data source under Recent scans, or you can view all scans on the Scans tab.
Select the scan that you want to manage. You can then:
- Edit the scan by selecting Edit scan.
- Cancel an in-progress scan by selecting Cancel scan run.
- Delete your scan by selecting Delete scan.
- Deleting your scan does not delete catalog assets created from previous scans.
- The asset will no longer be updated with schema changes if your source table has changed and you re-scan the source table after editing the description on the Schema tab of Microsoft Purview.
Browse and search assets
After scanning your Azure Databricks, you can browse data catalog or search data catalog to view the asset details.
From the Databricks workspace asset, you can find the associated Hive Metastore and the tables/views, reversed applies too.
Refer to the supported capabilities section on the supported Azure Databricks scenarios. For more information about lineage in general, see data lineage and lineage user guide.
Go to the Hive table/view asset -> lineage tab, you can see the asset relationship when applicable. For relationship between table and external storage assets, you'll see Hive table asset and the storage asset are directly connected bi-directionally, as they mutually impact each other. If you use mount point in create table statement, you need to provide the mount point information in scan settings to extract such relationship.
Now that you've registered your source, use the following guides to learn more about Microsoft Purview and your data:
Submit and view feedback for