Attach an Azure Databricks compute that is secured in a virtual network (VNet)
Both Azure Machine Learning and Azure Databricks can be secured by using a VNet to restrict incoming and outgoing network communication. When both services are configured to use a VNet, you can use a private endpoint to allow Azure Machine Learning to attach Azure Databricks as a compute resource.
The information in this article assumes that your Azure Machine Learning workspace and Azure Databricks are configured for two separate Azure Virtual Networks. To enable communication between the two services, Azure Private Link is used. A private endpoint for each service is created in the VNet for the other service. A private endpoint for Azure Machine Learning is added to communicate with the VNet used by Azure Databricks. A private endpoint for Azure Databricks is added to communicate with the VNet used by Azure Machine Learning.
Prerequisites
An Azure Machine Learning workspace that is configured for network isolation.
An Azure Databricks deployment that is configured in a virtual network (VNet injection).
Important
Azure Databricks requires two subnets (sometimes called the private and public subnet). Both of these subnets are delegated, and cannot be used by the Azure Machine Learning workspace when creating a private endpoint. We recommend adding a third subnet to the VNet used by Azure Databricks and using this subnet for the private endpoint.
The VNets used by Azure Machine Learning and Azure Databricks must use a different set of IP address ranges.
Limitations
Scenarios where the Azure Machine Learning control plane needs to communicate with the Azure Databricks control plane are not supported. To work around this limitation, allows public access to your workspace. This can be either using a workspace that isn't configured with a private link or a workspace with a private link that is configured to allow public access.
Create a private endpoint for Azure Machine Learning
To allow the Azure Machine Learning workspace to communicate with the VNet that Azure Databricks is using, use the following steps:
From the Azure portal, select your Azure Machine Learning workspace.
From the sidebar, select Networking, Private endpoint connections, and then + Private endpoint.
From the Create a private endpoint form, enter a name for the new private endpoint. Adjust the other values as needed by your scenario.
Select Next until you arrive at the Virtual Network tab. Select the Virtual network that is used by Azure Databricks, and the Subnet to connect to using the private endpoint.
Select Next until you can select Create to create the resource.
Create a private endpoint for Azure Databricks
To allow Azure Databricks to communicate with the VNet that the Azure Machine Learning workspace is using, use the following steps:
From the Azure portal, select your Azure Databricks instance.
From the sidebar, select Networking, Private endpoint connections, and then + Private endpoint.
From the Create a private endpoint form, enter a name for the new private endpoint. Adjust the other values as needed by your scenario.
Select Next until you arrive at the Virtual Network tab. Select the Virtual network that is used by Azure Machine Learning, and the Subnet to connect to using the private endpoint.
Attach the Azure Databricks compute
From Azure Machine Learning studio, select your workspace and then select Compute from the sidebar. Select Attached computes, + New, and then Azure Databricks.
From the Attach Databricks compute form, provide the following information:
- Compute name: The name of the compute you're adding. This value can be different than the name of your Azure Databricks workspace.
- Subscription: The subscription that contains the Azure Databricks workspace.
- Databricks workspace: The Azure Databricks workspace that you're attaching.
- Databricks access token: For information on generating a token, see Azure Databricks personal access tokens.
Select Attach to complete the process.
