User-defined route settings for Azure Databricks

  • Article

If your Azure Databricks workspace is deployed to your own virtual network (VNet), you can use custom routes, also known as user-defined routes (UDR), to ensure that network traffic is routed correctly for your workspace. For example, if you connect the virtual network to your on-premises network, traffic may be routed through the on-premises network and unable to reach the Azure Databricks control plane. User-defined routes can solve that problem.

You need a UDR for every type of outbound connection from the VNet. You can use both Azure service tags and IP addresses to define network access controls on your user-defined routes. Databricks recommends using Azure service tags to prevent service outages due to IP changes.

Configure user-defined routes with Azure service tags

Databricks recommends that you use Azure service tags, which represent a group of IP address prefixes from a given Azure service. Microsoft manages the address prefixes encompassed by the service tag and automatically updates the service tag as addresses change. This helps to prevent service outages due to IP changes and removes the need to periodically look up these IPs and update them in your route table. However, if your organization policies disallow service tags, you can optionally specify the routes as IP addresses.

Using service tags, your user-defined routes should use the following rules and associate the route table to your virtual network’s public and private subnets.

Source Address prefix Next hop type
Default Azure Databricks service tag Internet
Default Azure SQL service tag Internet
Default Azure Storage service tag Internet
Default Azure Event Hub service tag Internet

Note

You can choose to add the Microsoft Entra ID (formerly Azure Active Directory) service tag to facilitate Microsoft Entra ID authentication from Azure Databricks clusters to Azure resources.

If Azure Private Link is enabled on your workspace, the Azure Databricks service tag is not required.

The Azure Databricks service tag represents IP addresses for the required outbound connections to the Azure Databricks control plane, the secure cluster connectivity (SCC), and the Azure Databricks web application.

The Azure SQL service tag represents IP addresses for the required outbound connections to the Azure Databricks metastore, and the Azure Storage service tag represents IP addresses for artifact Blob storage and log Blob storage. The Azure Event Hub service tag represents the required outbound connections for logging to Azure Event Hub.

Some service tags allow more granular control by restricting IP ranges to a specified region. For example, a route table for an Azure Databricks workspace in the West US regions might look like:

Name Address prefix Next hop type
adb-servicetag AzureDatabricks Internet
adb-metastore Sql.WestUS Internet
adb-storage Storage.WestUS Internet
adb-eventhub EventHub.WestUS Internet

To get the service tags required for user-defined routes, see Virtual network service tags.

Configure user-defined routes with IP addresses

Databricks recommends that you use Azure service tags, but if your organization policies don’t allow service tags, you can use IP addresses to define network access controls on your user-defined routes.

The details vary based on whether secure cluster connectivity (SCC) is enabled for the workspace:

  • If secure cluster connectivity is enabled for the workspace, you need a UDR to allow the clusters to connect to the secure cluster connectivity relay in the control plane. Be sure to include the systems marked as SCC relay IP for your region.
  • If secure cluster connectivity is disabled for the workspace, there is an inbound connection from the Control Plane NAT, but the low-level TCP SYN-ACK to that connection technically is outbound data that requires a UDR. Be sure to include the systems marked as Control Plane NAT IP for your region.

Your user-defined routes should use the following rules and associate the route table to your virtual network’s public and private subnets.

Source Address prefix Next hop type
Default Control plane NAT IP (if SCC is disabled) Internet
Default SCC relay IP (if SCC is enabled) Internet
Default Webapp IP Internet
Default Metastore IP Internet
Default Artifact Blob storage IP Internet
Default Log Blob storage IP Internet
Default DBFS root storage IP - Blob Storage endpoint Internet
Default DBFS root storage IP - ADLS gen2 (dfs) endpoint Internet
Default Event Hub IP Internet

If Azure Private Link is enabled on your workspace, your user-defined routes should use the following rules and associate the route table to your virtual network’s public and private subnets.

Source Address prefix Next hop type
Default Metastore IP Internet
Default Artifact Blob storage IP Internet
Default Log Blob storage IP Internet
Default Event Hub IP Internet

To get the IP addresses required for user-defined routes, use the tables and instructions in Azure Databricks regions, specifically: