Custom DNS best practices for Azure Databricks workspace virtual networks

When you deploy a Azure Databricks workspace in your own virtual network (VNet injection), you can configure the VNet to use custom DNS servers instead of the Azure-provided DNS. Custom DNS is common when a workspace must resolve on-premises hostnames, use a centralized DNS forwarder in a hub-and-spoke topology, or apply DNS-level filtering.

Custom DNS gives you control over name resolution, but it also makes name resolution your responsibility. If your custom DNS servers can't resolve the names that classic compute needs, clusters fail to start and lose outbound connectivity to external data sources and library repositories. This article describes the recommended custom DNS configuration for VNet-injected workspaces, the most common misconfigurations, and how to troubleshoot them.

Note

Custom DNS applies to the classic compute plane in a VNet-injected workspace. Serverless compute runs in the Azure Databricks network and doesn't use your VNet's DNS configuration.

How Azure Databricks uses DNS in a VNet-injected workspace

Classic compute nodes run in your VNet's container (private) and host (public) subnets. During cluster startup and operation, those nodes must resolve and reach several Azure Databricks and Azure endpoints. If secure cluster connectivity (SCC) is enabled, each cluster opens an outbound connection to the SCC relay in the control plane during cluster creation, so name resolution must succeed before a cluster can come up.

The endpoints that classic compute depends on include:

Endpoint How it's reached
Azure Databricks control plane and web application Regional IP ranges, represented by the AzureDatabricks service tag
Secure cluster connectivity (SCC) relay The fully qualified domain name (FQDN) tunnel.<region>.azuredatabricks.net
Legacy Hive metastore consolidated-<region>-prod-metastore.mysql.database.azure.com
Workspace root (DBFS) storage The workspace storage account's dfs and blob endpoints
Artifact and log Blob storage dbartifactsprod<region>.blob.core.windows.net, dblogprod<region>.blob.core.windows.net
Event Hubs (logging) prod-<region>-observabilityeventhubs.servicebus.windows.net
External data sources and library repositories Public DNS names, such as PyPI, Maven, and CRAN repositories

Classic compute reaches the control plane and web application through regional IP ranges rather than a customer-facing hostname, so Azure Databricks recommends allowlisting them with the AzureDatabricks service tag in your route tables, network security groups, or firewall. Some firewalls, such as Palo Alto, can't consume Azure service tags. On those, allowlist the underlying IP ranges from the reference list instead.

The AzureDatabricks service tag already covers the SCC relay. If you allowlist specific IP addresses instead of the service tag, allowlist the SCC relay by its FQDN, because the IP addresses behind it change. Some regions fail over to a secondary region, so the reference list can show more than one tunnel FQDN for a region. Allowlist all of them. The metastore, storage, and Event Hubs endpoints belong to other Azure services with their own service tags, such as Sql, Storage, and EventHub. For the full, per-region list of addresses and FQDNs, see Metastore, artifact Blob storage, system tables storage, log Blob storage, and Event Hubs endpoint IP addresses.

Your custom DNS servers don't have to resolve every Azure and public name themselves. The recommended pattern is to resolve the names you own and forward everything else to Azure.

Forward unresolved queries to Azure DNS

Configure your custom DNS servers to conditionally forward the names they can't resolve to the Azure-provided DNS resolver at the virtual IP address 168.63.129.16. Azure DNS resolves the Azure Databricks control plane, storage, metastore, and other Azure service names for you, and it keeps working as those addresses change.

Important

Don't fully replace Azure-provided DNS without conditional forwarding. If your custom DNS servers become authoritative for Azure or Azure Databricks domains and don't forward the queries they can't answer, resolution of required endpoints fails, and clusters can't start.

At a minimum, conditionally forward the following domains to Azure DNS, and forward any storage and Azure service domains your workloads use:

  • *.azuredatabricks.net
  • *.privatelink.azuredatabricks.net
  • *.databricksapps.com

If your workspace uses Azure Private Link for the classic compute plane connection, name resolution must return the private endpoint IP addresses rather than public addresses. Azure creates the private DNS zone privatelink.azuredatabricks.net and adds address records for your workspace endpoints, including the databricks_ui_api endpoint and the browser_authentication endpoint used for single sign-on. The browser authentication record uses the format pl-auth.<region>.

When a workspace uses custom DNS, its classic compute nodes send DNS queries to your custom DNS servers instead of to Azure DNS. A private DNS zone linked to a VNet is consulted only through the Azure-provided resolver at 168.63.129.16, so linking the zone alone doesn't return the private endpoint addresses to compute that queries a custom resolver. Both of the following are required:

  • Link the privatelink.azuredatabricks.net private DNS zone to the VNet where Azure DNS resolves the query: your workspace VNet, or the hub VNet that hosts your resolver in a hub-and-spoke topology. This gives Azure DNS the private endpoint address records to return.
  • Configure your custom DNS servers to conditionally forward *.privatelink.azuredatabricks.net and *.azuredatabricks.net to Azure DNS at 168.63.129.16. Forwarding is what sends the query into the linked private DNS zone. In a hub-and-spoke topology, forward to the central resolver that reaches Azure DNS.

If the private DNS zone isn't linked, or is linked but missing the address record for an endpoint, Azure DNS falls back to public recursion and returns the endpoint's public IP address. Name resolution appears to succeed, but traffic goes to the public endpoint instead of the private endpoint. Confirm that the linked zone holds the expected address records for every private endpoint the workspace uses.

Other Azure private endpoints, such as storage, use their own Azure private DNS zones, for example privatelink.dfs.core.windows.net and privatelink.blob.core.windows.net. Apply the same pattern: link those zones and conditionally forward the corresponding privatelink domains to Azure DNS so the linked zones are consulted. Private endpoint DNS integration has several scenarios depending on your setup. For details, see Azure Private Endpoint DNS integration.

For the full front-end and back-end Private Link configuration, see Configure classic compute plane private connectivity to Azure Databricks and Configure inbound Private Link for workspaces.

Common custom DNS misconfigurations

Most custom DNS support cases come from one of the following misconfigurations:

  • No forwarding to Azure DNS. Custom DNS servers can't resolve Azure Databricks control plane, storage, or metastore names and don't forward the queries to 168.63.129.16. Clusters fail to start, or fail to reach storage and the metastore after starting.
  • Missing Private DNS zone link, forwarding, or records. The workspace uses Private Link, but the VNet isn't linked to the privatelink.azuredatabricks.net zone, doesn't forward to Azure DNS, or the linked zone is missing the correct address record for an endpoint. Names resolve to public IP addresses, the wrong address, or nothing, so private connectivity fails.
  • Custom DNS servers unreachable. The custom DNS server addresses on the VNet are wrong, or network rules block the compute subnets from reaching them, for example when the workspace VNet isn't peered with the VNet that hosts your custom DNS servers. All name resolution fails.
  • On-premises resolver without Azure forwarding. DNS queries route to an on-premises resolver that resolves internal names but can't resolve or forward Azure names. Add conditional forwarding for the Azure and Azure Databricks domains.
  • Blocked library repositories. DNS resolves the required Azure Databricks endpoints but not the public library repositories, so library installations from PyPI, Maven, or CRAN fail.

Troubleshoot custom DNS issues

If clusters fail to start or lose connectivity, check DNS resolution from a cluster.

  1. Create a cluster in the workspace. If the cluster fails to start and the event log shows an error such as Cluster terminated. Reason: Control Plane Request Failure, the cluster can't reach the control plane, which is often a name-resolution problem.

  2. From a notebook on a running cluster, resolve a required endpoint to confirm that your custom DNS returns the expected address. For a Private Link workspace, verify that the workspace URL resolves to the private endpoint IP address rather than a public address.

    %sh
nslookup adb-<workspace-id>.<n>.azuredatabricks.net
nslookup tunnel.<region>.azuredatabricks.net

    If clusters can't start at all, for example because of a back-end Private Link problem, run these checks from an Azure VM in the workspace VNet's public subnet instead of from a notebook.

  3. If resolution fails or returns the wrong address, verify the following. When a cluster fails to start because of DNS, the cluster event log names the network health check that failed, shown in parentheses:

    • The VNet's custom DNS server addresses are correct, and the compute subnets can reach those servers (X_NHC_DNS_SERVER_UNREACHABLE).
    • Your custom DNS servers conditionally forward the Azure and Azure Databricks domains to Azure DNS at 168.63.129.16 (X_NHC_MULTIPLE_COMPONENTS_DNS_ERROR, when the control plane or artifact storage names don't resolve).
    • For Private Link, the privatelink.azuredatabricks.net zone is linked to the VNet where Azure DNS resolves, your custom DNS servers forward the Azure Databricks domains to Azure DNS, and the expected address records exist (X_NHC_CONTROL_PLANE_HTTP_ERROR, when the control plane private endpoint doesn't resolve).

  4. If clusters start but can't reach on-premises or other Azure platform as a service (PaaS) resources, confirm that user-defined routes exist for those destinations, because routing and DNS are configured separately.

If you still can't resolve the issue, contact your Microsoft and Azure Databricks account teams.

Additional resources

  • Last updated on