Configure a private network gateway

Important

Private network gateway is in Private Preview. To request access, contact your Azure Databricks account team.

This article describes how to set up and manage a private network gateway using the account REST API. For an overview of what a private network gateway is and how it works, see Private network gateway.

Before you begin

Before you create a private network gateway, make sure you have the following:

  • An Azure Databricks account on the Premium plan tier, with at least one workspace that has serverless compute enabled.
  • Azure Databricks account admin privileges.
  • Permission in your Azure subscription to delegate a subnet to Microsoft.Databricks/workspaces in your target VNet.
  • A VNet with a dedicated subnet (a /28 CIDR block or larger) available for the gateway. This subnet must not be used by any other Azure resource, and must be in the same region as your NCC and workspaces.
  • The subscription ID of the VNet that contains the delegated subnet must be in the same Microsoft Entra ID tenant as the subscription ID of your Azure Databricks workspaces.
  • The IP address of a DNS resolver that is reachable from the gateway subnet.

Private network gateway is available in the following Azure regions during Private Preview: australiaeast, brazilsouth, canadacentral, centralindia, centralus, eastasia, eastus, eastus2, francecentral, germanywestcentral, japaneast, koreacentral, northcentralus, northeurope, southcentralus, southeastasia, swedencentral, switzerlandnorth, uaenorth, uksouth, westcentralus, westeurope, westus, westus2, and westus3. The gateway and its subnet must be in the same region as the NCC.

Enable the preview

To request access to the private network gateway Private Preview, contact your Azure Databricks account team and provide the following:

  • Your Azure Databricks account ID.
  • The regions where you intend to use the gateway.

Turn on the preview

Enrollment typically takes 1 to 2 business days. Once your account team confirms access, enable the preview in the account console:

  1. Go to your account console at account console.
  2. In the left navigation, click Previews.
  3. Find private network gateway and toggle it to On.

Set up a private network gateway

All private network gateway configuration uses the account REST API. There is no UI or Terraform support during Private Preview.

Create or select an NCC

If you already have an NCC in the region you want to use, skip this step.

First, obtain an OAuth access token using a service principal that has the account admin role. An account admin creates the service principal in the account console beforehand. To request the token, run the following:

curl --location 'https://accounts.azuredatabricks.net/oidc/accounts/{account_id}/v1/token' \
  --header 'Content-Type: application/x-www-form-urlencoded' \
  --header 'Authorization: Basic <base64(client_id:client_secret)>' \
  --data-urlencode 'grant_type=client_credentials' \
  --data-urlencode 'scope=all-apis'

To create an NCC, run the following:

curl --request POST \
  'https://accounts.azuredatabricks.net/api/2.0/accounts/{account_id}/network-connectivity-configs' \
  --header 'Authorization: Bearer <access_token>' \
  --header 'Content-Type: application/json' \
  --data '{ "name": "my-ncc", "region": "eastus2" }'

Save the network_connectivity_config_id from the response.

Delegate a subnet to Databricks

The gateway injects into a subnet in your VNet to establish the tunnel. Delegate this subnet to Microsoft.Databricks/workspaces before you create the gateway.

  1. In the Azure portal, go to your target VNet.

  2. Click Subnets in the left sidebar.

  3. Select a subnet dedicated to the gateway, or create a new one. This subnet must not be used by any other resource.

  4. Under Subnet delegation, select Microsoft.Databricks/workspaces.

  5. Click Save.

    The delegated subnet must be in the same Azure region as your NCC and target workspaces, and must have enough IP addresses for the gateway nodes. A /28 or larger is recommended.

Create the private network gateway

Creating a gateway from the account console REST API requires an OAuth access token from a service principal with client credentials, created beforehand by an account admin in the account console.

To create the gateway, send a POST request to the private-network-gateways endpoint under your NCC. The request body accepts the following fields:

  • gateway_name (required): A human-readable name for the gateway.
  • azure_cloud_connection.gateway_subnet.resource_id (required): The full Azure resource ID of the subnet you delegated in the previous step.
  • private_dns_resolvers (required): The IP address of the DNS resolver inside your VNet. Use 168.63.129.16 for Azure-provided DNS if you have no custom private DNS zone.
  • traffic_mode (required): SPECIFIC_DESTINATIONS or ALL_TRAFFIC. See Traffic modes.
  • destinations (required when traffic_mode is SPECIFIC_DESTINATIONS): The DNS names to route through the gateway.

To create a gateway in SPECIFIC_DESTINATIONS mode, run the following:

curl --request POST \
  'https://accounts.azuredatabricks.net/api/2.0/accounts/{account_id}/network-connectivity-configs/{network_connectivity_config_id}/private-network-gateways' \
  --header 'Authorization: Bearer <access_token>' \
  --header 'Content-Type: application/json' \
  --data '{
    "gateway_name": "my-azure-png",
    "traffic_mode": "SPECIFIC_DESTINATIONS",
    "azure_cloud_connection": {
      "gateway_subnet": {
        "resource_id": "/subscriptions/{subscription_id}/resourceGroups/{resource_group}/providers/Microsoft.Network/virtualNetworks/{vnet_name}/subnets/{subnet_name}"
      }
    },
    "private_dns_resolvers": [
      { "resolver_type": "IP_ADDRESS", "value": "10.0.0.4" }
    ],
    "destinations": [
      { "destination_type": "DNS_NAME", "value": "myserver.internal.contoso.com" }
    ]
  }'

The gateway is created in the CREATING state. It moves to ESTABLISHED after Azure Databricks successfully injects the gateway into your subnet, which typically takes 2 to 5 minutes.

Confirm the gateway is established

To check the gateway state, send a GET request until state is ESTABLISHED:

curl --request GET \
  'https://accounts.azuredatabricks.net/api/2.0/accounts/{account_id}/network-connectivity-configs/{network_connectivity_config_id}/private-network-gateways/{gateway_id}' \
  --header 'Authorization: Bearer <access_token>'

Confirm that state is ESTABLISHED before you continue.

Attach the NCC to your workspaces

If your workspace is already attached to the NCC, skip this step. An NCC is a regional object and can attach only to workspaces in the same region.

In the account console, go to Workspaces, select the workspace, click Update Workspace, and under network connectivity configuration, select your NCC. Repeat for each workspace that should use the gateway.

You can also attach the NCC using the account REST API:

curl --request PATCH \
  'https://accounts.azuredatabricks.net/api/2.0/accounts/{account_id}/workspaces/{workspace_id}' \
  --header 'Authorization: Bearer <access_token>' \
  --header 'Content-Type: application/json' \
  --data '{ "network_connectivity_config_id": "{network_connectivity_config_id}" }'

DNS configuration

A gateway uses the resolver you specify in private_dns_resolvers to resolve hostnames for the destinations you configure. The resolver must be reachable from the gateway subnet.

If your private resources are registered in an Azure private DNS zone, link that zone to the VNet that contains the gateway subnet so the hostnames resolve correctly. To resolve public FQDNs, such as for resources that egress through your firewall, use the Azure-provided resolver at 168.63.129.16. If you have private DNS zones with overlapping names, use your own resolver instead.

Manage a gateway

You can update, delete, and list gateways with the account REST API.

Update a gateway

You can update gateway_name, destinations, private_dns_resolvers, and traffic_mode in place with a PATCH request. The update_mask query parameter is required and specifies which fields to update. When you switch traffic_mode to ALL_TRAFFIC, clear destinations in the same request. When you switch to SPECIFIC_DESTINATIONS, include the destinations to route. To change the gateway subnet, delete the gateway and create a new one.

To update the destinations and traffic mode, run the following:

curl --request PATCH \
  'https://accounts.azuredatabricks.net/api/2.0/accounts/{account_id}/network-connectivity-configs/{network_connectivity_config_id}/private-network-gateways/{gateway_id}?update_mask=destinations,traffic_mode' \
  --header 'Authorization: Bearer <access_token>' \
  --header 'Content-Type: application/json' \
  --data '{ "destinations": [ { "destination_type": "DNS_NAME", "value": "onprem-db2.corp.contoso.com" } ], "traffic_mode": "SPECIFIC_DESTINATIONS" }'

Delete a gateway

Warning

Deleting a gateway cuts off connectivity to the private resources that depend on it. Confirm that no active workloads rely on the gateway before you delete it.

To delete the gateway, send a DELETE request to the gateway endpoint under your NCC:

curl --request DELETE \
  'https://accounts.azuredatabricks.net/api/2.0/accounts/{account_id}/network-connectivity-configs/{network_connectivity_config_id}/private-network-gateways/{gateway_id}' \
  --header 'Authorization: Bearer <access_token>'

List gateways

To list the private network gateways in an NCC, send a GET request to the gateway endpoint under your NCC:

curl --request GET \
  'https://accounts.azuredatabricks.net/api/2.0/accounts/{account_id}/network-connectivity-configs/{network_connectivity_config_id}/private-network-gateways' \
  --header 'Authorization: Bearer <access_token>'

Gateway states

A gateway reports one of the following states:

State Description
CREATING Azure Databricks is creating the gateway network interface in your subnet.
ESTABLISHED The gateway is ready to route traffic.
DELETING The gateway is being removed.
FAILED The gateway could not be provisioned. This state is terminal: delete the gateway and create a new one after fixing the underlying cause, such as an unsupported region or Availability Zone.

Troubleshooting

If a workload can't connect to a required endpoint through the gateway, use the following table to diagnose common issues.

Issue Cause Resolution
Gateway stays CREATING for more than 10 minutes Subnet delegation is missing or misconfigured, or there isn't enough IP space Verify the subnet is delegated to Microsoft.Databricks/workspaces and is at least /28. Contact your account team if the issue persists.
DNS resolution fails with SERVFAIL or NXDOMAIN The private DNS resolver isn't reachable from the gateway subnet, or the private DNS zone isn't linked to the correct VNet Confirm the resolver IP is reachable from within the delegated subnet, and that the private DNS zone is linked to the VNet that contains the gateway subnet.
DNS resolves but the connection times out NSG rules are blocking traffic on the gateway subnet or the destination subnet Check the NSG inbound and outbound rules on both the gateway subnet and the destination resource, and verify the destination is reachable from the same VNet.
A JDBC or database connection fails despite DNS resolving A database-side firewall or authentication policy is blocking connections from the gateway Verify the database allows connections from the gateway subnet's IP range, and check the database firewall and access control configuration.
The gateway is ESTABLISHED but traffic still doesn't flow The destination DNS_NAME isn't added to the gateway, or the gateway is attached to an NCC that isn't bound to the workspace Check the destinations configured on the gateway, and confirm the NCC is attached to the source workspace.
Traffic to an Azure-managed service, such as Azure SQL Database or Key Vault, doesn't go through the gateway The DNS_NAME matches an Azure service endpoint suffix, so traffic takes the service endpoint path instead Check the DNS_NAME against the service endpoint suffix list. For private-only resources, add a more specific DNS_NAME to the gateway.
Traffic is dropped even though the FQDN is configured on the gateway A serverless egress gateway (SEG) policy denies the destination and is evaluated before the private network gateway Check the SEG allow lists for overlapping workspaces.
  • Last updated on