Configure inbound Private Link for performance-intensive services

Important

This feature is in Public Preview.

This page shows how to configure Private Link for inbound connectivity to performance-intensive services on the Azure Databricks platform. This private connection allows external clients and users to access services on the Azure Databricks platform, such as Zerobus Ingest and Lakebase Autoscaling.

Benefits

  • Enhanced security: Traffic between your network and Databricks services remains within the Azure network infrastructure.
  • Access to performance-intensive services: Private connections to services like Zerobus Ingest and Lakebase Autoscaling.
  • Compliance requirements: Meet regulatory requirements that mandate private network connectivity.
  • Cost efficiency: Private Link costs less than public connectivity options such as NAT gateways.

Azure Databricks bills for certain networking costs associated with inbound Private Link connections to performance-intensive services. For more information, see Understand Databricks networking costs.

Requirements

  • Your Azure Databricks account must be on the Premium tier.
  • You must enable the Private connectivity for performance-intensive services Public Preview feature on your account. You can self-enroll from the account console. Without this feature enabled, private endpoints don't appear in the account console.
  • You must be an Azure Databricks account admin to register private endpoints.
  • You must have Network Contributor or equivalent permissions in Azure to create private endpoints.

Set up a private endpoint

Create a private endpoint

This step creates a private endpoint in the Azure portal that connects to your performance-intensive services on Azure Databricks.

Prepare VNet and subnet

  1. Prepare a VNet and subnet to host the private endpoint. You can create a new VNet or reuse an existing one (such as your workspace VNet).

    Note

    The inbound private endpoint can be hosted in a different VNet from the workspace (for example, a transit VNet). The VNet constraint applies only to backend private endpoints.

  2. Configure the Private endpoint network policy for your subnet:

    • Recommended (default): Disabled. This is the default setting and the configuration that Azure Databricks tests and supports.
    • Alternative (customer-managed): NetworkSecurityGroup or RouteTable. Enabling the policy also works and might be required for networks that use Azure Virtual WAN with ExpressRoute, where disabling the policy can cause asymmetric routing. If you enable the policy, you are responsible for ensuring that your network security group and route table rules permit traffic to the Service Direct private endpoint. Azure Databricks doesn't provide support for connectivity or performance issues that result from custom network security group or route table rules on the subnet.

    See Manage network policies for private endpoints for details.

  3. If you reuse an existing workspace VNet, you must use or create a different subnet from the one used by the workspace.

    Note

    The private endpoint for performance-intensive services can be placed in the same subnet as your existing inbound private endpoint. The "different subnet" requirement applies only to the workspace's compute injection subnets.

  4. If the VNet hosting the private endpoint differs from the VNet sending traffic, configure VNet peering or connectivity. See Verify VNet connectivity.

Deploy a private endpoint

  1. In the Azure portal, search for Private endpoints in Microsoft Marketplace and select Create.
  2. Enter a name and network interface name, and set the region to match your workspace VNet region.
  3. Click Next: Resource.
  4. Select Connect to an Azure resource by resource ID or alias.
  5. In the Resource ID or alias field, enter the Private Link Service resource ID for performance-intensive services for your region. See Private Link Service Resource IDs for performance-intensive services and general access for the list of resource IDs.
  6. In the Target sub-resource field, enter service_direct.
  7. Click Next: Virtual Network.
  8. Select the virtual network and subnet that you prepared in the Prepare VNet and subnet section.
  9. Click Next: DNS.
  10. Leave Integrate with private DNS zone set to No. You configure DNS manually in a later step.
  11. Click Next: Tags.
  12. Click Next: Review + create.
  13. Review the configuration and click Create to deploy the private endpoint.
  14. After the deployment completes, record these values:
    • Private endpoint name: The name of your private endpoint.
    • Resource GUID: Go to the private endpoint resource, click JSON view, and find the value in properties.resourceGuid. This is required when you register your private endpoint.
    • Private IP address: In JSON view, find the IP address in properties.customDnsConfigs[0].ipAddresses[0]. This is required when you configure DNS.

After deployment, the private endpoint connection state shows as Pending. This is expected. The endpoint remains in Pending state until you register your private endpoint.

Register your private endpoint

After you create your private endpoint in the Azure portal, register it with Azure Databricks.

  1. Go to the Azure Databricks account console.
  2. In the sidebar, click Security > Networking > Endpoints > Register endpoint.

Allowlist your private endpoint to workspaces with context-based ingress

Important

Private access to workspaces using context-based ingress is in Beta.

  1. In the account console, click Security in the sidebar.
  2. Click Context-based ingress & egress control in the sidebar.
  3. Under Workspace level policies, click New workspace policy.
  4. Under Ingress > Private Network Access, define your private access policy.
    1. By default, all registered endpoints are allowed: Allow access from all private endpoints. If this is acceptable, proceed to the next step. This default includes only the first 200 registered private endpoints; if your policy requires more, contact your account team for an increase.
    2. If you want to allowlist specific registered endpoints while denying all other endpoints, uncheck Allow access from all private endpoints and add an allow rule.
      1. Select the identities and workspace destinations you want to allow access to (by default, all identities and destinations are allowed).
      2. Then select source type = Selected private endpoints, and select the Service Direct private endpoint(s) you registered. The region of this private endpoint must match the region of your workspace. A Service Direct endpoint can only serve workspaces in the same region.
      3. Click Confirm.
      4. You can also add Deny rules in your policy, which define exceptions to your allow rules.
  5. When you are done configuring your private access policy, you can also configure your public access policy in Ingress > Public Network Access. You can disable all public IP access by unchecking Allow access from all public IPs. Azure Databricks recommends keeping public access enabled while testing your DNS configuration, then disabling it after DNS is finalized to enforce exclusive private connectivity.
  6. Test your ingress policy in Dry run mode first, before switching it to Enforced. Dry run mode access denials are logged in the system.access.inbound_network system table but do not block access.
  7. Make sure your Egress policy is correct.
  8. Attach your network policy to your workspace(s).
  9. Save your network policy. Context-based ingress policy updates take under 10 minutes to take effect.

Configure DNS

After your private endpoint is registered and allowlisted, configure DNS so that traffic routes through the private endpoint using the privatelink.azuredatabricks.net domain.

  • Databricks recommends a naming convention that includes the region and purpose, such as PE westus2 for inbound private link.
  1. Create an Azure private DNS zone named privatelink.azuredatabricks.net.
  2. Link the private DNS zone to the VNet hosting your private endpoint. See Link the virtual network.

Create DNS A record

  1. Go to your privatelink.azuredatabricks.net private DNS zone.
  2. Select the Recordsets tab under DNS Management.
  3. Click Add to add a record set.
  4. Configure the A record:
    • Name: <region>.service-direct (replace <region> with your Azure region, for example westus2.service-direct)
    • Type: A
    • IP address: The private IP address from your private endpoint (recorded when you created the private endpoint)
  5. Click OK to save the record.

Verify DNS resolution

From a machine in your VNet or from a workspace job attached to your private DNS zone, confirm that DNS queries resolve to the private endpoint IP:

nslookup westus2.service-direct.privatelink.azuredatabricks.net

Or use dig:

dig westus2.service-direct.privatelink.azuredatabricks.net

Both commands return the private IP address of your private endpoint.

Verify VNet connectivity

If the VNet generating traffic differs from the VNet hosting the private endpoint, configure VNet peering or connectivity between them. See Azure Private Endpoint DNS Integration Scenarios for detailed guidance.

Disable public access in Azure portal (optional)

Azure Databricks recommends disabling public access to your workspace with context-based ingress, as described in Allowlist your private endpoint to workspaces with context-based ingress. However, you can also disable public access in the Azure portal. Completing Private Link setup doesn't automatically block public internet access to your workspace. Public and private access are independent settings. To enforce private-only connectivity, disable public network access:

  1. In the Azure portal, go to your Azure Databricks workspace resource.
  2. Under Settings, set Allow Public Network Access to Disable.

Limitations

  • Each account is limited to 5 private endpoints for performance-intensive services per region and 100 per account. Contact your Azure Databricks account team for quota increases.

Additional resources

  • Last updated on