Harden intra-VNet network security group (NSG) rules for Azure Databricks

Important

This feature is in Public Preview.

Network security groups (NSGs) control traffic to and from your Azure Databricks workspace subnets. Intra-VNet NSG rule hardening restricts intra-VNet traffic to the workspace's host and worker subnet CIDRs instead of the entire virtual network. This feature only applies to VNet injection workspaces on the classic compute plane.

Overview

When Azure Databricks creates a workspace, it attaches a network security group (NSG) to each subnet. It also adds the following incoming and outbound rules with priority 100:

Protocol Source Source port Destination Destination port
Any VirtualNetwork Any VirtualNetwork Any

This rule allows the worker subnet to communicate with the host subnet within the same virtual network. It also allows resources in the same or peered virtual network to exchange traffic with workspace resources. Intra-VNet NSG rule hardening lets administrators further restrict this rule. Administrators replace the VirtualNetwork → VirtualNetwork source and destination with the specific CIDR ranges of the host and worker subnets. This restricts intra-VNet communication to the workspace subnets.

Requirements

Enable intra-VNet NSG rule hardening

Step 1: Enroll your subscription

Enroll your Azure subscription in the intra-VNet NSG rule hardening Public Preview through the Azure portal Feature Preview page. After you enroll, all new workspaces created in the subscription automatically use the hardened NSG rule.

To enroll:

  1. In the Azure portal, go to the Feature Preview page.
  2. In the Filter by name field, enter HardenSubnetCIDRInNetworkIntentPolicy, and then select the relevant subscription.
  3. Click the display name of the feature: Hardening Inbound / outbound rules for Databricks Subnets.
  4. In the dialog, click Register.

Step 2: Create new workspaces

After your enrollment is confirmed, create workspaces as you usually would. Azure Databricks automatically applies the hardened intra-VNet NSG rule to new workspaces in enrolled subscriptions.

Step 3: Validate the NSG configuration

After workspace creation, confirm that the intra-VNet NSG rule uses subnet CIDRs instead of VirtualNetwork:

  1. In the Azure portal, go to the managed resource group for your Azure Databricks workspace.
  2. Open the network security group associated with the workspace.
  3. In the incoming and outbound security rules, find the rule whose source and destination are both VirtualNetwork, and confirm that the source and destination now show your host and worker subnet CIDR ranges.

Step 4: Configure NSG rules for private endpoint connectivity

This step applies only if your private endpoints are deployed in a subnet different from the workspace host and worker subnets.

After hardening, the NSG no longer allows broad VirtualNetwork → VirtualNetwork traffic. If your workspace connects to private endpoints in another subnet, you must add explicit NSG rules to permit that traffic. Without those rules, you can lose access to those resources, including any required Azure services.

Warning

Add the required NSG rules before or immediately after enabling hardening. If you skip this step, workspaces might lose connectivity.

For each private endpoint in a separate subnet that your workspace must reach, add an NSG allow rule that permits traffic from the workspace subnets to the private endpoint IP addresses or CIDR ranges.

Backfill existing workspaces

After you enroll your subscription, you can request a one-time backfill to update the intra-VNet NSG rule on existing workspaces.

Important

The backfill can cause a network disruption of up to 10 minutes. Schedule the backfill during a maintenance window and stop scheduled or ad hoc workloads before proceeding.

To backfill existing workspaces:

  1. Compile a list of workspace IDs to backfill.
  2. File a Azure Databricks support ticket with the list of workspace IDs and request the intra-VNet NSG rule hardening backfill.
  3. Azure Databricks completes the backfill, which can take up to 5 business days.

After backfill completes, the workspace NSG replaces the VirtualNetwork → VirtualNetwork rule in the incoming and outbound rules with the host and worker subnet CIDR ranges.

Limitations

  • Intra-VNet NSG rule hardening is supported only for VNet injection workspaces.
  • Backfill for existing workspaces is a one-time operation that you cannot reverse automatically. To return to the previous configuration, manually add a VirtualNetwork → VirtualNetwork allow rule to your NSG. See Backfill existing workspaces.
  • Backfill can cause a network disruption of up to 10 minutes. Azure Databricks recommends stopping scheduled and ad hoc workloads before proceeding.
  • If your workspace uses frontend or backend Azure Private Link with private endpoints in a subnet other than the workspace host or worker subnets, you must complete Step 4: Configure NSG rules for private endpoint connectivity to preserve connectivity.
  • Last updated on