I want to give a static public IP address to my Databricks clusters

Question

I could go over the "private" Microsoft network backbone using something like ExpressRoute.

Connecting to a firewall outside of Databricks.

But for now I just need to connect over the public internet to build a proof of concept.

Would need to give my Databricks clusters a stable egress public IP for the firewall to recognise.

I'm doing this by attaching the subnets for both worker and driver nodes to an Azure NAT gateway.
Then all nodes would share the public IP address attached to that Azure NAT gateway.

This is actually preventing clusters from being provisioned now and I have to un-attach the NAT gateway.

Because when the NAT gateway is attached, new clusters are still trying to create their own public IP addresses.

That is a conflict with the public IP address on the Azure NAT gateway.

It is also a conflict with Secure Cluster Connectivity (SCC), a Databricks feature I turned on because I thought it prevents my clusters from creating their own dedicated public IP addresses.

For context, my Databricks clusters are being created to our own virtual network (VNet Injection) rather than the out-of-the-box network managed by Databricks.

Below is a diagram of the network layout and I have outlined examples of public IP addresses being created (despite enabling SCC) and therefore preventing a stable public egress IP through a NAT gateway.

Answer 1

First thing to is to set Up an Azure NAT Gateway :

1. Create a NAT Gateway:
   • Navigate to the Azure portal.
   • Go to Create a resource > Networking > NAT gateway.
   • Configure the NAT gateway with a public IP address.
2. Attach NAT Gateway to Subnets:
   • In the NAT gateway configuration, attach it to the subnets used by your Databricks clusters. Make sure both worker and driver subnets are included.

Then, you need to configure the Databricks cluster network :

1. Create a Virtual Network (VNet):
   • Ensure your Databricks workspace is configured to use your custom VNet (VNet Injection).
2. Configure Subnets:
   • Make sure the subnets used by Databricks clusters are the ones attached to the NAT gateway.

Then, enable the Secure Cluster Connectivity which routes all traffic through the Azure backbone network.

Then, modify the NSGs associated with your subnets to allow outbound traffic to the internet through the NAT gateway.

Next step, enable Azure Private Link to ensure that your Databricks clusters do not create their own public IPs.

Example Configuration Steps:

1. Create a NAT Gateway:

   
      az network nat gateway create --resource-group <your_resource_group> --name <your_nat_gateway_name> --location <your_location> --public-ip-addresses <your_public_ip_address>
   
   

2. Attach NAT Gateway to Subnet:

   
      az network vnet subnet update --resource-group <your_resource_group> --vnet-name <your_vnet_name> --name <your_subnet_name> --nat-gateway <your_nat_gateway_name>
   
   

3. Enable SCC in Databricks:
   • Go to the Databricks workspace settings.
   • Enable Secure Cluster Connectivity (SCC).
4. Update NSG Rules:
   • Ensure outbound rules allow traffic to the internet through the NAT gateway.

Diagram:

Azure Virtual Network (VNet)

    |

    |-- Subnet A (Driver Nodes)

    |     |-- NAT Gateway (Public IP)

    |

    |-- Subnet B (Worker Nodes)

          |-- NAT Gateway (Public IP)