NAT gateway for outbound traffic from Azure Databricks cluster without enabling SCC (stable public IP)

Question

Hi Team,

We have a Azure Databricks cluster with VNET injection and Secure cluster connectivity (SCC) is disabled for our databricks cluster.

Currently each of the VMs gets a dynamic public IP from Azure for outbound traffic but we want to ensure that all outbound traffic to the internet goes through a stable public IP. So, we are looking for options to achieve this.

One thing we found is associating Azure NAT gateway having a public IP to the subnet of Azure databricks cluster. But does it require SCC to be enabled for databricks cluster mandatorily? We didn't find any clear documentation about this.

We just want to ensure the outbound traffic from cluster should originate from a stable public IP, which can be used by external systems for communication etc. and we do not want to enable SCC for our cluster as of now.

Looking forward to hear from you.

Thank you!

Answer 1

@VD - Thanks for the question and using MS Q&A forum.

To ensure that all outbound traffic from your Azure Databricks cluster originates from a stable public IP without enabling Secure Cluster Connectivity (SCC), you can indeed use an Azure NAT Gateway. Azure NAT Gateway provides a simple and scalable solution to provide outbound connectivity for resources in a virtual network.

Here are the steps to achieve this:

1. Create a NAT Gateway:
   • Go to the Azure portal.
   • Navigate to "Create a resource" and search for "NAT Gateway".
   • Click on "Create" and follow the prompts to create a NAT Gateway.
   • Ensure you associate a public IP address with the NAT Gateway. This public IP will be the stable IP address used for outbound traffic.
2. Associate the NAT Gateway with the Subnet:
   • After creating the NAT Gateway, you need to associate it with the subnet where your Azure Databricks cluster is deployed.
   • Navigate to the Virtual Network that contains your Databricks cluster.
   • Go to the "Subnets" section and select the specific subnet used by your Databricks cluster.
   • In the subnet settings, you will find an option to associate a NAT Gateway. Select the NAT Gateway you created.

By associating the NAT Gateway with the subnet, all outbound traffic from resources in that subnet, including your Databricks cluster, will go through the NAT Gateway and therefore will have the public IP associated with the NAT Gateway.

Important Points to Note:

• SCC Requirement: Enabling Secure Cluster Connectivity (SCC) is not a requirement for using a NAT Gateway. You can use NAT Gateway independently of SCC to manage outbound traffic and ensure it uses a stable public IP.
• VNET Configuration: Ensure your Databricks cluster is properly configured to use the VNET and subnet where the NAT Gateway is associated.
• Firewall and Network Security: Make sure your network security groups, route tables, and firewall settings allow the necessary outbound traffic through the NAT Gateway.

This setup will allow your Databricks cluster to have a stable public IP for outbound traffic, which can be used for consistent communication with external systems.

For more information refer to this: NAT gateway with public IP for SCC disabled Databricks cluster

Hope this helps. Do let us know if you have any further queries.

---

If this answers your query, do click `Accept Answer` and `Yes` for was this answer helpful. And, if you have any further query do let us know.