Share via

How do I figure out what public IP ranges my Databricks workspace clusters are coming from?

McDonald, Matthew 101 Reputation points
2024-05-08T22:13:53.72+00:00

Relatively new to Databricks. I have an existing workspace that was created years ago. It is vnet-injected but it has secured cluster connectivity (SCC) disabled. I need to know the outbound IP addresses/ranges the clusters would communicate on to whitelist on another separate public-facing service.

I found the following article that supposedly outlines the IP ranges for outbound NAT ranges within Databricks. https://learn.microsoft.com/en-us/azure/databricks/resources/supported-regions#outbound

However, my cluster does not seem to come from any of those documented ranges.

I created test instance configured the same way and what I have found, is that every time a cluster is started, the VM automatically gets its own dedicated public IP and it communicates out on that. Also, this IP changes every time the cluster starts.

Interestingly, if I enable SCC, the public IPs are no longer generated, and the cluster traffic does seem to then come from those documented ranges.

So I'm trying to understand how outbound connectivity works in this existing configuration without SCC. Do these dedicated public IPs come from a standard range? Is that documented anywhere?

Unfortunately, I simply can't enable SCC so that it goes through the ranges documented above.

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,955 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. PRADEEPCHEEKATLA-MSFT 78,986 Reputation points Microsoft Employee
    2024-05-20T05:42:40.1833333+00:00

    @McDonald, Matthew - Thanks for the question and using MS Q&A platform.

    Based on the information you provided, it seems that your Databricks workspace clusters are not using the documented outbound NAT ranges. This could be because SCC is disabled and the clusters are using their own dedicated public IP addresses.

    In this case, you can try the following steps to determine the public IP addresses/ranges that your clusters are using:

    1. Check the Azure portal for the public IP addresses associated with the virtual machines (VMs) that are running your Databricks clusters. You can find this information by navigating to the VMs in the Azure portal and looking at the "Public IP address" field.

    Once you have the public IP addresses, you can use a tool like IP2Location to determine the IP ranges that they belong to. This will give you an idea of the IP ranges that your clusters are communicating on.

    If the IP ranges you find are not the same as the documented outbound NAT ranges, you can try whitelisting the specific IP addresses instead of the entire range. Alternatively, you can try enabling SCC to use the documented ranges.

    Hope this helps. Do let us know if you any further queries.

    If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

    0 comments