Additional ports for Azure Databricks traffic to the control plane

RE: Important notice: We're opening new ports for Azure Databricks in April

Please elaborate on this: need to ensure that no NSGs on your subnets block the new ports before 1 April 2024. The notice only mentions the 10 new ports: 3306 and 8443–8451. What protocol, source, and destination? We have the ports opened only for a source and destination of VirtualNetwork. Is that sufficient?

Bhargava-MSFT 31,196 Reputation points Microsoft Employee

2024-03-08T19:16:06.31+00:00

Hello Bratcher, Danny (Tarpon Springs),

Welcome to the Microsoft Q&A forum.

Since the notice didn't explicitly mention about the protocol, they will generally use TCP unless otherwise specified.

you will need to ensure that the NSGs on your subnets allow outbound traffic over the new ports to the Azure Databricks control plane.

Please review your network configuration and update your NSG rules as needed to ensure that outbound traffic over the new ports is allowed.

I hope this answers your question.
Gundreddy, Sravani 0 Reputation points

2024-03-11T18:47:08.1866667+00:00

Does Microsoft or Databricks apply these additional ports to existing NSG's automatically. Does customer have to make changes if the workspace is vnet integrated.
Bhargava-MSFT 31,196 Reputation points Microsoft Employee

2024-03-11T19:56:06.14+00:00

Hello Gundreddy, Sravani,

The customer needs to ensure that the necessary changes are made to accommodate the new ports. (Microsoft or Databricks does not automatically apply these new ports to existing Network Security Groups)

If your Azure Databricks workspace is VNet-integrated, you will need to review and update your NSG rules manually.

I hope this answers your question.
oscar van zelst 0 Reputation points

2024-03-13T07:18:58.7766667+00:00

Thank you BhargavaGunnam-MSFT , this helps but updating the current nsg on vnet injected databricks is not possible as of >> you do not have permission to update or delete these NSG rules; any attempt to do so is blocked by the subnet delegation. Thus we should add a new nsg rule manual ??? will databricks this deploy automaticly in the near future so thats it's part of the currnet nsg and blocked afterwards ? And I have to make a firewall rule also for this new ports. What should I set as destination in the firewall application rule? thank you !

Accepted answer

Bhargava-MSFT 31,196 Reputation points Microsoft Employee

2024-03-13T22:19:59.8266667+00:00

Hello oscar van zelst, Bratcher, Danny (Tarpon Springs),

Here, the only action required from your end would be to check the available Databricks NSGs and verify if any of these ports have been blocked. If these ports are not blocked on your end, then no further action is necessary.

How to verify: Click on the NSG attached to the subnet:

Look for outbound rules that deny traffic from VirtualNetwork sources to the AzureDatabricks service tag destination on ports 8443-8451 and 3306.

I hope this helps.
Please sign in to rate this answer.
oscar van zelst 0 Reputation points

2024-03-14T08:34:24.24+00:00

Hi Thank you,

We have our default nsg rule;

name = "deny_all_internet"

priority = 300

direction = "Outbound"

access = "Deny"

So we need to open additonal ports.

Databricks is also behind a firewall. Can you tell me how the (azure) application firewall rule should look like ? what should be destination (fqdn) in de firewall ? something like this? rule {

**name = "Allow_Databricks_additional_ports"** **destination_addresses = ["*.azuredatabricks.net"]** **destination_ports = ["3306","8443-8451"]**

Bhargava-MSFT 31,196 Reputation points Microsoft Employee

2024-03-14T20:07:24.81+00:00

Your NSG rule named “deny_all_internet” with a priority of 300, which denies all outbound traffic, will indeed block the necessary ports for Databricks unless additional rules are added to allow the required traffic.

For Azure Databricks, you should allow outbound traffic to the service tag destination for AzureDatabricks on ports 8443-8451 and 3306. The firewall rule you’ve outlined seems correct, where you’re allowing traffic to the destination addresses matching “*.azuredatabricks.net” on the specified ports.

Bhargava-MSFT 31,196 Reputation points Microsoft Employee

2024-03-14T20:15:18.0766667+00:00

If Databricks resources are being blocked by the firewall rules, you should allow HTTPS and WebSocket traffic to “.azuredatabricks.net". This can be added to the “Application Rule Collection” with your DNS proxy enabled. The target FQDNs should be ".azuredatabricks.net” with the source type as IP Address and the source as the IP range for the public subnet in the Databricks virtual network that you created.

https://learn.microsoft.com/en-us/azure/databricks/resources/firewall-rules

Target FQDNs

Name : any

Source type : IP Address

Source : IP range for the public subnet in the databricks virtual network that you created

Port: HTTPS; HTTP

Target FQDNs : *.azuredatabricks.net

oscar van zelst 0 Reputation points

2024-03-15T06:35:48.07+00:00

Thank you this help me very much !

AA 0 Reputation points

2024-03-21T16:28:07.9233333+00:00

apart from scalability and new feature, any other reason why port 8443 - 84 51 should be open? Just asking from security point of view
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Your answer

Share via

Additional ports for Azure Databricks traffic to the control plane

0 additional answers

Your answer