Databricks: Connection error between nodes — Network configuration impact

Question

Problem description

I'm experiencing intermittent connection errors between nodes in my Azure Databricks environment. The issue started after I made recent changes to the virtual network, specifically expanding the CIDR address space to address IP exhaustion.

Environment

Azure Databricks workspace with virtual network integration, region not specified, workloads include Spark jobs with shuffle stages.

What I've already tried

I verified that nodes were shut down before subnet changes. After recreating clusters, the problem persisted. I checked the subnet configurations, NSG rules, route tables, VNet peering settings, and firewall rules. I confirmed the workspace is bound to the current subnets and that the new address space is permitted in network controls. Recreating clusters with fresh nodes did not resolve the issue.

Current status

I am seeking guidance on whether the subnet CIDR modifications could have caused the network connectivity issues, and what steps I should take to verify and resolve this problem.

Answer 1

Hi Gustavo,

Yes, changing subnet CIDR ranges can affect communication between Databricks nodes if routing or security rules are not fully updated.

Since recreating the clusters did not help, I recommend checking effective NSG rules, route tables, and driver-to-worker connectivity using Azure Network Watcher. Also review Spark executor logs for connection timeouts during shuffle operations.

If the issue persists, raise an Azure Databricks support case with the cluster logs and network change details.

Thank you, and feel free to respond back for more assistance.