Serverless compute plane networking

This guide introduces tools to secure network access between the compute resources in the Azure Databricks serverless compute plane and customer resources. To learn more about the control plane and the serverless compute plane, see Azure Databricks architecture overview.

To learn more about classic compute and serverless compute, see Types of compute.

Dôležité

Effective December 4, 2024, Azure Databricks began charging for networking costs associated with serverless workloads connecting to customer resources. You are currently billed for private endpoint per-hour charges to your resources. Data processing charges for Private Link connections are waived indefinitely. Billing for other networking costs will be rolled out gradually, including:

• Public connectivity to your resources, like over a NAT gateway.
• Data transfer charges, such as when serverless compute and the target resource reside in different regions.

Charges will not be applied retroactively.

Serverless compute plane networking overview

Serverless compute resources run in the serverless compute plane, which is managed by Azure Databricks. Account admins can configure secure connectivity between the serverless compute plane and their resources. This network connection is labeled as 2 on the diagram below:

Connectivity between the control plane and the serverless compute plane is always over the cloud network backbone and not the public internet. For more information on configuring security features on the other network connections in the diagram, see Networking.

What is serverless egress control?

Serverless egress control allows you to manage outbound network connections from your serverless compute resources.

Using network policies, you can:

• Enhance security: Mitigate data exfiltration risks by restricting outbound connections.
• Define precise rules: Control outbound connections by specifying allowed locations, connections, FQDNs, and Azure storage accounts.
• Simplify management: Easily configure and manage egress policies in your serverless environment.

See What is serverless egress control?

What is a network connectivity configuration (NCC)?

Serverless network connectivity is managed with network connectivity configurations (NCC). NCCs are account-level regional constructs that are used to manage private endpoints creation and firewall enablement at scale.

Account admins create NCCs in the account console and an NCC can be attached to one or more workspaces. An NCC enables firewalls and private endpoints:

• Resource firewall enablement by subnets: An NCC enables Databricks-managed stable Azure service subnets for adding service endpoints to your resource firewalls for secure access to Azure resources from serverless SQL warehouses. When an NCC is attached to a workspace, serverless compute in that workspace uses one of those networks to connect the Azure resource using service endpoints. You can allow list those networks on your Azure resource firewall. The network rules are automatically added to the workspace storage account. See Configure a firewall for serverless compute access.
• Private endpoints: When you add a private endpoint in an NCC, Azure Databricks creates a private endpoint request to your Azure resource. Once the request is accepted on the resource side, the private endpoint is used to access your Azure resource from the serverless compute plane. See Configure private connectivity from serverless compute.

Poznámka

Databricks uses service endpoints, private IPs, and public IPs to connect to resources based on their location and type. These connectivity methods are generally available unless explicitly stated otherwise.