Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Important
This feature is in Public Preview.
This page describes how providers set up OpenSharing SecureConnect to share data from cloud storage that is behind a firewall or private endpoint, without needing to allowlist each recipient's network.
How SecureConnect works
Before enabling SecureConnect on a Azure Databricks account, a provider makes a one-time configuration. This configuration allows Azure Databricks recipients to access the provider's storage behind a firewall or private endpoint. Azure Databricks then routes recipient requests through a managed proxy, so the provider does not need to update their storage firewall when adding a new recipient.
Recipients access shared data using their existing OpenSharing setup:
- Azure Databricks recipients on serverless compute access shares with no per-provider firewall changes.
- Azure Databricks recipients on classic compute and open recipients allowlist a single set of Azure Databricks control plane IPs for the provider's region.
Without SecureConnect, a provider must add each recipient's network identifier to their storage firewall, coordinating with the recipient and a cloud platform administrator for every new recipient.
Requirements
- The OpenSharing SecureConnect preview in the Account Console must be enabled. See Manage Azure Databricks previews.
Set up SecureConnect as a provider
Setting up SecureConnect involves configuring your storage firewall to allow access and enabling SecureConnect for your metastores and recipients.
Step 1: Configure your storage firewall
For the lowest networking costs, keep the region of your shared assets the same as your provider metastore region.
The following instructions assume that your shared assets and provider metastore are in the same region.
SecureConnect accesses your storage through the serverless data plane. To allow Azure Databricks to access your resources, associate your Azure resource with a network security perimeter in transition mode and allowlist the AzureDatabricksServerless service tag. See Configure an Azure network security perimeter for Azure resources.
(Optional) Configure private connectivity with a network connectivity configuration (NCC)
If your shared storage is behind a private endpoint and is not reachable from the public network, an account admin must configure a network connectivity configuration (NCC) and attach it to the metastore that hosts your shared data. For more about NCCs, see What is a network connectivity configuration (NCC)?.
An NCC attached to a workspace can't be attached to a metastore. An NCC applied to a metastore for OpenSharing applies to all shares attached to the metastore.
Create an NCC and a private endpoint rule for your storage account but do not attach the NCC to a workspace. See Configure private connectivity to Azure resources for NCC and private endpoint setup.
Attach the NCC to your OpenSharing metastore:
- As an Azure Databricks account administrator, go to the account console.
- In the sidebar, click
Catalog. - Click the name of the OpenSharing metastore to open its details.
- Under OpenSharing Network connectivity configuration (NCC) click Edit.
- Search for and select the NCC you created for OpenSharing.
- Click Save.
Important
If you are unable to attach an NCC to a metastore, contact your Databricks account team to enable private connectivity for OpenSharing SecureConnect using an NCC.
Step 2: Enable SecureConnect on a metastore
An account administrator or metastore administrator can configure the metastore so new recipients automatically use SecureConnect. By default, new and existing recipients are not enrolled in SecureConnect. You must configure existing recipients separately. See Step 3: Enable SecureConnect for individual recipients.
An account administrator or metastore administrator can configure SecureConnect:
Account admin
- Log in to the account console.
- In the sidebar, click Catalog.
- Click the name of a metastore to open its details.
- Toggle the SecureConnect setting:
- On: New recipients in the metastore have SecureConnect enabled by default at creation time. Existing recipients are not affected.
- Off (default): New recipients are not enrolled in SecureConnect. Enable per recipient individually.
Metastore admin
In your Azure Databricks workspace, click
Catalog to open Catalog Explorer.At the top of the Catalog pane, click the
gear icon and select OpenSharing.Alternatively, in the upper-right corner, click Share > OpenSharing.
Click Settings in the upper-right corner.
Turn on the setting for Enable SecureConnect for new recipients.
Click Save.
Step 3: Enable SecureConnect for individual recipients
Recipient owners, and users with the USE_RECIPIENT privilege, toggle SecureConnect on or off for each recipient. SecureConnect is disabled on a recipient by default, unless the metastore was set to enable it for all new recipients when the recipient was created.
To configure SecureConnect on a recipient:
In your Azure Databricks workspace, click
Catalog.At the top of the Catalog pane, click the
gear icon and select OpenSharing.Alternatively, in the upper-right corner, click Share > OpenSharing.
On the Shared by me tab, click the Recipients tab.
Turn on SecureConnect for each desired recipient.
(Optional) Step 4: Restrict open recipient access with IP ACLs
For open recipients, you can restrict which client IP addresses are allowed to reach SecureConnect using IP access lists. IP ACLs apply only to open recipients.
With SecureConnect, IP ACLs apply to both OpenSharing endpoint access and storage access. Without SecureConnect, IP ACLs restrict only OpenSharing endpoint access; storage URLs remain reachable from any client IP.
For setup instructions, see Restrict OpenSharing recipient access using IP access lists (Databricks-to-Open sharing).
Note
IP ACL changes for SecureConnect-enabled open recipients can take up to 10 minutes to take effect.
Supported sharing scenarios
Important
Any unsupported feature falls back to direct access from the recipient compute to the storage. The provider must manually grant access to recipient IPs in their storage firewall. See What is the OpenSharing Databricks-to-Databricks protocol? or What is the OpenSharing Databricks-to-Open sharing protocol?.
SecureConnect supports sharing to AWS, Azure, and GCP.
mTLS to SecureConnect is supported for only serverless recipient clusters.
Feature support
| Feature | D2O (token) | D2O (OIDC)* | D2O (Iceberg) | D2D (serverless) | D2D (classic) |
|---|---|---|---|---|---|
| Tables with history and without partitions | ✓ | ✓ | ✗ | ✓ ** | ✓ ** |
| Tables without history or with partitions | ✓ | ✓ | ✗ | ✓ | ✓ |
| Views | ✓ | ✓ | ✗ | ✗ | ✓ |
| Foreign tables | ✓ | ✓ | ✗ | ✓ | ✓ |
| Materialized views | ✓ | ✓ | ✗ | ✗ | ✓ |
| Streaming tables | ✓ | ✓ | ✗ | ✗ | ✓ |
| Volumes | ✗ | ✗ | ✗ | ✗ | ✗ |
| Notebooks | ✗ | ✗ | ✗ | ✗ | ✗ |
| AI models | ✗ | ✗ | ✗ | ✗ | ✗ |
* OIDC sharing does not currently work when the recipient is also on Azure Databricks.
** Cloud token optimization is not available for SecureConnect.
Limitations
- Your assets can't be backed by Cloudflare R2 storage.
For recipient-side limitations, such as mTLS support and Databricks-to-Open sharing restrictions, see Limitations.
Unsupported regions
SecureConnect is not available in Azure China, Azure Government, or the following Azure regions:
australiacentralaustraliacentral2australiasoutheastcanadaeastfrancecentraljapanwestkoreacentralmexicocentralnorthcentralusnorwayeastqatarcentralsouthafricanorthsouthindiaswitzerlandwestuaenorthukwestwestcentraluswestindiawestus3
Billing
Azure Databricks does not currently charge for SecureConnect data transfer. See Upcoming Azure billing changes for OpenSharing SecureConnect for more details.