Connect Azure Databricks to a OneLake external location

Important

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Azure Databricks previews.

This page explains how to connect to OneLake using an external location. With the connection, you can create Unity Catalog objects (tables, volumes, views, Materialized views, and Streaming tables) and govern your data through Unity Catalog. If you want to read existing OneLake data from Azure Databricks without writing, see Enable OneLake catalog federation.

To connect to a OneLake path, you must have two Unity Catalog securable objects:

  • A storage credential, which specifies a managed identity or service principal that grants access to your OneLake storage.
  • An external location, which defines the ID-based ABFSS path to your OneLake storage and references the storage credential.

Requirements

  • Databricks Runtime 18.1 or above, or serverless compute
  • Active Microsoft Fabric workspace with capacity assigned
  • CREATE STORAGE CREDENTIAL privilege on the Unity Catalog metastore, and CREATE EXTERNAL LOCATION privilege on both the metastore and the storage credential
  • Your Azure Databricks workspace must be enabled for Unity Catalog
  • A Fabric administrator must enable Users can create Azure Databricks Storage items in Administrator Portal > Tenant Settings
  • The target Fabric workspace must have Authenticate with OneLake user-delegated SAS tokens enabled in Workspace settings > Delegated settings > OneLake Settings

Warning

Access to the Azure Databricks Storage item in a Fabric workspace is not restricted to the assigned managed identity. Any principal with a non-viewer role (Member, Contributor, or Administrator) in the Fabric workspace can read, write, and delete data in the Azure Databricks Storage item, regardless of whether they were granted access through Unity Catalog.

Until this limitation is addressed, treat the Fabric workspace containing your Azure Databricks Storage item as a privileged, restricted workspace:

  • Limit the number of principals assigned Member, Contributor, or Administrator roles in that Fabric workspace.
  • Do not add end users or broad service principals to this workspace unless strictly necessary.
  • Use a dedicated Fabric workspace solely for the Azure Databricks Storage item. Do not mix it with other Fabric assets or user-facing workspaces.

Step 1: Configure identity and access in Azure and Fabric

  1. In the Azure portal, create or identify an Access Connector for Azure Databricks with a managed identity, or a service principal.
  2. In Microsoft Fabric, go to your target workspace.
  3. Click the Workspace settings gear button, then click Manage access.
  4. Click Add people or groups and search for your managed identity or service principal.
  5. Assign the managed identity or service principal an Administrator, Member, or Contributor role.

Step 2: Provision storage in Fabric and get the path

  1. In your Fabric workspace, create a Azure Databricks Storage item.

  2. After the item is created, construct the ID-based ABFSS path:

    abfss://<WorkspaceID>@onelake.dfs.fabric.microsoft.com/<DatabricksStorageID>/Files/

    The workspace ID and Azure Databricks Storage item ID are available in the browser URL when the item is open in Fabric.

    Note

    The external location requires the ID-based path format (GUIDs), not display names. Name-based paths are rejected at creation time.

    Save this path. You use it in Step 4.

Step 3: Create a Unity Catalog storage credential

  1. Sign in to your Unity Catalog-enabled Azure Databricks workspace.

  2. In the sidebar, click Data icon. Catalog.

  3. Click Add or plus icon, then click Create a credential.

  4. Select a Credential Type of Azure Managed Identity.

  5. Enter a Storage credential name and an optional comment.

  6. Enter the Access Connector ID (the resource ID of the access connector associated with your managed identity).

    The resource ID is in the format:

    /subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Databricks/accessConnectors/<connector-name>

  7. (Optional) If you created the access connector using a user-assigned managed identity, enter the Managed Identity ID.

  8. Click Create.

Step 4: Create the OneLake external location

  1. In the sidebar, click Data icon. Catalog.

  2. Click Add or plus icon, then click Create an external location.

  3. Enter an External location name.

  4. Under Storage type, select OneLake.

  5. Under Storage credential, select the credential you created in Step 3.

  6. Under URL, enter the ID-based ABFSS path from Step 2. For example:

    abfss://<WorkspaceID>@onelake.dfs.fabric.microsoft.com/<DatabricksStorageID>/Files/

  7. Click Create.

  8. Click Test connection to verify the external location.

    Note

    A skipped file events read warning during the connection test is expected behavior. File event notifications are not yet supported on OneLake paths. The external location is functioning correctly.

Step 5: Create Unity Catalog objects on the OneLake external location

After you create the external location, you can create a catalog that uses it as managed storage. OneLake stores all the managed tables that you create in that catalog.

-- Create a catalog with the OneLake external location as managed storage
CREATE CATALOG my_onelake_catalog
MANAGED LOCATION '<onelake_external_location_path>';

-- Create a managed table
CREATE TABLE my_onelake_catalog.default.my_table (
  id INT,
  name STRING,
  created_at TIMESTAMP
);

-- Insert data
INSERT INTO my_onelake_catalog.default.my_table
VALUES (1, 'example', current_timestamp());

Replace <onelake_external_location_path> with the external location URL or a sub-path within it.

A OneLake external location supports the following Unity Catalog object types: managed tables (Delta and Iceberg), volumes, views, Materialized views, and Streaming tables.

Troubleshoot

Error Resolution
403 Forbidden when accessing OneLake path Verify that (1) the managed identity or service principal has the correct workspace role in Fabric, and (2) the Fabric workspace has an active capacity assignment.
Skipped file events read warning during connection test This warning is normal. OneLake paths don't yet enable file event notifications. The external location is functioning correctly.
Storage credential creation or test fails Verify that: (1) the managed identity resource ID is correct; (2) the managed identity has been granted Administrator, Member, or Contributor in the target Fabric workspace; (3) the path uses the correct ID-based format (abfss://<WorkspaceID>@onelake.dfs.fabric.microsoft.com/<DatabricksStorageID>/Files/); (4) the path specifies the /Files folder.

Next steps

  • Last updated on