Training
Certification
Microsoft Certified: Fabric Data Engineer Associate - Certifications
As a fabric data engineer, you should have subject matter expertise with data loading patterns, data architectures, and orchestration processes.
This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
This article describes how to set up Lakehouse Federation to run federated queries on Databricks data in another Databricks workspace. To learn more about Lakehouse Federation, see What is Lakehouse Federation?.
Important
Databricks-to-Databricks Lakehouse Federation is a good tool for running queries on data managed by another Databricks workspace’s Hive or AWS Glue metastore. For most other scenarios, other Azure Databricks workflows are more efficient:
There is no need to set up Lakehouse Federation in either of these scenarios.
To connect to a Databricks catalog in another workspace using Lakehouse Federation, you must create the following in your Azure Databricks Unity Catalog metastore:
Workspace requirements:
Compute requirements:
Permissions required:
CREATE CONNECTION
privilege on the Unity Catalog metastore attached to the workspace.CREATE CATALOG
permission on the metastore and be either the owner of the connection or have the CREATE FOREIGN CATALOG
privilege on the connection.Additional permission requirements are specified in each task-based section that follows.
You must also have an active cluster or SQL warehouse in the Azure Databricks workspace that you are using to configure the connection.
A connection specifies a path and credentials for accessing an external database system. To create a connection, you can use Catalog Explorer or the CREATE CONNECTION
SQL command in an Azure Databricks notebook or the Databricks SQL query editor.
Note
You can also use the Databricks REST API or the Databricks CLI to create a connection. See POST /api/2.1/unity-catalog/connections and Unity Catalog commands.
Permissions required: Metastore admin or user with the CREATE CONNECTION
privilege.
In your Azure Databricks workspace, click Catalog.
At the top of the Catalog pane, click the Add icon and select Add a connection from the menu.
Alternatively, from the Quick access page, click the External data > button, go to the Connections tab, and click Create connection.
Enter a user-friendly Connection name.
Select a Connection type of Databricks.
Enter the following connection properties for the other Databricks instance.
(Optional) Click Test connection to confirm that it works.
(Optional) Add a comment.
Click Create.
Run the following command in a notebook or the Databricks SQL query editor, replacing the following:
<connection-name>
: User-friendly name for the connection you’re creating.<workspace-instance>
: The target workspace instance. To learn how to get the workspace instance name, see Get identifiers for workspace objects.<sql-warehouse-path>
: The HTTP path for your SQL warehouse. To get the path, go to SQL > SQL Warehouses in the sidebar, select the SQL warehouse, go to the Connection details tab, and copy the value for HTTP path.<personal-access-token>
: A Azure Databricks personal access token that enables access to the target workspace. To learn how to get a token, see Azure Databricks personal access token authentication. For connections, Databricks recommends that you use a service principal’s personal access token.CREATE CONNECTION <connection-name> TYPE databricks
OPTIONS (
host '<workspace-instance>',
httpPath '<sql-warehouse-path>',
personalAccessToken '<personal-access-token>'
);
We recommend that you use Azure Databricks secrets instead of plaintext strings for sensitive values like credentials. For example:
CREATE CONNECTION <connection-name> TYPE databricks
OPTIONS (
host '<workspace-instance>',
httpPath '<sql-warehouse-path>',
personalAccessToken secret ('<secret-scope>','<secret-key-password>')
)
For information about setting up secrets, see Secret management.
A foreign catalog mirrors a catalog in the external Databricks workspace so that you can query and manage access to data in that external Databricks catalog as if it were a catalog in your own workspsace. To create a foreign catalog, you use a connection to the external Databricks workspace that has already been defined.
To create a foreign catalog, you can use Catalog Explorer or the CREATE FOREIGN CATALOG
SQL command in an Azure Databricks notebook or the Databricks SQL query editor.
Note
You can also use the Databricks REST API or the Databricks CLI to create a catalog. See POST /api/2.1/unity-catalog/catalogs and Unity Catalog commands.
Permissions required: CREATE CATALOG
permission on the metastore and either ownership of the connection or the CREATE FOREIGN CATALOG
privilege on the connection.
In your Azure Databricks workspace, click Catalog to open Catalog Explorer.
At the top of the Catalog pane, click the Add icon and select Add a catalog from the menu.
Alternatively, from the Quick access page, click the Catalogs button, and then click the Create catalog button.
Follow the instructions for creating foreign catalogs in Create catalogs.
Run the following SQL command in a notebook or Databricks SQL editor. Items in brackets are optional. Replace the placeholder values:
<catalog-name>
: Name for the foreign catalog that you are creating.<connection-name>
: The connection object that specifies the data source, path, and access credentials.<external-catalog-name>
: Name of the catalog in the external Databricks workspace that you are mirroring.CREATE FOREIGN CATALOG [IF NOT EXISTS] <catalog-name> USING CONNECTION <connection-name>
OPTIONS (catalog '<external-catalog-name>');
The following pushdowns are supported on all compute:
The following pushdowns are supported on Databricks Runtime 13.3 LTS and above and SQL warehouse compute:
The following pushdowns are not supported:
Training
Certification
Microsoft Certified: Fabric Data Engineer Associate - Certifications
As a fabric data engineer, you should have subject matter expertise with data loading patterns, data architectures, and orchestration processes.