Palantir Foundry catalog federation

Important

This feature is in Public Preview.

Palantir Foundry catalog federation lets Unity Catalog read Palantir Foundry Iceberg tables directly from cloud storage.

With catalog federation, Unity Catalog accesses the Palantir Foundry Iceberg tables in cloud storage and the query runs entirely on Azure Databricks compute. Data teams can discover, govern, and query Palantir Foundry data from Azure Databricks without building ETL pipelines. Data access is read-only.

Before you begin

Review the following requirements before you set up Palantir Foundry catalog federation.

Workspace requirements:

Compute requirements:

  • Azure Databricks compute must use Databricks Runtime 18 or above.
  • SQL warehouses must be pro or serverless.
  • Dedicated clusters (formerly single-user clusters) are not supported.

Networking requirements:

Permissions required:

  • To create a connection, you must be a metastore admin or a user with the CREATE CONNECTION privilege on the Unity Catalog metastore attached to the workspace.
  • To create a foreign catalog, you must have the CREATE CATALOG permission on the metastore. You must also own the connection or have the CREATE FOREIGN CATALOG privilege on it.

Palantir Foundry requirements:

  • The Palantir Foundry tables must be Iceberg tables.
  • A Palantir Foundry third-party application for authentication. Configure the application according to the authentication method:
    • OAuth User to Machine (U2M):
      • The application must support the api:iceberg-read and offline_access scopes.
      • The application must have the OAuth redirect URL https://<databricks-workspace-url>/login/oauth/lakehousefederation.html configured, where <databricks-workspace-url> is the hostname of your workspace.
      • The user who authenticates the connection must have access to the federated root folder.
    • OAuth Machine to Machine (M2M):
      • The application must support the client credentials grant and the api:iceberg-read scope.
      • The service user behind the application must have access to the federated root folder.

Step 1: Create a connection

Permissions required: Metastore admin or user with the CREATE CONNECTION privilege.

A connection specifies a path and credentials for accessing an external system. It identifies the Palantir Foundry host and the credentials to authenticate with, not the folder to federate. You select the root folder when you create the foreign catalog in Step 2: Create a foreign catalog.

Palantir Foundry federation supports two authentication methods, described in the following subsections. Both require a Palantir Foundry third-party application.

Note

You can also use the Databricks REST API or the Databricks CLI to create a connection. See POST /api/2.1/unity-catalog/connections and Unity Catalog commands.

OAuth User to Machine (U2M)

With OAuth User to Machine (U2M), Unity Catalog accesses Foundry as the user who authenticates the connection. Create this connection using Catalog Explorer. This method can't be set up using SQL, because the OAuth sign-in flow requires the UI.

  1. In your Azure Databricks workspace, click Data icon. Catalog.
  2. At the top of the Catalog pane, click the Add or plus icon Add icon and select Create a connection from the menu.
  3. On the Connection basics page of the Set up connection wizard, enter a user-friendly Connection name.
  4. Select a Connection type of Palantir Foundry and an Auth type of OAuth User to Machine, then click Next.
  5. On the Authentication page, enter the Host of your Palantir Foundry instance, for example your-company.palantirfoundry.com, and complete the OAuth sign-in flow to authorize the connection.
  6. (Optional) Add a comment.
  7. Click Create connection.

OAuth Machine to Machine (M2M)

With OAuth Machine to Machine (M2M), Unity Catalog accesses Foundry as the service user behind the third-party application, using the client credentials grant. Create this connection using Catalog Explorer or the CREATE CONNECTION SQL command in an Azure Databricks notebook or the Databricks SQL query editor.

Catalog Explorer

  1. In your Azure Databricks workspace, click Data icon. Catalog.
  2. At the top of the Catalog pane, click the Add or plus icon Add icon and select Create a connection from the menu.
  3. On the Connection basics page of the Set up connection wizard, enter a user-friendly Connection name.
  4. Select a Connection type of Palantir Foundry and an Auth type of OAuth Machine to Machine, then click Next.
  5. On the Authentication page, enter the following:
    • Host: The hostname of your Palantir Foundry instance, for example your-company.palantirfoundry.com.
    • Client ID: The client ID from your Palantir Foundry third-party application.
    • Client secret: The client secret from your Palantir Foundry third-party application.
  6. (Optional) Add a comment.
  7. Click Create connection.

SQL

Run the following command in a notebook or the Databricks SQL query editor. Replace the placeholder values:

  • <connection-name>: Name for the connection in Azure Databricks.
  • <palantir-host>: The hostname of your Palantir Foundry instance, for example your-company.palantirfoundry.com.
  • <client-id>: The client ID from your Palantir Foundry third-party application.
  • <secret-scope> and <secret-key>: The secret scope and key that store the client secret from your Palantir Foundry third-party application.
CREATE CONNECTION <connection-name> TYPE PALANTIR
OPTIONS (
  host '<palantir-host>',
  client_id '<client-id>',
  client_secret secret('<secret-scope>','<secret-key>')
);

Databricks recommends that you use secrets instead of plaintext strings for sensitive values like the client secret. For information about setting up secrets, see Secret management.

Step 2: Create a foreign catalog

Permissions required: CREATE CATALOG permission on the metastore and either ownership of the connection or the CREATE FOREIGN CATALOG privilege on the connection.

A foreign catalog mirrors your Palantir Foundry root folder so that you can query and manage access to its tables using Azure Databricks and Unity Catalog. To create a foreign catalog, use the connection that you created in Step 1: Create a connection.

Create a foreign catalog by using Catalog Explorer or the CREATE FOREIGN CATALOG SQL command in an Azure Databricks notebook or the Databricks SQL query editor.

Catalog Explorer

  1. In your Azure Databricks workspace, click Data icon. Catalog to open Catalog Explorer.
  2. At the top of the Catalog pane, click the Add or plus icon Add icon and select Add a catalog from the menu.
  3. Enter a Catalog name and select a catalog Type of Foreign.
  4. Select the Connection that you created in Step 1: Create a connection from the drop-down menu.
  5. For Root folder, enter the Palantir Foundry folder where Unity Catalog looks for schemas and tables to federate, for example /space-name/project-name/folder-name. See Root folder and namespace mapping.
  6. In the Storage location field, specify a writable cloud storage location that stores metadata for the Iceberg tables in this catalog. The Azure Databricks-managed service account must have read-write access to it. This location can be any storage bucket and does not need to be related to the table buckets.
  7. Click Create catalog.
  8. Assign workspace access, an owner, and privileges as prompted.

SQL

Run the following command in a notebook or the Databricks SQL query editor. Items in brackets are optional. Replace the placeholder values:

  • <catalog-name>: Name for the catalog in Azure Databricks.
  • <connection-name>: The name of the connection that you created in Step 1: Create a connection.
  • <root-folder>: The Palantir Foundry folder where Unity Catalog looks for schemas and tables to federate, for example /space-name/project-name/folder-name. See Root folder and namespace mapping.
  • <storage-location>: A writable cloud storage location that stores metadata for the Iceberg tables in this catalog. The Azure Databricks-managed service account must have read-write access to it. This location can be any storage bucket and does not need to be related to the table buckets.
CREATE FOREIGN CATALOG [IF NOT EXISTS] <catalog-name>
USING CONNECTION <connection-name>
OPTIONS (
  root_folder '<root-folder>',
  storage_root '<storage-location>'
);

Unity Catalog discovers the folders and tables under the root folder:

  • Palantir Foundry subfolders become Unity Catalog schemas.
  • Palantir Foundry Iceberg tables become Unity Catalog foreign tables.

Step 3: Grant permissions and query the catalog

After you set up catalog federation, users must have the appropriate Unity Catalog permissions to access federated tables:

  • All users need USE CATALOG and USE SCHEMA permissions on the catalog and schema respectively.
  • To read from a federated table, users need the SELECT permission.

For more information about Unity Catalog privileges and how to grant them, see Manage privileges in Unity Catalog.

After you grant permissions, users can query the foreign tables from Azure Databricks:

SELECT * FROM <catalog-name>.<schema>.<table>;

Root folder and namespace mapping

When you create a foreign catalog, you specify a Palantir Foundry folder (for example /space-name/project-name/folder-name) as the federation root. Unity Catalog maps its subfolders to foreign schemas and the Iceberg tables in those subfolders become foreign tables in Unity Catalog. This maps the deeper Foundry hierarchy onto a consistent three-level structure.

For example, if you federate the root folder /sales/orders:

Palantir Foundry object                       Unity Catalog object
────────────────────────────────────────────  ──────────────────────────────────────────────────
sales                    (Foundry Space)
└── orders               (Foundry Project)    <foreign-catalog>                        (Catalog)
    └── shipments        (Foundry Folder)     <foreign-catalog>.shipments              (Schema)
        └── line_items   (Iceberg Table)      <foreign-catalog>.shipments.line_items   (Table)

The folder you federate becomes the catalog, each subfolder becomes a schema, and each Iceberg table becomes a foreign table.

Limitations

Palantir Foundry catalog federation has the following limitations:

  • Palantir Foundry federation can access only Iceberg tables. Views are not supported.
  • Private connectivity to the Palantir Foundry catalog is not supported.
  • Schema and table names follow standard Unity Catalog naming limitations. Azure Databricks does not support names that contain a period (.), space ( ), or forward slash (/). See Securable object naming requirements.
  • A root folder that is a single-level folder in Foundry (a folder that contains only tables and no subfolders to map to schemas) surfaces no schemas or tables. The federation requires at least one level of subfolders to map to Unity Catalog schemas.

See Limitations for Iceberg-related limitations.

Troubleshooting

The following section describes common errors and their resolutions.

Failed to create request URI or invalid root folder

  • The root_folder option doesn't match a folder path in your Palantir Foundry instance. Verify the path.

Authentication failed when creating the connection

The Palantir Foundry third-party application credentials are not valid, or the authenticating identity does not have access to the federated root folder. Confirm the client ID and secret (for OAuth M2M) or re-authenticate (for OAuth U2M), and verify that the identity has permission on the root folder. See Before you begin.

  • Last updated on