Share via

Access Azure Databricks tables from Apache Iceberg clients

Important

Unity Catalog Apache Iceberg REST Catalog API is in Public Preview in Databricks Runtime 16.4 LTS and above.

Unity Catalog has a read-only implementation of the Iceberg REST Catalog API, which is generally available. This endpoint is recommended for reading Delta tables with Iceberg reads enabled. See Read Databricks tables from Apache Iceberg clients (legacy) more information.

The Apache Iceberg REST catalog lets supported clients, such as Apache Spark, Apache Flink, and Trino, read from and write to Unity Catalog–registered Iceberg tables on Azure Databricks. Snowflake is supported for read access only.

For a full list of supported integrations, see Unity Catalog integrations.

Use the Unity Catalog Iceberg catalog endpoint

Unity Catalog provides an implementation of the Iceberg REST catalog API specification.

Configure access using the endpoint /api/2.1/unity-catalog/iceberg-rest. See the Iceberg REST API spec for details on using this REST API.

Note

Azure Databricks has introduced credential vending for some Iceberg reader clients. Databricks recommends using credential vending to control access to cloud storage locations for supported systems. See Unity Catalog credential vending for external system access.

If credential vending is unsupported for your client, you must configure access from the client to the storage location containing the files and metadata for the Delta or Iceberg table. Refer to documentation for your Iceberg client for configuration details.

Requirements

Azure Databricks supports Iceberg REST catalog access to tables as part of Unity Catalog. You must have Unity Catalog enabled in your workspace to use these endpoints. The following table types are accessible via the Iceberg REST Catalog:

Topic Read Write
Managed Iceberg Yes Yes
Foreign Iceberg Yes No
Managed Delta (with Iceberg reads enabled) Yes No
External Delta (with Iceberg read enabled) Yes No

Foreign Iceberg tables are not automatically refreshed when reading via the Iceberg REST Catalog API. To refresh, you must run REFRESH FOREIGN TABLE to read the latest snapshot. Credential vending on Foreign Iceberg tables is not supported.

Note

You must configure Delta tables to be accessible via the Iceberg REST Catalog API. See Read Delta tables with Iceberg clients.

You must complete the following configuration steps to configure access to read or write to Azure Databricks tables from Iceberg clients using the Iceberg REST catalog:

Use Iceberg tables with Apache Spark

The following is an example how to configure Apache Spark to access Azure Databricks tables via the Iceberg REST Catalog API using OAuth authentication:

"spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions",

# Configuration for accessing Uniform tables in Unity Catalog
"spark.sql.catalog.<spark-catalog-name>": "org.apache.iceberg.spark.SparkCatalog",
"spark.sql.catalog.<spark-catalog-name>.type": "rest",
"spark.sql.catalog.<spark-catalog-name>.rest.auth.type": "oauth2",
"spark.sql.catalog.<spark-catalog-name>.uri": "<workspace-url>/api/2.1/unity-catalog/iceberg-rest",
"spark.sql.catalog.<spark-catalog-name>.oauth2-server-uri": "<workspace-url>/oidc/v1/token",
"spark.sql.catalog.<spark-catalog-name>.credential":"<oauth_client_secret>",
"spark.sql.catalog.<spark-catalog-name>.warehouse":"<uc-catalog-name>"
"spark.sql.catalog.<spark-catalog-name>.scope":"all-apis"

Replace the following variables:

  • <uc-catalog-name>: The name of the catalog in Unity Catalog that contains your tables.
  • <spark-catalog-name>: The name you want to assign the catalog in your Spark session.
  • <workspace-url>: URL of the Azure Databricks workspace.
  • <oauth_client_id>: OAuth client ID for the authenticating principal.
  • <oauth_client_secret>: OAuth client secret for the authenticating principal.

With these configurations, you can query tables in Unity Catalog using Apache Spark. To access tables across multiple catalogs, you must configure each catalog separately.

When you query tables in Unity Catalog using Spark configurations, keep the following in mind:

  • You need "spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions" only if you are running Iceberg-specific stored procedures.

  • Azure Databricks uses cloud object storage for all tables. You must add the the iceberg-spark-runtime JAR as Spark packages:

    • AWS: org.apache.iceberg:iceberg-aws-bundle:<iceberg-version>
    • Azure: org.apache.iceberg:iceberg-azure-bundle:<iceberg-version>
    • GCP: org.apache.iceberg:iceberg-gcp-bundle:<iceberg-version>

    For details, see the documentation for the Iceberg AWS integration for Spark.

    Note

    These configurations are not required when accessing Iceberg tables from Azure Databricks. Loading external Iceberg JARs onto Azure Databricks clusters is not supported.

Read Azure Databricks tables with Snowflake

The following is an example of the recommended configuration settings to allow Snowflake to read Azure Databricks tables by connecting to the Iceberg REST Catalog in Unity Catalog:

CREATE OR REPLACE CATALOG INTEGRATION <catalog-integration-name>
  CATALOG_SOURCE = ICEBERG_REST
  TABLE_FORMAT = ICEBERG
  CATALOG_NAMESPACE = '<uc-schema-name>'
  REST_CONFIG = (
    CATALOG_URI = '<workspace-url>/api/2.1/unity-catalog/iceberg-rest',
    WAREHOUSE = '<uc-catalog-name>'
    ACCESS_DELEGATION_MODE = VENDED_CREDENTIALS
  )
  REST_AUTHENTICATION = (
    TYPE = BEARER
    BEARER_TOKEN = '<token>'
  )
  ENABLED = TRUE;

Replace the following variables:

  • <catalog-integration-name>: The name you want to assign the catalog registered to Snowflake.
  • <uc-schema-name>: The name of the schema in Unity Catalog you need to access.
  • <uc-catalog-name>: The name of the catalog in Unity Catalog you need to access.
  • <workspace-url>: URL of the Azure Databricks workspace.
  • <token>: PAT token for the principal configuring the integration.

Use Azure Databricks tables with PyIceberg

The following is an example of the configuration settings to allow PyIceberg to access Azure Databricks tables by connecting to the Iceberg REST Catalog in Unity Catalog:

catalog:
  unity_catalog:
    uri: https://<workspace-url>/api/2.1/unity-catalog/iceberg-rest
    warehouse: <uc-catalog-name>
    token: <token>

Replace the following variables:

  • <workspace-url>: URL of the Azure Databricks workspace.
  • <uc-catalog-name>: The name of the catalog in Unity Catalog you need to access.
  • <token>: PAT token for the principal configuring the integration.

See the documentation for the PyIceberg REST catalog configuration.

REST API curl example

You can also use a REST API call like the one in this curl example to load a table:

curl -X GET -H "Authorization: Bearer $OAUTH_TOKEN" -H "Accept: application/json" \
https://<workspace-instance>/api/2.1/unity-catalog/iceberg-rest/v1/catalogs/<uc_catalog_name>/namespaces/<uc_schema_name>/tables/<uc_table_name>

You should then receive a response like this:

{
  "metadata-location": "abfss://my-container@my-storage-account.dfs.core.windows.net/path/to/iceberg/table/metadata/file",
  "metadata": <iceberg-table-metadata-json>,
  "config": {
    "expires-at-ms": "<epoch-ts-in-millis>",
    "adls.sas-token.<storage-account-name>.dfs.core.windows.net": "<temporary-sas-token>"
  }
}

Note

The expires-at-ms field in the response indicates the expiration time of the credentials and has a default expiry time of one hour. For better performance, have the client cache the credentials until the expiration time before requesting a new one.