Access Databricks tables from Delta clients

This page describes how to use the Unity REST API to create, read, and write to Unity Catalog managed and external tables from external Delta clients. For a full list of supported integrations, see Unity Catalog integrations.

Tip

For information about how to read Azure Databricks data using Microsoft Fabric, see Use Microsoft Fabric to read data that is registered in Unity Catalog.

Create, read, and write using the Unity REST API

Important

Creating and writing to Unity Catalog managed tables from Delta clients is in Beta. External client support is limited.

The Unity REST API provides external clients create, read, and write access to tables registered to Unity Catalog. Configure access using the workspace URL as the endpoint. The following table types are accessible:

Table type	Read	Write	Create
Managed Delta	Yes	Yes*	Yes*
External Delta	Yes	Yes	Yes

* Supported for managed Delta tables with catalog commits.

Requirements

Azure Databricks supports Unity REST API access to tables as part of Unity Catalog. You must have Unity Catalog enabled in your workspace to use these endpoints.

You must also complete the following configuration steps to configure access to tables from Delta clients using the Unity REST API:

Enable External data access for your metastore. See Enable external data access on the metastore.
Grant the principal accessing data externally the EXTERNAL USE SCHEMA privilege on the schema containing the objects. See Grant a principal Unity Catalog privileges.
For external tables accessed by path: Grant the principal the EXTERNAL USE LOCATION privilege on the external location containing the table path. See Grant a principal Unity Catalog privileges.
Make sure that the principal has relevant privileges:
- SELECT on the table for reads
- MODIFY on the table for writes
- CREATE on the schema for table creation
- For external writes to managed Delta tables, verify that the table being written to has catalog commits enabled.
Authenticate using one of the following methods:
- Personal access token (PAT): See Authorize access to Azure Databricks resources.
- OAuth machine-to-machine (M2M) authentication: Supports automatic credential and token refresh for long-running Spark jobs (>1 hour). See Authorize service principal access to Azure Databricks with OAuth.

Limitations

External access to UniForm tables with IcebergCompatV3 is not currently supported. After externally writing to a UniForm table, you must run MSCK REPAIR TABLE in Databricks to generate Iceberg metadata.
Schema changes (for example, ALTER TABLE), table property updates, and table feature changes are not currently supported on managed tables from external clients.
External clients can't perform table maintenance operations, such as OPTIMIZE, VACUUM, and ANALYZE, on managed Delta tables.
External clients cannot create shallow clones.
External clients cannot create tables with generated columns, default columns, or constraint columns.
When creating external tables, Azure Databricks recommends using Apache Spark to ensure that column definitions are in a format compatible with Apache Spark. The API does not validate the correctness of the column specification. If the specification is not compatible with Apache Spark, then Databricks Runtime might be unable to read the tables.

Access Delta tables with Apache Spark using PAT authentication

The following configuration is required to read or write to Unity Catalog managed and external Delta tables with Apache Spark using PAT authentication:

"spark.sql.extensions": "io.delta.sql.DeltaSparkSessionExtension",
"spark.sql.catalog.spark_catalog": "io.unitycatalog.spark.UCSingleCatalog",
"spark.sql.catalog.<uc-catalog-name>": "io.unitycatalog.spark.UCSingleCatalog",
"spark.sql.catalog.<uc-catalog-name>.uri": "<workspace-url>",
"spark.sql.catalog.<uc-catalog-name>.token": "<token>",
"spark.sql.defaultCatalog": "<uc-catalog-name>",
"spark.jars.packages": "io.delta:delta-spark_4.1_2.13:4.2.0,io.unitycatalog:unitycatalog-spark_2.13:0.4.1,org.apache.hadoop:hadoop-azure:3.4.2"

Substitute the following variables:

<uc-catalog-name>: The name of the catalog in Unity Catalog that contains your tables.
<token>: Personal access token (PAT) for the principal configuring the integration.

<workspace-url>: The Azure Databricks workspace URL, including the workspace ID. For example, adb-1234567890123456.12.azuredatabricks.net.

Note

The package versions shown above are current as of the last update to this page. Newer versions might be available. Verify that package versions are compatible with your Spark version.

For additional details about configuring Apache Spark for cloud object storage, see the Unity Catalog OSS documentation.

Important

Databricks Runtime 16.4 and above is required to read from, write to, or create tables with catalog commits enabled. Databricks Runtime 18.0 and above is required to enable or disable catalog commits on existing tables.

To create managed Delta tables with catalog commits, use the following SQL:

CREATE TABLE <uc-catalog-name>.<schema-name>.<table-name> (id INT, desc STRING)
TBLPROPERTIES ('delta.feature.catalogManaged' = 'supported') USING delta;

To create external Delta tables, use the following SQL:

CREATE TABLE <uc-catalog-name>.<schema-name>.<table-name> (id INT, desc STRING)
USING delta
LOCATION <path>;

Access Delta tables with Apache Spark using OAuth authentication

Azure Databricks also supports OAuth machine-to-machine (M2M) authentication. OAuth automatically handles token and credential renewal for Unity Catalog authentication.

OAuth authentication for external Spark clients requires:

Unity Catalog Spark client version 0.4.1 or later (io.unitycatalog:unitycatalog-spark)
Apache Spark 4.0 or later
Delta Spark 4.2.0 or later
An OAuth M2M service principal with appropriate permissions. See Authorize service principal access to Azure Databricks with OAuth.

The following configuration is required to create, read, or write to Unity Catalog managed tables and external Delta tables with Apache Spark using OAuth authentication:

"spark.sql.extensions": "io.delta.sql.DeltaSparkSessionExtension",
"spark.sql.catalog.spark_catalog": "io.unitycatalog.spark.UCSingleCatalog",
"spark.sql.catalog.<uc-catalog-name>": "io.unitycatalog.spark.UCSingleCatalog",
"spark.sql.catalog.<uc-catalog-name>.uri": "<workspace-url>",
"spark.sql.catalog.<uc-catalog-name>.auth.type": "oauth",
"spark.sql.catalog.<uc-catalog-name>.auth.oauth.uri": "<oauth-token-endpoint>",
"spark.sql.catalog.<uc-catalog-name>.auth.oauth.clientId": "<oauth-client-id>",
"spark.sql.catalog.<uc-catalog-name>.auth.oauth.clientSecret": "<oauth-client-secret>",
"spark.sql.defaultCatalog": "<uc-catalog-name>",
"spark.jars.packages": "io.delta:delta-spark_4.1_2.13:4.2.0,io.unitycatalog:unitycatalog-spark_2.13:0.4.1,org.apache.hadoop:hadoop-azure:3.4.2"

Substitute the following variables:

<uc-catalog-name>: The name of the catalog in Unity Catalog that contains your tables.
<oauth-token-endpoint>: OAuth token endpoint URL. To construct this URL:
1. Locate your Azure Databricks account ID. See Locate your account ID.
2. Use the format: https://accounts.cloud.databricks.com/oidc/accounts/<account-id>/v1/token
<oauth-client-id>: OAuth client ID for your service principal. See Authorize service principal access to Azure Databricks with OAuth.
<oauth-client-secret>: OAuth client secret for your service principal. See Authorize service principal access to Azure Databricks with OAuth.

<workspace-url>: The Azure Databricks workspace URL, including the workspace ID. For example, adb-1234567890123456.12.azuredatabricks.net.

Note

The package versions shown above are current as of the last update to this page. Newer versions might be available. Verify that package versions are compatible with your Spark version.

Create Delta tables using the API

To create an external Delta table using the Unity Catalog REST API, follow these steps:

Step 1: Make a POST request to the Create Table API

Use the following API request to register the table metadata in Unity Catalog:

curl --location --request POST 'https://<workspace-url>/api/2.0/unity-catalog/tables/' \
--header 'Authorization: Bearer <token>' \
--header 'Content-Type: application/json' \
--data '{
  "name": "<table-name>",
  "catalog_name": "<uc-catalog-name>",
  "schema_name": "<schema-name>",
  "table_type": "EXTERNAL",
  "data_source_format": "DELTA",
  "storage_location": "<path>",
  "columns": [
    {
      "name": "id",
      "type_name": "LONG",
      "type_text": "bigint",
      "type_json": "\"long\"",
      "type_precision": 0,
      "type_scale": 0,
      "position": 0,
      "nullable": true
    },
    {
      "name": "name",
      "type_name": "STRING",
      "type_text": "string",
      "type_json": "\"string\"",
      "type_precision": 0,
      "type_scale": 0,
      "position": 1,
      "nullable": true
    }
  ]
}'

Substitute the following variables:

<workspace-url>: URL of the Azure Databricks workspace
<token>: Token for the principal making the API call
<uc-catalog-name>: Name of the catalog in Unity Catalog that will contain the external table
<schema-name>: Name of the schema within the catalog where the table will be created
<table-name>: Name of the external table
<path>: Fully qualified path to the table data

Step 2: Initialize the Delta table location

The API call above registers the table in :[UC], but it does not create the Delta files at the storage location. To initialize the table location, write an empty Delta table using Spark:

The schema used in this step must exactly match the column definitions provided in the API request.


from pyspark.sql.types import StructType, StructField, StringType, LongType

# Define schema matching your API call
schema = StructType([
    StructField("id", LongType(), True),
    StructField("name", StringType(), True)
])

# Create an empty DataFrame and initialize the Delta table
empty_df = spark.createDataFrame([], schema)
empty_df.write \
    .format("delta") \
    .mode("overwrite") \
    .save("<path>")

Note

The Create Table API for external clients has the following limitations:

Only external Delta tables are supported ("table_type": "EXTERNAL" and "data_source_format": "DELTA").
Only the following fields are allowed:
- name
- catalog_name
- schema_name
- table_type
- data_source_format
- columns
- storage_location
- properties
Column masks are not supported.

Pripomienky

Bola táto stránka užitočná?

Last updated on 2026-04-14

Zdieľať cez

Access Databricks tables from Delta clients

Create, read, and write using the Unity REST API

Requirements

Limitations

Access Delta tables with Apache Spark using PAT authentication

Access Delta tables with Apache Spark using OAuth authentication

Create Delta tables using the API

Step 1: Make a POST request to the Create Table API

Step 2: Initialize the Delta table location

Pripomienky

Ďalšie zdroje informácií