Secrets in Unity Catalog

This page describes how to create, read, govern, and manage secrets in Unity Catalog. A Unity Catalog secret is a securable object that stores sensitive material, such as a password, token, or API key. Your notebooks and jobs can reference the secret without exposing the value in code.

Unity Catalog secrets use the three-level namespace (catalog.schema.secret) and are available across the workspaces attached to a metastore. Unity Catalog privileges govern them. This enables you to apply the same access model and auditing that you use for other data assets to your secrets.

Note

Unity Catalog secrets are distinct from workspace-level Azure Databricks secrets, which are organized into secret scopes. Use Unity Catalog secrets when you want to govern secrets with Unity Catalog privileges and reference them with the three-level namespace.

How Unity Catalog secrets work

A Unity Catalog secret is a securable object under a schema, with the fully qualified name catalog.schema.secret. Like other Unity Catalog securable objects, secrets support privilege inheritance from the catalog and schema. For more information about securable objects and inheritance, see Unity Catalog securable objects reference.

You can use a Unity Catalog secret in the following ways:

Retrieve the value in code. With READ SECRET access, users can retrieve a secret value from notebooks and jobs using dbutils or the Unity Catalog REST API. They can then use it to authenticate with external systems or to encrypt and decrypt data.
Use the value in a session-scoped Python or Scala UDF. See Session-scoped UDFs.
Use the value in a Unity Catalog Python UDF. A scalar or Batch UDF declares each secret in its SECRETS clause. See Python UDFs.
Use the value in a Unity Catalog Scala UDF. A scalar UDF declares each secret in its SECRETS clause. See Scala UDFs.

For requirements and permission behavior across UDF types, see UDF requirements and permissions.

UDFs can use Azure Databricks-managed secrets or secrets backed by an external secret manager. See External secrets in Unity Catalog.

UDF requirements and permissions

Requirements and permission behavior differ between session-scoped and Unity Catalog UDFs.

Session-scoped UDFs

A session-scoped Python UDF retrieves a secret with databricks.secrets.get(), and a session-scoped Scala UDF retrieves one with com.databricks.Secrets.get(). Secret access uses the caller's permissions.

Compute requirements for secret access depend on the UDF language:

On serverless compute, the notebook or job session must use environment version 6 or above for Python and Scala UDFs.
On classic compute, session-scoped Python UDFs require Databricks Runtime 19 or above in standard or dedicated access mode.
On classic compute, session-scoped Scala UDFs require Databricks Runtime 19 or above with standard access mode.

Unity Catalog UDFs

Scalar and Batch Unity Catalog Python UDFs and scalar Unity Catalog Scala UDFs declare secrets in the SECRETS clause and must explicitly set environment_version to 6 or above. They support serverless compute, serverless SQL warehouses, and classic compute running Databricks Runtime 19 or above with standard access mode.

Pro SQL warehouses support scalar and Batch Unity Catalog Python UDFs that use secrets. Unity Catalog Scala UDFs that use secrets are not supported on pro SQL warehouses.

To create or replace a UDF that declares a secret, the principal executing the statement must have READ SECRET on the secret and USE CATALOG and USE SCHEMA on its parent catalog and schema. At runtime, the UDF uses the current function owner's permissions. Callers need the usual function privileges, including EXECUTE, but do not need direct access to the declared secrets. If the function owner loses permission to read a declared secret, the UDF fails.

Use secret-enabled UDFs in column masks on dedicated compute

You cannot invoke a Unity Catalog Python or Scala UDF that uses the SECRETS clause directly on dedicated access mode compute. However, you can create a Unity Catalog SQL function that calls the secret-enabled UDF and use the SQL function as an attribute-based access control (ABAC) column mask. When a query runs on dedicated compute, Azure Databricks delegates column mask enforcement to serverless compute. This exception applies only while enforcing the column mask; it does not enable direct invocation of the secret-enabled UDF on dedicated compute. See Unsupported compute versions.

Warning

Use secret values only within the UDF implementation. Do not return secret values or include them in UDF results. Secret redaction helps reduce accidental exposure in errors and logs, but it does not prevent UDF code from exposing secret material in query results.

Azure Databricks stores Unity Catalog secret values encrypted and applies secret redaction to reduce accidental exposure in outputs and logs. To rotate a secret, periodically update its value in the UI or with the Unity Catalog REST API.

Privileges for Unity Catalog secrets

The following privileges govern secrets. You can grant them at the catalog, schema, or individual secret level, and they follow Unity Catalog privilege inheritance.

Privilege	Description
`CREATE SECRET`	Allows a user to create a secret in a schema. Granted at the catalog or schema level.
`READ SECRET`	Allows a user to retrieve a secret value.
`WRITE SECRET`	Allows a user to update a secret value.
`REFERENCE SECRET`	Allows a user to reference a secret without access to the value.

To create a secret in a schema, a user must have USE CATALOG permission and either own the schema or have CREATE SECRET and USE SCHEMA on the schema. To learn how to grant privileges, see Manage privileges in Unity Catalog.

Before you begin

To use Unity Catalog secrets, you must meet the following requirements:

The workspace must be enabled for Unity Catalog. For an introduction, see What is Unity Catalog?.
You must access secrets from Unity Catalog-enabled compute. Azure Databricks recommends one of the following:
- Serverless jobs and notebooks using environment version 4 or above.
- Classic compute in standard access mode running Databricks Runtime 17.3 LTS or above.
To retrieve secrets with dbutils, the compute must run Databricks Runtime 17.3 LTS or above, or serverless environment version 4 or above.

Create a secret

Creating a secret requires that you have USE CATALOG permission and own the schema or have CREATE SECRET and USE SCHEMA on the schema. See Privileges for Unity Catalog secrets.

Catalog Explorer

In your Azure Databricks workspace, click Catalog to open Catalog Explorer.
Go to the schema where you want to create the secret.
Click Create > Secret.
Enter a name and value. Optionally, add a comment and an expiration date. If a secret expires, Catalog Explorer shows a warning.
Click Create.

REST API

Run the following cURL command using the /api/2.1/unity-catalog/secrets endpoint:

curl -X POST \
  -H "Authorization: Bearer $DATABRICKS_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{
    "catalog_name": "main",
    "schema_name": "default",
    "name": "example_secret",
    "value": "your_secret_value",
    "comment": "your secret description"
  }' \
  "$DATABRICKS_HOST/api/2.1/unity-catalog/secrets"

Read a secret

To read a secret value, you must have READ SECRET on the secret or on a parent catalog or schema.

Secrets utility (dbutils.secrets)

Azure Databricks recommends dbutils to read secrets, because it applies secret redaction. This option requires Databricks Runtime 17.3 LTS or above, or serverless environment version 4 or above.

# Read a specific secret
my_secret = dbutils.secrets.get(catalog="main", schema="default", key="example_secret")

For more information, see Secrets utility (dbutils.secrets).

REST API

Warning

Secret values retrieved with the Unity Catalog REST API are not subject to secret redaction, though access is still recorded in audit logs. Azure Databricks recommends dbutils instead.

To return the value, set include_value=true and read the effective_value field in the response:

curl -G \
  -H "Authorization: Bearer $DATABRICKS_TOKEN" \
  --data-urlencode "include_value=true" \
  "$DATABRICKS_HOST/api/2.1/unity-catalog/secrets/main.default.example_secret"

Use a secret in your code

After you read a Unity Catalog secret with dbutils.secrets.get, pass the returned value to your application code. dbutils redacts the value in cell output and logs, so you can use it without exposing it.

The following example uses a secret as a bearer token to call an external API:

import requests

api_key = dbutils.secrets.get(catalog="main", schema="default", key="service_api_key")

response = requests.get(
    "https://api.example.com/v1/resource",
    headers={"Authorization": f"Bearer {api_key}"},
)
response.raise_for_status()

The following example retrieves a secret value and passes it to dbutils.credentials.getServiceCredentialsProvider to configure a boto3 session for the AWS SDK. A service credential name is not itself sensitive, so this example stores one in a secret only to illustrate chaining a retrieved secret value into another SDK call. For more information about service credentials, see Use Unity Catalog service credentials to connect to external cloud services.

import boto3

credential_name = dbutils.secrets.get(catalog="main", schema="default", key="service_credential_name")

boto3_session = boto3.Session(
    botocore_session=dbutils.credentials.getServiceCredentialsProvider(credential_name),
    region_name="your-aws-region",
)
sm = boto3_session.client("secretsmanager")

Manage permissions on secrets

Grant CREATE SECRET at the catalog or schema level to control who can create secrets. Grant READ SECRET, WRITE SECRET, or REFERENCE SECRET at the catalog, schema, or individual secret level to control access. Privilege inheritance applies. To learn more about granting and revoking privileges, see Manage privileges in Unity Catalog.

Grant the ability to create secrets

Catalog Explorer

In Catalog Explorer, go to the schema.
Click the Permissions tab.
Click Grant.
Select the principals to grant access to, then select CREATE SECRET.

If a principal does not have USE SCHEMA, a warning prompts you to grant it. USE SCHEMA is also required to create secrets in the schema.
Click Confirm.

SQL

GRANT CREATE SECRET, USE SCHEMA ON SCHEMA main.default TO `user@example.com`;

REST API

Run the following cURL command using the /api/2.1/unity-catalog/permissions/schema/{schema_name} endpoint:

curl -X PATCH \
  -H "Authorization: Bearer $DATABRICKS_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{
    "changes": [{
      "principal": "user@example.com",
      "add": ["CREATE_SECRET", "READ_SECRET", "REFERENCE_SECRET", "WRITE_SECRET"]
    }]
  }' \
  "$DATABRICKS_HOST/api/2.1/unity-catalog/permissions/schema/{schema_name}"

Grant access to a secret

Catalog Explorer

In Catalog Explorer, go to the secret and click it.
Click the Permissions tab.
Click Grant.
Select the principals and the privileges to grant, then click Confirm.

SQL

GRANT READ SECRET ON SECRET main.default.example_secret TO `user@example.com`;

REST API

Run the following cURL command using the /api/2.1/unity-catalog/permissions/secret/{catalog.schema.secret} endpoint:

curl -X PATCH \
  -H "Authorization: Bearer $DATABRICKS_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{
    "changes": [{
      "principal": "user@example.com",
      "add": ["READ_SECRET", "REFERENCE_SECRET", "WRITE_SECRET"]
    }]
  }' \
  "$DATABRICKS_HOST/api/2.1/unity-catalog/permissions/secret/{catalog.schema.secret}"

List, update, and delete secrets

List secrets

Catalog Explorer

In Catalog Explorer, go to the schema.
In the Overview pane, click Secrets to see all secrets in the schema.

Secrets utility (dbutils.secrets)

# List all secrets in a schema
all_secrets = dbutils.secrets.list(catalog="main", schema="default")

REST API

Use page_size to control the number of results per page. If more results are available, the response includes a next_page_token:

curl -G \
  -H "Authorization: Bearer $DATABRICKS_TOKEN" \
  --data-urlencode "catalog_name=main" \
  --data-urlencode "schema_name=default" \
  --data-urlencode "page_size=100" \
  "$DATABRICKS_HOST/api/2.1/unity-catalog/secrets"

To retrieve the next page, pass the next_page_token value from the previous response as the page_token parameter:

curl -G \
  -H "Authorization: Bearer $DATABRICKS_TOKEN" \
  --data-urlencode "catalog_name=main" \
  --data-urlencode "schema_name=default" \
  --data-urlencode "page_token=<next_page_token>" \
  "$DATABRICKS_HOST/api/2.1/unity-catalog/secrets"

Update a secret

To update a secret value, you must have WRITE SECRET on the secret.

Catalog Explorer

In Catalog Explorer, go to the schema and click Secrets in the Overview pane.
Click the secret to update.
In the top-right corner, click the kebab menu (vertical dots) and select Edit.
Enter a new value or expiration date, then click Confirm.

REST API

Update requests require the update_mask parameter. Only fields included in both update_mask and the request body are updated:

curl -X PATCH \
  -H "Authorization: Bearer $DATABRICKS_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"value": "new_secret_value"}' \
  "$DATABRICKS_HOST/api/2.1/unity-catalog/secrets/main.default.example_secret?update_mask=*"

Delete a secret

Catalog Explorer

In Catalog Explorer, go to the schema and click Secrets in the Overview pane.
Click the secret to delete.
In the top-right corner, click the kebab menu (vertical dots) and select Delete.
Enter the full name of the secret, then click Delete.

REST API

curl -X DELETE \
  -H "Authorization: Bearer $DATABRICKS_TOKEN" \
  "$DATABRICKS_HOST/api/2.1/unity-catalog/secrets/main.default.example_secret"

Audit events for Unity Catalog secrets

The system.access.audit system table records events related to Unity Catalog secrets. For example, to see all secret events for a user on a specific date, run the following query:

SELECT * FROM system.access.audit
WHERE
  user_identity.email = "user@example.com"
  AND event_date = "2026-02-20"
  AND service_name = "unityCatalog"
  AND action_name LIKE "%Secret%";

For more information about audit logs, see Audit log system table reference.

Encrypt secret values with customer-managed keys

By default, Azure Databricks encrypts secret values with Databricks-managed keys. You can instead use customer-managed keys (CMK). If you enable the CMK-encrypted managed catalog feature and attach a CMK configuration to your account, Azure Databricks uses the CMK to encrypt secret values. For more information, see Customer-managed keys for Unity Catalog.

Back secrets with an external secret manager

Instead of Azure Databricks storing secret values, you can back a schema with an external secret manager so the values stay in your cloud secret manager while remaining governable in Unity Catalog. AWS Secrets Manager on AWS and Azure Key Vault on Azure are supported. For how external secrets work, see External secrets in Unity Catalog. To back a schema with an external secret manager, see Configure external secrets in Unity Catalog.

Limitations

Unity Catalog secrets have the following limitations:

Limited direct access from SQL warehouses. You cannot retrieve Unity Catalog secrets directly from SQL warehouses. Scalar and Batch Unity Catalog Python UDFs can access secrets declared in their SECRETS clauses on pro and serverless SQL warehouses. Scalar Unity Catalog Scala UDFs can access declared secrets on serverless SQL warehouses only.
No direct SECRETS UDF invocation on dedicated compute. You cannot directly invoke scalar or Batch Unity Catalog Python UDFs or scalar Unity Catalog Scala UDFs that use the SECRETS clause on dedicated access mode compute. For the column-mask exception, see Use secret-enabled UDFs in column masks on dedicated compute.
No global discovery. Unity Catalog secrets do not appear in global search.
No BROWSE permission support. BROWSE on a catalog doesn't apply to Unity Catalog secrets. To make a secret discoverable, grant READ SECRET or REFERENCE SECRET on the individual secret or its schema.
No init scripts. You cannot use Unity Catalog secrets in global or cluster init scripts. Azure Databricks recommends using dedicated features instead of init scripts where possible.
No information schema. Information schema tables for secrets are not yet available. Use Catalog Explorer or the REST API for discovery.
dbutils runtime scope. dbutils retrieval is supported on Databricks Runtime-backed notebooks and jobs. Non-Databricks Runtime contexts, such as remote development or compiled JAR run modes, are not supported.
OAuth API scope. The Unity Catalog secrets API is accessible only with the unity-catalog OAuth API scope. Use the secrets API scope only for workspace-level Azure Databricks secrets.
Quota limits. Up to 100 secrets per schema and 1,000 per metastore.

Σχόλια

Ήταν χρήσιμη αυτή η σελίδα;

Last updated on 2026-09-30

Secrets in Unity Catalog

How Unity Catalog secrets work

UDF requirements and permissions

Session-scoped UDFs

Unity Catalog UDFs

Use secret-enabled UDFs in column masks on dedicated compute

Privileges for Unity Catalog secrets

Before you begin

Create a secret

Catalog Explorer

REST API

Read a secret

Secrets utility (dbutils.secrets)

REST API

Use a secret in your code

Manage permissions on secrets

Grant the ability to create secrets

Catalog Explorer

SQL

REST API

Grant access to a secret

Catalog Explorer

SQL

REST API

List, update, and delete secrets

List secrets

Catalog Explorer

Secrets utility (dbutils.secrets)

REST API

Update a secret

Catalog Explorer

REST API

Delete a secret

Catalog Explorer

REST API

Audit events for Unity Catalog secrets

Encrypt secret values with customer-managed keys

Back secrets with an external secret manager

Limitations

Σχόλια

Πρόσθετοι πόροι