Configure external secrets in Unity Catalog

Important

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Azure Databricks previews.

This page shows how to connect a Unity Catalog schema to an external secret manager so its secret values remain in Azure Key Vault.

For how external secrets work and their limitations, see External secrets in Unity Catalog.

Before you begin

  • Meet the requirements for Unity Catalog secrets.
  • The external secrets Beta must be enabled for your workspace.
  • The schema you want to back externally must not contain any Azure Databricks-managed secrets. Delete existing secrets before you switch the backend.
  • You must have a Unity Catalog connection to your external secret manager and a Unity Catalog service credential that the connection uses to authenticate. To create these, you must have USE CONNECTION on an existing connection or CREATE CONNECTION on the metastore to create one, and access to a service credential, or CREATE CREDENTIAL on the metastore to create one.
  • Azure Key Vault backing is available only on Azure Databricks on Azure, using an AZURE_KEY_VAULT connection.

Set up external secrets

Create a service credential

The connection authenticates to your external secret manager with a Unity Catalog service credential.

Create a service credential backed by a managed identity or service principal, and grant it permission to list and read secrets from the target Key Vault. For example, assign the Key Vault Secrets User role, or an equivalent access policy with get and list secret permissions. For instructions, see Create service credentials.

Create a connection

Create a Unity Catalog connection that references the service credential from the previous step and points to your external secret manager.

Catalog Explorer

  1. In Azure Databricks, open Catalog Explorer, click the + menu, and select Create a connection.
  2. Enter a Connection name and select the Azure Key Vault connection type.
  3. Select the service credential to authenticate with, and set the vault name of your Key Vault.
  4. Click Create.

Databricks CLI

Pass the connection body with --json, using the AZURE_KEY_VAULT connection type. Set vault_name to the name of your Key Vault and credential to the service credential name:

databricks connections create --json '{
  "name": "my_key_vault_connection",
  "connection_type": "AZURE_KEY_VAULT",
  "options": {
    "vault_name": "my-vault",
    "credential": "my_key_vault_credential"
  }
}'

REST API

Use the /api/2.1/unity-catalog/connections endpoint with the AZURE_KEY_VAULT connection type. Azure Databricks resolves vault_name to https://<vault_name>.vault.azure.net:

curl -X POST \
  -H "Authorization: Bearer $DATABRICKS_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "my_key_vault_connection",
    "connection_type": "AZURE_KEY_VAULT",
    "options": {
      "vault_name": "my-vault",
      "credential": "my_key_vault_credential"
    }
  }' \
  "$DATABRICKS_HOST/api/2.1/unity-catalog/connections"

Back a schema with the connection

Configure the schema's secret backend in Catalog Explorer.

  1. In Azure Databricks, open Catalog Explorer and go to the schema.

  2. In the schema's details, locate the external secrets manager setting and click Enable.

    The schema must not contain any Azure Databricks-managed secrets. If it does, remove them first. Enable stays disabled until the schema has no secrets.

  3. Select the connection you created in the previous step, then confirm.

The schema is now backed by your external secret manager, and its secrets appear in Unity Catalog.

To point the schema at a different connection, use Edit. To return the schema to Azure Databricks-managed storage, edit the schema and disable external secrets.

Setting or changing a schema's secret backend requires USE CATALOG on the parent catalog, ownership of the schema or MANAGE on it, and USE CONNECTION on the connection.

Read external secrets

After the schema is backed externally, its secrets appear in Unity Catalog and you read them like any other Unity Catalog secret. Listing a schema triggers an import, so a newly added secret appears only after the next list. The read commands are the same on both clouds.

Dbutils

dbutils applies secret redaction and is the recommended way to read a value. Requires Databricks Runtime 17.3 LTS or above, or serverless environment version 4 or above.

# List the secrets in the schema
all_secrets = dbutils.secrets.list(catalog="main", schema="default")

# Read a specific secret value from the external secret manager
my_secret = dbutils.secrets.get(catalog="main", schema="default", key="example_secret")

REST API

Set include_value=true and read the effective_value field to return the value. The REST API does not redact returned values, though it still audits access; Azure Databricks recommends dbutils instead.

curl -G \
  -H "Authorization: Bearer $DATABRICKS_TOKEN" \
  --data-urlencode "include_value=true" \
  "$DATABRICKS_HOST/api/2.1/unity-catalog/secrets/main.default.example_secret"

You can also browse and list external secrets in Catalog Explorer, the same way as Azure Databricks-managed secrets. See Read a secret.

If a secret does not appear in Unity Catalog yet, you can still read it by name. Provide the name as it appears in the external secret manager, with the character substitutions from Naming restrictions applied.

Additional resources

Feature Description
External secrets in Unity Catalog Learn how Unity Catalog imports, governs, and reads externally backed secrets, and review the limitations.
Secrets in Unity Catalog Create, govern, and manage secrets that Azure Databricks stores in Unity Catalog.
Create service credentials Create the Unity Catalog service credential that the connection uses to authenticate.
  • Last updated on