Secrets

Article
11/10/2023

A secret is a key-value pair that stores secret material, with a key name unique within a secret scope. Each scope is limited to 1000 secrets. The maximum allowed secret value size is 128 KB.

Create a secret

Secret names are case insensitive.

The method for creating a secret depends on whether you are using an Azure Key Vault-backed scope or a Databricks-backed scope.

Create a secret in an Azure Key Vault-backed scope

To create a secret in Azure Key Vault you use the Azure Set Secret REST API or Azure portal UI.

Azure Key Vault

Create a secret in a Databricks-backed scope

To create a secret in a Databricks-backed scope using the Databricks CLI (version 0.205 and above):

databricks secrets put-secret <scope-name> <key-name>

An editor opens and displays content like this:

# ----------------------------------------------------------------------
# Do not edit the above line. Everything that follows it will be ignored.
# Please input your secret value above the line. Text will be stored in
# UTF-8 (MB4) form and any trailing new line will be stripped.
# Exit without saving will abort writing secret.

Paste your secret value above the line and save and exit the editor. Your input is stripped of the comments and stored associated with the key in the scope.

If you issue a write request with a key that already exists, the new value overwrites the existing value.

You can provide the secret value in the CLI command by using the --string-value flag. For example:

databricks secrets put-secret <scope-name> <key-name> --string-value <secret>

You can also provide a secret from a file or from the command line. For more information about writing secrets, see What is the Databricks CLI?.

List secrets

To list secrets in a given scope:

databricks secrets list-secret <scope-name>

The response displays metadata information about the secret, such as the secret key name and last updated at timestamp (in milliseconds since epoch). You use the Secrets utility (dbutils.secrets) in a notebook or job to read a secret. For example:

databricks secrets list --scope jdbc

Key name    Last updated
----------  --------------
password    1531968449039
username    1531968408097

Read a secret

You create secrets using the REST API or CLI, but you must use the Secrets utility (dbutils.secrets) in a notebook or job to read a secret.

Delete a secret

To delete a secret from a scope with the Databricks CLI:

databricks secrets delete-secret <scope-name> <key-name>

You can also use the Secrets API.

To delete a secret from a scope backed by Azure Key Vault, use the Azure SetSecret REST API or Azure portal UI.

Use a secret in a Spark configuration property or environment variable

Important

This feature is in Public Preview.

Note

Available in Databricks Runtime 6.4 Extended Support and above.

You can reference a secret in a Spark configuration property or environment variable. Retrieved secrets are redacted from notebook output and Spark driver and executor logs.

Important

Keep the following security implications in mind when referencing secrets in a Spark configuration property or environment variable:

If table access control is not enabled on a cluster, any user with Can Attach To permissions on a cluster or Run permissions on a notebook can read Spark configuration properties from within the notebook. This includes users who do not have direct permission to read a secret. Databricks recommends enabling table access control on all clusters or managing access to secrets using secret scopes.
Even when table access control is enabled, users with Can Attach To permissions on a cluster or Run permissions on a notebook can read cluster environment variables from within the notebook. Databricks does not recommend storing secrets in cluster environment variables if they must not be available to all users on the cluster.
Secrets are not redacted from the Spark driver log stdout and stderr streams. By default, Spark driver logs are viewable by users with any of the following cluster level permissions:
- Can Attach To
- Can Restart
- Can Manage
You can optionally limit who can read Spark driver logs to users with the Can Manage permission by setting the cluster’s Spark configuration property spark.databricks.acl.needAdminPermissionToViewLogs true

Requirements and limitations

The following requirements and limitations apply to referencing secrets in Spark configuration properties and environment variables:

Cluster owners must have Can Read permission on the secret scope.
Only cluster owners can add a reference to a secret in a Spark configuration property or environment variable and edit the existing scope and name. Owners change a secret using the Secrets API. You must restart your cluster to fetch the secret again.
Users with the Can Manage permission on the cluster can delete a secret Spark configuration property or environment variable.

Syntax for referencing secrets in a Spark configuration property or environment variable

You can refer to a secret using any valid variable name or Spark configuration property. Azure Databricks enables special behavior for variables referencing secrets based on the syntax of the value being set, not the variable name.

The syntax of the Spark configuration property or environment variable value must be {{secrets/<scope-name>/<secret-name>}}. The value must start with {{secrets/ and end with }}.

The variable portions of the Spark configuration property or environment variable are:

<scope-name>: The name of the scope in which the secret is associated.
<secret-name>: The unique name of the secret in the scope.

For example, {{secrets/scope1/key1}}.

Note

There should be no spaces between the curly brackets. If there are spaces, they are treated as part of the scope or secret name.

Reference a secret with a Spark configuration property

You specify a reference to a secret in a Spark configuration property in the following format:

spark.<property-name> {{secrets/<scope-name>/<secret-name>}}

Any Spark configuration <property-name> can reference a secret. Each Spark configuration property can only reference one secret, but you can configure multiple Spark properties to reference secrets.

Example

You set a Spark configuration to reference a secret:

spark.password {{secrets/scope1/key1}}

To fetch the secret in the notebook and use it:

Python

spark.conf.get("spark.password")

SQL

SELECT ${spark.password};

Reference a secret in an environment variable

You specify a secret path in an environment variable in the following format:

<variable-name>={{secrets/<scope-name>/<secret-name>}}

You can use any valid variable name when you reference a secret. Access to secrets referenced in environment variables is determined by the permissions of the user who configured the cluster. Secrets stored in environmental variables are accessible by all users of the cluster, but are redacted from plaintext display like secrets referenced elsewhere.

Environment variables that reference secrets are accessible from a cluster-scoped init script. See Set and use environment variables with init scripts.