Secrets API 2.0
The Secrets API allows you to manage secrets, secret scopes, and access permissions. To manage secrets, you must:
- Create a secret scope.
- Add your secrets to the scope.
- If you have the Premium Plan, assign access control to the secret scope.
To learn more about creating and managing secrets, see Secret management and Secret access control. You access and reference secrets in notebooks and jobs by using Secrets utility (dbutils.secrets).
Important
To access Databricks REST APIs, you must authenticate. To use the Secrets API with Azure Key Vault secrets, you must authenticate using an Azure Active Directory token.
Create secret scope
| Endpoint | HTTP Method |
|---|---|
2.0/secrets/scopes/create |
POST |
You can either:
- Create an Azure Key Vault-backed scope in which secrets are stored in Azure-managed storage and encrypted with a cloud-based specific encryption key.
- Create a Databricks-backed secret scope in which secrets are stored in Databricks-managed storage and encrypted with a cloud-based specific encryption key.
Create an Azure Key Vault-backed scope
The scope name:
- Must be unique within a workspace.
- Must consist of alphanumeric characters, dashes, underscores, and periods, and may not exceed 128 characters.
The names are considered non-sensitive and are readable by all users in the workspace. By default, a workspace is limited to a maximum of 100 secret scopes. To increase this maximum for a workspace, contact your Databricks representative.
Example
curl --netrc --request POST \
https://<databricks-instance>/api/2.0/secrets/scopes/create \
--header "Content-Type: application/json" \
--header "Authorization: Bearer <token>" \
--header "X-Databricks-Azure-SP-Management-Token: <management-token>" \
--data @create-scope.json
create-scope.json:
{
"scope": "my-simple-azure-keyvault-scope",
"scope_backend_type": "AZURE_KEYVAULT",
"backend_azure_keyvault":
{
"resource_id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/azure-rg/providers/Microsoft.KeyVault/vaults/my-azure-kv",
"dns_name": "https://my-azure-kv.vault.azure.net/"
},
"initial_manage_principal": "users"
}
Replace:
<databricks-instance>with the Azure Databricks workspace instance name, for exampleadb-1234567890123456.7.azuredatabricks.net.<token>with your Azure Databricks personal access token. For more information, see Authentication using Azure Databricks personal access tokens.<management-token>with your Azure Active Directory token. For more information, see Get Azure AD tokens for users by using MSAL.- The contents of
create-scope.jsonwith fields that are appropriate for your solution.
This example uses a .netrc file.
If initial_manage_principal is specified, the initial ACL applied to the scope is applied to the supplied principal (user, service principal, or group) with MANAGE permissions. The only supported principal for this option is the group users, which contains all users in the workspace. If initial_manage_principal is not specified, the initial ACL with MANAGE permission applied to the scope is assigned to the API request issuer’s user identity.
Throws RESOURCE_ALREADY_EXISTS if a scope with the given name already exists.
Throws RESOURCE_LIMIT_EXCEEDED if maximum number of scopes in the workspace is exceeded.
Throws INVALID_PARAMETER_VALUE if the scope name is invalid.
For more information, see Create an Azure Key Vault-backed secret scope using the Databricks CLI.
Create a Databricks-backed secret scope
The scope name:
- Must be unique within a workspace.
- Must consist of alphanumeric characters, dashes, underscores, and periods, and may not exceed 128 characters.
The names are considered non-sensitive and are readable by all users in the workspace. A workspace is limited to a maximum of 100 secret scopes.
Example
curl --netrc --request POST \
https://<databricks-instance>/api/2.0/secrets/scopes/create \
--data @create-scope.json
create-scope.json:
{
"scope": "my-simple-databricks-scope",
"initial_manage_principal": "users"
}
Replace:
<databricks-instance>with the Azure Databricks workspace instance name, for exampleadb-1234567890123456.7.azuredatabricks.net.- The contents of
create-scope.jsonwith fields that are appropriate for your solution.
This example uses a .netrc file.
Throws RESOURCE_ALREADY_EXISTS if a scope with the given name already exists.
Throws RESOURCE_LIMIT_EXCEEDED if maximum number of scopes in the workspace is exceeded.
Throws INVALID_PARAMETER_VALUE if the scope name is invalid.
Request structure
| Field Name | Type | Description |
|---|---|---|
| scope | STRING |
Scope name requested by the user. Scope names are unique. This field is required. |
| initial_manage_principal | STRING |
This field is optional. If not specified, only the API request issuer’s identity is granted MANAGE permissions on the new scope. If the string users is specified, all users in the workspace are granted MANAGE permissions. |
Delete secret scope
| Endpoint | HTTP Method |
|---|---|
2.0/secrets/scopes/delete |
POST |
Delete a secret scope.
Example
curl --netrc --request POST \
https://<databricks-instance>/api/2.0/secrets/scopes/delete \
--data @delete-scope.json
delete-scope.json:
{
"scope": "my-secret-scope"
}
Replace:
<databricks-instance>with the Azure Databricks workspace instance name, for exampleadb-1234567890123456.7.azuredatabricks.net.- The contents of
delete-scope.jsonwith fields that are appropriate for your solution.
This example uses a .netrc file.
Throws RESOURCE_DOES_NOT_EXIST if the scope does not exist.
Throws PERMISSION_DENIED if the user does not have permission to make this API call.
Request structure
| Field Name | Type | Description |
|---|---|---|
| scope | STRING |
Name of the scope to delete. This field is required. |
List secret scopes
| Endpoint | HTTP Method |
|---|---|
2.0/secrets/scopes/list |
GET |
List all secret scopes available in the workspace.
Example
Request
curl --netrc --request GET \
https://<databricks-instance>/api/2.0/secrets/scopes/list \
| jq .
Replace <databricks-instance> with the Azure Databricks workspace instance name, for example adb-1234567890123456.7.azuredatabricks.net.
This example uses a .netrc file and jq.
Response
{
"scopes": [
{
"name": "my-databricks-scope",
"backend_type": "DATABRICKS"
},
{
"name": "mount-points",
"backend_type": "DATABRICKS"
}
]
}
Throws PERMISSION_DENIED if you do not have permission to make this API call.
Response structure
| Field Name | Type | Description |
|---|---|---|
| scopes | An array of SecretScope | The available secret scopes. |
Put secret
The method for creating or modifying a secret depends on the type of scope backend. To create or modify a secret in a scope backed by Azure Key Vault, use the Azure SetSecret REST API. To create or modify a secret from a Databricks-backed scope, use the following endpoint:
| Endpoint | HTTP Method |
|---|---|
2.0/secrets/put |
POST |
Insert a secret under the provided scope with the given name. If a secret already
exists with the same name, this command overwrites the existing secret’s value.
The server encrypts the secret using the secret scope’s encryption settings before
storing it. You must have WRITE or MANAGE permission on the secret scope.
The secret key must consist of alphanumeric characters, dashes, underscores, and periods, and cannot exceed 128 characters. The maximum allowed secret value size is 128 KB. The maximum number of secrets in a given scope is 1000.
You can read a secret value only from within a command on a cluster (for example, through a notebook);
there is no API to read a secret value outside of a cluster. The permission applied is based on
who is invoking the command and you must have at least READ permission.
Example
curl --netrc --request POST \
https://<databricks-instance>/api/2.0/secrets/put \
--data @put-secret.json
put-secret.json:
{
"scope": "my-databricks-scope",
"key": "my-string-key",
"string_value": "my-value"
}
Replace:
<databricks-instance>with the Azure Databricks workspace instance name, for exampleadb-1234567890123456.7.azuredatabricks.net.- The contents of
put-secret.jsonwith fields that are appropriate for your solution.
This example uses a .netrc file.
The input fields “string_value” or “bytes_value” specify the type of the secret, which will determine the value returned when the secret value is requested. Exactly one must be specified.
Throws RESOURCE_DOES_NOT_EXIST if no such secret scope exists.
Throws RESOURCE_LIMIT_EXCEEDED if maximum number of secrets in scope is exceeded.
Throws INVALID_PARAMETER_VALUE if the key name or value length is invalid.
Throws PERMISSION_DENIED if the user does not have permission to make this API call.
Request structure
| Field Name | Type | Description |
|---|---|---|
| string_value OR bytes_value | STRING OR BYTES |
If string_value, if specified, the value will be stored in UTF-8 (MB4) form. If bytes_value, if specified, value will be stored as bytes. |
| scope | STRING |
The name of the scope to which the secret will be associated with. This field is required. |
| key | STRING |
A unique name to identify the secret. This field is required. |
Delete secret
The method for deleting a secret depends on the type of scope backend. To delete a secret from a scope backed by Azure Key Vault, use the Azure SetSecret REST API. To delete a secret from a Databricks-backed scope, use the following endpoint:
| Endpoint | HTTP Method |
|---|---|
2.0/secrets/delete |
POST |
Delete the secret stored in this secret scope. You must have WRITE or MANAGE
permission on the secret scope.
Example
curl --netrc --request POST \
https://<databricks-instance>/api/2.0/secrets/delete \
--data @delete-secret.json
delete-secret.json:
{
"scope": "my-secret-scope",
"key": "my-secret-key"
}
Replace:
<databricks-instance>with the Azure Databricks workspace instance name, for exampleadb-1234567890123456.7.azuredatabricks.net.- The contents of
delete-secret.jsonwith fields that are appropriate for your solution.
This example uses a .netrc file.
Throws RESOURCE_DOES_NOT_EXIST if no such secret scope or secret exists.
Throws PERMISSION_DENIED if you do not have permission to make this API call.
Request structure
| Field Name | Type | Description |
|---|---|---|
| scope | STRING |
The name of the scope that contains the secret to delete. This field is required. |
| key | STRING |
Name of the secret to delete. This field is required. |
List secrets
| Endpoint | HTTP Method |
|---|---|
2.0/secrets/list |
GET |
List the secret keys that are stored at this scope. This is a metadata-only
operation; you cannot retrieve secret data using this API. You must have READ
permission to make this call.
Example
Request
curl --netrc --request GET \
'https://<databricks-instance>/api/2.0/secrets/list?scope=<scope-name>' \
| jq .
Or:
curl --netrc --get \
https://<databricks-instance>/api/2.0/secrets/list \
--data scope=<scope-name> \
| jq .
Replace:
<databricks-instance>with the Azure Databricks workspace instance name, for exampleadb-1234567890123456.7.azuredatabricks.net.<scope-name>with the name of the secrets scope, for examplemy-scope.
This example uses a .netrc file and jq.
Response
{
"secrets": [
{
"key": "my-string-key",
"last_updated_timestamp": 1520467595000
},
{
"key": "my-byte-key",
"last_updated_timestamp": 1520467595000
}
]
}
The last_updated_timestamp returned is in milliseconds since epoch.
Throws RESOURCE_DOES_NOT_EXIST if no such secret scope exists.
Throws PERMISSION_DENIED if you do not have permission to make this API call.
Request structure
| Field Name | Type | Description |
|---|---|---|
| scope | STRING |
The name of the scope whose secrets you want to list. This field is required. |
Response structure
| Field Name | Type | Description |
|---|---|---|
| secrets | An array of SecretMetadata | Metadata information of all secrets contained within the given scope. |
Put secret ACL
| Endpoint | HTTP Method |
|---|---|
2.0/secrets/acls/put |
POST |
Create or overwrite the ACL associated with the given principal (user, service principal, or group) on the specified scope point. In general, a user, service principal, or group will use the most powerful permission available to them, and permissions are ordered as follows:
MANAGE- Allowed to change ACLs, and read and write to this secret scope.WRITE- Allowed to read and write to this secret scope.READ- Allowed to read this secret scope and list what secrets are available.
You must have the MANAGE permission to invoke this API.
Example
curl --netrc --request POST \
https://<databricks-instance>/api/2.0/secrets/acls/put \
--data @put-secret-acl.json
put-secret-acl.json:
{
"scope": "my-secret-scope",
"principal": "data-scientists",
"permission": "READ"
}
Replace:
<databricks-instance>with the Azure Databricks workspace instance name, for exampleadb-1234567890123456.7.azuredatabricks.net.- The contents of
put-secret-acl.jsonwith fields that are appropriate for your solution.
This example uses a .netrc file.
The principal field specifies an existing Azure Databricks principal to be granted or revoked access
using the unique identifier of that principal. A user is specified with their email,
a service principal with its applicationId value, and a group with its group name.
Throws RESOURCE_DOES_NOT_EXIST if no such secret scope exists.
Throws RESOURCE_ALREADY_EXISTS if a permission for the principal already exists.
Throws INVALID_PARAMETER_VALUE if the permission is invalid.
Throws PERMISSION_DENIED if you do not have permission to make this API call.
Request structure
| Field Name | Type | Description |
|---|---|---|
| scope | STRING |
The name of the scope to apply permissions to. This field is required. |
| principal | STRING |
The principal to which the permission is applied. This field is required. |
| permission | AclPermission | The permission level applied to the principal. This field is required. |
Delete secret ACL
| Endpoint | HTTP Method |
|---|---|
2.0/secrets/acls/delete |
POST |
Delete the given ACL on the given scope.
You must have the MANAGE permission to invoke this API.
Example
curl --netrc --request POST \
https://<databricks-instance>/api/2.0/secrets/acls/delete \
--data @delete-secret-acl.json
delete-secret-acl.json:
{
"scope": "my-secret-scope",
"principal": "data-scientists"
}
Replace:
<databricks-instance>with the Azure Databricks workspace instance name, for exampleadb-1234567890123456.7.azuredatabricks.net.- The contents of
delete-secret-acl.jsonwith fields that are appropriate for your solution.
This example uses a .netrc file.
Throws RESOURCE_DOES_NOT_EXIST if no such secret scope, principal, or ACL exists.
Throws PERMISSION_DENIED if you do not have permission to make this API call.
Request structure
| Field Name | Type | Description |
|---|---|---|
| scope | STRING |
The name of the scope to remove permissions from. This field is required. |
| principal | STRING |
The principal to remove an existing ACL from. This field is required. |
Get secret ACL
| Endpoint | HTTP Method |
|---|---|
2.0/secrets/acls/get |
GET |
Describe the details about the given ACL, such as the group and permission.
You must have the MANAGE permission to invoke this API.
Example
Request
curl --netrc --request GET \
'https://<databricks-instance>/api/2.0/secrets/acls/get?scope=<scope-name>&principal=<principal-name>' \
| jq .
Or:
curl --netrc --get \
https://<databricks-instance>/api/2.0/secrets/acls/get \
--data 'scope=<scope-name>&principal=<principal-name>' \
| jq .
Replace:
<databricks-instance>with the Azure Databricks workspace instance name, for exampleadb-1234567890123456.7.azuredatabricks.net.<scope-name>with the name of the secrets scope, for examplemy-scope.<principal-name>with the name of the principal, for exampleusers.
This example uses a .netrc file and jq.
Response
{
"principal": "data-scientists",
"permission": "READ"
}
Throws RESOURCE_DOES_NOT_EXIST if no such secret scope exists.
Throws PERMISSION_DENIED if you do not have permission to make this API call.
Request structure
| Field Name | Type | Description |
|---|---|---|
| scope | STRING |
The name of the scope to fetch ACL information from. This field is required. |
| principal | STRING |
The principal to fetch ACL information for. This field is required. |
Response structure
| Field Name | Type | Description |
|---|---|---|
| principal | STRING |
The principal to which the permission is applied. This field is required. |
| permission | AclPermission | The permission level applied to the principal. This field is required. |
List secret ACLs
| Endpoint | HTTP Method |
|---|---|
2.0/secrets/acls/list |
GET |
List the ACLs set on the given scope.
You must have the MANAGE permission to invoke this API.
Example
Request
curl --netrc --request GET \
'https://<databricks-instance>/api/2.0/secrets/acls/list?scope=<scope-name>' \
| jq .
Or:
curl --netrc --get \
https://<databricks-instance>/api/2.0/secrets/acls/list \
--data scope=<scope-name> \
| jq .
Replace:
<databricks-instance>with the Azure Databricks workspace instance name, for exampleadb-1234567890123456.7.azuredatabricks.net.<scope-name>with the name of the secrets scope, for examplemy-scope.
This example uses a .netrc file and jq.
Response
{
"items": [
{
"principal": "admins",
"permission": "MANAGE"
},
{
"principal": "data-scientists",
"permission": "READ"
}
]
}
Throws RESOURCE_DOES_NOT_EXIST if no such secret scope exists.
Throws PERMISSION_DENIED if you do not have permission to make this API call.
Request structure
| Field Name | Type | Description |
|---|---|---|
| scope | STRING |
The name of the scope to fetch ACL information from. This field is required. |
Response structure
| Field Name | Type | Description |
|---|---|---|
| items | An array of AclItem | The associated ACLs rule applied to principals in the given scope. |
Data structures
In this section:
AclItem
An item representing an ACL rule applied to the given principal (user, service principal, or group) on the associated scope point.
| Field Name | Type | Description |
|---|---|---|
| principal | STRING |
The principal to which the permission is applied. This field is required. |
| permission | AclPermission | The permission level applied to the principal. This field is required. |
SecretMetadata
The metadata about a secret. Returned when listing secrets. Does not contain the actual secret value.
| Field Name | Type | Description |
|---|---|---|
| key | STRING |
A unique name to identify the secret. |
| last_updated_timestamp | INT64 |
The last updated timestamp (in milliseconds) for the secret. |
SecretScope
An organizational resource for storing secrets. Secret scopes can be different types, and ACLs can be applied to control permissions for all secrets within a scope.
| Field Name | Type | Description |
|---|---|---|
| name | STRING |
A unique name to identify the secret scope. |
| backend_type | ScopeBackendType | The type of secret scope backend. |
AclPermission
The ACL permission levels for secret ACLs applied to secret scopes.
| Permission | Description |
|---|---|
| READ | Allowed to perform read operations (get, list) on secrets in this scope. |
| WRITE | Allowed to read and write secrets to this secret scope. |
| MANAGE | Allowed to read/write ACLs, and read/write secrets to this secret scope. |
ScopeBackendType
The type of secret scope backend.
| Type | Description |
|---|---|
| AZURE_KEYVAULT | A secret scope in which secrets are stored in an Azure Key Vault. |
| DATABRICKS | A secret scope in which secrets are stored in Databricks managed storage and encrypted with a cloud-based specific encryption key. |
Feedback
Submit and view feedback for