Manage personal access tokens

  • Article
  • 13 minutes to read

To authenticate to the Azure Databricks REST API, a user can create a personal access token and use it in their REST API request. Tokens have an optional expiration date and can be revoked. See Authentication using Azure Databricks personal access tokens.

The number of personal access tokens per user is limited to 600 per workspace.

The ability to create personal access tokens is enabled by default for all Azure Databricks workspaces that were created in 2018 or later. Workspace administrators can enable or disable the ability to create personal token access for any workspace, regardless of when the workspace was created.

When the ability to create personal access tokens is enabled for a workspace, workspace administrators can also monitor tokens, control which non-admin users can create tokens and use tokens, and set maximum lifetimes for new tokens.

Note

You can also enable Azure Databricks users to use Azure Active Directory tokens for REST API access instead of Azure Databricks personal access tokens. If your workspace uses Azure Active Directory tokens, the instructions in this article do not apply.

When the ability to generate personal access tokens is enabled for your workspace, by default all users in your Azure Databricks workspace can generate personal access tokens to access Azure Databricks REST APIs, and they can generate these tokens with any expiration date they like, including an indefinite lifetime.

As an Azure Databricks admin, you can use the Token Management API 2.0 and Permissions API 2.0 to control token usage at a more fine-grained level. The APIs are published on each workspace instance. To learn how to access and authenticate to the API, see Authentication using Azure Databricks personal access tokens. You must access the API as an Azure Databricks admin.

For some tasks, you can also use the Admin Console.

The following table indicates the tasks you can perform using the web application and those you can perform using the REST APIs. For the cells that say Yes, click the word to view the related documentation.

Task Description Admin Console REST API
Enable/disable Enable or disable all tokens for this workspace Yes Yes
Control who can use tokens Limit personal access token creation and usage to specified users and groups in this workspace. If you revoke a user’s permission to create and use tokens, that user’s existing tokens are also revoked. Yes Yes
Max lifetime for new tokens. Set the maximum lifetime of new tokens in this workspace No Yes
Manage existing tokens For existing tokens, get the token creator, expiration date, and user-provided token description. Revoke tokens for users who should no longer have access to Azure Databricks APIs. By monitoring and controlling token creation, you reduce the risk of lost tokens or long-lasting tokens that could lead to data exfiltration from the workspace. No Yes

See the following diagram for the typical flow of token management using the REST API for administration and token usage. See the table in this topic for the set of tasks that you could alternatively use the Admin Console.

Token management

Requirements

This feature requires the Premium Plan.

Enable or disable token-based authentication for the workspace

Token-based authentication is enabled by default for all Azure Databricks workspaces that were created in 2018 or later. You can change this setting in the Admin Console. To specify which users are allowed to use tokens, see Control who can use or create tokens.

Warning

Partner Connect and partner integrations require personal access tokens to be enabled on a workspace.

To disable the ability to create personal access tokens for the workspace:

  1. Go to the Admin Console.
  2. Click the Workspace Settings tab.
  3. Click the Personal Access Tokens toggle.
  4. Click Confirm. This change may take a few seconds to take effect.

To use token-based authentication for a REST API request, see Authentication using Azure Databricks personal access tokens.

No tokens are deleted when you disable token-based authentication for a workspace. If tokens are re-enabled later, any non-expired tokens are immediately available for use.

If you want to disable token access for a subset of users, you can keep token-based authentication enabled for the workspace and then set fine-grained permissions for users and groups. See Control who can use or create tokens.

You can also use the REST API to make this change. To enable or disable the token management feature for a workspace, call the workspace configuration for tokens API (PATCH /workspace-conf). In a JSON request body, specify enableTokensConfig as true (enabled) or false (disabled). To render the OpenAPI in HTML or import into other tools, see Token Management API 2.0.

For example, to enable the feature:

curl -X PATCH -n \
  https://<databricks-instance>/api/2.0/workspace-conf \
  -d '{
    "enableTokensConfig": "true",
    }'

Control who can use or create tokens

A user can have one of the following token permissions:

  • No permissions

  • Can Use – Whether a user can use personal access tokens. For workspaces created after the release of Azure Databricks platform version 3.28 (Sept 9-15, 2020) by default no non-admin user or group is granted the Can Use permission. Admins must explicitly grant the Can Use permission to a user, a group, or to all users by granting it to the users group.

    Important

    Workspaces created before 3.28 was released maintain the permissions that were already in place. The previous default was for all users to have the Can Use permission. Admins can revoke that group permission and grant it to other groups or to individual non-admin users. See Remove permissions.

  • Can Manage – Whether a user or group can manage personal access tokens and permissions for them. Users in the admins group have this permission by default and you cannot revoke it. No other groups can be granted this permission. The API enforces these rules.

This table lists the permissions required for each token-related task:

Task No permission Can Use Can Manage
Create a token – Yes Yes
Use a token for authentication – Yes Yes
Revoke your own token – Yes Yes
Revoke any user’s token ** – – Yes
List all tokens ** – – Yes
Modify token permissions *** – – Yes

Actions that are marked with ** require the Token Management API 2.0.

Actions that are marked with *** can be performed in the Admin Console or with the Permissions API 2.0. See Control who can use or create tokens.

Manage token permissions using the admin console

To manage token permissions for the workspace using the admin console:

  1. Go to the admin console.

  2. Click the Workspace Settings tab.

  3. Click the Permissions button next to Personal Access Tokens to open the token permissions editor.

  4. Add, remove, or update permissions. For information about the meaning of a permission and the default permissions for a workspace, see Control who can use or create tokens.

    If the users group has the Can Use permission and you want to apply more fine-grained access for non-admin users, remove the Can Use permissions from the users group by clicking the X next to the permission drop-down in the users row.

    To grant the permission to other entities, select each user or group to whom you want to grant access. Select a user or group from the Select User or Group… drop-down, select Can Use, and click the + Add button. In the following example, the admin has removed access for the users group and is granting access to the Data Science B2 group.

    Add group permissions

  5. Click Save to save your changes.

    Warning

    After saving your changes, any users who previously had either Can Use or Can Manage permission but no longer have either permission are denied access to token-based authentication and their active tokens are immediately deleted (revoked). Deleted tokens cannot be retrieved.

Manage token permissions using the Permissions API

Get all token permissions for the workspace

To get token permissions for all Azure Databricks users, Azure Databricks groups, and Azure service principals for the workspace, call the get all token permissions for the workspace API (GET /permissions/authorization/tokens). To render the OpenAPI in HTML or import into other tools, see Token Management API 2.0.

The response includes an access_control_list array. Each element is a user object, a group object, or a service principal object. They each have an identity field appropriate to the type: users have a user_name field, groups have a group_name field, and service principals have a service_principal_name field. All elements have an all_permissions field that specifies which permission levels (CAN_USE or CAN_MANAGE) are granted.

For example:

curl -n -X GET "https://<databricks-instance>/api/2.0/preview/permissions/authorization/tokens"

Example response:

{
  "object_id": "authorization/tokens",
  "object_type": "tokens",
  "access_control_list": [
    {
      "user_name": "jsmith@example.com",
      "all_permissions": [
        {
          "permission_level": "CAN_USE",
          "inherited": false
        }
      ]
    }
  ]
}

Set token permissions

You can set permissions on one or more users, groups, or service principals. For each user, you need to know the email address, which is specified in the user_name request property. For each group, specify the group name in the group_name property. For a service principal, specify the service principal applicationId value in the service_principal_name property.

To set token permissions, call the set token permissions API (PATCH /permissions/authorization/tokens). To render the OpenAPI in HTML or import into other tools, see Token Management API 2.0.

You can set permissions on one or more user, groups, or Azure service principals. For each user, you need to know the email address, which is specified in the user_name request property. For each group, specify the group name in the group_name property. For an Azure service principal, specify the service principal applicationId value in the service_principal_name property.

Entities (such as users or groups) not mentioned explicitly are not directly affected by this request, although changes to group membership can indirectly affect user access.

You can only grant, not revoke, permissions with this API.

For example, the following example grants access to user jsmith@example.com and the group mygroup.

curl -n -X PATCH "https://<databricks-instance>/api/2.0/preview/permissions/authorization/tokens"
  -d '{
    "access_control_list": [
      {
        "user_name": "jsmith@example.com",
        "permission_level": "CAN_USE",
      },
      {
        "group_name": "mygroup",
        "permission_level": "CAN_USE",
      }
    ]
  }'

Example response:

{
  "access_control_list": [
    {
      "user_name": "jsmith@example.com",
      "all_permissions": [
        {
          "permission_level": "CAN_USE",
          "inherited": false
        }
      ]
    },
    {
      "group_name": "mygroup",
      "all_permissions": [
        {
          "permission_level": "CAN_USE",
          "inherited": false
        }
      ]
    }
  ]
}

If you want to set token permissions for all entities in the workspace in one request, use the Replace all token permissions for workspace API (PUT /permissions/authorization/tokens). See Remove permissions. To render the OpenAPI in HTML or import into other tools, see Token Management API 2.0.

Remove permissions

Note

For workspaces created after the release of Azure Databricks platform version 3.28 (Sept 9-15, 2020) the default is for no users to have the Can Use permission. Admins must explicitly grant those permissions, whether to the entire users group or on a user-by-user or group-by-group basis. Workspaces created before 3.28 was released maintain the permissions that were already in place. The default was for all users to have the Can Use permission. Admins can revoke that group permission assignment and grant it to other groups or to individual non-admin users.

To remove permissions from all or some non-admin users, use the update all permissions API (PUT /permissions/authorization/tokens), which requires that you specify the complete set of permissions for all objects that are granted permissions for the entire workspace. To render the OpenAPI in HTML or import into other tools, see Token Management API 2.0.

If you want to authorize a subset of non-admin users to create and use tokens, do all three of the following:

  • Grant the users, groups, and Azure service principals the CAN_USE permission.
  • Do not grant the CAN_USE permission to the built-in users group if you want to only authorize some non-admin users. You can optionally choose to assign the permission to this group, in which case all non-admin users can create and use tokens.
  • Grant the built-in admins group the CAN_MANAGE permission. This is a requirement of the API.

Warning

After a successful request, if a user or Azure service principal does not have token permissions directly or indirectly through a group, their tokens are immediately deleted. Deleted tokens cannot be retrieved.

The following example grants Can Use tokens permission to group field-automation-group, omits permissions for the users (all users) group, and grants CAN_MANAGE permission to the admins group as required by the API. Any non-admin users that are not in the group field-support-engineers will lose access to token creation and their existing tokens are immediately deleted (revoked).

curl -n -X PUT "https://<databricks-instance>/api/2.0/preview/permissions/authorization/tokens"
  -d '{
    "access_control_list": [
      {
        "group_name": "field-automation-group",
        "permission_level": "CAN_USE",
      },
      {
        "group_name": "admins",
        "permission_level": "CAN_MANAGE",
      },
    ]
  }'

Set maximum lifetime of new tokens (REST API only)

Use the token lifetime management APIs to manage the maximum lifetime of new tokens in this workspace.

To set the maximum lifetime for new tokens, call the Set the maximum token lifetime for new tokens API (PATCH /workspace-conf). Set maxTokenLifetimeDays to the maximum token lifetime of new tokens in days, as an integer. If you set it to zero, new tokens are permitted to have no lifetime limit. To render the OpenAPI in HTML or import into other tools, see Token Management API 2.0.

For example:

curl -n -X PATCH "https://<databricks-instance>/api/2.0/workspace-conf"
  -d '{
  "maxTokenLifetimeDays": "90"
  }'

Warning

This limit applies only to new tokens. To review existing tokens, see the get tokens API. To render the OpenAPI in HTML or import into other tools, see Token Management API 2.0.

To get the workspace’s maximum lifetime for new tokens, call the get the maximum token lifetime for new tokens API (GET /workspace-conf) and pass keys=maxTokenLifetimeDays as a query parameter. The response includes an maxTokenLifetimeDays property that is the maximum token lifetime of new tokens in days, as an integer. If it is zero, new tokens are permitted to have no lifetime limit.

For example:

curl -n -X GET "https://<databricks-instance>/api/2.0/workspace-conf?keys=maxTokenLifetimeDays"

Example response:

{
    "maxTokenLifetimeDays": "90"
}

Monitor and revoke tokens (REST API only)

Use the token management APIs to manage existing tokens in the workspace.

Get tokens for the workspace

To get the workspace’s tokens, call the get all tokens API (GET /token-management/tokens). The response includes a token_infos array. Each element represents a token and includes fields for ID (token_id), creation time (creation_time), expiry time (expiry_time), description (comment), and the user that created it (the ID is created_by_id and the username is created_by_username). To render the OpenAPI in HTML or import into other tools, see Token Management API 2.0.

You can learn more about the user using the SCIM get user API (GET /scim/v2/Users/{id}).

To filter results by a user, set the request body property created_by_id (for the ID) or created_by_username (for the username). You can get a user ID from a display name using the SCIM get users API (GET /scim/v2/Users)

For example:

curl -n -X GET "https://<databricks-instance>/api/2.0/token-management/tokens"
  -d '{
  "created_by_id": "1234567890"
  }'

Example response:

{
  "token_infos": [
    {
      "token_id": "<token-id>",
      "creation_time": 1580265020299,
      "expiry_time": 1580265020299,
      "comment": "This is for ABC division's automation scripts.",
      "created_by_id": 1234567890,
      "created_by_username": "jsmith@example.com"
    }
  ]
}

Alternatively, get a specific token using the get a token API (GET /token-management/tokens/{token_id}). To render the OpenAPI in HTML or import into other tools, see Token Management API 2.0.

Delete (revoke) a token

  1. Find the token ID. See Get tokens for the workspace.
  2. Call the delete a token API (DELETE /token-management/tokens). Pass the token ID in the path.

For example:

curl -n -X DELETE "https://<databricks-instance>/api/2.0/token-management/tokens/<token-id>"

Terraform integration

You can manage permissions in a fully automated setup using Databricks Terraform provider and databricks_permissions as follows.

Warning

The following configuration contains the statement authorization = "tokens". There can be only one authorization = "tokens" permissions resource per Azure Databricks workspace, otherwise there will be a permanent configuration drift. After applying the following changes, users who previously had either CAN_USE or CAN_MANAGE permission but no longer have either permission have their access to token-based authentication revoked. Their active tokens are immediately deleted (revoked).

resource "databricks_group" "auto" {
  display_name = "Automation"
}

resource "databricks_group" "eng" {
  display_name = "Engineering"
}

resource "databricks_permissions" "token_usage" {
  authorization = "tokens"

  access_control {
    group_name       = databricks_group.auto.display_name
    permission_level = "CAN_USE"
  }

  access_control {
    group_name       = databricks_group.eng.display_name
    permission_level = "CAN_USE"
  }
}