Authentication for the Databricks CLI

Note

Databricks CLI use is subject to the Databricks License and Databricks Privacy Notice, including any Usage Data provisions.

This article explains how to set up authentication between the Databricks CLI and your Azure Databricks accounts and workspaces. It assumes you already installed the Databricks CLI. See Install or update the Databricks CLI.

Before you run Databricks CLI commands, you must configure authentication for the accounts or workspaces that you plan to use. The required setup depends on whether you want to run workspace-level commands, account-level commands, or both.

To view available CLI command groups, run databricks -h. For the list of corresponding REST API operations, see Databricks REST API.

For information about Microsoft Entra authentication to Databricks with Azure DevOps specifically, see Authenticate with Azure DevOps on Azure Databricks.

OAuth machine-to-machine (M2M) authentication

Machine-to-machine (M2M) authentication with OAuth allows services, scripts, or applications to access Databricks resources without interactive user sign-in. Instead of relying on personal access tokens (PATs) or user credentials, M2M authentication uses a service principal and an OAuth client credential flow to request and manage tokens.

To configure and use OAuth M2M authentication:

  1. Complete the OAuth M2M authentication setup steps. See Authorize service principal access to Azure Databricks with OAuth.

  2. Create a Azure Databricks configuration profile with the following fields in your .databrickscfg file.

    For account-level commands

    [<some-unique-configuration-profile-name>]
host          = <account-console-url>
account_id    = <account-id>
client_id     = <service-principal-client-id>
client_secret = <service-principal-oauth-secret>

    For workspace-level commands

    [<some-unique-configuration-profile-name>]
host          = <workspace-url>
client_id     = <service-principal-client-id>
client_secret = <service-principal-oauth-secret>

To use the profile, pass it with the --profile or -p flag in CLI commands. For example:

databricks account groups list -p <profile-name>

Press Tab after --profile or -p to show a list of available profiles.

OAuth user-to-machine (U2M) authentication

With OAuth user-to-machine (U2M) authentication, you log in interactively and the CLI manages short-lived tokens on your behalf. OAuth tokens expire in under an hour, which reduces risk if a token is accidentally exposed. See Authorize user access to Azure Databricks with OAuth.

To log in:

For account-level commands

databricks auth login --host <account-console-url> --account-id <account-id>

For workspace-level commands

databricks auth login --host <workspace-url>

The CLI guides you through a browser-based login flow. When you finish, the CLI saves a configuration profile with your workspace host and profile name. You can accept the suggested profile name or enter your own.

To use the profile, pass it with the --profile or -p flag in CLI commands. For example:

databricks clusters list -p <profile-name>

Press Tab after --profile or -p to show a list of available profiles.

Token storage

By default starting in Databricks CLI version 1.0.0, U2M tokens are stored in OS-native secure storage: Keychain on macOS, Credential Manager on Windows, and D-Bus Secret Service on Linux. Your .databrickscfg file holds only non-secret configuration such as host and profile name. The token itself is never written there.

Each profile has its own cached token. If you have multiple profiles pointing at the same workspace, logging out of one does not log out of the others.

If your environment does not support OS-native secure storage (for example, a headless server or CI environment), you can fall back to plaintext token storage.

For troubleshooting secure storage errors, see Troubleshoot the Databricks CLI.

Use the plaintext fallback

If your environment does not support secure storage, or if you depend on file-based token storage (for example, syncing the credentials file over SSH), you can configure the CLI to use the plain JSON file with either of the following options. After setting either option, run databricks auth login again.

  • Configure the DATABRICKS_AUTH_STORAGE environment variable.

    export DATABRICKS_AUTH_STORAGE=plaintext

  • Set the auth_storage setting in the Databricks configuration profile file ~/.databrickscfg.

    [__settings__]
auth_storage = plaintext

The environment variable takes precedence over the configuration profile setting.

Azure managed identities authentication

Azure managed identities authentication uses managed identities for Azure resources (formerly Managed Service Identities (MSI)) for authentication. See What are managed identities for Azure resources?. See also Authenticate with Azure managed identities.

To create an Azure user-assigned managed identity, do the following:

  1. Create or identify an Azure VM and install the Databricks CLI on it, then assign your managed identity to your Azure VM and your target Azure Databricks accounts, workspaces, or both. See Use Azure managed identities with Azure Databricks.

  2. On the Azure VM, create or identify an Azure Databricks configuration profile with the following fields in your .databrickscfg file. If you create the profile, replace the placeholders with the appropriate values.

    For account-level commands, set the following values in your .databrickscfg file:

    [<some-unique-configuration-profile-name>]
host            = <account-console-url>
account_id      = <account-id>
azure_client_id = <azure-managed-identity-application-id>
azure_use_msi   = true

    For workspace-level commands, set the following values in your .databrickscfg file:

    [<some-unique-configuration-profile-name>]
host            = <workspace-url>
azure_client_id = <azure-managed-identity-application-id>
azure_use_msi   = true

    Databricks recommends using host and explicitly assigning the identity to the workspace. Alternatively, use azure_workspace_resource_id with the Azure resource ID. This approach requires Contributor or Owner permissions on the Azure resource, or a custom role with specific Azure Databricks permissions.

  3. On the Azure VM, use the Databricks CLI's --profile or -p option followed by the name of your configuration profile to set the profile for Databricks to use, for example, databricks account groups list -p <configuration-profile-name> or databricks clusters list -p <configuration-profile-name>.

    Tip

    You can press Tab after --profile or -p to display a list of existing available configuration profiles to choose from, instead of entering the configuration profile name manually.

Microsoft Entra ID service principal authentication

Microsoft Entra ID service principal authentication uses the credentials of a Microsoft Entra ID service principal to authenticate. To create and manage service principals for Azure Databricks, see Service principals. See also Authenticate with Microsoft Entra service principals.

To configure and use Microsoft Entra ID service principal authentication, you must have the Authenticate with the Azure CLI installed locally. You must also do the following:

  1. Create or identify an Azure Databricks configuration profile with the following fields in your .databrickscfg file. If you create the profile, replace the placeholders with the appropriate values.

    For account-level commands, set the following values in your .databrickscfg file:

    [<some-unique-configuration-profile-name>]
host                = <account-console-url>
account_id          = <account-id>
azure_tenant_id     = <azure-service-principal-tenant-id>
azure_client_id     = <azure-service-principal-application-id>
azure_client_secret = <azure-service-principal-client-secret>

    For workspace-level commands, set the following values in your .databrickscfg file:

    [<some-unique-configuration-profile-name>]
host                = <workspace-url>
azure_tenant_id     = <azure-service-principal-tenant-id>
azure_client_id     = <azure-service-principal-application-id>
azure_client_secret = <azure-service-principal-client-secret>

    Databricks recommends using host and explicitly assigning the Microsoft Entra ID service principal to the workspace. Alternatively, use azure_workspace_resource_id with the Azure resource ID. This approach requires Contributor or Owner permissions on the Azure resource, or a custom role with specific Azure Databricks permissions.

  2. Use the Databricks CLI's --profile or -p option followed by the name of your configuration profile, as part of the Databricks CLI command call, for example databricks account groups list -p <configuration-profile-name> or databricks clusters list -p <configuration-profile-name>.

    Tip

    You can press Tab after --profile or -p to display a list of existing available configuration profiles to choose from, instead of entering the configuration profile name manually.

Azure CLI authentication

Azure CLI authentication uses the Azure CLI to authenticate the signed-in entity. See also Authenticate with the Azure CLI.

To configure Azure CLI authentication, you must do the following:

  1. Have the Azure CLI installed locally.

  2. Use the Azure CLI to log in to Azure Databricks by running the az login command. See Sign in with the Azure CLI.

  3. Create or identify an Azure Databricks configuration profile with the following fields in your .databrickscfg file. If you create the profile, replace the placeholders with the appropriate values.

    For account-level commands, set the following values in your .databrickscfg file:

    [<some-unique-configuration-profile-name>]
host       = <account-console-url>
account_id = <account-id>

    For workspace-level commands, set the following values in your .databrickscfg file:

    [<some-unique-configuration-profile-name>]
host = <workspace-url>

  4. Use the Databricks CLI's --profile or -p option followed by the name of your configuration profile, as part of the Databricks CLI command call, for example databricks account groups list -p <configuration-profile-name> or databricks clusters list -p <configuration-profile-name>.

    Tip

    You can press Tab after --profile or -p to display a list of existing available configuration profiles to choose from, instead of entering the configuration profile name manually.

Authentication order of evaluation

Whenever the Databricks CLI authenticates to a Azure Databricks workspace or account, it looks for required settings in the following order:

  1. Bundle settings files, for commands run from a bundle working directory. Bundle settings files can't contain credential values directly.
  2. Environment variables, as listed in this article and in Environment variables and fields for unified authentication.
  3. Configuration profiles in the .databrickscfg file.

As soon as the CLI finds the required setting, it stops searching other locations.

Examples:

  • If a DATABRICKS_TOKEN environment variable is set, the CLI uses it, even if multiple tokens exist in .databrickscfg.
  • If no DATABRICKS_TOKEN is set, and a bundle environment references a profile name such as dev → profile DEV, the CLI uses the credentials from that profile in .databrickscfg.
  • If no DATABRICKS_TOKEN is set, and a bundle environment specifies a host value, the CLI searches for a profile in .databrickscfg with a matching host and uses its token.

Personal access token authentication (legacy)

Important

Where possible, Azure Databricks recommends using OAuth instead of PATs for user account authentication because OAuth provides stronger security. Consider the following authentication methods:

Azure Databricks personal access token authentication uses an Azure Databricks personal access token to authenticate the target Azure Databricks entity, such as an Azure Databricks user account. See Authenticate with Azure Databricks personal access tokens (legacy).

To create a personal access token, follow the steps in Create personal access tokens for workspace users.

  • Last updated on