Share via

Azure managed identities authentication

  • Article

Azure managed identities authentication uses managed identities for Azure resources (formerly Managed Service Identities (MSI)) to authenticate with Azure Databricks. Programmatic calls to Azure Databricks account and workspace operations use this managed identity when working with Azure resources that support managed identities, such as Azure VMs.

Note

Managed identities for Azure resources are different than Microsoft Entra ID service principals, which Azure Databricks also supports for authentication. To learn how to use Microsoft Entra ID service principals for Azure Databricks authentication instead of managed identities for Azure resources, see:

Azure managed identities authentication is supported only between properly configured resources that support managed identities–such as Azure virtual machines (Azure VMs)–and Azure Databricks accounts and workspaces.

To configure Azure managed identities authentication with Azure Databricks, you must set the following associated environment variables, .databrickscfg fields, Terraform fields, or Config fields on a properly supported Azure VM:

  • The Azure Databricks host.
    • For account operations, specify https://accounts.azuredatabricks.net.
    • For workspace operations, specify the per-workspace URL, for example https://adb-1234567890123456.7.azuredatabricks.net.
  • For account operations, the Azure Databricks account ID.
  • The tenant ID of the managed identity.
  • The client ID of the managed identity.
  • The Azure resource ID.
  • Set Azure use managed identities to true.

To perform Azure managed identities authentication with Azure Databricks, integrate the following within your code, based on the participating tool or SDK:

Environment

To use environment variables for a specific Azure Databricks authentication type with a tool or SDK, see Supported authentication types by Azure Databricks tool or SDK or the tool’s or SDK’s documentation. See also Environment variables and fields for client unified authentication and the Default order of evaluation for client unified authentication methods and credentials.

For account-level operations, set the following environment variables:

  • DATABRICKS_HOST, set to the value of your Azure Databricks account console URL, https://accounts.azuredatabricks.net.
  • DATABRICKS_ACCOUNT_ID
  • ARM_CLIENT_ID
  • ARM_USE_MSI, set to true.

For workspace-level operations, set the following environment variables:

  • DATABRICKS_HOST, set to the value of your Azure Databricks per-workspace URL, for example https://adb-1234567890123456.7.azuredatabricks.net.
  • ARM_CLIENT_ID
  • ARM_USE_MSI, set to true.

For workspace-level operations, if the target identity has not already been added to the workspace, then specify DATABRICKS_AZURE_RESOURCE_ID along with the Azure resource ID for the Azure Databricks workspace, instead of DATABRICKS_HOST along with the workspace URL. In this case, the target identity must have at least Contributor or Owner permissions on the Azure resource for the Azure Databricks workspace.

Profile

Create or identify an Azure Databricks configuration profile with the following fields in your .databrickscfg file. If you create the profile, replace the placeholders with the appropriate values. To use the profile with a tool or SDK, see Supported authentication types by Azure Databricks tool or SDK or the tool’s or SDK’s documentation. See also Environment variables and fields for client unified authentication and the Default order of evaluation for client unified authentication methods and credentials.

For account-level operations, set the following values in your .databrickscfg file. In this case, the Azure Databricks account console URL is https://accounts.azuredatabricks.net:

[<some-unique-configuration-profile-name>]
host            = <account-console-url>
account_id      = <account-id>
azure_client_id = <azure-managed-identity-application-id>
azure_use_msi   = true

For workspace-level operations, set the following values in your .databrickscfg file. In this case, the host is the Azure Databricks per-workspace URL, for example https://adb-1234567890123456.7.azuredatabricks.net:

[<some-unique-configuration-profile-name>]
host            = <workspace-url>
azure_client_id = <azure-managed-identity-application-id>
azure_use_msi   = true

For workspace-level operations, if the target identity has not already been added to the workspace, then specify azure_workspace_resource_id along with the Azure resource ID for the Azure Databricks workspace, instead of host along with the workspace URL. In this case, the target identity must have at least Contributor or Owner permissions on the Azure resource for the Azure Databricks workspace.

Cli

For the Databricks CLI, do one of the following:

  • Set the environment variables as specified in this article’s “Environment” section.
  • Set the values in your .databrickscfg file as specified in this article’s “Profile” section.

Environment variables always take precedence over values in your .databrickscfg file.

See also Azure managed identities authentication.

Connect

Note

Databricks Connect relies on the Databricks SDK for Python for authentication. The Databricks SDK for Python has not yet implemented Azure managed identities authentication.

Vs code

Note

The Databricks extension for Visual Studio Code does not yet support Azure managed identities authentication.

Terraform

For account-level operations, for default authentication:

provider "databricks" {
  alias = "accounts"
}

For direct configuration (replace the retrieve placeholders with your own implementation to retrieve the values from the console or some other configuration store, such as HashiCorp Vault. See also Vault Provider). In this case, the Azure Databricks account console URL is https://accounts.azuredatabricks.net:

provider "databricks" {
  alias           = "accounts"
  host            = <retrieve-account-console-url>
  account_id      = <retrieve-account-id>
  azure_client_id = <retrieve-azure-client-id>
  azure_use_msi   = true
}

For workspace-level operations, for default authentication:

provider "databricks" {
  alias = "workspace"
}

For direct configuration (replace the retrieve placeholders with your own implementation to retrieve the values from the console or some other configuration store, such as HashiCorp Vault. See also Vault Provider). In this case, the host is the Azure Databricks per-workspace URL, for example https://adb-1234567890123456.7.azuredatabricks.net:

provider "databricks" {
  alias           = "workspace"
  host            = <retrieve-workspace-url>
  azure_client_id = <retrieve-azure-client-id>
  azure_use_msi   = true
}

For workspace-level operations, if the target identity has not already been added to the workspace, then specify azure_workspace_resource_id along with the Azure resource ID for the Azure Databricks workspace, instead of host along with the workspace URL. In this case, the target identity must have at least Contributor or Owner permissions on the Azure resource for the Azure Databricks workspace.

For more information about authenticating with the Databricks Terraform provider, see Authentication.

Python

Note

The Databricks SDK for Python has not yet implemented Azure managed identities authentication.

Java

Note

The Databricks SDK for Java has not yet implemented Azure managed identities authentication.

Go

For account-level operations, for default authentication:

import (
  "github.com/databricks/databricks-sdk-go"
)
// ...
a := databricks.Must(databricks.NewAccountClient())
// ...

For direct configuration (replace the retrieve placeholders with your own implementation to retrieve the values from the console or some other configuration store, such as Azure KeyVault). In this case, the Azure Databricks account console URL is https://accounts.azuredatabricks.net:

import (
  "github.com/databricks/databricks-sdk-go"
)
// ...
a := databricks.Must(databricks.NewAccountClient(&databricks.Config{
  Host:          retrieveAccountConsoleUrl(),
  AccountId:     retrieveAccountId(),
  AzureClientId: retrieveAzureClientId(),
  AzureUseMSI:   true,
}))
// ...

For workspace-level operations, for default authentication:

import (
  "github.com/databricks/databricks-sdk-go"
)
// ...
w := databricks.Must(databricks.NewWorkspaceClient())
// ...

For direct configuration (replace the retrieve placeholders with your own implementation to retrieve the values from the console or some other configuration store, such as Azure KeyVault). In this case, the host is the Azure Databricks per-workspace URL, for example https://adb-1234567890123456.7.azuredatabricks.net:

import (
  "github.com/databricks/databricks-sdk-go"
)
// ...
w := databricks.Must(databricks.NewWorkspaceClient(&databricks.Config{
  Host:          retrieveWorkspaceUrl(),
  AzureClientId: retrieveAzureClientId(),
  AzureUseMSI:   true,
}))
// ...

For workspace-level operations, if the target identity has not already been added to the workspace, then specify AzureResourceID along with the Azure resource ID for the Azure Databricks workspace, instead of Host along with the workspace URL. In this case, the target identity must have at least Contributor or Owner permissions on the Azure resource for the Azure Databricks workspace.

For more information about authenticating with Databricks tools and SDKs that use Go and that implement Databricks client unified authentication, see Authenticate the Databricks SDK for Go with your Azure Databricks account or workspace.