Share via

Microsoft Entra ID service principal authentication

  • Article

Microsoft Entra ID service principal authentication uses the credentials of a Microsoft Entra ID service principal to authenticate. To create and manage service principals for Azure Databricks, see:

Note

Databricks recommends that you use OAuth machine-to-machine (M2M) authentication in most scenarios instead of Microsoft Entra ID service principal authentication. This is because OAuth M2M authentication uses Azure Databricks OAuth access tokens that are more robust when authenticating only with Azure Databricks.

You should only use Microsoft Entra ID service principal authentication in cases where you need to authenticate with Azure Databricks and other Azure resources at the same time, which requires Microsoft Entra ID tokens.

To use OAuth M2M authentication instead of Microsoft Entra ID service principal authentication, skip this article and see OAuth machine-to-machine (M2M) authentication.

Microsoft Entra ID service principals are different than managed identities for Azure resources, which Azure Databricks also supports for authentication. To learn how to use managed identities for Azure resources instead of Microsoft Entra ID service principals for Azure Databricks authentication, see Set up and use Azure managed identities authentication for Azure Databricks automation.

To configure Microsoft Entra ID service principal authentication with Azure Databricks, you must set the following associated environment variables, .databrickscfg fields, Terraform fields, or Config fields:

  • The Azure Databricks host.

    • For account operations, specify https://accounts.azuredatabricks.net.

    • For workspace operations, specify the per-workspace URL, for example https://adb-1234567890123456.7.azuredatabricks.net.

      If the Microsoft Entra ID service principal has not already been added to the workspace, then specify the Azure resource ID instead. In this case, the Microsoft Entra ID service principal must have at least Contributor or Owner permissions on the Azure resource.

  • For account operations, the Azure Databricks account ID.

  • The Azure resource ID.

  • The tenant ID of the Microsoft Entra ID service principal.

  • The client ID of the Microsoft Entra ID service principal.

  • The client secret of the Microsoft Entra ID service principal.

To perform Microsoft Entra ID service principal authentication with Azure Databricks, integrate the following within your code, based on the participating tool or SDK:

Environment

To use environment variables for a specific Azure Databricks authentication type with a tool or SDK, see Supported authentication types by Azure Databricks tool or SDK or the tool’s or SDK’s documentation. See also Environment variables and fields for client unified authentication and the Default order of evaluation for client unified authentication methods and credentials.

For account-level operations, set the following environment variables:

  • DATABRICKS_HOST, set to the value of your Azure Databricks account console URL, https://accounts.azuredatabricks.net.
  • DATABRICKS_ACCOUNT_ID
  • ARM_TENANT_ID
  • ARM_CLIENT_ID
  • ARM_CLIENT_SECRET

For workspace-level operations, set the following environment variables:

  • DATABRICKS_HOST, set to the value of your Azure Databricks per-workspace URL, for example https://adb-1234567890123456.7.azuredatabricks.net.
  • ARM_TENANT_ID
  • ARM_CLIENT_ID
  • ARM_CLIENT_SECRET

For workspace-level operations, if the Microsoft Entra ID service principal has not already been added to the workspace, then specify DATABRICKS_AZURE_RESOURCE_ID along with the Azure resource ID for the Azure Databricks workspace, instead of HOST along with the workspace URL. In this case, the Microsoft Entra ID service principal must have at least Contributor or Owner permissions on the Azure resource for the Azure Databricks workspace.

Profile

Create or identify an Azure Databricks configuration profile with the following fields in your .databrickscfg file. If you create the profile, replace the placeholders with the appropriate values. To use the profile with a tool or SDK, see Supported authentication types by Azure Databricks tool or SDK or the tool’s or SDK’s documentation. See also Environment variables and fields for client unified authentication and the Default order of evaluation for client unified authentication methods and credentials.

For account-level operations, set the following values in your .databrickscfg file. In this case, the Azure Databricks account console URL is https://accounts.azuredatabricks.net:

[<some-unique-configuration-profile-name>]
host                = <account-console-url>
account_id          = <account-id>
azure_tenant_id     = <azure-service-principal-tenant-id>
azure_client_id     = <azure-service-principal-application-id>
azure_client_secret = <azure-service-principal-client-secret>

For workspace-level operations, set the following values in your .databrickscfg file. In this case, the host is the Azure Databricks per-workspace URL, for example https://adb-1234567890123456.7.azuredatabricks.net:

[<some-unique-configuration-profile-name>]
host                = <workspace-url>
azure_tenant_id     = <azure-service-principal-tenant-id>
azure_client_id     = <azure-service-principal-application-id>
azure_client_secret = <azure-service-principal-client-secret>

For workspace-level operations, if the Microsoft Entra ID service principal has not already been added to the workspace, then specify azure_workspace_resource_id along with the Azure resource ID for the Azure Databricks workspace, instead of host along with the workspace URL. In this case, the Microsoft Entra ID service principal must have at least Contributor or Owner permissions on the Azure resource for the Azure Databricks workspace.

Cli

For the Databricks CLI, do one of the following:

  • Set the environment variables as specified in this article’s “Environment” section.
  • Set the values in your .databrickscfg file as specified in this article’s “Profile” section.

Environment variables always take precedence over values in your .databrickscfg file.

See also Microsoft Entra ID service principal authentication.

Connect

Note

Microsoft Entra ID service principal authentication is supported on the following Databricks Connect versions:

  • For Python, Databricks Connect for Databricks Runtime 13.1 and above.
  • For Scala, Databricks Connect for Databricks Runtime 13.3 LTS and above.

For Databricks Connect, you can do one of the following:

  • Set the values in your .databrickscfg file for Azure Databricks workspace-level operations as specified in this article’s “Profile” section. Also set the cluster_id environment variable in your profile to your per-workspace URL, for example https://adb-1234567890123456.7.azuredatabricks.net.
  • Set the environment variables for Azure Databricks workspace-level operations as specified in this article’s “Environment” section. Also set the DATABRICKS_CLUSTER_ID environment variable to your per-workspace URL, for example https://adb-1234567890123456.7.azuredatabricks.net.

Values in your .databrickscfg file always take precedence over environment variables.

To initialize the Databricks Connect client with these environment variables or values in your .databrickscfg file, see one of the following:

Vs code

For the Databricks extension for Visual Studio Code, do the following:

  1. Set the values in your .databrickscfg file for Azure Databricks workspace-level operations as specified in this article’s “Profile” section.
  2. In the Configuration pane of the Databricks extension for Visual Studio Code, click Configure Databricks.
  3. In the Command Palette, for Databricks Host, enter your per-workspace URL, for example https://adb-1234567890123456.7.azuredatabricks.net, and then press Enter.
  4. In the Command Palette, select your target profile’s name in the list for your URL.

For more details, see Authentication setup for the Databricks extension for VS Code.

Terraform

For account-level operations, for default authentication:

provider "databricks" {
  alias = "accounts"
}

For direct configuration (replace the retrieve placeholders with your own implementation to retrieve the values from the console or some other configuration store, such as HashiCorp Vault. See also Vault Provider). In this case, the Azure Databricks account console URL is https://accounts.azuredatabricks.net:

provider "databricks" {
  alias               = "accounts"
  host                = <retrieve-account-console-url>
  account_id          = <retrieve-account-id>
  azure_tenant_id     = <retrieve-azure-tenant-id>
  azure_client_id     = <retrieve-azure-client-id>
  azure_client_secret = <retrieve-azure-client-secret>
}

For workspace-level operations, for default authentication:

provider "databricks" {
  alias = "workspace"
}

For direct configuration (replace the retrieve placeholders with your own implementation to retrieve the values from the console or some other configuration store, such as HashiCorp Vault. See also Vault Provider). In this case, the host is the Azure Databricks per-workspace URL, for example https://adb-1234567890123456.7.azuredatabricks.net:

provider "databricks" {
  alias               = "workspace"
  host                = <retrieve-workspace-url>
  azure_tenant_id     = <retrieve-azure-tenant-id>
  azure_client_id     = <retrieve-azure-client-id>
  azure_client_secret = <retrieve-azure-client-secret>
}

For workspace-level operations, if the Microsoft Entra ID service principal has not already been added to the workspace, then specify azure_workspace_resource_id along with the Azure resource ID for the Azure Databricks workspace, instead of host along with the workspace URL. In this case, the Microsoft Entra ID service principal must have at least Contributor or Owner permissions on the Azure resource for the Azure Databricks workspace.

For more information about authenticating with the Databricks Terraform provider, see Authentication.

Python

For account-level operations, for default authentication:

from databricks.sdk import AccountClient

a = AccountClient()
# ...

For direct configuration (replace the retrieve placeholders with your own implementation to retrieve the values from the console or some other configuration store, such as Azure KeyVault). In this case, the Azure Databricks account console URL is https://accounts.azuredatabricks.net:

from databricks.sdk import AccountClient

a = AccountClient(
  host                = retrieve_account_console_url(),
  account_id          = retrieve_account_id(),
  azure_tenant_id     = retrieve_azure_tenant_id(),
  azure_client_id     = retrieve_azure_client_id(),
  azure_client_secret = retrieve_azure_client_secret()
)
# ...

For workspace-level operations, for default authentication:

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()
# ...

For direct configuration (replace the retrieve placeholders with your own implementation to retrieve the values from the console or some other configuration store, such as Azure KeyVault). In this case, the host is the Azure Databricks per-workspace URL, for example https://adb-1234567890123456.7.azuredatabricks.net:

from databricks.sdk import WorkspaceClient

w = WorkspaceClient(
  host                = retrieve_workspace_url(),
  azure_tenant_id     = retrieve_azure_tenant_id(),
  azure_client_id     = retrieve_azure_client_id(),
  azure_client_secret = retrieve_azure_client_secret()
)
# ...

For workspace-level operations, if the Microsoft Entra ID service principal has not already been added to the workspace, then specify azure_workspace_resource_id along with the Azure resource ID for the Azure Databricks workspace, instead of host along with the workspace URL. In this case, the Microsoft Entra ID service principal must have at least Contributor or Owner permissions on the Azure resource for the Azure Databricks workspace.

For more information about authenticating with Databricks tools and SDKs that use Python and that implement Databricks client unified authentication, see:

Java

For account-level operations, for default authentication:

import com.databricks.sdk.AccountClient;
// ...
AccountClient a = new AccountClient();
// ...

For direct configuration (replace the retrieve placeholders with your own implementation to retrieve the values from the console or some other configuration store, such as Azure KeyVault). In this case, the Azure Databricks account console URL is https://accounts.azuredatabricks.net:

import com.databricks.sdk.AccountClient;
import com.databricks.sdk.core.DatabricksConfig;
// ...
DatabricksConfig cfg = new DatabricksConfig()
  .setHost(retrieveAccountConsoleUrl())
  .setAccountId(retrieveAccountId())
  .setAzureTenantId(retrieveAzureTenantId())
  .setAzureClientId(retrieveAzureClientId())
  .setAzureClientSecret(retrieveAzureClientSecret())
AccountClient a = new AccountClient(cfg);
// ...

For workspace-level operations, for default authentication:

import com.databricks.sdk.WorkspaceClient;
// ...
WorkspaceClient w = new WorkspaceClient();
// ...

For direct configuration (replace the retrieve placeholders with your own implementation to retrieve the values from the console or some other configuration store, such as Azure KeyVault). In this case, the host is the Azure Databricks per-workspace URL, for example https://adb-1234567890123456.7.azuredatabricks.net:

import com.databricks.sdk.WorkspaceClient;
import com.databricks.sdk.core.DatabricksConfig;
// ...
DatabricksConfig cfg = new DatabricksConfig()
  .setHost(retrieveWorkspaceUrl())
  .setAzureTenantId(retrieveAzureTenantId())
  .setAzureClientId(retrieveAzureClientId())
  .setAzureClientSecret(retrieveAzureClientSecret())
WorkspaceClient w = new WorkspaceClient(cfg);
// ...

For workspace-level operations, if the Microsoft Entra ID service principal has not already been added to the workspace, then specify setAzureWorkspaceResourceId along with the Azure resource ID for the Azure Databricks workspace, instead of setHost along with the workspace URL. In this case, the Microsoft Entra ID service principal must have at least Contributor or Owner permissions on the Azure resource for the Azure Databricks workspace.

For more information about authenticating with Databricks tools and SDKs that use Java and that implement Databricks client unified authentication, see:

Go

For account-level operations, for default authentication:

import (
  "github.com/databricks/databricks-sdk-go"
)
// ...
a := databricks.Must(databricks.NewAccountClient())
// ...

For direct configuration (replace the retrieve placeholders with your own implementation to retrieve the values from the console or some other configuration store, such as Azure KeyVault). In this case, the Azure Databricks account console URL is https://accounts.azuredatabricks.net:

import (
  "github.com/databricks/databricks-sdk-go"
)
// ...
a := databricks.Must(databricks.NewAccountClient(&databricks.Config{
  Host:              retrieveAccountConsoleUrl(),
  AccountId:         retrieveAccountId(),
  AzureTenantId:     retrieveAzureTenantId(),
  AzureClientId:     retrieveAzureClientId(),
  AzureClientSecret: retrieveAzureClientSecret(),
}))
// ...

For workspace-level operations, for default authentication:

import (
  "github.com/databricks/databricks-sdk-go"
)
// ...
w := databricks.Must(databricks.NewWorkspaceClient())
// ...

For direct configuration (replace the retrieve placeholders with your own implementation to retrieve the values from the console or some other configuration store, such as Azure KeyVault). In this case, the host is the Azure Databricks per-workspace URL, for example https://adb-1234567890123456.7.azuredatabricks.net:

import (
  "github.com/databricks/databricks-sdk-go"
)
// ...
w := databricks.Must(databricks.NewWorkspaceClient(&databricks.Config{
  Host:              retrieveWorkspaceUrl(),
  AzureTenantId:     retrieveAzureTenantId(),
  AzureClientId:     retrieveAzureClientId(),
  AzureClientSecret: retrieveAzureClientSecret(),
}))
// ...

For workspace-level operations, if the Microsoft Entra ID service principal has not already been added to the workspace, then specify AzureWorkspaceResourceId along with the Azure resource ID for the Azure Databricks workspace, instead of Host along with the workspace URL. In this case, the Microsoft Entra ID service principal must have at least Contributor or Owner permissions on the Azure resource for the Azure Databricks workspace.

For more information about authenticating with Databricks tools and SDKs that use Go and that implement Databricks client unified authentication, see Authenticate the Databricks SDK for Go with your Azure Databricks account or workspace.