Manage service principals

This article explains how to create and manage service principals for your Azure Databricks account and workspaces.

For an overview of the Azure Databricks identity model, see Azure Databricks identities.

To manage access for service principals, see Authentication and access control.

What is a service principal?

A service principal is an identity that you create in Azure Databricks for use with automated tools, jobs, and applications. Service principals give automated tools and scripts API-only access to Azure Databricks resources, providing greater security than using users or groups.

You can grant and restrict a service principal’s access to resources in the same way as you can an Azure Databricks user. For example, you can do the following:

• Give a service principal account admin and workspace admin roles.
• Give a service principal access to data, either at the account level using Unity Catalog, or at the workspace level.
• Add a service principal to a group at both the account and workspace level, including the workspace admins group.

You can also grant Azure Databricks users, service principals, and groups permissions to use a service principal. This allows users to run jobs as the service principal, instead of as their identity. This prevents jobs from failing if a user leaves your organization or a group is modified.

Unlike an Azure Databricks user, a service principal is an API-only identity; it cannot be used to access the Azure Databricks UI.

Databricks recommends that you enable your workspaces for identity federation. Identity federation enables you to configure service principals in the account console, and then assign them access to specific workspaces. This simplifies Azure Databricks administration and data governance.

Important

Databricks began to enable new workspaces for identity federation and Unity Catalog automatically on November 9, 2023, with a rollout proceeding gradually across accounts. If your workspace is enabled for identity federation by default, it cannot be disabled. For more information, see Automatic enablement of Unity Catalog.

Databricks and Microsoft Entra ID service principals

Service principals can either be Azure Databricks managed service principals or Microsoft Entra ID managed service principals.

Azure Databricks managed service principals can authenticate to Azure Databricks using Databricks OAuth authentication and personal access tokens. Microsoft Entra ID managed service principals can authenticate to Azure Databricks using Databricks OAuth authentication and Microsoft Entra ID tokens. For more information on authentication for service principals, see Manage tokens for a service principal.

Azure Databricks managed service principals are managed directly within Azure Databricks. Microsoft Entra ID managed service principals are managed in Microsoft Entra ID, which requires additional permissions. Databricks recommends that you use Azure Databricks managed service principals for Azure Databricks automation and that you use Microsoft Entra ID managed service principals in cases where you must authenticate with Azure Databricks and other Azure resources at the same time.

To create a Azure Databricks managed service principal, skip this section and continue reading with Who can manage and use service principals?.

To use Microsoft Entra ID managed service principals in Azure Databricks, an admin user must create a Microsoft Entra ID application in Azure. To create a Microsoft Entra ID managed service principal, follow these instructions:

1. Sign in to the Azure portal.

   Note

   The portal to use is different depending on whether your Microsoft Entra ID application runs in the Azure public cloud or in a national or sovereign cloud. For more information, see National clouds.

2. If you have access to multiple tenants, subscriptions, or directories, click the Directories + subscriptions (directory with filter) icon in the top menu to switch to the directory in which you want to provision the service principal.

3. In Search resources, services, and docs, search for and select Microsoft Entra ID.

4. Click + Add and select App registration.

5. For Name, enter a name for the application.

6. In the Supported account types section, select Accounts in this organizational directory only (Single tenant).

7. Click Register.

8. On the application page’s Overview page, in the Essentials section, copy the following values:

   • Application (client) ID
   • Directory (tenant) ID

   

9. To generate a client secret, within Manage, click Certificates & secrets.

   Note

   You use this client secret to generate Microsoft Entra ID tokens for authenticating Microsoft Entra ID service principals with Azure Databricks. To determine whether an Azure Databricks tool or SDK can use Microsoft Entra ID tokens, see the tool’s or SDK’s documentation.

10. On the Client secrets tab, click New client secret.

    

11. In the Add a client secret pane, for Description, enter a description for the client secret.

12. For Expires, select an expiry time period for the client secret, and then click Add.

13. Copy and store the client secret’s Value in a secure place, as this client secret is the password for your application.

Who can manage and use service principals?

To manage service principals in Azure Databricks, you must have one of the following: the account admin role, the workspace admin role, or the manager or user role on a service principal.

• Account admins can add service principals to the account and assign them admin roles. They can also assign service principals to workspaces, as long as those workspaces use identity federation.
• Workspace admins can add service principals to an Azure Databricks workspace, assign them the workspace admin role, and manage access to objects and functionality in the workspace, such as the ability to create clusters or access specified persona-based environments.
• Service Principal Managers can manage roles on a service principal. The creator of a service principal becomes the service principal manager. Account admins are service principal managers on all service principals in an account.

Note

If a service principal was created before June 13, 2023, then the creator of the service principal does not have the service principal manager role by default. Ask an account admin to grant you the service principal manager role.

Users with the Service Principal Manager role do not inherit the Service Principal User role. If you want to use the service principal to execute jobs, you need to explicitly assign yourself the service principal user role, even after creating the service principal.

• Service Principal Users can run jobs as the service principal. The job runs using the identity of the service principal, instead of the identity of the job owner. For more information, see Manage identities, permissions, and privileges for Databricks Jobs.

For information on how to grant the service principal manager and user roles, see Roles for managing service principals.

Manage service principals in your account

Account admins can add service principals to your Azure Databricks account using the account console.

Add service principals to your account using the account console

Service principals can either be created in Azure Databricks or linked from an existing Microsoft Entra ID service principal. See Databricks and Microsoft Entra ID service principals.

1. As an account admin, log in to the account console.
2. In the sidebar, click User management.
3. On the Service principals tab, click Add service principal.
4. Under Management, choose Databricks managed or Microsoft Entra ID managed.
5. If you chose Microsoft Entra ID managed, under Microsoft Entra application ID, paste the application (client) ID for the service principal.
6. Enter a name for the service principal.
7. Click Add.

Assign account admin roles to a service principal

1. As an account admin, log in to the account console.
2. In the sidebar, click User management.
3. On the Service principals tab, find and click the username.
4. On the Roles tab, turn on Account admin or Marketplace admin.

Assign a service principal to a workspace using the account console

To add users to a workspace using the account console, the workspace must be enabled for identity federation. Workspace admins can also assign service principals to workspaces using the workspace admin settings page. For details, see Add a service principal to a workspace using the workspace admin settings.

1. As an account admin, log in to the account console.
2. In the sidebar, click Workspaces.
3. Click your workspace name.
4. On the Permissions tab, click Add permissions.
5. Search for and select the service principal, assign the permission level (workspace User or Admin), and click Save.

Remove a service principal from a workspace using the account console

To remove service principals from a workspace using the account console, the workspace must be enabled for identity federation. When a service principal is removed from a workspace, the service principal can no longer access the workspace, however permissions are maintained on the service principal. If the service principal is later added back to the workspace, it regains its previous permissions.

1. As an account admin, log in to the account console
2. In the sidebar, click Workspaces.
3. Click your workspace name.
4. On the Permissions tab, find the service principal.
5. Click the kebab menu at the far right of the service principal row and select Remove.
6. On the confirmation dialog, click Remove.

Deactivate a service principal in your Azure Databricks account

Account admins can deactivate service principals across an Azure Databricks account. A deactivated service principal cannot authenticate to the Azure Databricks account or workspaces. However, all of the service principal’s permissions and workspace objects remain unchanged. When a service principal is deactivated the following is true:

• The service principal cannot authenticate to the account or any of their workspaces from any method.
• Applications or scripts that use the tokens generated by the service principal are no longer able to access the Databricks API. The tokens remain but cannot be used to authenticate while a service principal is deactivated.
• Clusters owned by the service principal remain running.
• Scheduled jobs created by the service principal fail unless they are assigned to a new owner.

When a service principal is reactivated, it can login to Azure Databricks with the same permissions. Databricks recommends deactivating service principals from the account instead of removing them because removing a service principal is a destructive action. A deactivated service principal’s status is labeled Inactive in the account console. You can also deactivate a service principal from a specific workspace. See Deactivate a service principal in your Azure Databricks workspace.

You cannot deactivate a service principal using the account console. Instead, use the Account Service Principals API.

For example:

curl --netrc -X PATCH \
https://${DATABRICKS_HOST}/api/2.1/accounts/{account_id}/scim/v2/ServicePrincipals/{id} \
--header 'Content-type: application/scim+json' \
--data @update-sp.json \
| jq .

update-sp.json:

{
  "schemas": [ "urn:ietf:params:scim:api:messages:2.0:PatchOp" ],
  "Operations": [
    {
      "op": "replace",
      "path": "active",
      "value": [
        {
          "value": "false"
        }
      ]
    }
  ]
}

Remove service principals from your Azure Databricks account

Account admins can delete service principals from an Azure Databricks account. Workspace admins cannot. When you delete a service principal from the account, that principal is also removed from their workspaces.

Important

When you remove a service principal from the account, that service principal is also removed from their workspaces, regardless of whether or not identity federation has been enabled. We recommend that you refrain from deleting account-level service principals unless you want them to lose access to all workspaces in the account. Be aware of the following consequences of deleting service principals:

• Applications or scripts that use the tokens generated by the service principal can no longer access Databricks APIs
• Jobs owned by the service principal fail
• Clusters owned by the service principal stop
• Queries or dashboards created by the service principal and shared using the Run as Owner credential have to be assigned to a new owner to prevent sharing from failing

When a Microsoft Entra ID service principal is removed from an account, the service principal can no longer access the account or its workspaces, however permissions are maintained on the service principal. If the service principal is later added back to the account, it regains its previous permissions.

To remove a service principal using the account console, do the following:

1. As an account admin, log in to the account console.
2. In the sidebar, click User management.
3. On the Service principals tab, find and click the username.
4. On the Principal Information tab, click the kebab menu in the upper-right corner and select Delete.
5. In the confirmation dialog box, click Confirm delete.

Manage service principals in your workspace

Workspace admins can manage service principals in their workspaces using the workspace admin settings page.

Add a service principal to a workspace using the workspace admin settings

Service principals can either be created in Azure Databricks or linked from an existing Microsoft Entra ID service principal. See Databricks and Microsoft Entra ID service principals.

1. As a workspace admin, log in to the Azure Databricks workspace.

2. Click your username in the top bar of the Azure Databricks workspace and select Settings.

3. Click on the Identity and access tab.

4. Next to Service principals, click Manage.

5. Click Add service principal.

6. Select an existing service principal to assign to the workspace or click Add new to create a new one.

   To add a new service principal, choose Databricks managed or Microsoft Entra ID managed. If you choose Microsoft Entra ID managed, paste the Application (client) ID for the service principal and enter a display name.

7. Click Add.

Note

If your workspace is not enabled for identity federation, you cannot assign existing account service principals to your workspace.

Assign the workspace admin role to a service principal using the workspace admin settings page

1. As a workspace admin, log in to the Azure Databricks workspace.
2. Click your username in the top bar of the Azure Databricks workspace and select Settings.
3. Click on the Identity and access tab.
4. Next to Groups, click Manage.
5. Select the admins system group.
6. Click Add members.
7. Select the service principal and click Confirm.

To remove the workspace admin role from a service principal, remove the service principal from the admin group.

Deactivate a service principal in your Azure Databricks workspace

Workspace admins can deactivate service principals in a Azure Databricks workspace. A deactivated service principal cannot access the workspace from Azure Databricks APIs, however all of the service principal’s permissions and workspace objects remain unchanged. When a service principal is deactivated:

• The service principal cannot authenticate to the workspaces from any method.
• The service principal’s status shows as Inactive in the workspace admin setting page.
• Applications or scripts that use the tokens generated by the service principal can no longer access the Databricks API. The tokens remain but cannot be used to authenticate while a service principal is deactivated.
• Clusters owned by the service principal remain running.
• Scheduled jobs created by the service principal have to be assigned to a new owner to prevent them from failing.

When a service principal is reactivated, it can authenticate to the workspace with the same permissions. Databricks recommends deactivating service principals instead of removing them because removing a service principal is a destructive action.

1. As a workspace admin, log in to the Azure Databricks workspace.
2. Click your username in the top bar of the Azure Databricks workspace and select Settings.
3. Click on the Identity and access tab.
4. Next to Service principals, click Manage.
5. Select the service principal you want to deactivate.
6. Under Status, uncheck Active.

To set a service principal to active, perform the same steps, but check the checkbox instead.

Remove a service principal from a workspace using the workspace admin settings page

Removing a service principal from a workspace does not remove the service principal from the account. To remove a service principal from your account, see Remove service principals from your Azure Databricks account.

When a service principal is removed from a workspace, the service principal can no longer access the workspace, however permissions are maintained on the service principal. If the service principal is later added back to a workspace, it regains its previous permissions.

1. As a workspace admin, log in to the Azure Databricks workspace.
2. Click your username in the top bar of the Azure Databricks workspace and select Settings.
3. Click on the Identity and access tab.
4. Next to Service principals, click Manage.
5. Select the service principal.
6. In the upper-right corner, click Delete.
7. Click Delete to confirm.

Manage service principals using the API

Account admins and workspace admins can manage service principals in the Azure Databricks account and workspaces using Databricks APIs. To manage roles on a service principal using the API, see Manage service principal roles using the Databricks CLI.

Manage service principals in the account using the API

Admins can add and manage service principals in the Azure Databricks account using the Account Service Principals API. Account admins and workspace admins invoke the API using a different endpoint URL:

• Account admins use {account-domain}/api/2.1/accounts/{account_id}/scim/v2/.
• Workspace admins use {workspace-domain}/api/2.0/account/scim/v2/.

For details, see the Account Service Principals API.

Manage service principals in the workspace using the API

Account and workspace admins can use the Workspace Assignment API to assign service principals to workspaces enabled for identity federation. The Workspace Assignment API is supported through the Azure Databricks account and workspaces.

• Account admins use {account-domain}/api/2.0/accounts/{account_id}/workspaces/{workspace_id}/permissionassignments.
• Workspace admins use {workspace-domain}/api/2.0/preview/permissionassignments/principals/{principal_id}.

See Workspace Assignment API.

If your workspace is not enabled for identity federation, a workspace admin can use the workspace-level APIs to assign service principals to their workspaces. See Workspace Service Principals API.

Manage tokens for a service principal

Service principals can authenticate to APIs on Azure Databricks by using Azure Databricks OAuth tokens or Azure Databricks personal access tokens, as follows:

• Azure Databricks OAuth tokens can be used to authenticate to Azure Databricks account-level and workspace-level APIs.
  • Azure Databricks OAuth tokens that are created at the Azure Databricks account level can be used to authenticate to Azure Databricks account-level and workspace-level APIs.
  • Azure Databricks OAuth tokens that are created at the Azure Databricks workspace level can be used to authenticate only to Azure Databricks workspace-level APIs.
• Azure Databricks personal access tokens can be used to authenticate only to Azure Databricks workspace-level APIs.

Service principals can also authenticate to APIs on Azure Databricks by using Microsoft Entra ID tokens.

Manage Databricks OAuth authentication for a service principal

To authenticate to account-level and workspace-level Databricks REST APIs, account admins can use Azure Databricks OAuth tokens for service principals. You can request an OAuth token using the client ID and a client secret for the service principal. For more information, see Authenticate access to Azure Databricks with a service principal using OAuth (OAuth M2M).

Manage personal access tokens for a service principal

To authenticate a service principal to workspace-level APIs on Azure Databricks only, a service principal can create Databricks personal access tokens for itself, as follows:

This procedure assumes that you are using OAuth machine-to-machine (M2M) authentication or Microsoft Entra ID service principal authentication to set up the Databricks CLI for authenticating the service principal to generate Azure Databricks personal access tokens for itself. See OAuth machine-to-machine (M2M) authentication or Microsoft Entra ID service principal authentication.

1. Use the Databricks CLI to run the following command, which generates another access token for the service principal.

   Run the following command:

   databricks tokens create --comment <comment> --lifetime-seconds <lifetime-seconds> -p <profile-name>
   

   • --comment: Replace <comment> with a meaningful comment about the access token’s purpose. If the --comment option is not specified, then no comment is generated.
   • --lifetime-seconds: Replace <lifetime-seconds> with the number of seconds that the access token is valid for. For example, 1 day is 86400 seconds. If the --lifetime-seconds option is not specified, the access token is set to never expire (not recommended).
   • --profile-name: Replace <profile-name> with the name of an Azure Databricks configuration profile that contains authentication information for the service principal and the target workspace. If the -p option is not specified, the Databricks CLI will attempt to find and use a configuration profile named DEFAULT.

2. In the response, copy the value of token_value, which is the access token for the service principal.

   Be sure to save the copied token in a secure location. Do not share your copied token with others. If you lose the copied token, you cannot regenerate that exact same token. Instead, you must repeat this procedure to create a new token.

   If you are not able to create or use tokens in your workspace, this might be because your workspace administrator has disabled tokens or has not given you permission to create or use tokens. See your workspace administrator or the following:

   • Enable or disable personal access token authentication for the workspace
   • Personal access token permissions

Manage Microsoft Entra ID authentication for a service principal

Only Microsoft Entra ID managed service principals can authenticate to APIs using Microsoft Entra ID tokens. To create a Microsoft Entra ID access token, see Get Microsoft Entra ID tokens for service principals.

The Microsoft Entra ID access token can be used to call the Tokens API to create a Databricks personal access token for the service principal.