Sync users and groups from Microsoft Entra ID
This article describes how to configure your identity provider (IdP) and Azure Databricks to provision users and groups to Azure Databricks using SCIM, or System for Cross-domain Identity Management, an open standard that allows you to automate user provisioning.
About SCIM provisioning in Azure Databricks
SCIM lets you use an identity provider (IdP) to create users in Azure Databricks, give them the proper level of access, and remove access (deprovision them) when they leave your organization or no longer need access to Azure Databricks.
You can use a SCIM provisioning connector in your IdP or invoke the SCIM Groups API to manage provisioning. You can also use these APIs to manage identities in Azure Databricks directly, without an IdP.
Account-level and workspace-level SCIM provisioning
You can either configure one SCIM provisioning connector from Microsoft Entra ID to your Azure Databricks account, using account-level SCIM provisioning, or configure separate SCIM provisioning connectors to each workspace, using workspace-level SCIM provisioning.
- Account-level SCIM provisioning: Databricks recommends that you use account-level SCIM provisioning to create, update, and delete all users from the account. You manage the assignment of users and groups to workspaces within Azure Databricks. Your workspaces must be enabled for identity federation to manage users’ workspace assignments.
Workspace-level SCIM provisioning (legacy and Public Preview): For workspaces that are not enabled for identity federation, you must manage account-level and workspace-level SCIM provisioning in parallel. You don’t need workspace-level SCIM provisioning for any workspaces that are enabled for identity federation.
If you already have workspace-level SCIM provisioning set up for a workspace, Databricks recommends that you enable the workspace for identity federation, set up account-level SCIM provisioning, and turn off the workspace-level SCIM provisioner. See Migrate workspace-level SCIM provisioning to the account level.
Requirements
To provision users and groups to Azure Databricks using SCIM:
- Your Azure Databricks account must have the Premium plan.
- To provision users to your Azure Databricks account using SCIM (including the SCIM REST APIs), you must be an Azure Databricks account admin.
- To provision users to an Azure Databricks workspace using SCIM (including the SCIM REST APIs), you must be an Azure Databricks workspace admin.
For more information about admin privileges, see Manage users, service principals, and groups.
You can have a maximum of 10,000 combined users and service principals and 5000 groups in an account. Each workspace can have a maximum of 10,000 combined users and service principals and 5000 groups.
Provision identities to your Azure Databricks account
You can use SCIM to provision users and groups from Microsoft Entra ID to your Azure Databricks account using a SCIM provisioning connector or directly using the SCIM APIs.
Add users and groups to your Azure Databricks account using Microsoft Entra ID (formerly Azure Active Directory)
You can sync account-level identities from your Microsoft Entra ID tenant to Azure Databricks using a SCIM provisioning connector.
Important
If you already have SCIM connectors that sync identities directly to your workspaces, you must disable those SCIM connectors when the account-level SCIM connector is enabled. See Migrate workspace-level SCIM provisioning to the account level.
For complete instructions, see Provision identities to your Azure Databricks account using Microsoft Entra ID.
Note
When you remove a user from the account-level SCIM connector, that user is deactivated from the account and all of their workspaces, regardless of whether or not identity federation has been enabled. When you remove a group from the account-level SCIM connector, all users in that group are deactivated from the account and from any workspaces they had access to, (unless they are members of another group or have been directly granted access to the account-level SCIM connector).
Add users, service principals, and groups to your account using the SCIM API
Account admins can add users, service principals, and groups to the Azure Databricks account using the Account SCIM API. Account admins call the API on accounts.azuredatabricks.net ({account_domain}/api/2.0/accounts/{account_id}/scim/v2/) and can use either a SCIM token or a Microsoft Entra ID token to authenticate.
Note
The SCIM token is restricted to the Account SCIM API /api/2.0/accounts/{account_id}/scim/v2/ and cannot be used to authenticate to other Databricks REST APIs.
To get the SCIM token, do the following:
As an account admin, log in to the account console.
In the sidebar, click Settings.
Click User Provisioning.
If provisioning isn’t enabled, click Enable user provisioning and copy the token.
If provisioning is already enabled, click Regenerate token and copy the token.
To use a Microsoft Entra ID token to authenticate, see Microsoft Entra ID service principal authentication.
Workspace admins can add users and service principals using the same API. Workspace admins call the API on the workspace domain {workspace-domain}/api/2.0/account/scim/v2/.
Rotate the account-level SCIM token
If the account-level SCIM token is compromised or if you have business requirements to rotate authentication tokens periodically, you can rotate the SCIM token.
- As an Azure Databricks account admin, log in to the account console.
- In the sidebar, click Settings.
- Click User Provisioning.
- Click Regenerate token. Make a note of the new token. The previous token will continue to work for 24 hours.
- Within 24 hours, update your SCIM application to use the new SCIM token.
Migrate workspace-level SCIM provisioning to the account level
If you are enabling account-level SCIM provisioning and you already have workspace-level SCIM provisioning set up for some workspaces, Databricks recommends that you turn off the workspace-level SCIM provisioner and instead sync users and group to the account level.
Create a group in Microsoft Entra ID that includes all of the users and groups that you are currently provisioning to Azure Databricks using your workspace-level SCIM connectors.
Databricks recommends that this group include all users in all workspaces in your account.
Configure a new SCIM provisioning connector to provision users and groups to your account, using the instructions in Provision identities to your Azure Databricks account.
Use the group or groups that you created in step 1. If you add a user that shares a username (email address) with an existing account user, those users are merged. Existing groups in the account are not affected.
Confirm that the new SCIM provisioning connector is successfully provisioning users and groups to your account.
Shut down the old workspace-level SCIM connectors that were provisioning users and groups to your workspaces.
Do not remove users and groups from the workspace-level SCIM connectors before shutting them down. Revoking access from a SCIM connector deactivates the user in the Azure Databricks workspace. For more information, see Deactivate a user in your Azure Databricks workspace.
Migrate workspace-local groups to account groups.
If you have legacy groups in your workspaces, they are known as workspace-local groups. You cannot manage workspace-local groups using account-level interfaces. Databricks recommends that you convert them to account groups. See Migrate workspace-local groups to account groups
Provision identities to an Azure Databricks workspace (legacy)
Important
This feature is in Public Preview.
If you want to use an IdP connector to provision users and groups and you have a workspace that is not identity federated, you must configure SCIM provisioning at the workspace level.
Note
Workspace-level SCIM does not recognize account groups that are assigned to your identity federated workspace and workspace-level SCIM API calls will fail if they involve account groups. If your workspace is enabled for identity federation, Databricks recommends that you use the account-level SCIM API instead of the workspace-level SCIM API and that you set up account-level SCIM provisioning and turn off the workspace-level SCIM provisioner. For detailed instructions, see Migrate workspace-level SCIM provisioning to the account level.
Add users and groups to your workspace using an IdP provisioning connector
Follow the instructions in the appropriate IdP-specific article:
Add users, groups, and service principals to your workspace using the SCIM API
Workspace admins can add users, groups, and service principals to the Azure Databricks account using workspace-level SCIM APIs. See Workspace Users API, Workspace Groups API, and Workspace Service Principals API
Feedback
Coming soon: Throughout 2024 we will be phasing out GitHub Issues as the feedback mechanism for content and replacing it with a new feedback system. For more information see: https://aka.ms/ContentUserFeedback.
Submit and view feedback for