Sync users and groups from Azure Active Directory

This article describes how to configure your identity provider (IdP) and Azure Databricks to provision users and groups to Azure Databricks using SCIM, or System for Cross-domain Identity Management, an open standard that allows you to automate user provisioning.

About SCIM provisioning in Azure Databricks

SCIM lets you use an identity provider (IdP) to create users in Azure Databricks, give them the proper level of access, and remove access (deprovision them) when they leave your organization or no longer need access to Azure Databricks.

You can use a SCIM provisioning connector in your IdP or invoke the SCIM APIs to manage provisioning. You can also use these APIs to manage identities in Azure Databricks directly, without an IdP.

Account-level and workspace-level SCIM provisioning

You can either configure one SCIM provisioning connector from Azure Active Directory to your Azure Databricks account, using account-level SCIM provisioning, or configure separate SCIM provisioning connectors to each workspace, using workspace-level SCIM provisioning.

  • Account-level SCIM provisioning: Azure Databricks recommends you use account-level SCIM provisioning to create, update, and delete all users from the account. You will manage the assignment of users and groups to workspaces within Databricks. Your workspaces must be enabled for identity federation, in order to manage the assignment of users to workspaces.

Account-level SCIM diagram

  • Workspace-level SCIM provisioning (public preview): If none of your workspaces is enabled for identity federation, or if you have a mix of workspaces, some enabled for identity federation and others not, you must manage account-level and workspace-level SCIM provisioning in parallel. In a mixed scenario, you don’t need workspace-level SCIM provisioning for any workspaces that are enabled for identity federation.

    If you already have workspace-level SCIM provisioning set up for workspaces that you are enabling for identity federation, you should set up account-level SCIM provisioning and turn off the workspace-level SCIM provisioner. See Migrate workspace-level SCIM provisioning to the account level.

Requirements

To provision users and groups to Azure Databricks using SCIM:

  • Your Azure Databricks account must have the Premium Plan.
  • To provision users to your Azure Databricks account using SCIM (including the SCIM REST APIs), you must be an Azure Databricks account admin.
  • To provision users to a Azure Databricks workspace using SCIM (including the SCIM REST APIs), you must be an Azure Databricks workspace admin.

For more information about admin privileges, see Manage users, service principals, and groups.

You can have a maximum of 10,000 combined users and service principals and 5000 groups in an account. Each workspace can have a maximum of 10,000 combined users and service principals and 5000 groups.

Note

When you use SCIM provisioning, user and group attributes stored in Azure Active Directory can override changes you make using the Azure Databricks admin console, account console, or SCIM (Groups) API.

For example, if a user is assigned the Allow Cluster Creation entitlement in Azure Active Directory and you remove that entitlement using the Azure Databricks Admin Console, the user will be re-granted that entitlement the next time the IdP syncs with Azure Databricks, if the IdP is configured to provision that entitlement. The same behavior applies to groups.

Provision identities to your Azure Databricks account

You can use SCIM to provision users and groups from Azure Active Directory to your Azure Databricks account using a SCIM provisioning connector or directly using the SCIM APIs.

Add users and groups to your Azure Databricks account using Azure Active Directory (Azure AD)

You can sync account-level identities from your Azure Active Directory (Azure AD) tenant to Azure Databricks using a SCIM provisioning connector.

Important

If you already have SCIM connectors that sync users and groups directly to your workspaces and those workspaces are enabled for identity federation, you should disable those SCIM connectors when the account-level SCIM connector is enabled. If you have workspaces that are not identity federated, you should continue to use any SCIM connectors you have configured for those workspaces, running in parallel with the account-level SCIM connector.

For complete instructions, see Provision identities to your Azure Databricks account using Azure Active Directory (Azure AD).

Add users, service principals, and groups to your account using the SCIM API

Account admins can add users, service principals, and groups to the Azure Databricks account using the SCIM API for Accounts. Account admins call the API on accounts.azuredatabricks.net ({account_domain}/api/2.0/accounts/{account_id}/scim/v2/) and use a SCIM token.

To get the SCIM token, do the following:

  1. As an account admin, log in to the Azure Databricks account console.

  2. Click User Settings Icon Settings.

  3. Click User Provisioning.

    If provisioning isn’t enabled, click Enable user provisioning and copy the token.

    If provisioning is already enabled, click Regenerate token and copy the token.

See SCIM API 2.0 (Accounts).

Rotate the account-level SCIM token

If the account-level SCIM token is compromised or if you have business requirements to rotate authentication tokens periodically, you can rotate the SCIM token.

  1. As a Azure Databricks account admin, log in to the Azure Databricks account console.
  2. Click User Settings Icon Settings.
  3. Click User Provisioning.
  4. Click Regenerate token. Make a note of the new token. The previous token will continue to work for 24 hours.
  5. Within 24 hours, update your SCIM application to use the new SCIM token.

Provision identities to a Azure Databricks workspace

Important

This feature is in Public Preview.

If you want to use an IdP connector to provision users and groups and you have a workspace that is not identity federated, you must configure SCIM provisioning at the workspace level.

Add users and groups to your workspace using an IdP provisioning connector

Follow the instructions in the appropriate IdP-specific article:

Add users, groups, and service principals to your workspace using the SCIM API

Workspace admins can add users, groups, and service principals to the Azure Databricks account using the SCIM APIs for workspaces. See SCIM API 2.0.

Migrate workspace-level SCIM provisioning to the account level

If you already have workspace-level SCIM provisioning set up for workspaces that you are enabling for identity federation, you should set up account-level SCIM provisioning and turn off the workspace-level SCIM provisioner.

  1. Create a group in Azure Active Directory that includes all of the users and groups that you are currently provisioning to Azure Databricks using your workspace-level SCIM connectors.

    This group should include all users in all workspaces in your account.

  2. Configure a new SCIM provisioning connector to provision users and groups to your account, using the instructions in Provision identities to your Azure Databricks account.

    Use the group or groups that you created in step 1.

  3. Confirm that the new SCIM provisioning connector is successfully provisioning users and groups to your account.

  4. Shut down the old workspace-level SCIM connectors that were provisioning users and groups to your workspaces.

    Shut down only the SCIM connectors that are provisioning users and groups to workspaces that are enabled for identity federation. Keep the provisioning connectors in service for any workspaces that are not enabled for identity federation, but ensure that any identity that you add using the workspace-level connector is also being added using the account-level connector. IdP groups can help you manage this parallel provisioning scenario.

  5. Migrate workspace-local groups to account groups.

    If you have existing groups in your identity-federated workspaces, they are known as workspace-local groups and you cannot manage them using account-level interfaces. Databricks recommends that you convert them to account groups. See Migrate workspace-local groups to account groups

Important

When you remove a user from the account-level SCIM connector, that user is also removed from the account and all of their workspaces, regardless of whether or not identity federated has been enabled. When you remove a group from the account-level SCIM connector, all users in that group are deleted from the account and lose access to any workspaces they had access to, unless they are members of another group or have been directly granted access to the account or any workspaces. You should refrain from removing users and groups unless you want them to lose access to all workspaces in the account. Be aware of the following consequences of deleting users:

  • Applications or scripts that use the tokens generated by the user will no longer be able to access the Databricks API
  • Jobs owned by the user will fail
  • Clusters owned by the user will stop
  • Queries or dashboards created by the user and shared using the Run as Owner credential will have to be assigned to a new owner to prevent sharing from failing