Share via

Configure SCIM provisioning using Microsoft Entra ID (Azure Active Directory)

  • Article

This article describes how to set up provisioning to the Azure Databricks account using Microsoft Entra ID.

Databricks recommends that you provision users, service principals, and groups to the account level and manage the assignment of users and groups to workspaces within Azure Databricks. Your workspaces must be enabled for identity federation, in order to manage the assignment of users to workspaces.

Note

The way provisioning is configured is entirely separate from configuring authentication and conditional access for Azure Databricks workspaces or accounts. Authentication for Azure Databricks is handled automatically by Microsoft Entra ID, using the OpenID Connect protocol flow. You can configure conditional access, which lets you create rules to require multi-factor authentication or restrict logins to local networks, at the service level.

Provision identities to your Azure Databricks account using Microsoft Entra ID

You can sync account-level users and groups from your Microsoft Entra ID tenant to Azure Databricks using a SCIM provisioning connector.

Important

If you already have SCIM connectors that sync identities directly to your workspaces, you must disable those SCIM connectors when the account-level SCIM connector is enabled. See Migrate workspace-level SCIM provisioning to the account level.

Requirements

  • Your Azure Databricks account must have the Premium plan.
  • You must have the Cloud Application Administrator role in Microsoft Entra ID.
  • Your Microsoft Entra ID account must be a Premium edition account to provision groups. Provisioning users is available for any Microsoft Entra ID edition.
  • You must be an Azure Databricks account admin.

Note

To enable the account console and establish your first account admin, see Establish your first account admin.

Step 1: Configure Azure Databricks

  1. As an Azure Databricks account admin, log in to the Azure Databricks account console.
  2. Click User Settings Icon Settings.
  3. Click User Provisioning.
  4. Click Set up user provisioning.

Copy the SCIM token and the Account SCIM URL. You will use these to configure your Microsoft Entra ID application.

Note

The SCIM token is restricted to the Account SCIM API /api/2.1/accounts/{account_id}/scim/v2/ and cannot be used to authenticate to other Databricks REST APIs.

Step 2: Configure the enterprise application

These instructions tell you how to create an enterprise application in the Azure portal and use that application for provisioning. If you have an existing enterprise application, you can modify it to automate SCIM provisioning using Microsoft Graph. This removes the need for a separate provisioning application in the Azure Portal.

Follow these steps to enable Microsoft Entra ID to sync users and groups to your Azure Databricks account. This configuration is separate from any configurations you have created to sync users and groups to workspaces.

  1. In your Azure portal, go to Microsoft Entra ID > Enterprise Applications.
  2. Click + New Application above the application list. Under Add from the gallery, search for and select Azure Databricks SCIM Provisioning Connector.
  3. Enter a Name for the application and click Add.
  4. Under the Manage menu, click Provisioning.
  5. Set Provisioning Mode to Automatic.
  6. Set the SCIM API endpoint URL to the Account SCIM URL that you copied earlier.
  7. Set Secret Token to the Azure Databricks SCIM token that you generated earlier.
  8. Click Test Connection and wait for the message that confirms that the credentials are authorized to enable provisioning.
  9. Click Save.

Step 3: Assign users and groups to the application

Users and groups assigned to the SCIM application will be provisioned to the Azure Databricks account. If you have existing Azure Databricks workspaces, Databricks recommends that you add all existing users and groups in those workspaces to the SCIM application.

Note

Microsoft Entra ID does not support the automatic provisioning of service principals to Azure Databricks. You can add service principals your Azure Databricks account following Manage service principals in your account.

Microsoft Entra ID does not support the automatic provisioning of nested groups to Azure Databricks. Microsoft Entra ID can only read and provision users that are immediate members of the explicitly assigned group. As a workaround, explicitly assign (or otherwise scope in) the groups that contain the users who need to be provisioned. For more information, see this FAQ.

  1. Go to Manage > Properties.
  2. Set Assignment required to No. Databricks recommends this option, which allows all users to sign in to the Azure Databricks account.
  3. Go to Manage > Provisioning.
  4. To start synchronizing Microsoft Entra ID users and groups to Azure Databricks, set the Provisioning Status toggle to On.
  5. Click Save.
  6. Go to Manage > Users and groups.
  7. Click Add user/group, select the users and groups, and click the Assign button.
  8. Wait a few minutes and check that the users and groups exist in your Azure Databricks account.

Users and groups that you add and assign will automatically be provisioned to the Azure Databricks account when Microsoft Entra ID schedules the next sync.

Note

If you remove a user from the account-level SCIM application, that user is deactivated from the account and from their workspaces, regardless of whether or not identity federation has been enabled.

Provisioning tips

  • Users and groups that existed in the Azure Databricks workspace prior to enabling provisioning exhibit the following behavior upon provisioning sync:
    • Are merged if they also exist in Microsoft Entra ID
    • Are ignored if they don’t exist in Microsoft Entra ID
  • Individually assigned user permissions that are duplicated by membership in a group remain even after the group membership is removed for the user.
  • Users removed from an Azure Databricks workspace directly, using the Azure Databricks workspace admin settings page:
    • Lose access to that Azure Databricks workspace but may still have access to other Azure Databricks workspaces.
    • Will not be synced again using Microsoft Entra ID provisioning, even if they remain in the enterprise application.
  • The initial Microsoft Entra ID sync is triggered immediately after you enable provisioning. Subsequent syncs are triggered every 20-40 minutes, depending on the number of users and groups in the application. See Provisioning summary report in the Microsoft Entra ID documentation.
  • You cannot update the username or email address of an Azure Databricks workspace user.
  • The admins group is a reserved group in Azure Databricks and cannot be removed.
  • You can use the Azure Databricks Groups API or the Groups UI to get a list of members of any Azure Databricks workspace group.
  • You cannot sync nested groups or Microsoft Entra ID service principals from the Azure Databricks SCIM Provisioning Connector application. Databricks recommends using the enterprise application to sync users and groups and manage nested groups and service principals within Azure Databricks. However, you can also use the Databricks Terraform provider or custom scripts that target the Azure Databricks SCIM API to sync nested groups or Microsoft Entra ID service principals.
  • Updates to group names in Microsoft Entra ID do not sync into Azure Databricks.
  • The parameters userName and emails.value must match. A mismatch can lead to Azure Databricks rejecting user creation requests from the Microsoft Entra ID SCIM application. For cases such as external users or aliased emails, you might need to change the enterprise application’s default SCIM mapping to use userPrincipalName rather than mail.

(Optional) Automate SCIM provisioning using Microsoft Graph

Microsoft Graph includes authentication and authorization libraries that you can integrate into your application to automate provisioning of users and groups to your Azure Databricks account or workspaces, instead of configuring a SCIM provisioning connector application.

  1. Follow the instructions for registering an application with Microsoft Graph. Make a note of the Application ID and the Tenant ID for the application
  2. Go to the applications’s Overview page. On that page:
    1. Configure a client secret for the application, and make a note of the secret.
    2. Grant the application these permissions:
      • Application.ReadWrite.All
      • Application.ReadWrite.OwnedBy
  3. Ask a Microsoft Entra ID administrator to grant admin consent.
  4. Update your application’s code to add support for Microsoft Graph.

Troubleshooting

Users and groups do not sync

  • If you are using the Azure Databricks SCIM Provisioning Connector application:
    • In the account console verify that the Azure Databricks SCIM token that was used to set up provisioning is still valid.
  • Do not attempt to sync nested groups, which are not supported by Microsoft Entra ID automatic provisioning. For more information, see this FAQ.

Microsoft Entra ID service principals do not sync

  • The Azure Databricks SCIM Provisioning Connector application does not support syncing service principals.

After initial sync, the users and groups stop syncing

If you are using the Azure Databricks SCIM Provisioning Connector application: After the initial sync, Microsoft Entra ID does not sync immediately after you change user or group assignments. It schedules a sync with the application after a delay, based on the number of users and groups. To request an immediate sync, go to Manage > Provisioning for the enterprise application and select Clear current state and restart synchronization.

Microsoft Entra ID provisioning service IP range not accessible

The Microsoft Entra ID provisioning service operates under specific IP ranges. If you need to restrict network access, you must allow traffic from the IP addresses for AzureActiveDirectory in this IP range file. For more information, see IP Ranges.