Migrate to automatic identity management

This page describes how to migrate an Azure Databricks account to automatic identity management.

For an overview of automatic identity management, see Automatic identity management.

Prerequisites

Before enabling automatic identity management, confirm that your environment meets the following requirements.

Requirement	Details	How to verify
Premium or Enterprise tier	Automatic identity management is not available on Standard tier accounts.	Account console > Account settings > Subscription tier
Single Microsoft Entra ID tenant	Automatic identity management does not support cross-tenant identities. All users, groups, and service principals must belong to a single Microsoft Entra ID tenant. Environments with cross-tenant identities must remain on SCIM provisioning.	Confirm with your identity team.
Identity federation enabled on at least one workspace	Automatic identity management requires at least one identity-federated workspace. Non-federated workspaces continue to function but do not use automatic identity management.	In the account console, open Workspaces and look for the Identity federation badge. To enable it, see Enable identity federation.
Account admin role	Only account admins can enable automatic identity management.	Account console > User Management. Your role chip must show Account admin.

Roles and responsibilities

Role	Responsibilities	Behavior after automatic identity management is enabled
Account admin	Enables automatic identity management. Configures, pauses, or disables account-level SCIM. Runs validation and external ID checks. Assigns groups to workspaces. Manages account-level identities and Unity Catalog permissions. Reviews identity-related audit logs.	Owns enabling, validating, and rolling back automatic identity management. Runs the external ID misconfiguration mitigation script if needed.
Workspace admin	Does not control automatic identity management or SCIM. Manages workspace-level ACLs using identities already assigned to the workspace. Runs post-migration tests and reports issues.	Can search for and provision Microsoft Entra ID identities directly from the workspace (Settings > Identity and Access > Manage users). Cannot enable or disable automatic identity management.
Workspace user	Signs in, runs notebooks and jobs, uses assigned compute. Shares assets with provisioned identities.	No change to active sessions. First sign-in after enabling automatic identity management provisions the user via JIT. Permissions resolve through the same group memberships as before. Can share assets with Microsoft Entra ID identities.

What stays the same

When you enable automatic identity management, the following are preserved:

SCIM-synced group memberships: Automatic identity management does not remove group memberships that were originally synced by SCIM. This is by design to avoid breaking jobs and permissions that depend on those memberships.
Existing permissions: Workspace and Unity Catalog permissions continue to resolve against the same principals.
Active sessions: Automatic identity management does not force user logouts or interrupt running sessions.
Running jobs: Service principals authenticated with current credentials continue to work.
Workspace-local groups: Automatic identity management only manages account-level identities. Workspace-local groups continue to work but are not synced from Microsoft Entra ID.
Databricks-only service principals: Service principals not registered in Microsoft Entra ID are unaffected.

Migrate without existing SCIM provisioning

Use this path if your Azure Databricks account has no account-level SCIM provisioning. This is the simplest migration path.

Audit automations that reference service principals by display name.

When automatic identity management is enabled, Azure Databricks treats Microsoft Entra ID as the authoritative source and overwrites custom service principal display names with those from Microsoft Entra ID. To prevent broken workflows, update any automation (such as Terraform configurations or scripts) to reference service principals by their application ID instead of display name.
Enable automatic identity management:
1. As an account admin, log in to the account console.
2. Click Security.
3. On the User provisioning tab, toggle Automatic identity management to Enabled.
4. Allow 5 to 10 minutes for the change to propagate.
(Optional) Configure the account access denylist. Use the account access denylist to restrict specific Microsoft Entra ID identities from accessing your Azure Databricks account. See Deny identities access to your account.
(Optional) Notify workspace admins. Share the automatic identity management overview with workspace admins so they understand the expected behavior changes, including just-in-time provisioning.

After enabling automatic identity management, complete the validation steps.

Migrate from existing SCIM provisioning

Use this path if your Azure Databricks account currently uses account-level SCIM provisioning. Databricks recommends enabling automatic identity management alongside your existing SCIM provisioning and running both in parallel. You can disable SCIM after validating that automatic identity management is working correctly.

What changes when you enable automatic identity management

Behavior	Before automatic identity management	After automatic identity management
Identity origin	SCIM pushes users, groups, and service principals from Microsoft Entra ID to the account SCIM endpoint on a schedule.	Azure Databricks reads identities directly from Microsoft Entra ID using the Graph API. Provisioning is just-in-time (JIT) on first use or login.
Group memberships	Direct members only. Nested groups must be flattened in Microsoft Entra ID or in SCIM scope rules.	Transitive group memberships are expanded automatically.
Group membership sync latency	Wait for the next SCIM cycle, typically 40 minutes.	Browser sign-in: up to 5 minutes. Non-browser (jobs, CLI, service principals): up to 40 minutes.
Service principal provisioning	Service principals are pushed by SCIM scope rules.	Service principals are provisioned on first authenticated use. No SCIM push is required.
Group renames in Microsoft Entra ID	SCIM updates the display name on the next sync cycle.	Group names are not proactively re-synced. An account admin opening the group detail page in the account console triggers a refresh. You can also call the `resolveByExternalId` API to trigger a sync.

Prepare for migration

Run the automatic identity management enablement prep script from the Databricks Knowledge Base.

This script identifies and resolves external ID mismatches between Azure Databricks and Microsoft Entra ID, and identifies which workspaces are identity-federated.

Note

Automatic identity management uses the Microsoft Entra ID objectId as the authoritative link for syncing identities. If a principal's externalId in Azure Databricks does not match its Microsoft Entra ID objectId, automatic identity management may create a duplicate principal. Run the discovery script before enabling automatic identity management to identify and fix any mismatches.
Audit automations that reference service principals by display name.

When automatic identity management is enabled, Azure Databricks treats Microsoft Entra ID as the authoritative source and overwrites custom service principal display names with those from Microsoft Entra ID. To prevent broken workflows, update any automation (such as Terraform configurations or scripts) to reference service principals by their application ID instead of display name.
(Optional) Configure the account access denylist. See Deny identities access to your account.
(Optional) Notify workspace admins. Share the automatic identity management overview with workspace admins so they understand the expected behavior changes, including just-in-time provisioning.

Enable automatic identity management

As an account admin, log in to the account console.
Click Security.
On the User provisioning tab, toggle Automatic identity management to Enabled.
Allow 5 to 10 minutes for the change to propagate.

Automatic identity management and SCIM provisioning run in parallel while both are active. SCIM continues to manage identities that it originally provisioned. After you validate automatic identity management, you can disable SCIM. See Disable SCIM provisioning.

Validate automatic identity management

After enabling automatic identity management, verify that it is syncing identities correctly.

Test by role

Have a representative user from each role run the following tests.

Account admin

Provision a user from the account console:
1. Go to Account console > User Management > Users and click Add user.
2. Search for a user in Microsoft Entra ID who has not been provisioned in Azure Databricks
3. Select the user and click Add user.
4. Confirm the user appears in Azure Databricks with a status of Active.
Provision a group from the account console:
1. Go to Account console > User Management > Groups and click Add group.
2. Search for a group in Microsoft Entra ID that has not been provisioned in Azure Databricks.
3. Confirm the group appears in Azure Databricks. Group members are provisioned on a just-in-time basis when they log in.
Verify workspace access:
1. Assign the group from the previous step to a workspace.
2. Have a member of that group authenticate to the workspace.
3. Confirm that authentication succeeds and that the user is provisioned in Azure Databricks and added to the workspace.

Workspace admin

Provision a user from the workspace:
1. Go to Settings > Identity and Access > Manage users and click Add user.
2. Search for a user in Microsoft Entra ID who has not been provisioned in Azure Databricks.
3. Confirm the user appears with a status of Active.
Provision a group from the workspace:
1. Open a workspace asset such as a query or dashboard and click Share.
2. Search for a group in Microsoft Entra ID that has not been provisioned in Azure Databricks and share the asset with that group.
3. Confirm that the group is provisioned in Azure Databricks.
4. Have members of that group authenticate to the workspace and verify that they can access the shared asset.

Verify automatic identity management in audit logs

Query the system.access.audit table to confirm that automatic identity management is active. A non-zero count for events such as add, addPrincipalToGroup, createGroup, and updateUser confirms that identities are being synced.

SELECT action_name, count(*) AS n
FROM system.access.audit
WHERE request_params.endpoint = 'autoUserCreation'
  AND event_time > current_timestamp() - INTERVAL 2 DAYS
GROUP BY action_name
ORDER BY n DESC;

For more information on automatic identity management audit events, see Audit automatic identity management events.

Disable automatic identity management

For steps to disable automatic identity management, see Disable automatic identity management.

When you disable automatic identity management after migrating from SCIM:

Identities created by automatic identity management remain in Azure Databricks but are no longer synced with Microsoft Entra ID.
Group memberships that were synced by automatic identity management are no longer used to resolve permissions. Users who previously inherited permissions through those group memberships lose access.
Users who authenticated to Azure Databricks while automatic identity management was active can continue to access dashboards they previously viewed, but their group memberships are not refreshed from Microsoft Entra ID.
Users who have never accessed Azure Databricks cannot access it after automatic identity management is disabled, even if they belong to an assigned group in Microsoft Entra ID.

Databricks recommends setting up SCIM provisioning before disabling automatic identity management. See Configure SCIM provisioning using Microsoft Entra ID (Azure Active Directory).

Disable SCIM provisioning

After validating automatic identity management, you can optionally disable SCIM provisioning. Disable SCIM only after confirming that all of the following conditions are met in your environment:

No nested groups: Your identity structure does not rely on nested group memberships in Microsoft Entra ID.
External ID alignment: Groups were replicated from Microsoft Entra ID using the SCIM Connector app, so externalId values in Azure Databricks match the corresponding Microsoft Entra ID objectId values. Use the Customer Instance Discovery Script to identify and resolve any mismatches.
No local membership modifications: Group memberships have been managed only through SCIM and have not been manually modified in Azure Databricks. The Customer Instance Discovery Script can detect locally modified memberships.

If your environment does not meet all of these conditions, Databricks recommends continuing to run SCIM provisioning and automatic identity management in parallel.

Known limitations after disabling SCIM

Persistent SCIM-synced group memberships

Group memberships that were synced by SCIM persist after SCIM is disabled. For example, if a child group C is nested under parent group P in Microsoft Entra ID, and the C to P relationship is later removed in Microsoft Entra ID, the user may still inherit permissions from P because the membership remains in Azure Databricks. Databricks recommends cleaning up direct group memberships from external groups before disabling SCIM.

Environments with nested group structures

For environments that use nested group structures, Databricks recommends continuing to run SCIM provisioning alongside automatic identity management to keep nested group memberships in sync.

Manually modified group memberships

If group memberships have been manually edited in Azure Databricks, those memberships are not synchronized with Microsoft Entra ID after you disable SCIM and may become stale over time. Follow the instructions in the automatic identity management enablement prep script from the Databricks Knowledge Base to identify and fix manually modified group memberships before disabling SCIM.

Migration considerations

Nested groups must be explicitly assigned to a workspace

When using nested Microsoft Entra ID groups, child groups are not automatically available for resource sharing within a workspace. To grant a child group permissions on workspace objects such as notebooks or queries, you must assign that child group directly to the workspace, even if its parent group is already assigned.

For example, if parent group P contains child group C, and only P is assigned to the workspace, C cannot be granted permissions on workspace resources until C itself is also assigned.

Group members are provisioned on a just-in-time basis

Unlike SCIM, automatic identity management does not proactively sync group memberships into Azure Databricks. Group members are provisioned when they have activity in Azure Databricks (for example, by logging in) or when an account admin or workspace admin explicitly adds them.

Deleted users are not immediately deactivated

When a user is deleted from Microsoft Entra ID, they continue to appear as active in Azure Databricks. They cannot log in, but their status is not automatically updated.

Email address changes create new users

If a user's email address changes in Microsoft Entra ID (for example, alice@example.com becomes alice-new@example.com), automatic identity management creates a new user in Azure Databricks rather than updating the existing record. This is consistent with current SCIM behavior. For email address changes, contact Databricks Support.

SCIM API does not return members provisioned through automatic identity management

The SCIM GET /groups/{id} API returns only group members provisioned through the Azure Databricks UI or SCIM. It does not return users who were provisioned through automatic identity management or users who exist in Microsoft Entra ID but have not yet been provisioned in Azure Databricks.

Unity Catalog permissions require provisioned identities

Unity Catalog permission-granting APIs cannot reference identities that have not yet been provisioned in Azure Databricks. Before granting permissions to an identity, provision it using the resolveByExternalId API. After the identity is provisioned, you can grant permissions as usual.

Functions that reference a principal by name, such as is_account_group_member(), also require the principal to be provisioned first.

Workspace-local groups are not managed by automatic identity management

Automatic identity management does not manage workspace-local groups. Workspace-local groups and their assigned permissions continue to work, but membership updates must be handled outside of automatic identity management.

Databricks recommends migrating workspace-local groups to account-level groups so that automatic identity management can manage their memberships. See Migrate workspace-local groups to account groups.

Feedback

Was this page helpful?

Last updated on 2026-05-12