Unable to create or bring up the cluster on azure databricks. - Failed to perform resource identity operation

Question

Hi,

We have set up an Azure Databricks service along supporting services and it was working fine until the below changes were performed on Azure subscription.

Details of changes:

--> subscriptions was moved to a different directory

Post this change, when we are trying to create or start the existing cluster in Databricks workspace, we are receiving the below error:

"Azure Operation Not Allowed Exception: Failed to launch the VM because the operation is not permitted, please contact your system administrator. [details] FailedIdentityOperation: Identity operation for resource '/subscriptions/xxxxxxxxxxxxxx/resourceGroups/databricks-rg-AZR-CDP-pcat2snjugnem/providers/Microsoft.Compute/virtualMachines/xxxxxxxx' failed with error 'Failed to perform resource identity operation. Status: 'BadRequest'. Response: '{"error":{"code":"CrossTenantUserAssignmentRequestForbidden","message":"It is not permitted to assign a user assigned managed identity to an Azure resource in a different Azure AD tenant."}}'.'.(OnDemand)"

Thanks,

Sandeep

Answer 1

Hi Sandeep !

Thank you for posting on Microsoft Learn.

Did you move your subscription to a different AAD tenant recently ?

If yes, it may be the issue in your case as Databricks clusters often use UAMIs for secure resource access which are bound to the original tenant. After you move the subscription to a new tenant, the identities will be logically orphaned (still exist but you can't use them in the new context).

As a workaround, if you're not using specific permissions tied to a UAMI you can then switch to system assigned managed identity which is automatically scoped to the new tenant. Be careful it may have an impact on VM creation, access to ADLS (Gen2), Key Vault, and other services that you integrated via identity.

Or you simply recreate UAMI in the new tenant and subscription and then assign it to the Databricks workspace. You will make extra effort if you have any role assignments or policies since you need to reconfigure them.

If my answer helped you, don't hesitate to accept it ! :D

Answer 2

Hi Sandeep Jidagi, Thank you so much for your query.

After digging into the issue, I found that the error you're running into "CrossTenantUserAssignmentRequestForbidden: It is not permitted to assign a user assigned managed identity to an Azure resource in a different Azure AD tenant" is a known limitation in Azure, especially after moving a subscription to a different Azure Active Directory (AAD) tenant.

What’s Causing This: When you move an Azure subscription from one AAD tenant to another, a few important things break:

• User Assigned Managed Identities (UAMIs) don’t carry over. They are tied to the original tenant, and after the move, you can’t assign them to resources under the new tenant. If you try, you'll hit this exact error.
• Role-Based Access Control (RBAC) permissions are wiped during the transfer. You’ll need to reassign all roles in the new tenant manually.
• System Assigned Managed Identities (SAMIs) might still work, but they could need reconfiguring depending on how your resources are set up.
• Azure Key Vault access policies that pointed to identities in the old tenant will break. Those identities no longer exist from the new tenant’s perspective.
• And finally, Azure Databricks workspaces can’t be moved between tenants at all. If the workspace was created in the old tenant, you’ll likely need to recreate it from scratch in the new one.

What You Can Do to Fix It: Here are a few ways to resolve it

• Recreate the UAMI in the new tenant
  • Create a new User Assigned Managed Identity under the new tenant.
  • Assign it the right roles (e.g., for networking or storage).
  • Update your Databricks workspace and clusters to use this new identity.
• Switch to a System Assigned Managed Identity
  • If you don’t specifically need a UAMI, consider switching to SAMI.
  • Enable the system-assigned identity for your Databricks workspace.
  • Assign the required roles and test your cluster start-up.
• Rebuild your Azure Databricks workspace
  • If nothing else works, you might have to recreate the workspace in the new tenant.
  • That means reconfiguring your environment and migrating any data or notebooks you had.

You can refer Documentation Also Documentation

Let me know if any of this won't help or if you’re seeing any specific error messages! Happy to help you.

Please do not forget to click "Accept the answer” and Yes wherever the information provided helps you, this can be beneficial to other community members. If you have any other questions or still running into more issues, let me know in the "comments" and I would be happy to help you.