Raising the limit in Azure Databricks identities

Question

Context:

The Azure Databricks documentation states that "You can have a maximum of 10,000 combined users and service principals and 5,000 groups in an account. Each workspace can have a maximum of 10,000 combined users and service principals and 5,000 groups."

I work for an organization that considers including all of their 84,000 employees into the Azure Databricks account and into the already deployed Databricks DEV/TEST/PROD workspaces.

This leads to a few questions:

1. Is there any reason for having a 10,000 identities limit in Databricks Account and Workspaces ?
2. Would it be possible to get Azure Databricks support team to increase those Databricks identities limits for our Unity Account and Workspaces to say 90.000 (to also allow for future growth) ?
3. Would there be consequences to consider before raising that 10,000 principals limit to e.g. 90,000 ?

Answer 1

Hello @Kim Stig Hansen

Welcome to the Microsoft Q&A and thank you for posting your questions here.

Problem

I understand that you are having trouble with the identity limits in Azure Databricks as you're think about adding 84,000 employees to the account and current workspaces. You would want to know why there are limits in the first place, if it's possible to raise them to fit 90,000 people, and what might happen if they do and there are more questions arises:

Scenario

Kim Stig Hansen, a member of an organization considering Azure Databricks for their data analytics needs, is evaluating the platform's scalability. They discover that Azure Databricks imposes a limit of 10,000 combined users and service principals and 5,000 groups per account and workspace. With 84,000 employees to include and existing DEV/TEST/PROD workspaces, Kim seeks clarity on the reasons for these limits and the possibility of raising them to support their workforce adequately. They are particularly concerned about potential performance, manageability, security, and cost implications of such a change.

Solution

This solution was based on your unique scenario given and your questions, while focusing on the problem statement.

Raising the identity limits in Azure Databricks to accommodate 84,000 employees is indeed a significant consideration. Let's address your questions:

Is there any reason for having a 10,000 identities limit in Databricks Account and Workspaces?

Reason for the limit: The limit on identities likely exists to maintain performance and manageability within Azure Databricks. Handling a large number of identities can impact system performance and increase complexity in managing access controls and permissions.

Would it be possible to get Azure Databricks support team to increase those Databricks identities limits for our Unity Account and Workspaces to say 90.000 (to also allow for future growth)?

Requesting an increase: It's possible to request an increase in the identity limits from the Azure Databricks support team. However, before doing so, it's essential to evaluate the potential impact on performance, manageability, and security. You should provide a thorough justification for the increase, including anticipated usage patterns and growth projections. Microsoft Azure Databrick help center will be glad to assist here.

My advice is to contact your solution architect, or direct sale support or use the link to contact Databricks Help Center. https://help.databricks.com/s/

Finally

Would there be consequences to consider before raising that 10,000 principals limit to e.g. 90,000?

Consequences of raising the limit: Increasing the identity limit to accommodate 90,000 principals could have several consequences to consider:

• Performance
• Manageability
• Security
• Cost

Before proceeding with raising the limit, it's advisable to engage with the Azure Databricks support team as stated in the previous answer to discuss your requirements and assess any potential implications thoroughly. Additionally, consider implementing best practices for identity and access management to ensure security and compliance with organizational policies.

References

To read more information about resource limit in Azure Databricks, kindly use the link below:

Azure Databricks documentation

Resource limits

Identity best practices

You can also use the additional resources available by the right of this page for more reading and leaning.

Accept Answer

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

** Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful ** so that others in the community facing similar issues can easily find the solution.

Best Regards,

Sina Salam