Unable to create a Metastore in Azure Databricks

Question

Hello awesome people!

We are trying to implment Azure data bricks for our Proof of concept work, which involves the following.

• Gen 2 storage account with Hierarchical namespace enabled.

• Azure data bricks connector with the "Storage blob data contributor" permission assigned to it as shown in the image below.

We have populated the details with the correct format for ADLS gen2 path and Azure connector ID :

But when We are try to create an Azure metastore, we get the following error (Access validation failed - Internal server error).

Note : all of our resources (Resource group, storage account, Azure data bricks connector which is used as system managed identity) are in the same same region (West-Europe).

We also tried creating it in a different regions according to the supported regions documentation (https://learn.microsoft.com/en-us/azure/databricks/resources/supported-regions) but the same issue persists.

Any suggestions / valuable feedback to fix this issue is appreciated!

Answer 1

Hello Ajinkya,

I am having the exact same issue, and also there is another thread open for the same - https://community.databricks.com/t5/data-governance/metastore-creation-azure-databricks-internal-server-error/td-p/39514 (opened on 10-08-2023 which is pretty recent as per the date of creation for this thread).

The only difference I noted from this thread and when I tried creating Metastore first time is that the Access connector for databricks requires a "Storage Blob Data Owner" role in the IAM of the storage account. Although doing that did not resolve my problem.

You can try the resolutions mentioned in the above thread, even though I tried them I am still facing the same Internal Server Error. I highly doubt this error is on our side (role issues), and its recent as well. I went ahead and force created the metastore, and up until now I was able to create a catalog in the metastore.

Will update on my findings for this in future.

Answer 2

Hello Ajinkya Joshi,

Welcome to the Microsoft Q&A platform.

I am sure the issue you are seeing is due to the permissions.

Please see the below requirements to create metastore.

<copied from the documentation page>

To create a metastore:

• You must be an Azure Databricks account admin.

The first Azure Databricks account admin must be an Azure Active Directory Global Administrator at the time that they first log in to the Azure Databricks account console. Upon first login, that user becomes an Azure Databricks account admin and no longer needs the Azure Active Directory Global Administrator role to access the Azure Databricks account. The first account admin can assign users in the Azure Active Directory tenant as additional account admins (who can themselves assign more account admins). Additional account admins do not require specific roles in Azure Active Directory.

• The workspaces that you attach to the metastore must be on the Azure Databricks Premium plan.
• In your Azure tenant, you must have permission to create:
  • A storage account to use with Azure Data Lake Storage Gen2. See Create a storage account to use with Azure Data Lake Storage Gen2.
    • A new resource to hold a system-assigned managed identity. This requires that you be a Contributor or Owner of a resource group in any subscription in the tenant.

https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/create-metastore#requirements

I hope this helps. Please let me know if you have any further questions.

Answer 3

I am having the exact same issue. Did you find a resolution?

Answer 4

@ BhargavaGunnam-MSFT I am getting below error. Any help?