Require Databricks to Connect to Storage Account via Private Endpoint

Question

Hello,

I would like to have Azure Databricks connect to its storage account only through private endpoints for security purposes. Azure Databricks is currently configured with Private Endpoints, but the Storage Account is not. When I try to create a private endpoint on the storage account, I get an error: DenyAssignmentAuthorizationFailed .

My search results haven't found anything specific to this error and creating the storage account private endpoints.

I believe that I need to create 2 private endpoints. 1 for blob and 1 for dfs.

What prerequisite am I missing to accomplish this task? Thank you.

Error:

"The client with object id '' has permission to perform action 'Microsoft.Resources/deployments/validate/action' on scope '/subscriptions//resourceGroups/databricks/providers/Microsoft.Resources/deployments/Microsoft.PrivateEndpoint-20231127090825'; however, the access is denied because of the deny assignment with name 'System deny assignment created by Azure Databricks /subscriptions//resourceGroups/main/providers/Microsoft.Databricks/workspaces/databricks' and Id '15ba378cda7641e9ba2ccac' at scope '/subscriptions//resourceGroups/main'."

Answer 1

@Anon4343 - Thanks for the details.

This is an excepted behaviour in Azure Databricks.

You cannot enabled private endpoint or make any changes to the storage account associated to the Azure Databricks workspace.

Reason: Azure Storage account associated by Azure Databricks even if you're owner - it's a resource managed by Databricks, and it prevents direct access to the data because it stores some system information inside storage account.

For more details, refer to https://learn.microsoft.com/en-us/azure/databricks/storage/

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 2

Original answer is now outdated: https://learn.microsoft.com/en-us/azure/databricks/security/network/storage/firewall-support

Not sure if this was available when OP did their question but similar error will be encountered if you try to re-use managed resource group created during workspace creation. Linked documentation states that you need to use your own resource group and subnets for private endpoint creation:

• "The resource group must not be the same as the managed resource group that your workspace storage account is in."
• "You must have a separate subnet for the private endpoints for the storage account. This is in addition to the main two subnets for basic Azure Databricks functionality.

The subnet must be in the same VNet as the workspace or in a separate VNet that the workspace can access. Use the minimum size /28 in CIDR notation."

"This is an excepted behaviour in Azure Databricks."

Well, it should not be if publicly open storage accounts triggers Defender. Luckily that is fixable now.

Answer 3

You're getting the error because Databricks adds a deny assignment on its managed resource group, blocking changes like private endpoint creation.

Create the blob and dfs private endpoints in your own resource group, not the Databricks-managed one.