Managed storage account's compliance

Hello Chitra Gurumurthy,

Greetings! Welcome to Microsoft Q&A Platform.

Unfortunately, it’s not currently possible to disable public access for Databricks Managed Storage Accounts (DBFS root) directly. The managed resource group configuration prevents such changes. Public access is required for Databricks to function properly, as logs, models, and other essential data are stored in the DBFS root and accessed by the Azure Databricks control plane. But if it comes to the data security, here are the relevant information. The storage account has the deny policy that prevents from making changes & accessing the data for anyone except the Databricks application. So even it has public access, you can't generate SAS, use storage account key, etc.

Deny assignments are a type of Azure Policy assignment that explicitly denies access to a specific action in Azure. In your case, the deny assignment is blocking the action 'Microsoft.Storage/storageAccounts/listKeys/action' on the specified scope. Unfortunately, you can't override this deny assignment on the managed resource group created by Azure Databricks, even if you're the owner. This is because it's a resource managed by Databricks, and it prevents direct access to the data because it stores some system information inside the storage account. If you need to access data in the Databricks managed storage account, it's recommended to use DBFS for temporary data or mount a new storage account .

References:

1. Azure Policy - Deny assignments: https://docs.microsoft.com/en-us/azure/role-based-access-control/deny-assignments
2. https://stackoverflow.com/questions/73064767/how-to-override-deny-assignment-so-that-i-can-access-the-databricks-managed-stor
3. https://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/security-compliance-and-privacy/best-practices

Alternatively, you can try creating a custom solution as updated above to achieve the ask, A custom policy definition allows customers to define their own rules for using Azure.

Hope this answer helps! Please let us know if you have any further queries. I’m happy to assist you further.

Please "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

Hello Chitra Gurumurthy (NON EA SC ALT),

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that you would like Azure Databricks managed storage accounts key access disable and compliant.

Solution

To make Azure Databricks managed storage accounts compliant by disabling key access, you'll need to follow a few steps.

If you don’t have a policy to enforce disabling key access, you can create a custom policy. Below code is an example of a policy definition that ensures key access is disabled:

{
  "properties": {
    "displayName": "Ensure that Azure Storage Accounts have key access disabled",
    "policyType": "Custom",
    "mode": "Indexed",
    "description": "This policy ensures that Azure Storage Accounts have key access disabled.",
    "parameters": {},
    "policyRule": {
      "if": {
        "allOf": [
          {
            "field": "type",
            "equals": "Microsoft.Storage/storageAccounts"
          },
          {
            "anyOf": [
              {
                "field": "Microsoft.Storage/storageAccounts/allowBlobPublicAccess",
                "exists": "true"
              }
            ]
          }
        ]
      },
      "then": {
        "effect": "deny"
      }
    }
  }
}

Also, below is optional to define policy that audits the use of storage account keys. This policy will help identify any usage of storage account keys and flag them for compliance checks:

{
  "properties": {
    "displayName": "Audit use of storage account keys",
    "policyType": "Custom",
    "mode": "All",
    "description": "This policy audits the use of storage account keys.",
    "parameters": {},
    "policyRule": {
      "if": {
        "allOf": [
          {
            "field": "type",
            "equals": "Microsoft.Storage/storageAccounts"
          },
          {
            "field": "Microsoft.Storage/storageAccounts/networkAcls.defaultAction",
            "equals": "Allow"
          }
        ]
      },
      "then": {
        "effect": "audit"
      }
    }
  }
}

To Implement:

Navigate to Azure Policy, In the left-hand menu, select "All services".

In the search box, type "Policy" and select "Policy" from the results.

Create and Assign the First Policy above (Ensure Key Access Disabled)

Create the Policy Definition:

In the Azure Policy dashboard, select "Definitions".

Click "+ Policy definition".

• Fill in the following details:
  • Name: Ensure that Azure Storage Accounts have key access disabled.
  • Description: This policy ensures that Azure Storage Accounts have key access disabled.
  • Category: Choose a category, or create a new one like "Custom Policies".
  • In the "Policy Rule" section, paste the JSON provided earlier. Click "Save" to create the policy definition.

Go back to the "Assignments" tab in the Azure Policy dashboard.

Click "+ Assign policy".

• In the "Basics" tab, fill in the following:
  • Scope: Select the subscription or resource group where your Databricks-managed storage accounts reside.
  • Policy definition: Select the policy you just created. Click "Review + create" and then "Create" to assign the policy.

Repeat the above for the second script to assign the second policy JSON above (Audit Storage Account Key Usage).

References

To read more and get more detail information, kindly use the links below:

Source: Best practices for Azure Databricks. Accessed, 6/19/2024.

Source: Azure Policy Samples and Definitions. Accessed, 6/19/2024.

Source: Creating and Implementing Custom Policies. Accessed, 6/19/2024.

Accept Answer

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

** Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful ** so that others in the community facing similar issues can easily find the solution.

Best Regards,

Sina Salam