Compliance security profile
This article describes the compliance security profile and its compliance controls.
Compliance security profile overview
The compliance security profile enables additional monitoring, a hardened compute image, and other features and controls on Azure Databricks workspaces. The compliance security profile includes controls that help meet the applicable security requirements of some compliance standards. Enabling the compliance security profile is required to use Azure Databricks to process data that is regulated under the following compliance standards:
You can also choose to enable the compliance security profile for its enhanced security features without the need to conform to a compliance standard.
Important
- You are solely responsible for ensuring your own compliance with all applicable laws and regulations.
- For PCI-DSS, you are solely responsible for ensuring that the compliance security profile and the appropriate compliance standards are configured before processing regulated data. For processing PHI data, Databricks strongly recommends use of the compliance security profile and selecting the HIPAA compliance standard.
If you enable this feature on any workspace, you are charged for the Enhanced Security and Compliance add-on as described on the pricing page.
Which compute resources get enhanced security
The compliance security profile enhancements apply to compute resources in the classic compute plane in all regions.
The compliance security profile enhancements for HIPAA also apply to compute resources in the serverless compute plane in all regions.
Azure Databricks does not allow starting serverless compute resources when PCI-DSS is enabled.
Note
Most Azure instance types are supported, but generation 2 (Gen2) and Arm64-based virtual machines are unsupported. Azure Databricks does not allow starting compute with those instance types when the compliance security profile is enabled.
For more information on compute plane architecture, see Azure Databricks architecture overview.
Compliance security profile features and technical controls
Security enhancements include:
An enhanced hardened operating system image based on Ubuntu Advantage.
Ubuntu Advantage is a package of enterprise security and support for open source infrastructure and applications that includes a CIS Level 1 hardened image.
Automatic cluster update is automatically enabled.
Clusters are restarted to get the latest updates periodically during a maintenance window that you can configure. See Automatic cluster update.
Enhanced securing monitoring is automatically enabled.
Security monitoring agents generate logs that you can review. For more information on the monitoring agents, see Monitoring agents in Azure Databricks compute plane images.
Communications within the cluster and for egress use TLS 1.2 encryption or higher, including connecting to the metastore.
Requirements
- Your Azure Databricks workspace is on the Premium pricing tier.
Step 1: Prepare a workspace for the compliance security profile
Follow these steps when you create a new workspaces with the security profile enabled or enable it on an existing workspace.
- If the workspace is configured to restrict outbound network access, you must configure your network to additionally allow traffic to port 2443. See Deploy Azure Databricks in your Azure virtual network (VNet injection).
- Run the following tests to verify that the changes were correctly applied:
- Launch a Databricks cluster with one driver, one worker, any DBR version, and any supported instance type.
- Confirm the cluster enters the Running state.
Step 2: Enable the compliance security profile on a workspace
Note
Databricks Assistant is disabled by default on workspaces that have enabled the compliance security profile. Workspace admins can enable it by following the instructions For an account: Disable or enable Databricks Assistant features.
Enable the compliance security profile.
To use the Azure portal to enable the compliance security profile on a workspace, see Use Azure portal to enable settings on a new workspace. You can also use an ARM template to enable the compliance security profile. See Use an ARM template.
Updates might take up to six hours to propagate to all environments. Workloads that are actively running continue with the settings that were active at the time of starting the compute resource, and new settings apply the next time these workloads are started.
Restart all running compute.
Step 3: Confirm that the compliance security profile is enabled for a workspace
To confirm that a workspace is using the compliance security profile, check that it has the yellow shield logo displayed in the user interface.
A shield logo appears in the top-right of the page, to the left of the workspace name:
Click the workspace name to see a list of the workspaces that you have access to. The workspaces that enable the compliance security profile have a shield icon followed by the text “Compliance security profile”.
You can also confirm a workspace is using the compliance security profile from the Security and compliance tab on the workspace page in the account console.
If the shield icons are missing for a workspace with the compliance security profile enabled, contact your Azure Databricks account team.