Enable HSM customer-managed keys for managed services

Note

This feature requires the Premium plan.

This article describes how to configure your own key from Azure Key Vault Managed HSM. For instructions on using a key from Azure Key Vault vaults, see Enable customer-managed keys for managed services.

Requirements

• To use the Azure CLI for these tasks, install the Azure CLI tool and install the Databricks extension:

  az extension add --name databricks
  

• To use Powershell for these tasks, install Azure PowerShell and install the Databricks Powershell module. You must also log in:

  Connect-AzAccount
  

  To log in to your Azure account as a user, see PowerShell login with an Azure Databricks user account. To log in to your Azure account as a service principal, see PowerShell login with a Microsoft Entra ID service principal.

Step 1: Create an Azure Key Vault Managed HSM and an HSM key

You can use an existing Azure Key Vault Managed HSM or create and activate a new one following the quickstarts in the Managed HSM documentation. See Quickstart: Provision and activate a Managed HSM using Azure CLI. The Azure Key Vault Managed HSM must have Purge Protection enabled.

Important

The Key Vault must be in the same Azure tenant as your Azure Databricks workspace.

To create an HSM key, follow Create an HSM key.

Step 2: Configure the Managed HSM role assignment

Configure a role assignment for the Key Vault Managed HSM so that your Azure Databricks workspace has permission to access it. You can configure a role assignment using the Azure portal, Azure CLI, or Azure Powershell.

Use the Azure portal

1. Go to your Managed HSM resource in the Azure portal.
2. In the left menu, under Settings, select Local RBAC.
3. Click Add.
4. In the Role field, select Managed HSM Crypto Service Encryption User.
5. In the Scope field, select All keys (/).
6. In the Security principal field, type AzureDatabricks and scroll to the Enterprise Application result that has an Application ID of 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d and select it.
7. Click Create.
8. In the left menu, under Settings, select Keys and select your key.
9. In the Key Identifier field, copy the text.

Use Azure CLI

1. Get the object ID of the AzureDatabricks application with the Azure CLI.

   az ad sp show --id "2ff814a6-3304-4ab8-85cb-cd0e6f879c1d" \
                   --query "id" \
                   --output tsv
   

2. Configure the Managed HSM role assignment. Replace <hsm-name> with the your Managed HSM name and replace <object-id> with the object ID of the AzureDatabricks application from the previous step.

   az keyvault role assignment create --role "Managed HSM Crypto Service Encryption User"
       --scope "/" --hsm-name <hsm-name>
       --assignee-object-id <object-id>
   

Use Azure Powershell

Replace <hsm-name> with the your Managed HSM name.

Connect-AzureAD
$managedService = Get-AzureADServicePrincipal \
-Filter "appId eq '2ff814a6-3304-4ab8-85cb-cd0e6f879c1d'"

New-AzKeyVaultRoleAssignment -HsmName <hsm-name> \
-RoleDefinitionName "Managed HSM Crypto Service Encryption User" \
-ObjectId $managedService.ObjectId

Step 3: Add a key to a workspace

You can create or update a workspace with a customer-managed key for managed services, using the Azure portal, Azure CLI, or Azure Powershell.

Use the Azure portal

1. Go to the Azure Portal homepage.

2. Click Create a resource in the top left corner of the page.

3. Within the search bar, type Azure Databricks and click the Azure Databricks option.

4. Click Create in the Azure Databricks widget.

5. Enter values for the input fields on the Basics and Networking tabs.

6. After you reach the Encryption tab:

   • For creating a workspace, enable Use your own key in the Managed Services section.
   • For updating a workspace, enable Managed Services.

7. Set the encryption fields.

   

   • In the Key Identifier field, paste the Key Identifier of your Managed HSM key.
   • In the Subscription dropdown, enter the subscription name of your Azure Key Vault key.

8. Complete the remaining tabs and click Review + Create (for new workspace) or Save (for updating a workspace).

Use the Azure CLI

Create or update a workspace:

For both creation and update, add these fields to the command:

• managed-services-key-name: Managed HSM name
• managed-services-key-vault: Managed HSM URI
• managed-services-key-version: Managed HSM version

Example creation of a workspace using these fields:

az databricks workspace create --name <workspace-name> \
--resource-group <resource-group-name> \
--location <location> \
--sku premium \
--managed-services-key-name <hsm-name> \
--managed-services-key-vault <hsm-uri> \
--managed-services-key-version <hsm-version>

Example update of a workspace using these fields:

az databricks workspace update --name <workspace-name> \
--resource-group <resource-group-name> \
--managed-services-key-name <hsm-name> \
--managed-services-key-vault <hsm-uri> \
--managed-services-key-version <hsm-version>

Important

If you rotate the key, you must keep the old key available for 24 hours.

Use Powershell

To create or update a workspace, add the following parameters to the command for your new key:

• ManagedServicesKeyVaultPropertiesKeyName: Managed HSM name
• ManagedServicesKeyVaultPropertiesKeyVaultUri: Managed HSM URI
• ManagedServicesKeyVaultPropertiesKeyVersion: Managed HSM version

Example workspace creation with these fields:

New-AzDatabricksWorkspace -Name <workspace-name> \
-ResourceGroupName <resource-group-name> \
-location $keyVault.Location \
-sku premium \
-ManagedServicesKeyVaultPropertiesKeyName $hsm.Name \
-ManagedServicesKeyVaultPropertiesKeyVaultUri $hsm.Uri \
-ManagedServicesKeyVaultPropertiesKeyVersion $hsm.Version

Example workspace update with these fields:

Update-AzDatabricksWorkspace -Name <workspace-name> \
-ResourceGroupName <resource-group-name> \
-sku premium \
-ManagedServicesKeyVaultPropertiesKeyName $hsm.Name \
-ManagedServicesKeyVaultPropertiesKeyVaultUri $hsm.VaultUri \
-ManagedServicesKeyVaultPropertiesKeyVersion $hsm.Version

Important

If you rotate the key, you must keep the old key available for 24 hours.

Step 4 (optional): Re-import notebooks

After you initially add a key for managed services for an existing workspace, only future write operations use your key. Existing data is not re-encrypted.

You can export all notebooks and then re-import them so the key that encrypts the data is protected and controlled by your key. You can use the Export and Import Workspace APIs.

Rotate the key at a later time

If you are already using a customer-managed key for managed services, you can update the workspace with a new key version, or an entirely new key. This is called key rotation.

1. Create a new key or rotate your existing key in the Managed HSM Vault.

   Ensure the new key has the proper permissions.

2. Update the workspace with your new key using the portal, CLI, or PowerShell. See Step 3: Add a key to a workspace and follow instructions for workspace update. Be sure that you use the same values for the resource group name and the workspace name so it updates the existing workspace, rather than creating a new workspace. Other than changes in the key-related parameters, use the same parameters that were used for creating the workspace.

   Important

   If you rotate the key, you must keep the old key available for 24 hours.

3. Optionally export and re-import existing notebooks to ensure that all your existing notebooks use your new key.