Customer-managed keys for Azure Machine Learning
Azure Machine Learning is built on top of multiple Azure services. While the data is stored securely using encryption keys that Microsoft provides, you can enhance security by also providing your own (customer-managed) keys. The keys you provide are stored securely using Azure Key Vault.
Customer-managed keys are used with the following services that Azure Machine Learning relies on:
|Service||What it’s used for|
|Azure Cosmos DB||Stores metadata for Azure Machine Learning|
|Azure Cognitive Search||Stores workspace metadata for Azure Machine Learning|
|Azure Storage Account||Stores workspace metadata for Azure Machine Learning|
|Azure Container Instance||Hosting trained models as inference endpoints|
|Azure Kubernetes Service||Hosting trained models as inference endpoints|
- Azure Cosmos DB, Cognitive Search, and Storage Account are secured using the same key. You can use a different key for Azure Kubernetes Service and Container Instance.
- To use a customer-managed key with Azure Cosmos DB, Cognitive Search, and Storage Account, the key is provided when you create your workspace. The key(s) used with Azure Container Instance and Kubernetes Service are provided when configuring those resources.
In addition to customer-managed keys, Azure Machine Learning also provides a hbi_workspace flag. Enabling this flag reduces the amount of data Microsoft collects for diagnostic purposes and enables extra encryption in Microsoft-managed environments. This flag also enables the following behaviors:
- Starts encrypting the local scratch disk in your Azure Machine Learning compute cluster, provided you haven’t created any previous clusters in that subscription. Else, you need to raise a support ticket to enable encryption of the scratch disk of your compute clusters.
- Cleans up your local scratch disk between jobs.
- Securely passes credentials for your storage account, container registry, and SSH account from the execution layer to your compute clusters using your key vault.
hbi_workspace flag does not impact encryption in transit, only encryption at rest.
An Azure subscription.
An Azure Key Vault instance. The key vault contains the key(s) used to encrypt your services.
The key vault instance must enable soft delete and purge protection.
The managed identity for the services secured by a customer-managed key must have the following permissions in key vault:
- wrap key
- unwrap key
For example, the managed identity for Azure Cosmos DB would need to have those permissions to the key vault.
- The customer-managed key for resources the workspace depends on can’t be updated after workspace creation.
- Resources managed by Microsoft in your subscription can’t transfer ownership to you.
- You can't delete Microsoft-managed resources used for customer-managed keys without also deleting your workspace.
How workspace metadata is stored
The following resources store metadata for your workspace:
|Service||How it’s used|
|Azure Cosmos DB||Stores job history data.|
|Azure Cognitive Search||Stores indices that are used to help query your machine learning content.|
|Azure Storage Account||Stores other metadata such as Azure Machine Learning pipelines data.|
Your Azure Machine Learning workspace reads and writes data using its managed identity. This identity is granted access to the resources using a role assignment (Azure role-based access control) on the data resources. The encryption key you provide is used to encrypt data that is stored on Microsoft-managed resources. It's also used to create indices for Azure Cognitive Search, which are created at runtime.
When you don't use a customer-managed key, Microsoft creates and manages these resources in a Microsoft owned Azure subscription and uses a Microsoft-managed key to encrypt the data.
When you use a customer-managed key, these resources are in your Azure subscription and encrypted with your key. While they exist in your subscription, these resources are managed by Microsoft. They're automatically created and configured when you create your Azure Machine Learning workspace.
When using a customer-managed key, the costs for your subscription will be higher because these resources are in your subscription. To estimate the cost, use the Azure pricing calculator.
These Microsoft-managed resources are located in a new Azure resource group is created in your subscription. This group is in addition to the resource group for your workspace. This resource group will contain the Microsoft-managed resources that your key is used with. The resource group will be named using the formula of
<Azure Machine Learning workspace resource group name><GUID>.
- The Request Units for the Azure Cosmos DB automatically scale as needed.
- If your Azure Machine Learning workspace uses a private endpoint, this resource group will also contain a Microsoft-managed Azure Virtual Network. This VNet is used to secure communications between the managed services and the workspace. You cannot provide your own VNet for use with the Microsoft-managed resources. You also cannot modify the virtual network. For example, you cannot change the IP address range that it uses.
If your subscription does not have enough quota for these services, a failure will occur.
Don't delete the resource group that contains this Azure Cosmos DB instance, or any of the resources automatically created in this group. If you need to delete the resource group or Microsoft-managed services in it, you must delete the Azure Machine Learning workspace that uses it. The resource group resources are deleted when the associated workspace is deleted.
How compute data is stored
Azure Machine Learning uses compute resources to train and deploy machine learning models. The following table describes the compute options and how data is encrypted by each one:
|Azure Container Instance||Data is encrypted by a Microsoft-managed key or a customer-managed key.For more information, see Encrypt data with a customer-managed key.|
|Azure Kubernetes Service||Data is encrypted by a Microsoft-managed key or a customer-managed key.For more information, see Bring your own keys with Azure disks in Azure Kubernetes Services.|
|Azure Machine Learning compute instance||Local scratch disk is encrypted if the
|Azure Machine Learning compute cluster||OS disk encrypted in Azure Storage with Microsoft-managed keys. Temporary disk is encrypted if the
Compute cluster The OS disk for each compute node stored in Azure Storage is encrypted with Microsoft-managed keys in Azure Machine Learning storage accounts. This compute target is ephemeral, and clusters are typically scaled down when no jobs are queued. The underlying virtual machine is de-provisioned, and the OS disk is deleted. Azure Disk Encryption isn't supported for the OS disk.
Each virtual machine also has a local temporary disk for OS operations. If you want, you can use the disk to stage training data. If the workspace was created with the
hbi_workspace parameter set to
TRUE, the temporary disk is encrypted. This environment is short-lived (only during your job) and encryption support is limited to system-managed keys only.
The OS disk for compute instance is encrypted with Microsoft-managed keys in Azure Machine Learning storage accounts. If the workspace was created with the
hbi_workspace parameter set to
TRUE, the local temporary disk on compute instance is encrypted with Microsoft managed keys. Customer managed key encryption isn't supported for OS and temp disk.
hbi_workspaceflag can only be set when a workspace is created. It can’t be changed for an existing workspace.
- When this flag is set to True, it may increase the difficulty of troubleshooting issues because less telemetry data is sent to Microsoft. There’s less visibility into success rates or problem types. Microsoft may not be able to react as proactively when this flag is True.
To enable the
hbi_workspace flag when creating an Azure Machine Learning workspace, follow the steps in one of the following articles:
- How to create and manage a workspace.
- How to create and manage a workspace using the Azure CLI.
- How to create a workspace using Hashicorp Terraform.
- How to create a workspace using Azure Resource Manager templates.