Configure customer-managed keys for data encryption in Azure AI Search
Azure AI Search automatically encrypts data at rest with service-managed keys. If more protection is needed, you can supplement default encryption with another encryption layer using keys that you create and manage in Azure Key Vault.
This article walks you through the steps of setting up customer-managed key (CMK) or "bring-your-own-key" (BYOK) encryption. Here are some points to keep in mind:
CMK encryption is enacted on individual objects. If you require CMK across your search service, set an enforcement policy.
CMK encryption depends on Azure Key Vault. You can create your own encryption keys and store them in a key vault, or you can use Azure Key Vault APIs to generate encryption keys. Azure Key Vault must be in the same subscription and tenant as Azure AI Search. Azure AI Search retrieves your managed key by connecting through a system or user-managed identity. This behavior requires both services to share the same tenant.
CMK encryption becomes operational when an object is created. You can't encrypt objects that already exist. CMK encryption occurs whenever an object is saved to disk, either data at rest for long-term storage or temporary data for short-term storage. With CMK, the disk never sees unencrypted data.
Note
If an index is CMK encrypted, it is only accessible if the search service has access to the key. If access is revoked, the index is unusable and the service cannot be scaled until the index is deleted or access to the key is restored.
CMK encrypted objects
Objects that can be encrypted include indexes, synonym lists, indexers, data sources, and skillsets. Encryption is computationally expensive to decrypt so only sensitive content is encrypted.
Encryption is performed over the following content:
All content within indexes and synonym lists, including descriptions.
For indexers, data sources, and skillsets, only those fields that store connection strings, descriptions, keys, and user inputs are encrypted. For example, skillsets have Azure AI services keys, and some skills accept user inputs, such as custom entities. In both cases, keys and user inputs into skills are encrypted.
Full double encryption
When you introduce CMK encryption, you're encrypting content twice. For the objects and fields noted in the previous section, content is first encrypted with your CMK, and secondly with the Microsoft-managed key. Content is doubly encrypted on data disks for long-term storage, and on temporary disks used for short-term storage.
Enabling CMK encryption increases index size and degrades query performance. Based on observations to date, you can expect to see an increase of 30-60 percent in query times, although actual performance varies depending on the index definition and types of queries. Because performance is diminished, we recommend that you only enable this feature on indexes that really require it.
Although double encryption is now available in all regions, support was rolled out in two phases:
The first rollout was on August 1, 2020 and included the five regions listed below. Search services created in the following regions supported CMK for data disks, but not temporary disks:
- West US 2
- East US
- South Central US
- US Gov Virginia
- US Gov Arizona
The second rollout on May 13, 2021 added encryption for temporary disks and extended CMK encryption to all supported regions.
If you're using CMK from a service created during the first rollout and you also want CMK encryption over temporary disks, you need to create a new search service in your region of choice and redeploy your content.
Prerequisites
The following tools and services are used in this scenario.
Azure AI Search on a billable tier (Basic or above, in any region).
Azure Key Vault, you can create a key vault using the Azure portal, Azure CLI, or Azure PowerShell. Create the resource in the same subscription as Azure AI Search. The key vault must have soft-delete and purge protection enabled.
Microsoft Entra ID. If you don't have one, set up a new tenant.
You should have a search client that can create the encrypted object. Into this code, you reference a key vault key and application registration information. This code could be a working app, or prototype code such as the C# code sample DotNetHowToEncryptionUsingCMK.
Tip
You can use a REST client or Azure PowerShell to create indexes and synonym maps that include an encryption key parameter. You can also use Azure SDKs. Portal support for adding a key to indexes or synonym maps isn't supported.
Key Vault tips
If you're new to Azure Key Vault, review this quickstart to learn about basic tasks: Set and retrieve a secret from Azure Key Vault using PowerShell. Here are some tips for using Key Vault:
Use as many key vaults as you need. Managed keys can be in different key vaults. A search service can have multiple encrypted objects, each one encrypted with a different customer-managed encryption key, stored in different key vaults.
Enable logging on Key Vault so that you can monitor key usage.
Remember to follow strict procedures during routine rotation of key vault keys and Active Directory application secrets and registration. Always update all encrypted content to use new secrets and keys before deleting the old ones. If you miss this step, your content can't be decrypted.
1 - Enable purge protection
As a first step, make sure soft-delete and purge protection are enabled on the key vault. Due to the nature of encryption with customer-managed keys, no one can retrieve your data if your Azure Key Vault key is deleted.
To prevent data loss caused by accidental Key Vault key deletions, soft-delete and purge protection must be enabled on the key vault. Soft-delete is enabled by default, so you'll only encounter issues if you purposely disabled it. Purge protection isn't enabled by default, but it's required for customer-managed key encryption in Azure AI Search.
You can set both properties using the portal, PowerShell, or Azure CLI commands.
Sign in to the Azure portal and open your key vault overview page.
On the Overview page under Essentials, enable Soft-delete and Purge protection.
2 - Create a key in Key Vault
Skip key generation if you already have a key in Azure Key Vault that you want to use, but collect the key identifier. You need this information when creating an encrypted object.
Sign in to the Azure portal and open your key vault overview page.
Select Keys on the left, and then select + Generate/Import.
In the Create a key pane, from the list of Options, choose the method that you want to use to create a key. You can Generate a new key, Upload an existing key, or use Restore Backup to select a backup of a key.
Enter a Name for your key, and optionally select other key properties.
Select Create to start the deployment.
Select the key, select the current version, and then make a note of the key identifier. It's composed of the key value Uri, the key name, and the key version. You need the identifier to define an encrypted index in Azure AI Search.
3 - Create a security principal
You have several options for accessing the encryption key at run time. The simplest approach is to retrieve the key using the managed identity and permissions of your search service. You can use either a system or user-managed identity. Doing so allows you to omit the steps for application registration and application secrets, and simplifies the encryption key definition.
Alternatively, you can create and register a Microsoft Entra application. The search service provides the application ID on requests.
A managed identity enables your search service to authenticate to Azure Key Vault without storing credentials (ApplicationID or ApplicationSecret) in code. The lifecycle of this type of managed identity is tied to the lifecycle of your search service, which can only have one managed identity. For more information about how managed identities work, see What are managed identities for Azure resources.
Make your search service a trusted service.
Conditions that prevent you from adopting this approach include:
You can't directly grant your search service access permissions to the key vault (for example, if the search service is in a different Microsoft Entra ID tenant than the Azure Key Vault).
A single search service is required to host multiple encrypted indexes or synonym maps, each using a different key from a different key vault, where each key vault must use a different identity for authentication. Because a search service can only have one managed identity, a requirement for multiple identities rules out the simplified approach for your scenario.
4 - Grant permissions
In this step, you create an access policy in Key Vault. This policy gives the application you registered with Microsoft Entra ID permission to use your customer-managed key.
Access permissions could be revoked at any given time. Once revoked, any search service index or synonym map that uses that key vault become unusable. Restoring key vault access permissions at a later time restores index and synonym map access. For more information, see Secure access to a key vault.
Still in the Azure portal, open your key vault Overview page.
Select the Access policies on the left, and select + Create to start the Create an access policy wizard.
On the Permissions page, select Get for Key permissions, Secret permissions, and Certificate Permissions. Select Unwrap Key and Wrap Key for ** cryptographic operations on the key.
Select Next.
On the Principle page, find and select the security principal used by the search service to access the encryption key. This will either be the system-managed or user-managed identity of the search service, or the registered application.
Select Next and Create.
Important
Encrypted content in Azure AI Search is configured to use a specific Azure Key Vault key with a specific version. If you change the key or version, the index or synonym map must be updated to use it before you delete the previous one. Failing to do so will render the index or synonym map unusable. You won't be able to decrypt the content if the key is lost.
5 - Encrypt content
Encryption keys are added when you create an object. To add a customer-managed key on an index, synonym map, indexer, data source, or skillset, use the Search REST API or an Azure SDK to create an object that has encryption enabled. The portal doesn't allow encryption properties on object creation.
Call the Create APIs to specify the encryptionKey property:
Insert the encryptionKey construct into the object definition. This property is a first-level property, on the same level as name and description. The following REST examples show property placement. If you're using the same vault, key, and version, you can paste in the same "encryptionKey" construct into each object definition.
The first example shows an "encryptionKey" for a search service that connects using a managed identity:
{ "encryptionKey": { "keyVaultUri": "https://demokeyvault.vault.azure.net", "keyVaultKeyName": "myEncryptionKey", "keyVaultKeyVersion": "eaab6a663d59439ebb95ce2fe7d5f660" } }The second example includes "accessCredentials", necessary if you registered an application in Microsoft Entra ID:
{ "encryptionKey": { "keyVaultUri": "https://demokeyvault.vault.azure.net", "keyVaultKeyName": "myEncryptionKey", "keyVaultKeyVersion": "eaab6a663d59439ebb95ce2fe7d5f660", "accessCredentials": { "applicationId": "00000000-0000-0000-0000-000000000000", "applicationSecret": "myApplicationSecret" } } }
Once you create the encrypted object on the search service, you can use it as you would any other object of its type. Encryption is transparent to the user and developer.
Note
None of these key vault details are considered secret and could be easily retrieved by browsing to the relevant Azure Key Vault page in Azure portal.
6 - Set up policy
Azure policies help to enforce organizational standards and to assess compliance at-scale. Azure AI Search has an optional built-in policy for service-wide CMK enforcement.
In this section, you set the policy that defines a CMK standard for your search service. Then, you set up your search service to enforce this policy.
Navigate to the built-in policy in your web browser. Select Assign
Set up the policy scope. In the Parameters section, uncheck Only show parameters... and set Effect to Deny.
During evaluation of the request, a request that matches a deny policy definition is marked as noncompliant. Assuming the standard for your service is CMK encryption, "deny" means that requests that don't specify CMK encryption are noncompliant.
Finish creating the policy.
Call the Services - Update API to enable CMK policy enforcement at the service level.
PATCH https://management.azure.com/subscriptions/[subscriptionId]/resourceGroups/[resourceGroupName]/providers/Microsoft.Search/searchServices/[serviceName]?api-version=2022-11-01
{
"properties": {
"encryptionWithCmk": {
"enforcement": "Enabled",
"encryptionComplianceStatus": "Compliant"
}
}
}
REST examples
This section shows the JSON for several objects so that you can see where to locate "encryptionKey" in an object definition.
Index encryption
The details of creating a new index via the REST API could be found at Create Index (REST API), where the only difference is specifying the encryption key details as part of the index definition:
{
"name": "hotels",
"fields": [
{"name": "HotelId", "type": "Edm.String", "key": true, "filterable": true},
{"name": "HotelName", "type": "Edm.String", "searchable": true, "filterable": false, "sortable": true, "facetable": false},
{"name": "Description", "type": "Edm.String", "searchable": true, "filterable": false, "sortable": false, "facetable": false, "analyzer": "en.lucene"},
{"name": "Description_fr", "type": "Edm.String", "searchable": true, "filterable": false, "sortable": false, "facetable": false, "analyzer": "fr.lucene"},
{"name": "Category", "type": "Edm.String", "searchable": true, "filterable": true, "sortable": true, "facetable": true},
{"name": "Tags", "type": "Collection(Edm.String)", "searchable": true, "filterable": true, "sortable": false, "facetable": true},
{"name": "ParkingIncluded", "type": "Edm.Boolean", "filterable": true, "sortable": true, "facetable": true},
{"name": "LastRenovationDate", "type": "Edm.DateTimeOffset", "filterable": true, "sortable": true, "facetable": true},
{"name": "Rating", "type": "Edm.Double", "filterable": true, "sortable": true, "facetable": true},
{"name": "Location", "type": "Edm.GeographyPoint", "filterable": true, "sortable": true}
],
"encryptionKey": {
"keyVaultUri": "https://demokeyvault.vault.azure.net",
"keyVaultKeyName": "myEncryptionKey",
"keyVaultKeyVersion": "eaab6a663d59439ebb95ce2fe7d5f660",
"accessCredentials": {
"applicationId": "00000000-0000-0000-0000-000000000000",
"applicationSecret": "myApplicationSecret"
}
}
}
You can now send the index creation request, and then start using the index normally.
Synonym map encryption
Create an encrypted synonym map using the Create Synonym Map Azure AI Search REST API. Use the "encryptionKey" property to specify which encryption key to use.
{
"name" : "synonymmap1",
"format" : "solr",
"synonyms" : "United States, United States of America, USA\n
Washington, Wash. => WA",
"encryptionKey": {
"keyVaultUri": "https://demokeyvault.vault.azure.net",
"keyVaultKeyName": "myEncryptionKey",
"keyVaultKeyVersion": "eaab6a663d59439ebb95ce2fe7d5f660",
"accessCredentials": {
"applicationId": "00000000-0000-0000-0000-000000000000",
"applicationSecret": "myApplicationSecret"
}
}
}
You can now send the synonym map creation request, and then start using it normally.
Data source encryption
Create an encrypted data source using the Create Data Source (REST API). Use the "encryptionKey" property to specify which encryption key to use.
{
"name" : "datasource1",
"type" : "azureblob",
"credentials" :
{ "connectionString" : "DefaultEndpointsProtocol=https;AccountName=datasource;AccountKey=accountkey;EndpointSuffix=core.windows.net"
},
"container" : { "name" : "containername" },
"encryptionKey": {
"keyVaultUri": "https://demokeyvault.vault.azure.net",
"keyVaultKeyName": "myEncryptionKey",
"keyVaultKeyVersion": "eaab6a663d59439ebb95ce2fe7d5f660",
"accessCredentials": {
"applicationId": "00000000-0000-0000-0000-000000000000",
"applicationSecret": "myApplicationSecret"
}
}
}
You can now send the data source creation request, and then start using it normally.
Skillset encryption
Create an encrypted skillset using the Create Skillset REST API. Use the "encryptionKey" property to specify which encryption key to use.
{
"name": "skillset1",
"skills": [ omitted for brevity ],
"cognitiveServices": { omitted for brevity },
"knowledgeStore": { omitted for brevity },
"encryptionKey": (optional) {
"keyVaultKeyName": "myEncryptionKey",
"keyVaultKeyVersion": "eaab6a663d59439ebb95ce2fe7d5f660",
"keyVaultUri": "https://demokeyvault.vault.azure.net",
"accessCredentials": {
"applicationId": "00000000-0000-0000-0000-000000000000",
"applicationSecret": "myApplicationSecret"}
}
}
You can now send the skillset creation request, and then start using it normally.
Indexer encryption
Create an encrypted indexer using the Create Indexer REST API. Use the "encryptionKey" property to specify which encryption key to use.
{
"name": "indexer1",
"dataSourceName": "datasource1",
"skillsetName": "skillset1",
"parameters": {
"configuration": {
"imageAction": "generateNormalizedImages"
}
},
"encryptionKey": {
"keyVaultUri": "https://demokeyvault.vault.azure.net",
"keyVaultKeyName": "myEncryptionKey",
"keyVaultKeyVersion": "eaab6a663d59439ebb95ce2fe7d5f660",
"accessCredentials": {
"applicationId": "00000000-0000-0000-0000-000000000000",
"applicationSecret": "myApplicationSecret"
}
}
}
You can now send the indexer creation request, and then start using it normally.
Important
While "encryptionKey" can't be added to existing search indexes or synonym maps, it may be updated by providing different values for any of the three key vault details (for example, updating the key version). When changing to a new Key Vault key or a new key version, any search index or synonym map that uses the key must first be updated to use the new key\version before deleting the previous key\version. Failing to do so will render the index or synonym map unusable, as it won't be able to decrypt the content once key access is lost. Although restoring key vault access permissions at a later time will restore content access.
Work with encrypted content
With customer-managed key encryption, you might notice latency for both indexing and queries due to the extra encrypt/decrypt work. Azure AI Search doesn't log encryption activity, but you can monitor key access through key vault logging. We recommend that you enable logging as part of key vault configuration.
Key rotation is expected to occur over time. Whenever you rotate keys, it's important to follow this sequence:
- Determine the key used by an index or synonym map.
- Create a new key in key vault, but leave the original key available.
- Update the encryptionKey properties on an index or synonym map to use the new values. Only objects that were originally created with this property can be updated to use a different value.
- Disable or delete the previous key in the key vault. Monitor key access to verify the new key is being used.
For performance reasons, the search service caches the key for up to several hours. If you disable or delete the key without providing a new one, queries continue to work on a temporary basis until the cache expires. However, once the search service can no longer decrypt content, you get this message: "Access forbidden. The query key used might have been revoked - please retry."
Next steps
If you're unfamiliar with Azure security architecture, review the Azure Security documentation, and in particular, this article:
Feedback
Coming soon: Throughout 2024 we will be phasing out GitHub Issues as the feedback mechanism for content and replacing it with a new feedback system. For more information see: https://aka.ms/ContentUserFeedback.
Submit and view feedback for