Edit

Add Key Management Service (KMS) etcd encryption to an Azure Kubernetes Service (AKS) cluster

  • Article
  • 9 minutes to read

This article shows you how to enable encryption at rest for your Kubernetes secrets in etcd using Azure Key Vault with the Key Management Service (KMS) plugin. The KMS plugin allows you to:

  • Use a key in Key Vault for etcd encryption.
  • Bring your own keys.
  • Provide encryption at rest for secrets stored in etcd.
  • Rotate the keys in Key Vault.

For more information on using the KMS plugin, see Encrypting Secret Data at Rest.

Prerequisites

Warning

KMS supports Konnectivity or API Server Vnet Integration. You can use kubectl get po -n kube-system to verify the results show that a konnectivity-agent-xxx pod is running. If there is, it means the AKS cluster is using Konnectivity. When using VNet integration, you can run the command az aks cluster show -g -n to verify the setting enableVnetIntegration is set to true.

Limitations

The following limitations apply when you integrate KMS etcd encryption with AKS:

  • Deletion of the key, Key Vault, or the associated identity isn't supported.
  • KMS etcd encryption doesn't work with system-assigned managed identity. The key vault access policy is required to be set before the feature is enabled. In addition, system-assigned managed identity isn't available until cluster creation, thus there's a cycle dependency.
  • Azure Key Vault with Firewall enabled to allow public access isn't supported because it blocks traffic from KMS plugin to the Key Vault.
  • The maximum number of secrets that a cluster enabled with KMS supports is 2,000.
  • Bring your own (BYO) Azure Key Vault from another tenant isn't supported.
  • With KMS enabled, you can't change associated Azure Key Vault model (public, private). To change associated key vault mode, you need to disable and enable KMS again.
  • If a cluster is enabled KMS with private key vault and not using the API Server VNet integration tunnel, then stop/start cluster is not allowed.

KMS supports public key vault and private key vault.

Enable KMS with public key vault

Create a key vault and key

Warning

Deleting the key or the Azure Key Vault is not supported and will cause the secrets to be unrecoverable in the cluster.

If you need to recover your Key Vault or key, see Azure Key Vault recovery management with soft delete and purge protection.

For non-RBAC key vault

Use az keyvault create to create a key vault.

az keyvault create --name MyKeyVault --resource-group MyResourceGroup

Use az keyvault key create to create a key.

az keyvault key create --name MyKeyName --vault-name MyKeyVault

Use az keyvault key show to export the key ID.

export KEY_ID=$(az keyvault key show --name MyKeyName --vault-name MyKeyVault --query 'key.kid' -o tsv)
echo $KEY_ID

The above example stores the key ID in KEY_ID.

For RBAC key vault

Use az keyvault create to create a key vault using Azure Role Based Access Control.

export KEYVAULT_RESOURCE_ID=$(az keyvault create --name MyKeyVault --resource-group MyResourceGroup  --enable-rbac-authorization true --query id -o tsv)

Assign yourself permission to create a key.

az role assignment create --role "Key Vault Crypto Officer" --assignee-object-id $(az ad signed-in-user show --query id --out tsv) --assignee-principal-type "User" --scope $KEYVAULT_RESOURCE_ID

Use az keyvault key create to create a key.

az keyvault key create --name MyKeyName --vault-name MyKeyVault

Use az keyvault key show to export the key ID.

export KEY_ID=$(az keyvault key show --name MyKeyName --vault-name MyKeyVault --query 'key.kid' -o tsv)
echo $KEY_ID

The above example stores the key ID in KEY_ID.

Create a user-assigned managed identity

Use az identity create to create a user-assigned managed identity.

az identity create --name MyIdentity --resource-group MyResourceGroup

Use az identity show to get the identity object ID.

IDENTITY_OBJECT_ID=$(az identity show --name MyIdentity --resource-group MyResourceGroup --query 'principalId' -o tsv)
echo $IDENTITY_OBJECT_ID

The above example stores the value of the identity object ID in IDENTITY_OBJECT_ID.

Use az identity show to get the identity resource ID.

IDENTITY_RESOURCE_ID=$(az identity show --name MyIdentity --resource-group MyResourceGroup --query 'id' -o tsv)
echo $IDENTITY_RESOURCE_ID

The above example stores the value of the identity resource ID in IDENTITY_RESOURCE_ID.

Assign permissions (decrypt and encrypt) to access key vault

For non-RBAC key vault

If your key vault is not enabled with --enable-rbac-authorization, you can use az keyvault set-policy to create an Azure key vault policy.

az keyvault set-policy -n MyKeyVault --key-permissions decrypt encrypt --object-id $IDENTITY_OBJECT_ID

For RBAC key vault

If your key vault is enabled with --enable-rbac-authorization, you need to assign the "Key Vault Crypto User" RBAC role which has decrypt, encrypt permission.

az role assignment create --role "Key Vault Crypto User" --assignee-object-id $IDENTITY_OBJECT_ID --assignee-principal-type "ServicePrincipal" --scope $KEYVAULT_RESOURCE_ID

Create an AKS cluster with KMS etcd encryption enabled

Create an AKS cluster using the az aks create command with the --enable-azure-keyvault-kms, --azure-keyvault-kms-key-vault-network-access and --azure-keyvault-kms-key-id parameters to enable KMS etcd encryption.

az aks create --name myAKSCluster --resource-group MyResourceGroup --assign-identity $IDENTITY_RESOURCE_ID --enable-azure-keyvault-kms --azure-keyvault-kms-key-vault-network-access "Public" --azure-keyvault-kms-key-id $KEY_ID

Update an existing AKS cluster to enable KMS etcd encryption

Use az aks update with the --enable-azure-keyvault-kms, --azure-keyvault-kms-key-vault-network-access and --azure-keyvault-kms-key-id parameters to enable KMS etcd encryption on an existing cluster.

az aks update --name myAKSCluster --resource-group MyResourceGroup --enable-azure-keyvault-kms --azure-keyvault-kms-key-vault-network-access "Public" --azure-keyvault-kms-key-id $KEY_ID

Use the following command to update all secrets. Otherwise, old secrets won't be encrypted. For larger clusters, you may want to subdivide the secrets by namespace or script an update.

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Rotate the existing keys

After changing the key ID (including key name and key version), you can use az aks update with the --enable-azure-keyvault-kms, --azure-keyvault-kms-key-vault-network-access and --azure-keyvault-kms-key-id parameters to rotate the existing keys of KMS.

Warning

Remember to update all secrets after key rotation. Otherwise, the secrets will be inaccessible if the old keys don't exist or aren't working.

Once you rotate the key, the old key (key1) is still cached and shouldn't be deleted. If you want to delete the old key (key1) immediately, you need to rotate the key twice. Then key2 and key3 are cached, and key1 can be deleted without impacting existing cluster.

az aks update --name myAKSCluster --resource-group MyResourceGroup  --enable-azure-keyvault-kms --azure-keyvault-kms-key-vault-network-access "Public" --azure-keyvault-kms-key-id $NEW_KEY_ID

Use the following command to update all secrets. Otherwise, old secrets will still be encrypted with the previous key. For larger clusters, you may want to subdivide the secrets by namespace or script an update.

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Enable KMS with private key vault

If you enable KMS with private key vault, AKS will create a private endpoint and private link in the node resource group automatically. The key vault will be added a private endpoint connection with the AKS cluster.

Create a private key vault and key

Warning

Deleting the key or the Azure Key Vault isn't supported and will cause the secrets to be unrecoverable in the cluster.

If you need to recover your key vault or key, see Azure Key Vault recovery management with soft delete and purge protection.

Use az keyvault create to create a private key vault.

az keyvault create --name MyKeyVault --resource-group MyResourceGroup --public-network-access Disabled

It's not supported to create or update keys in private key vault without private endpoint. To manage private key vaults, you can refer to Integrate Key Vault with Azure Private Link.

Create a user-assigned managed identity

Use az identity create to create a user-assigned managed identity.

az identity create --name MyIdentity --resource-group MyResourceGroup

Use az identity show to get the identity object ID.

IDENTITY_OBJECT_ID=$(az identity show --name MyIdentity --resource-group MyResourceGroup --query 'principalId' -o tsv)
echo $IDENTITY_OBJECT_ID

The above example stores the value of the identity object ID in IDENTITY_OBJECT_ID.

Use az identity show to get identity resource ID.

IDENTITY_RESOURCE_ID=$(az identity show --name MyIdentity --resource-group MyResourceGroup --query 'id' -o tsv)
echo $IDENTITY_RESOURCE_ID

The above example stores the value of the identity resource ID in IDENTITY_RESOURCE_ID.

Assign permissions (decrypt and encrypt) to access key vault

For non-RBAC key vault

If your key vault is not enabled with --enable-rbac-authorization, you can use az keyvault set-policy to create an Azure key vault policy.

az keyvault set-policy -n MyKeyVault --key-permissions decrypt encrypt --object-id $IDENTITY_OBJECT_ID

For RBAC key vault

If your key vault is enabled with --enable-rbac-authorization, you need to assign a RBAC role that contains decrypt, encrypt permission.

az role assignment create --role "Key Vault Crypto User" --assignee-object-id $IDENTITY_OBJECT_ID --assignee-principal-type "ServicePrincipal" --scope $KEYVAULT_RESOURCE_ID

For private key vaults, you need the Key Vault Contributor role to create a private link between the private key vault and the cluster.

az role assignment create --role "Key Vault Contributor" --assignee-object-id $IDENTITY_OBJECT_ID --assignee-principal-type "ServicePrincipal" --scope $KEYVAULT_RESOURCE_ID

Create an AKS cluster with private key vault and enable KMS etcd encryption

Create an AKS cluster using the az aks create command with the --enable-azure-keyvault-kms, --azure-keyvault-kms-key-id, --azure-keyvault-kms-key-vault-network-access and --azure-keyvault-kms-key-vault-resource-id parameters to enable KMS etcd encryption with private key vault.

az aks create --name myAKSCluster --resource-group MyResourceGroup --assign-identity $IDENTITY_RESOURCE_ID --enable-azure-keyvault-kms --azure-keyvault-kms-key-id $KEY_ID --azure-keyvault-kms-key-vault-network-access "Private" --azure-keyvault-kms-key-vault-resource-id $KEYVAULT_RESOURCE_ID

Update an existing AKS cluster to enable KMS etcd encryption with private key vault

Use az aks update with the --enable-azure-keyvault-kms, --azure-keyvault-kms-key-id, --azure-keyvault-kms-key-vault-network-access and --azure-keyvault-kms-key-vault-resource-id parameters to enable KMS etcd encryption on an existing cluster with private key vault.

az aks update --name myAKSCluster --resource-group MyResourceGroup --enable-azure-keyvault-kms --azure-keyvault-kms-key-id $KEY_ID --azure-keyvault-kms-key-vault-network-access "Private" --azure-keyvault-kms-key-vault-resource-id $KEYVAULT_RESOURCE_ID

Use the following command to update all secrets. Otherwise, old secrets won't be encrypted. For larger clusters, you may want to subdivide the secrets by namespace or script an update.

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Rotate the existing keys

After changing the key ID (including key name and key version), you can use az aks update with the --enable-azure-keyvault-kms, --azure-keyvault-kms-key-id, --azure-keyvault-kms-key-vault-network-access and --azure-keyvault-kms-key-vault-resource-id parameters to rotate the existing keys of KMS.

Warning

Remember to update all secrets after key rotation. Otherwise, the secrets will be inaccessible if the old keys are not existing or working.

Once you rotate the key, the old key (key1) is still cached and shouldn't be deleted. If you want to delete the old key (key1) immediately, you need to rotate the key twice. Then key2 and key3 are cached, and key1 can be deleted without impacting existing cluster.

az aks update --name myAKSCluster --resource-group MyResourceGroup  --enable-azure-keyvault-kms --azure-keyvault-kms-key-id $NewKEY_ID --azure-keyvault-kms-key-vault-network-access "Private" --azure-keyvault-kms-key-vault-resource-id $KEYVAULT_RESOURCE_ID

Use the following command to update all secrets. Otherwise, old secrets will still be encrypted with the previous key. For larger clusters, you may want to subdivide the secrets by namespace or script an update.

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Update key vault mode

Note

To change a different key vault with a different mode (public, private), you can run az aks update directly. To change the mode of attached key vault, you need to disable KMS and re-enable it with the new key vault IDs.

Below are the steps about how to migrate the attached public key vault to private mode.

Disable KMS on the cluster

Disable the KMS on existing cluster and release the key vault.

az aks update --name myAKSCluster --resource-group MyResourceGroup --disable-azure-keyvault-kms

Change key vault mode

Update the key vault from public to private.

az keyvault update --name MyKeyVault --resource-group MyResourceGroup --public-network-access Disabled

Enable KMS on the cluster with updated key vault

Re-enable the KMS with updated private key vault.

az aks update --name myAKSCluster --resource-group MyResourceGroup  --enable-azure-keyvault-kms --azure-keyvault-kms-key-id $NewKEY_ID --azure-keyvault-kms-key-vault-network-access "Private" --azure-keyvault-kms-key-vault-resource-id $KEYVAULT_RESOURCE_ID

After configuring KMS, you can enable diagnostic-settings for key vault to check the encryption logs.

Disable KMS

Use the following command to disable KMS on existing cluster.

az aks update --name myAKSCluster --resource-group MyResourceGroup --disable-azure-keyvault-kms

Use the following command to update all secrets. Otherwise, the old secrets will still be encrypted with the previous key and the encrypt/decrypt permission on key vault is still required. For larger clusters, you may want to subdivide the secrets by namespace or script an update.

kubectl get secrets --all-namespaces -o json | kubectl replace -f -