Rotate a customer-managed etcd encryption key on an Azure Red Hat OpenShift with hosted control planes cluster (preview)

You can rotate the customer-managed Key Management Service (KMS) key version on your Azure Red Hat OpenShift with hosted control planes (ARO HCP) cluster to meet your organization's security compliance requirements. After you update the key version, the system automatically re-encrypts all existing etcd data with the new key.

Important

If you use Azure Key Vault autorotation policies, new key versions created by the autorotation policy don't automatically propagate to your cluster. After each autorotation event, you must still update the cluster's KMS key version by following the steps in this article.

Prerequisites

  • An Azure Red Hat OpenShift with hosted control planes cluster running OpenShift 4.22 or later.
  • Azure CLI version 2.67.0 or later. To install or update, see Install the Azure CLI.
  • No cluster upgrade is in progress.
  • No re-encryption is currently in progress.

Set environment variables

Set the following environment variables if you haven't already:

SUBSCRIPTION=$(az account show --query id --output tsv)
CUSTOMER_RG_NAME="<resource-group-name>"
CLUSTER_NAME="<cluster-name>"

Grant yourself access to the key vault

When you create an Azure Red Hat OpenShift with hosted control planes cluster, the cluster's KMS managed identity gets access to the key vault, but the cluster administrator doesn't. To create new key versions for key rotation, you must assign yourself the Key Vault Crypto Officer role on the key vault.

  1. Get your Azure AD object ID:

    OBJECT_ID=$(az ad signed-in-user show --query id -o tsv)

  2. Get the key vault name:

    KEY_VAULT_NAME=$(az keyvault list \
  --resource-group ${CUSTOMER_RG_NAME} \
  --query '[].name' -o tsv)

    Note

    If your key vault is in a different resource group than your cluster, replace ${CUSTOMER_RG_NAME} with the resource group that contains the key vault.

  3. Assign the Key Vault Crypto Officer role to yourself:

    az role assignment create \
  --role "Key Vault Crypto Officer" \
  --assignee ${OBJECT_ID} \
  --scope /subscriptions/${SUBSCRIPTION}/resourceGroups/${CUSTOMER_RG_NAME}/providers/Microsoft.KeyVault/vaults/${KEY_VAULT_NAME}

    Note

    It can take up to two minutes for role assignments to propagate.

Create a new key version in Azure Key Vault

  1. Create a new key version.

    az keyvault key create \
  --vault-name <vault-name> \
  --name <key-name>

  2. Record the new key version identifier from the output. The version is the last segment of the kid (key identifier) URL.

    For example, if the kid is https://myvault.vault.azure.net/keys/mykey/abc123def456, the version is abc123def456.

Update the cluster encryption key version

After you create a new key version in Azure Key Vault, update your cluster to use it. The cluster doesn't detect new key versions automatically - you must update it manually. After the update, the system automatically re-encrypts all existing etcd data with the new key.

az aro hcp cluster update \
  --name "${CLUSTER_NAME}" \
  --resource-group "${CUSTOMER_RG_NAME}" \
  --set properties.etcd.dataEncryption.customerManaged.kms.activeKey.version="<new-key-version>"

Replace <new-key-version> with the version identifier you recorded in the previous step.

Important

Update only the key version. All other KMS encryption fields (name, vaultName, visibility, encryptionType, keyManagementMode) are immutable after cluster creation.

The following example output shows a successful update. The provisioningState value of Updating means the request was accepted and re-encryption has started.

{
  "id": "/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/<cluster-name>",
  "name": "<cluster-name>",
  "type": "microsoft.redhatopenshift/hcpopenshiftclusters",
  "properties": {
    "provisioningState": "Updating",
    "etcd": {
      "dataEncryption": {
        "customerManaged": {
          "kms": {
            "activeKey": {
              "version": "<new-key-version>"
            }
          }
        }
      }
    }
  }
}

Verify that key rotation is complete

After you update the key version, the cluster stays in the Updating state while the system re-encrypts all existing etcd data with the new key. During re-encryption, the old key is still in use to decrypt data that hasn't been re-encrypted yet.

Warning

You must wait for re-encryption to complete before performing any of the following operations:

  • Do not deactivate or delete the old key version in Azure Key Vault. The old key is still needed to read data that hasn't been re-encrypted yet. Removing it before re-encryption completes causes permanent data loss.
  • Do not initiate another key rotation. The system blocks concurrent rotations. Wait for re-encryption to complete before starting another rotation.
  • Do not take etcd backups. A backup taken during re-encryption might contain data encrypted with a mix of old and new keys, which complicates restoration.

Check the cluster's provisioning state to verify that re-encryption is complete.

az aro hcp cluster show \
  --name "${CLUSTER_NAME}" \
  --resource-group "${CUSTOMER_RG_NAME}" \
  --query "properties.provisioningState" --output tsv

The provisioning state indicates the current status of the operation:

  • Updating - Re-encryption is in progress. The old key is still in use to decrypt data that isn't re-encrypted yet.
  • Succeeded - Key rotation is complete and all etcd data is encrypted with the new key version.
  • Failed - The operation failed. Check the cluster's status for more details.

Disable the old key version (optional)

After you rotate the key, disable the old key version in Azure Key Vault. When you disable the old key version, you prevent it from being used for any cryptographic operations.

Tip

Disable the old key version instead of deleting it. You can re-enable a disabled key if you need to restore access during troubleshooting.

To disable the old key version, use the following steps:

az keyvault key set-attributes \
  --vault-name <vault-name> \
  --name <key-name> \
  --version <old-key-version> \
  --enabled false

Related content

  • Last updated on