Use the Azure Key Vault Provider for Secrets Store CSI Driver in an AKS cluster
The Azure Key Vault Provider for Secrets Store CSI Driver allows for the integration of an Azure key vault as a secret store with an Azure Kubernetes Service (AKS) cluster via a CSI volume.
Features
- Mounts secrets, keys, and certificates to a pod by using a CSI volume
- Supports CSI inline volumes
- Supports mounting multiple secrets store objects as a single volume
- Supports pod portability with the
SecretProviderClass
CRD - Supports Windows containers
- Syncs with Kubernetes secrets
- Supports autorotation of mounted contents and synced Kubernetes secrets
Limitations
A container using subPath volume mount won't receive secret updates when it's rotated. For more information, see Secrets Store CSI Driver known limitations.
Prerequisites
- If you don't have an Azure subscription, create a free account before you begin.
- Check that your version of the Azure CLI is 2.30.0 or later. If it's an earlier version, install the latest version.
- If you're restricting Ingress to the cluster, make sure ports 9808 and 8095 are open.
- The minimum recommended Kubernetes version is based on the rolling Kubernetes version support window. Make sure you're running version N-2 or later.
Create an AKS cluster with Azure Key Vault Provider for Secrets Store CSI Driver support
Create an Azure resource group.
az group create -n myResourceGroup -l eastus2
Create an AKS cluster with Azure Key Vault Provider for Secrets Store CSI Driver capability using the
az aks create
command with theazure-keyvault-secrets-provider
add-on.az aks create -n myAKSCluster -g myResourceGroup --enable-addons azure-keyvault-secrets-provider
A user-assigned managed identity, named
azureKeyvaultSecretsProvider
, is created by the add-on to access Azure resources. The following example uses this identity to connect to the Azure key vault where the secrets will be stored, but you can also use other identity access methods. Take note of the identity'sclientId
in the output...., "addonProfiles": { "azureKeyvaultSecretsProvider": { ..., "identity": { "clientId": "<client-id>", ... } }
Upgrade an existing AKS cluster with Azure Key Vault Provider for Secrets Store CSI Driver support
Upgrade an existing AKS cluster with Azure Key Vault Provider for Secrets Store CSI Driver capability using the
az aks enable-addons
command with theazure-keyvault-secrets-provider
add-on. The add-on creates a user-assigned managed identity you can use to authenticate to your Azure key vault.az aks enable-addons --addons azure-keyvault-secrets-provider --name myAKSCluster --resource-group myResourceGroup
Verify the Azure Key Vault Provider for Secrets Store CSI Driver installation
Verify the installation is finished using the
kubectl get pods
command to list all pods that have thesecrets-store-csi-driver
andsecrets-store-provider-azure
labels in the kube-system namespace, and ensure that your output looks similar to the following output:kubectl get pods -n kube-system -l 'app in (secrets-store-csi-driver,secrets-store-provider-azure)' NAME READY STATUS RESTARTS AGE aks-secrets-store-csi-driver-4vpkj 3/3 Running 2 4m25s aks-secrets-store-csi-driver-ctjq6 3/3 Running 2 4m21s aks-secrets-store-csi-driver-tlvlq 3/3 Running 2 4m24s aks-secrets-store-provider-azure-5p4nb 1/1 Running 0 4m21s aks-secrets-store-provider-azure-6pqmv 1/1 Running 0 4m24s aks-secrets-store-provider-azure-f5qlm 1/1 Running 0 4m25s
Verify that each node in your cluster's node pool has a Secrets Store CSI Driver pod and a Secrets Store Provider Azure pod running.
Create or use an existing Azure key vault
In addition to an AKS cluster, you'll need an Azure key vault resource that stores the secret content.
Create an Azure key vault using the
az keyvault create
command. The name of the key vault must be globally unique.az keyvault create -n <keyvault-name> -g myResourceGroup -l eastus2
Your Azure key vault can store keys, secrets, and certificates. In this example, use the
az keyvault secret set
command to set a plain-text secret calledExampleSecret
.az keyvault secret set --vault-name <keyvault-name> -n ExampleSecret --value MyAKSExampleSecret
Take note of the following properties for use in the next section:
- The name of the secret object in the key vault
- The object type (secret, key, or certificate)
- The name of your Azure key vault resource
- The Azure tenant ID that the subscription belongs to
Provide an identity to access the Azure key vault
The Secrets Store CSI Driver allows for the following methods to access an Azure key vault:
- An Azure Active Directory pod identity (preview)
- An Azure Active Directory workload identity (preview)
- A user-assigned or system-assigned managed identity
Follow the instructions in Provide an identity to access the Azure Key Vault Provider for Secrets Store CSI Driver for your chosen method.
Important
The rest of the examples on this page require that you've followed the instructions in Provide an identity to access the Azure Key Vault Provider for Secrets Store CSI Driver, chosen one of the identity methods, and configured a SecretProviderClass. Come back to this page after completing those steps.
Validate the secrets
After the pod starts, the mounted content at the volume path that you specified in your deployment YAML is available.
Use the following commands to validate your secrets and print a test secret.
## show secrets held in secrets-store kubectl exec busybox-secrets-store-inline -- ls /mnt/secrets-store/ ## print a test secret 'ExampleSecret' held in secrets-store kubectl exec busybox-secrets-store-inline -- cat /mnt/secrets-store/ExampleSecret
Obtain certificates and keys
The Azure Key Vault design makes sharp distinctions between keys, secrets, and certificates. The Key Vault service’s certificates features were designed to make use of its key and secret capabilities. When a key vault certificate is created, an addressable key and secret are also created with the same name. The key allows key operations, and the secret allows the retrieval of the certificate value as a secret.
A key vault certificate also contains public x509 certificate metadata. The key vault stores both the public and private components of your certificate in a secret. You can obtain each individual component by specifying the objectType
in SecretProviderClass
. The following table shows which objects map to the various resources associated with your certificate:
Object | Return value | Returns entire certificate chain |
---|---|---|
key |
The public key, in Privacy Enhanced Mail (PEM) format | N/A |
cert |
The certificate, in PEM format | No |
secret |
The private key and certificate, in PEM format | Yes |
Disable the Azure Key Vault Provider for Secrets Store CSI Driver on an existing AKS cluster
Note
Before you disable the add-on, ensure that no SecretProviderClass
is in use. Trying to disable the add-on while SecretProviderClass
exists will result in an error.
Disable the Azure Key Vault Provider for Secrets Store CSI Driver capability in an existing cluster using the
az aks disable-addons
command with theazure-keyvault-secrets-provider
add-on.az aks disable-addons --addons azure-keyvault-secrets-provider -g myResourceGroup -n myAKSCluster
Note
If the add-on is disabled, existing workloads will have no issues and will not see any updates in the mounted secrets. If the pod restarts or a new pod is created as part of scale-up event, the pod will fail to start because the driver is no longer running.
More configuration options
Enable and disable autorotation
Note
When the Azure Key Vault Provider for Secrets Store CSI Driver is enabled, it updates the pod mount and the Kubernetes secret that's defined in the secretObjects
field of SecretProviderClass
. It does so by polling for changes periodically, based on the rotation poll interval you've defined. The default rotation poll interval is 2 minutes.
Note
When a secret is updated in an external secrets store after initial pod deployment, the Kubernetes Secret and the pod mount will be periodically updated depending on how the application consumes the secret data.
Mount the Kubernetes Secret as a volume: Use the autorotation and Sync K8s secrets features of Secrets Store CSI Driver. The application will need to watch for changes from the mounted Kubernetes Secret volume. When the Kubernetes Secret is updated by the CSI Driver, the corresponding volume contents are automatically updated.
Application reads the data from the container’s filesystem: Use the rotation feature of Secrets Store CSI Driver. The application will need to watch for the file change from the volume mounted by the CSI driver.
Use the Kubernetes Secret for an environment variable: Restart the pod to get the latest secret as an environment variable. Use a tool such as Reloader to watch for changes on the synced Kubernetes Secret and perform rolling upgrades on pods.
Enable autorotation on a new AKS cluster
Enable autorotation of secrets using the
enable-secret-rotation
parameter when you create your cluster.az aks create -n myAKSCluster2 -g myResourceGroup --enable-addons azure-keyvault-secrets-provider --enable-secret-rotation
Enable autorotation on an existing AKS cluster
Update an existing cluster to enable autorotation of secrets using the
az aks addon update
command and theenable-secret-rotation
parameter.az aks addon update -g myResourceGroup -n myAKSCluster2 -a azure-keyvault-secrets-provider --enable-secret-rotation
Specify a custom rotation interval
Specify a custom rotation interval using the
rotation-poll-interval
parameter.az aks addon update -g myResourceGroup -n myAKSCluster2 -a azure-keyvault-secrets-provider --enable-secret-rotation --rotation-poll-interval 5m
Disable autorotation
To disable autorotation, first disable the addon. Then, re-enable the addon without the
enable-secret-rotation
parameter.# disable the addon az aks addon disable -g myResourceGroup -n myAKSCluster2 -a azure-keyvault-secrets-provider # re-enable the addon without the `enable-secret-rotation` parameter az aks addon enable -g myResourceGroup -n myAKSCluster2 -a azure-keyvault-secrets-provider
Sync mounted content with a Kubernetes secret
Note
The YAML examples here are incomplete. You'll need to modify them to support your chosen method of access to your key vault identity. For details, see Provide an identity to access the Azure Key Vault Provider for Secrets Store CSI Driver.
You might want to create a Kubernetes secret to mirror your mounted secrets content. Your secrets will sync after you start a pod to mount them. When you delete the pods that consume the secrets, your Kubernetes secret will also be deleted.
To sync mounted content with a Kubernetes secret, use the secretObjects
field when creating a SecretProviderClass
to define the desired state of the Kubernetes secret, as shown in the following example.
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: azure-sync
spec:
provider: azure
secretObjects: # [OPTIONAL] SecretObjects defines the desired state of synced Kubernetes secret objects
- data:
- key: username # data field to populate
objectName: foo1 # name of the mounted content to sync; this could be the object name or the object alias
secretName: foosecret # name of the Kubernetes secret object
type: Opaque # type of Kubernetes secret object (for example, Opaque, kubernetes.io/tls)
Note
Make sure the objectName
in the secretObjects
field matches the file name of the mounted content. If you use objectAlias
instead, it should match the object alias.
Set an environment variable to reference Kubernetes secrets
After creating the Kubernetes secret, you can reference it by setting an environment variable in your pod, as shown in the following example code.
Note
The example YAML demonstrates access to a secret through env variables and through volume/volumeMount. This is for illustrative purposes; a typical application would use one method or the other. However, be aware that in order for a secret to be available through env variables, it first must be mounted by at least one pod.
kind: Pod
apiVersion: v1
metadata:
name: busybox-secrets-store-inline
spec:
containers:
- name: busybox
image: k8s.gcr.io/e2e-test-images/busybox:1.29-1
command:
- "/bin/sleep"
- "10000"
volumeMounts:
- name: secrets-store01-inline
mountPath: "/mnt/secrets-store"
readOnly: true
env:
- name: SECRET_USERNAME
valueFrom:
secretKeyRef:
name: foosecret
key: username
volumes:
- name: secrets-store01-inline
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "azure-sync"
Access metrics
The Azure Key Vault Provider
Metrics are served via Prometheus from port 8898, but this port isn't exposed outside the pod by default.
Access the metrics over localhost using
kubectl port-forward
.kubectl port-forward -n kube-system ds/aks-secrets-store-provider-azure 8898:8898 & curl localhost:8898/metrics
Metrics provided by the Azure Key Vault Provider for Secrets Store CSI Driver
Metric | Description | Tags |
---|---|---|
keyvault_request | The distribution of how long it took to get from the key vault | os_type=<runtime os> , provider=azure , object_name=<keyvault object name> , object_type=<keyvault object type> , error=<error if failed> |
grpc_request | The distribution of how long it took for the gRPC requests | os_type=<runtime os> , provider=azure , grpc_method=<rpc full method> , grpc_code=<grpc status code> , grpc_message=<grpc status message> |
The Secrets Store CSI Driver
Metrics are served from port 8095, but this port isn't exposed outside the pod by default.
Access the metrics over localhost using
kubectl port-forward
.kubectl port-forward -n kube-system ds/aks-secrets-store-csi-driver 8095:8095 & curl localhost:8095/metrics
Metrics provided by the Secrets Store CSI Driver
Metric | Description | Tags |
---|---|---|
total_node_publish | The total number of successful volume mount requests | os_type=<runtime os> , provider=<provider name> |
total_node_unpublish | The total number of successful volume unmount requests | os_type=<runtime os> |
total_node_publish_error | The total number of errors with volume mount requests | os_type=<runtime os> , provider=<provider name> , error_type=<error code> |
total_node_unpublish_error | The total number of errors with volume unmount requests | os_type=<runtime os> |
total_sync_k8s_secret | The total number of Kubernetes secrets synced | os_type=<runtime os , provider=<provider name> |
sync_k8s_secret_duration_sec | The distribution of how long it took to sync the Kubernetes secret | os_type=<runtime os> |
total_rotation_reconcile | The total number of rotation reconciles | os_type=<runtime os> , rotated=<true or false> |
total_rotation_reconcile_error | The total number of rotation reconciles with error | os_type=<runtime os> , rotated=<true or false> , error_type=<error code> |
total_rotation_reconcile_error | The distribution of how long it took to rotate secrets-store content for pods | os_type=<runtime os> |
Troubleshooting
For generic troubleshooting steps, see Azure Key Vault Provider for Secrets Store CSI Driver troubleshooting.
Next steps
In this article, you learned how to use the Azure Key Vault Provider for Secrets Store CSI Driver with an AKS cluster. To learn more about the Azure Key Vault Provider for Secrets Store CSI Driver, see:
Feedback
Submit and view feedback for