Use the Azure Key Vault Provider for Secrets Store CSI Driver in an AKS cluster

Article
02/23/2023
10 minutes to read

The Azure Key Vault Provider for Secrets Store CSI Driver allows for the integration of an Azure key vault as a secret store with an Azure Kubernetes Service (AKS) cluster via a CSI volume.

Features

Mounts secrets, keys, and certificates to a pod by using a CSI volume
Supports CSI inline volumes
Supports mounting multiple secrets store objects as a single volume
Supports pod portability with the SecretProviderClass CRD
Supports Windows containers
Syncs with Kubernetes secrets
Supports autorotation of mounted contents and synced Kubernetes secrets

Limitations

A container using subPath volume mount won't receive secret updates when it's rotated. For more information, see Secrets Store CSI Driver known limitations.

Prerequisites

If you don't have an Azure subscription, create a free account before you begin.
Check that your version of the Azure CLI is 2.30.0 or later. If it's an earlier version, install the latest version.
If you're restricting Ingress to the cluster, make sure ports 9808 and 8095 are open.
The minimum recommended Kubernetes version is based on the rolling Kubernetes version support window. Make sure you're running version N-2 or later.

Create an AKS cluster with Azure Key Vault Provider for Secrets Store CSI Driver support

Create an Azure resource group.

az group create -n myResourceGroup -l eastus2

Create an AKS cluster with Azure Key Vault Provider for Secrets Store CSI Driver capability using the az aks create command with the azure-keyvault-secrets-provider add-on.
```
az aks create -n myAKSCluster -g myResourceGroup --enable-addons azure-keyvault-secrets-provider
```
A user-assigned managed identity, named azureKeyvaultSecretsProvider, is created by the add-on to access Azure resources. The following example uses this identity to connect to the Azure key vault where the secrets will be stored, but you can also use other identity access methods. Take note of the identity's clientId in the output.
```
...,
 "addonProfiles": {
    "azureKeyvaultSecretsProvider": {
      ...,
      "identity": {
        "clientId": "<client-id>",
        ...
      }
    }
```

Upgrade an existing AKS cluster with Azure Key Vault Provider for Secrets Store CSI Driver support

Upgrade an existing AKS cluster with Azure Key Vault Provider for Secrets Store CSI Driver capability using the az aks enable-addons command with the azure-keyvault-secrets-provider add-on. The add-on creates a user-assigned managed identity you can use to authenticate to your Azure key vault.
```
az aks enable-addons --addons azure-keyvault-secrets-provider --name myAKSCluster --resource-group myResourceGroup
```

Verify the Azure Key Vault Provider for Secrets Store CSI Driver installation

Verify the installation is finished using the kubectl get pods command to list all pods that have the secrets-store-csi-driver and secrets-store-provider-azure labels in the kube-system namespace, and ensure that your output looks similar to the following output:

kubectl get pods -n kube-system -l 'app in (secrets-store-csi-driver,secrets-store-provider-azure)'

NAME                                     READY   STATUS    RESTARTS   AGE
aks-secrets-store-csi-driver-4vpkj       3/3     Running   2          4m25s
aks-secrets-store-csi-driver-ctjq6       3/3     Running   2          4m21s
aks-secrets-store-csi-driver-tlvlq       3/3     Running   2          4m24s
aks-secrets-store-provider-azure-5p4nb   1/1     Running   0          4m21s
aks-secrets-store-provider-azure-6pqmv   1/1     Running   0          4m24s
aks-secrets-store-provider-azure-f5qlm   1/1     Running   0          4m25s

Verify that each node in your cluster's node pool has a Secrets Store CSI Driver pod and a Secrets Store Provider Azure pod running.

Create or use an existing Azure key vault

In addition to an AKS cluster, you'll need an Azure key vault resource that stores the secret content.

Create an Azure key vault using the az keyvault create command. The name of the key vault must be globally unique.
```
az keyvault create -n <keyvault-name> -g myResourceGroup -l eastus2
```
Your Azure key vault can store keys, secrets, and certificates. In this example, use the az keyvault secret set command to set a plain-text secret called ExampleSecret.
```
az keyvault secret set --vault-name <keyvault-name> -n ExampleSecret --value MyAKSExampleSecret
```
Take note of the following properties for use in the next section:
- The name of the secret object in the key vault
- The object type (secret, key, or certificate)
- The name of your Azure key vault resource
- The Azure tenant ID that the subscription belongs to

Provide an identity to access the Azure key vault

The Secrets Store CSI Driver allows for the following methods to access an Azure key vault:

An Azure Active Directory pod identity (preview)
An Azure Active Directory workload identity (preview)
A user-assigned or system-assigned managed identity

Follow the instructions in Provide an identity to access the Azure Key Vault Provider for Secrets Store CSI Driver for your chosen method.

Important

The rest of the examples on this page require that you've followed the instructions in Provide an identity to access the Azure Key Vault Provider for Secrets Store CSI Driver, chosen one of the identity methods, and configured a SecretProviderClass. Come back to this page after completing those steps.

Validate the secrets

After the pod starts, the mounted content at the volume path that you specified in your deployment YAML is available.

Use the following commands to validate your secrets and print a test secret.

## show secrets held in secrets-store
kubectl exec busybox-secrets-store-inline -- ls /mnt/secrets-store/

## print a test secret 'ExampleSecret' held in secrets-store
kubectl exec busybox-secrets-store-inline -- cat /mnt/secrets-store/ExampleSecret

Obtain certificates and keys

The Azure Key Vault design makes sharp distinctions between keys, secrets, and certificates. The Key Vault service’s certificates features were designed to make use of its key and secret capabilities. When a key vault certificate is created, an addressable key and secret are also created with the same name. The key allows key operations, and the secret allows the retrieval of the certificate value as a secret.

A key vault certificate also contains public x509 certificate metadata. The key vault stores both the public and private components of your certificate in a secret. You can obtain each individual component by specifying the objectType in SecretProviderClass. The following table shows which objects map to the various resources associated with your certificate:

Object	Return value	Returns entire certificate chain
`key`	The public key, in Privacy Enhanced Mail (PEM) format	N/A
`cert`	The certificate, in PEM format	No
`secret`	The private key and certificate, in PEM format	Yes

Disable the Azure Key Vault Provider for Secrets Store CSI Driver on an existing AKS cluster

Note

Before you disable the add-on, ensure that no SecretProviderClass is in use. Trying to disable the add-on while SecretProviderClass exists will result in an error.

Disable the Azure Key Vault Provider for Secrets Store CSI Driver capability in an existing cluster using the az aks disable-addons command with the azure-keyvault-secrets-provider add-on.
```
az aks disable-addons --addons azure-keyvault-secrets-provider -g myResourceGroup -n myAKSCluster
```

Note

If the add-on is disabled, existing workloads will have no issues and will not see any updates in the mounted secrets. If the pod restarts or a new pod is created as part of scale-up event, the pod will fail to start because the driver is no longer running.

More configuration options

Enable and disable autorotation

Note

When the Azure Key Vault Provider for Secrets Store CSI Driver is enabled, it updates the pod mount and the Kubernetes secret that's defined in the secretObjects field of SecretProviderClass. It does so by polling for changes periodically, based on the rotation poll interval you've defined. The default rotation poll interval is 2 minutes.

Note

When a secret is updated in an external secrets store after initial pod deployment, the Kubernetes Secret and the pod mount will be periodically updated depending on how the application consumes the secret data.

Mount the Kubernetes Secret as a volume: Use the autorotation and Sync K8s secrets features of Secrets Store CSI Driver. The application will need to watch for changes from the mounted Kubernetes Secret volume. When the Kubernetes Secret is updated by the CSI Driver, the corresponding volume contents are automatically updated.

Application reads the data from the container’s filesystem: Use the rotation feature of Secrets Store CSI Driver. The application will need to watch for the file change from the volume mounted by the CSI driver.

Use the Kubernetes Secret for an environment variable: Restart the pod to get the latest secret as an environment variable. Use a tool such as Reloader to watch for changes on the synced Kubernetes Secret and perform rolling upgrades on pods.

Enable autorotation on a new AKS cluster

Enable autorotation of secrets using the enable-secret-rotation parameter when you create your cluster.

az aks create -n myAKSCluster2 -g myResourceGroup --enable-addons azure-keyvault-secrets-provider --enable-secret-rotation

Enable autorotation on an existing AKS cluster

Update an existing cluster to enable autorotation of secrets using the az aks addon update command and the enable-secret-rotation parameter.
```
az aks addon update -g myResourceGroup -n myAKSCluster2 -a azure-keyvault-secrets-provider --enable-secret-rotation
```

Specify a custom rotation interval

Specify a custom rotation interval using the rotation-poll-interval parameter.

az aks addon update -g myResourceGroup -n myAKSCluster2 -a azure-keyvault-secrets-provider --enable-secret-rotation --rotation-poll-interval 5m

Disable autorotation

To disable autorotation, first disable the addon. Then, re-enable the addon without the enable-secret-rotation parameter.

# disable the addon
az aks addon disable -g myResourceGroup -n myAKSCluster2 -a azure-keyvault-secrets-provider

# re-enable the addon without the `enable-secret-rotation` parameter
az aks addon enable -g myResourceGroup -n myAKSCluster2 -a azure-keyvault-secrets-provider

Sync mounted content with a Kubernetes secret

Note

The YAML examples here are incomplete. You'll need to modify them to support your chosen method of access to your key vault identity. For details, see Provide an identity to access the Azure Key Vault Provider for Secrets Store CSI Driver.

You might want to create a Kubernetes secret to mirror your mounted secrets content. Your secrets will sync after you start a pod to mount them. When you delete the pods that consume the secrets, your Kubernetes secret will also be deleted.

To sync mounted content with a Kubernetes secret, use the secretObjects field when creating a SecretProviderClass to define the desired state of the Kubernetes secret, as shown in the following example.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-sync
spec:
  provider: azure                             
  secretObjects:                              # [OPTIONAL] SecretObjects defines the desired state of synced Kubernetes secret objects
  - data:
    - key: username                           # data field to populate
      objectName: foo1                        # name of the mounted content to sync; this could be the object name or the object alias
    secretName: foosecret                     # name of the Kubernetes secret object
    type: Opaque                              # type of Kubernetes secret object (for example, Opaque, kubernetes.io/tls)

Note

Make sure the objectName in the secretObjects field matches the file name of the mounted content. If you use objectAlias instead, it should match the object alias.

Set an environment variable to reference Kubernetes secrets

After creating the Kubernetes secret, you can reference it by setting an environment variable in your pod, as shown in the following example code.

Note

The example YAML demonstrates access to a secret through env variables and through volume/volumeMount. This is for illustrative purposes; a typical application would use one method or the other. However, be aware that in order for a secret to be available through env variables, it first must be mounted by at least one pod.

kind: Pod
apiVersion: v1
metadata:
  name: busybox-secrets-store-inline
spec:
  containers:
    - name: busybox
      image: k8s.gcr.io/e2e-test-images/busybox:1.29-1
      command:
        - "/bin/sleep"
        - "10000"
      volumeMounts:
      - name: secrets-store01-inline
        mountPath: "/mnt/secrets-store"
        readOnly: true
      env:
      - name: SECRET_USERNAME
        valueFrom:
          secretKeyRef:
            name: foosecret
            key: username
  volumes:
    - name: secrets-store01-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "azure-sync"

Access metrics

The Azure Key Vault Provider

Metrics are served via Prometheus from port 8898, but this port isn't exposed outside the pod by default.

Access the metrics over localhost using kubectl port-forward.

kubectl port-forward -n kube-system ds/aks-secrets-store-provider-azure 8898:8898 & curl localhost:8898/metrics

Metrics provided by the Azure Key Vault Provider for Secrets Store CSI Driver

Metric	Description	Tags
keyvault_request	The distribution of how long it took to get from the key vault	`os_type=<runtime os>`, `provider=azure`, `object_name=<keyvault object name>`, `object_type=<keyvault object type>`, `error=<error if failed>`
grpc_request	The distribution of how long it took for the gRPC requests	`os_type=<runtime os>`, `provider=azure`, `grpc_method=<rpc full method>`, `grpc_code=<grpc status code>`, `grpc_message=<grpc status message>`

The Secrets Store CSI Driver

Metrics are served from port 8095, but this port isn't exposed outside the pod by default.

Access the metrics over localhost using kubectl port-forward.

kubectl port-forward -n kube-system ds/aks-secrets-store-csi-driver 8095:8095 &
curl localhost:8095/metrics

Metrics provided by the Secrets Store CSI Driver

Metric	Description	Tags
total_node_publish	The total number of successful volume mount requests	`os_type=<runtime os>`, `provider=<provider name>`
total_node_unpublish	The total number of successful volume unmount requests	`os_type=<runtime os>`
total_node_publish_error	The total number of errors with volume mount requests	`os_type=<runtime os>`, `provider=<provider name>`, `error_type=<error code>`
total_node_unpublish_error	The total number of errors with volume unmount requests	`os_type=<runtime os>`
total_sync_k8s_secret	The total number of Kubernetes secrets synced	`os_type=<runtime os`, `provider=<provider name>`
sync_k8s_secret_duration_sec	The distribution of how long it took to sync the Kubernetes secret	`os_type=<runtime os>`
total_rotation_reconcile	The total number of rotation reconciles	`os_type=<runtime os>`, `rotated=<true or false>`
total_rotation_reconcile_error	The total number of rotation reconciles with error	`os_type=<runtime os>`, `rotated=<true or false>`, `error_type=<error code>`
total_rotation_reconcile_error	The distribution of how long it took to rotate secrets-store content for pods	`os_type=<runtime os>`

Troubleshooting

For generic troubleshooting steps, see Azure Key Vault Provider for Secrets Store CSI Driver troubleshooting.

Next steps

In this article, you learned how to use the Azure Key Vault Provider for Secrets Store CSI Driver with an AKS cluster. To learn more about the Azure Key Vault Provider for Secrets Store CSI Driver, see:

Use the Azure Key Vault Provider for Secrets Store CSI Driver in an AKS cluster

Features

Limitations

Prerequisites

Create an AKS cluster with Azure Key Vault Provider for Secrets Store CSI Driver support

Upgrade an existing AKS cluster with Azure Key Vault Provider for Secrets Store CSI Driver support

Verify the Azure Key Vault Provider for Secrets Store CSI Driver installation

Create or use an existing Azure key vault

Provide an identity to access the Azure key vault

Validate the secrets

Obtain certificates and keys

Disable the Azure Key Vault Provider for Secrets Store CSI Driver on an existing AKS cluster

More configuration options

Enable and disable autorotation

Enable autorotation on a new AKS cluster

Enable autorotation on an existing AKS cluster

Specify a custom rotation interval

Disable autorotation

Sync mounted content with a Kubernetes secret

Set an environment variable to reference Kubernetes secrets

Access metrics

The Azure Key Vault Provider

Metrics provided by the Azure Key Vault Provider for Secrets Store CSI Driver

The Secrets Store CSI Driver

Metrics provided by the Secrets Store CSI Driver

Troubleshooting

Next steps

Feedback

Additional resources

Additional resources