Edit

Manage and rotate certificates in Azure Kubernetes Service (AKS)

This article explains how to manage and rotate certificates in Azure Kubernetes Service (AKS) clusters to maintain secure communication between components and ensure compliance.

Prerequisites

  • Azure CLI version 2.0.77 or later. Check your version by using the az --version command. To install or upgrade, see Install Azure CLI.

Check cluster certificate authority (CA) certificate expiration date

Check the expiration date of the cluster CA certificate using the kubectl config view command.

kubectl config view --raw -o jsonpath="{.clusters[?(@.name == '')].cluster.certificate-authority-data}" | base64 -d | openssl x509 -text | grep -A2 Validity

Check API server certificate expiration date

Check the expiration date of the API server certificate using the following curl command:

curl https://{apiserver-fqdn} -k -v 2>&1 | grep expire

Check VMSS agent node certificate expiration date

Check the expiration date of the virtual machine scale set agent node certificate using the az vmss run-command invoke command.

az vmss run-command invoke \
    --resource-group <node-resource-group> \
    --name <vmss-name> \
    --command-id RunShellScript \
    --instance-id 1 \
    --scripts "openssl x509 -in  /var/lib/kubelet/pki/kubelet-client-current.pem -noout -enddate" \
    --query "value[0].message"

Check standalone VM certificate expiration date

Check the expiration date of a standalone VM using the az vm run-command invoke command.

az vm run-command invoke \
    --resource-group <resource-group> \
    --name <vm-name> \
    --command-id RunShellScript \
    --scripts "openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -enddate" \
    --query "value[0].message"

Manually rotate cluster CA certificates

When you rotate the cluster CA certificate, the following child certificates are also refreshed:

  • Service account (SA) tokens
  • API server certificates
  • Kubelet client certificates
  • Kubelet server certificates (if kubelet serving certificate rotation is enabled on the cluster)

Warning

Rotating your certificates using az aks rotate-certs recreates all of your nodes, virtual machine scale sets, and disks. This process can cause up to 30 minutes of downtime for your AKS cluster. If the command fails before completing, use the az aks show command to verify the status of the cluster shows Certificate Rotating. If the cluster is in a failed state, rerun az aks rotate-certs to rotate your certificates again.

  1. Connect to your cluster using the az aks get-credentials command.

    az aks get-credentials --resource-group <resource-group> --name <cluster-name>

  2. Rotate all certificates, CAs, and SAs on your cluster using the az aks rotate-certs command.

    az aks rotate-certs --resource-group <resource-group> --name <cluster-name>

  3. Verify the old certificates are no longer valid using any kubectl command, such as kubectl get nodes.

    kubectl get nodes

    If you didn't update the certificates used by kubectl, you see an error similar to the following example output:

    Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "ca")

Update kubectl client certificates

  1. Update the kubectl client certificate using the az aks get-credentials command with the --overwrite-existing flag.

    az aks get-credentials --resource-group <resource-group> --name <cluster-name> --overwrite-existing

  2. Verify the certificates are updated using the kubectl get command.

    kubectl get nodes

    Note

    If you have any services that run on top of AKS, you might need to update their certificates.

Verify kubelet serving certificate rotation is enabled

Each node with the feature enabled is automatically given the label kubernetes.azure.com/kubelet-serving-ca=cluster. Verify the labels are set by using the kubectl get nodes -L kubernetes.azure.com/kubelet-serving-ca command.

kubectl get nodes -L kubernetes.azure.com/kubelet-serving-ca

Verify kubelet goes through TLS bootstrapping process

When you enable this feature, each kubelet running your nodes goes through the serving TLS bootstrapping process.

Verify the bootstrapping process is taking place using the kubectl get command to get the current CSR objects within your cluster.

kubectl get csr --field-selector=spec.signerName=kubernetes.io/kubelet-serving

All serving CSRs are in the Approved,Issued state, which indicates the CSR was approved and issued a signed certificate. Serving CSRs have a signer name of kubernetes.io/kubelet-serving. For example:

NAME        AGE    SIGNERNAME                                    REQUESTOR                    REQUESTEDDURATION   CONDITION
csr-1ab2c   113s   kubernetes.io/kube-apiserver-client-kubelet   system:bootstrap:abcd1e      none              Approved,Issued
csr-defgh   111s   kubernetes.io/kubelet-serving                 system:node:akswinp1000000   none              Approved,Issued
csr-ij3kl   46m    kubernetes.io/kubelet-serving                 system:node:akswinp2000000   none              Approved,Issued
csr-mn4op   46m    kubernetes.io/kube-apiserver-client-kubelet   system:bootstrap:ab1cde      none              Approved,Issued

Verify kubelet is using a certificate obtained from server TLS bootstrapping

Confirm whether the node's kubelet is using a serving certificate signed by the cluster CA using the [kubectl debug][kubectl-debug] command to examine the contents of the kubelet's PKI directory.

kubectl debug node/<node> -ti --image=mcr.microsoft.com/azurelinux/base/core:3.0 -- ls -l /host/var/lib/kubelet/kubelet-server-current.pem

If a kubelet-server-current.pem symlink exists, the kubelet bootstraps and rotates its own serving certificate through the TLS bootstrapping process and is signed by the cluster CA.

Disable kubelet serving certificate rotation

  1. Disable kubelet serving certificate rotation by updating the node pool using the az aks nodepool update command and specify the tag aks-disable-kubelet-serving-certificate-rotation=true.

    az aks nodepool update \
    --cluster-name <cluster-name> \
    --resource-group <resource-group> \
    --name <node-pool-name> \
    --tags aks-disable-kubelet-serving-certificate-rotation=true

  2. Reimage your nodes using a node image upgrade or by scaling the pool to zero instances and then back up to the desired value.

Related content

For more information on AKS security, see the following articles:

  • Last updated on