AKS PVC is not attached

Question

Hi folks.
I have one AKS cluster disk driver enabled and autoscale enabled.
I have deployed a statefulset and it triggered ndoe scaling up. Then a pod of this statefulset has been scheduled on the new node. The PVC and PV creaed and all are in Bound status. But it is not attached to the pod.
Here is the pod describe result.

  Warning  FailedMount         7m15s (x5 over 27m)  kubelet                  Unable to attach or mount volumes: unmounted volumes=[state], unattached volumes=[kube-api-access-tl55s state]: timed out waiting for the condition
  Warning  FailedMount         31s (x11 over 34m)   kubelet                  Unable to attach or mount volumes: unmounted volumes=[state], unattached volumes=[state kube-api-access-tl55s]: timed out waiting for the condition
  Warning  FailedAttachVolume  13s (x13 over 34m)   attachdetach-controller  AttachVolume.Attach failed for volume "pvc-27274f7e-9690-45e0-8ada-6e031b51d07f" : timed out waiting for external-attacher of disk.csi.azure.com CSI driver to attach volume /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/mc_xxx_yyy-aks_eastus/providers/Microsoft.Compute/disks/pvc-27274f7e-9690-45e0-8ada-6e031b51d07f

I checkd the azuredisk container log of csi-azuredisk-node pods but there is neither error nor warn log.
All nodes are ready and I checked the cluster's node resource group and there is disk volume created correctly. Also all instances are healthy in VM scale set but there is no mounted volume in the instances.
I tried to find any related logs from AKS log viewer but I am not sure how to check external-attacher's log.

I tried several times to create PV but same result.

I used the default storage class managed-csi and the k8s version is 1.26.3.

THanks.

Answer 1

@shiva patpi thanks for your answers.

I fixed this issue finally.
Set your cluster name as CLUSETER_NAME variable in the following script.

AKS_SP_ID=$(az aks show --name ${CLUSTER_NAME} --query identity.principalId --output tsv)
AKS_SP_APP_ID=$(az ad sp show --id ${AKS_SP_ID} --query appId --output tsv)
az role assignment create --assignee ${AKS_MI_ID} --scope ${AKS_SP_APP_ID} --role Contributor

Answer 2

@Jose Barato I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Thanks to Shiva for helping with the troubleshooting.

Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to "Accept " the answer. Accepted answers show up at the top, resulting in improved discoverability for others.

Issue: AKS PVC is not attached to the pod

Solution: Customer shared - Setting cluster name as CLUSTER_NAME variable in the following script, resolved the issue.

AKS_SP_ID=$(az aks show --name ${CLUSTER_NAME} --query identity.principalId --output tsv) AKS_SP_APP_ID=$(az ad sp show --id ${AKS_SP_ID} --query appId --output tsv) az role assignment create --assignee ${AKS_MI_ID} --scope ${AKS_SP_APP_ID} --role Contributor

If your issue remains unresolved or have further questions, please let us know in the comments how we can assist. We are here to help you and strive to make your experience better and greatly value your feedback.

Answer 3

@Jose Barato

Can you validate if you are making use of DiskEncryptionSet ?

https://learn.microsoft.com/en-us/azure/aks/azure-disk-customer-managed-keys

If yes, make sure the corresponding AKS Managed Identity has got read permission of DiskEncryptionSet.

Add the required role assignment. As per our documentation: https://docs.microsoft.com/en-us/azure/aks/azure-disk-customer-managed-keys#encrypt-your-aks-cluster-data-diskoptional

Regards,

Shiva.