Share via

Pods unable to mount persistent volumes on some nodes

Fayaaz Ahmed 1 Reputation point
2021-08-16T13:43:37.2+00:00

We have had a problem with pods restarting and not being able to mount persistent volumes they were able to before. The pods get stuck creating with: 

error "Unable to attach or mount volumes: timed out waiting for the condition"

We have found some node pools work while others do not - this has caused some specific parts of our system running on those nodes to fail when they were deployed and restarted. Running Kubernetes v1.21

Azure Kubernetes Service
Azure Kubernetes Service
An Azure service that provides serverless Kubernetes, an integrated continuous integration and continuous delivery experience, and enterprise-grade security and governance.
2,456 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. shiva patpi 13,366 Reputation points Microsoft Employee Moderator
    2021-08-17T23:10:24.657+00:00

    Hello @Fayaaz Ahmed ,
    This happens sometimes due to container runtime not properly cleaning up the resources which are not needed.
    There might be couple of pending operations in the backend , do you still see the mount failures ?

    Can you please get the output of : (Check the detailed error message in that output)

    • kubectl describe pod <podname>
    • kubectl get pvc

    Also check the kubelet logs on the node where you are experiencing this issue .
    Check the status of CSI-AzureDisk pods. Basically those PODS are responsible to ensure that disk attach/detach from the cluster.

    Some times PVC takes time to attach to the volume due to application issues : https://github.com/kubernetes/kubernetes/issues/69699
    Also check out the work arounds mentioned at https://github.com/Azure/AKS/issues/1663

    Kindly let us know if you need additional help

    Regards,
    Shiva.

    1 person found this answer helpful.
    0 comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.