Forever Pending PVC in AKS Automatic (preview)

Question

I'm having some trouble with a PVC in Azure Kubernetes Automatic.

I have a simple PVC manifest like this:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shared-data-pvc
  namespace: test
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: default
  resources:
    requests:
      storage: 1Gi

Then I create a simple test pod that uses that PVC so that it has a consumer and that will trigger the volume creation:

kind: Pod
apiVersion: v1
metadata:
  name: nginx-azuredisk
spec:
  nodeSelector:
    kubernetes.io/os: linux
  containers:
    - image: mcr.microsoft.com/mirror/docker/library/nginx:1.23
      name: nginx-azuredisk
      command:
        - "/bin/sh"
        - "-c"
        - while true; do echo $(date) >> /mnt/azuredisk/outfile; sleep 1; done
      volumeMounts:
        - name: azuredisk01
          mountPath: "/mnt/azuredisk"
          readOnly: false
  volumes:
    - name: azuredisk01
      persistentVolumeClaim:
        claimName: shared-data-pvc

This all seem very simple but the PVC stays pending forever.

  kubectl get pvc 

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
shared-data-pvc Pending default <unset> 17m

  kubectl describe pvc

Name:          shared-data-pvc
Namespace:     test
StorageClass:  default
Status:        Pending
Volume:
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner: disk.csi.azure.com
               volume.kubernetes.io/selected-node: aks-nodepool01-97879719-vmss00000i
               volume.kubernetes.io/storage-provisioner: disk.csi.azure.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Used By:       nginx-azuredisk
Events:
  Type     Reason                Age                   From                                                                                               Message
  ----     ------                ----                  ----                                                                                               -------
  Normal   WaitForPodScheduled   19m                   persistentvolume-controller                                                                        waiting for pod nginx-azuredisk to be scheduled
  Warning  ProvisioningFailed    14m (x2 over 18m)     disk.csi.azure.com_csi-azuredisk-controller-597b7fd7f8-r4dbm_70a9a1d9-1407-437e-b044-d824fabf7b07  failed to provision volume with StorageClass "default": rpc error: code = Internal desc = context deadline exceeded
  Normal   ExternalProvisioning  4m11s (x63 over 19m)  persistentvolume-controller                                                                        Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Normal   Provisioning          117s (x11 over 19m)   disk.csi.azure.com_csi-azuredisk-controller-597b7fd7f8-r4dbm_70a9a1d9-1407-437e-b044-d824fabf7b07  External provisioner is provisioning volume for claim "mitre/mitre-shared-data-pvc"
  Warning  ProvisioningFailed    87s (x9 over 18m)     disk.csi.azure.com_csi-azuredisk-controller-597b7fd7f8-r4dbm_70a9a1d9-1407-437e-b044-d824fabf7b07  failed to provision volume with StorageClass "default": rpc error: code = DeadlineExceeded desc = context deadline exceeded

It sits like this forever. The Azure portal also shows it as pending

Answer 1

Hi Sapan Ganguly,

While WaitForFirstConsumer mode which will delay the binding and provisioning of a PersistentVolume until a Pod using the PersistentVolumeClaim is created.

Check the PersistentVolume status using kubectl get command

Delete the PVC and then the StorageClass, change the volumeBindingMode value in the StorageClass manifest file to “Immediate,” create the two objects again and check the status.

Afterthat try to create a pod to consume that volume claim.

Then, check the status of PVC and PV using kubectl command to see the status as bound.

Ensure that your Kubernetes cluster has sufficient permissions and quotas to provision the requested storage: kubectl get resourcequotas

And also try to verify node selector is correctly configured to schedule the pod.

If the information is helpful, please click on "Upvote"

If you have any queries, please do let us know, we will help you.

Thank You.