VolumeResizeFailed Event for file.csi.azure.com volume

Question

Hello! I keep getting VolumeResizeFailed event in my Kubernetes cluster.

I suspect this problem started when this volume run out of quota size. After the incident I increased quota for affected File Share from 100GB to 300GB. And updated PVC accordingly. Even restarted pods that were using the PVC, but this event keeps popping up even for new pods:

NodeExpandVolume.NodeExpandVolume failed for volume "pvc-71f2c2fa-71c0-44f9-916a-2be92a3a068d" : Expander.NodeExpand found CSI plugin kubernetes.io/csi/file.csi.azure.com to not support node expansion

Kubernetes version for control plane and Nodes: v1.24.3 (Ubuntu 18.04.6 LTS / 5.4.0-1094-azure)

Besides the annoying event, everything seems to be working fine.

Perhaps CSIDriver needs to be updated somehow?

---

PVC:

# k describe pvc <OMITTED> -n <OMITTED>
Name:          <OMITTED>
Namespace:     <OMITTED>
StorageClass:  azurefile
Status:        Bound
Volume:        pvc-71f2c2fa-71c0-44f9-916a-2be92a3a068d
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: file.csi.azure.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      300Gi
Access Modes:  ROX
VolumeMode:    Filesystem
Used By:       <OMITTED>
Events:        <none>

PV:

k get pv pvc-71f2c2fa-71c0-44f9-916a-2be92a3a068d -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: file.csi.azure.com
  creationTimestamp: "2022-05-31T11:02:40Z"
  finalizers:
  - kubernetes.io/pv-protection
  - external-attacher/file-csi-azure-com
  name: pvc-71f2c2fa-71c0-44f9-916a-2be92a3a068d
  resourceVersion: "275913721"
  uid: b1083355-b1f2-4bbe-b99d-2c8a7a9ea277
spec:
  accessModes:
  - ReadOnlyMany
  capacity:
    storage: 300Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: <OMITTED>
    namespace: <OMITTED>
    resourceVersion: "50352907"
    uid: 71f2c2fa-71c0-44f9-916a-2be92a3a068d
  csi:
    driver: file.csi.azure.com
    volumeAttributes:
      csi.storage.k8s.io/pv/name: pvc-71f2c2fa-71c0-44f9-916a-2be92a3a068d
      csi.storage.k8s.io/pvc/name: <OMITTED>
      csi.storage.k8s.io/pvc/namespace: <OMITTED>
      secretnamespace: <OMITTED>
      skuName: Standard_LRS
      storage.kubernetes.io/csiProvisionerIdentity: 1653913219296-8081-file.csi.azure.com
    volumeHandle: mc_<OMITTED>#<OMITTED>#pvc-71f2c2fa-71c0-44f9-916a-2be92a3a068d###<OMITTED>
  mountOptions:
  - mfsymlinks
  - actimeo=30
  persistentVolumeReclaimPolicy: Delete
  storageClassName: azurefile
  volumeMode: Filesystem
status:
  phase: Bound

SC:

# k describe sc azurefile
Name:                  azurefile
IsDefaultClass:        No
Annotations:           <none>
Provisioner:           file.csi.azure.com
Parameters:            skuName=Standard_LRS
AllowVolumeExpansion:  True
MountOptions:
  mfsymlinks
  actimeo=30
ReclaimPolicy:      Delete
VolumeBindingMode:  Immediate
Events:             <none>

CSIDriver:

# k describe csidriver file.csi.azure.com    
Name:         file.csi.azure.com
Namespace:    
Labels:       addonmanager.kubernetes.io/mode=Reconcile
              kubernetes.io/cluster-service=true
Annotations:  csiDriver: v1.24.0
              snapshot: v5.0.1
API Version:  storage.k8s.io/v1
Kind:         CSIDriver
Metadata:
  Creation Timestamp:  2022-06-30T12:03:56Z
  Managed Fields:
    API Version:  storage.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:csiDriver:
          f:kubectl.kubernetes.io/last-applied-configuration:
          f:snapshot:
        f:labels:
          .:
          f:addonmanager.kubernetes.io/mode:
          f:kubernetes.io/cluster-service:
      f:spec:
        f:attachRequired:
        f:fsGroupPolicy:
        f:podInfoOnMount:
        f:requiresRepublish:
        f:storageCapacity:
        f:volumeLifecycleModes:
          .:
          v:"Ephemeral":
          v:"Persistent":
    Manager:         kubectl-client-side-apply
    Operation:       Update
    Time:            2022-06-30T12:03:56Z
  Resource Version:  266738744
  UID:               40d3049b-9c03-4e4c-82ea-a93e1bf42f21
Spec:
  Attach Required:     false
  Fs Group Policy:     ReadWriteOnceWithFSType
  Pod Info On Mount:   true
  Requires Republish:  false
  Storage Capacity:    false
  Volume Lifecycle Modes:
    Persistent
    Ephemeral
Events:  <none>

Answer 1

Hi @Guntiss

Thanks for reaching Microsoft Q&A

Please delete your Deamonset "csi-azurefile-node" inside of the kube-system namespace and also the Storageclass "azurefile" and wait both for around 5min to be recreated again. After, scale your application deployment to 0 and then back to the required replicas to get running. Check then if the msg will be popping up.

Hope this helps. Please "Accept as Answer" if it helped, so that it can help others in the community looking for help on similar topics.