AKS File Share Persistent Mounting with Managed Identity - Having issue after key Rotation

Question

AKS File Share Persistent Mounting with Managed Identity - Having issue after key Rotation

Kalirajan Natarajan 6

Mounted Azure File shares in AKS deployments using Cluster UAMI with Reader & Storage account key operator service role. It was successfully mounted in all the POD replicas and able to create the files/list all the files of Azure file share from a pod. But, it is not working after key rotation. Also, I tried to create a new deployment, storage class, PVC. Still, facing permission issues while PODs are getting created.

Stage 1: (First Time Process)
Created AKS Cluster , Storage File share, User managed Identity
Assigned the UAMI to Cluster and provided the Reader & Storage account key operator service roles in the new storage scope.
Created new Custom Storage class, PVC, deployments.
Result: All functionalities were working as expected.

Stage 2: (Failure Process)
Created new deployment after key rotation as existing PODs were unable to access the Azure File Share. Permission issue.
Then, Created a new Storage Class/PVC/deployment - Still the same permission issue.

Error:
default 13s Warning FailedMount pod/myapp-deploymentkey1-67465fb9df-9xcrz MountVolume.SetUp failed for volume "xx" : mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t cifs -o file_mode=0777,dir_mode=0777,vers=3.0,actimeo=30,mfsymlinks,<masked> //{StorageName}.file.core.windows.net/sample1 /var/lib/kubelet/pods/xx8/volumes/kubernetes.io~azure-file/pvc-cxx
Output: mount error(13): Permission denied
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
default 13s Warning FailedMount pod/myapp-deploymentkey1-67465fb9df-jwmcc MountVolume.SetUp failed for volume "xx" : mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t cifs -o file_mode=0777,dir_mode=0777,vers=3.0,actimeo=30,mfsymlinks,<masked> //{StorageName}.file.core.windows.net/sample1 /var/lib/kubelet/pods/xxx/volumes/kubernetes.io~azure-file/pvc-xx
Output: mount error(13): Permission denied

Kindly help to resolve this.

SRIJIT-BOSE-MSFT 4,346 Reputation points Microsoft Employee Moderator

2021-09-22T09:11:48.43+00:00
@Kalirajan Natarajan , Thank you for sharing your concern.

Please find the general causes and respective solutions for "Mount error(13): Permission denied" when you mount an Azure file share
Can you please SSH into the AKS node where the consumer pod is currently scheduled and check if the following yields a successful result?

chroot /host sudo mkdir /mnt/MyAzureFileShare sudo mount -t cifs //$STORAGEACCT.file.core.windows.net/myshare /mnt/MyAzureFileShare -o vers=3.0,username=$STORAGEACCT,password=$STORAGEKEY,dir_mode=0777,file_mode=0777,serverino

You might also want to check this.