Share via

Azure AKS/Fileshare: Kubernetes pod can't mount pv using pvc and azure-file storage class

Mateusz Pawlik 6 Reputation points
2021-08-04T12:53:38.713+00:00

On AKS cluster we can't use autmatically created azurefile storage class. Azure-disk storage class is working without problems. PVC is bound successfully but during pod initialization we get this error: 

Warning  FailedMount  1s    kubelet            MountVolume.SetUp failed for volume "pvc-aa8ebcff-05a1-4395-9d82-6fcde7a400a6" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/07cba7ea-7a25-4662-81b6-26df37a2b7f4/volumes/kubernetes.io~azure-file/pvc-aa8ebcff-05a1-4395-9d82-6fcde7a400a6 --scope -- mount -t cifs -o actimeo=30,mfsymlinks,file_mode=0777,dir_mode=0777,vers=3.0,gid=1337,<masked> //<sa-name>.file.core.windows.net/kubernetes-dynamic-pvc-aa8ebcff-05a1-4395-9d82-6fcde7a400a6 /var/lib/kubelet/pods/07cba7ea-7a25-4662-81b6-26df37a2b7f4/volumes/kubernetes.io~azure-file/pvc-aa8ebcff-05a1-4395-9d82-6fcde7a400a6
Output: Running scope as unit: run-r689f9606e1ba48698b00aed3e01e03d3.scope
mount error(2): No such file or directory
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)

We were trying both methods described in official Azure docs:

  • Manually create and use a volume with Azure Files share in Azure Kubernetes Service (AKS)
  • Dynamically create and use a persistent volume with Azure Files in Azure Kubernetes Service (AKS)
  • it seems like not connectivity problem because I can connect to file share services from pod and node: nc -zvw3 <sa-name>.file.core.windows.net 445
  • cifs vers=2.1 also tested and not working
  • I was trying do debug nodes with kubectl debug, and it seems like there is no folder to mount this file system

Kubernetes version:
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.17", GitCommit:"68e3e47dad98b808ba41087c0c62e80cd4fcec36", GitTreeState:"clean", BuildDate:"2021-05-25T02:41:29Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

Azure Files
Azure Files
An Azure service that offers file shares in the cloud.
1,425 questions
Azure Kubernetes Service
Azure Kubernetes Service
An Azure service that provides serverless Kubernetes, an integrated continuous integration and continuous delivery experience, and enterprise-grade security and governance.
2,456 questions

4 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. deherman-MSFT 38,021 Reputation points Microsoft Employee Moderator
    2021-08-05T18:36:34.177+00:00

    @Mateusz Pawlik
    I believe this is related to a known issue with the kernel version. Can you try upgrading to the latest kernel or reverting to see if it works for you? If not please let me know and we can investigate this further.

    1 person found this answer helpful.
    0 comments
  2. Reza Manouchehri 21 Reputation points
    2021-08-05T06:54:23.01+00:00

    We have the same issue. We can create dynamically Azure disk but we can't use Azure file.
    The PVC is created and is bound but the pod can't mount it.

    We followed the below link:
    azure-files-dynamic-pv

    And below is the error:

    MountVolume.SetUp failed for volume "pvc-cf416a30-11c3-4f42-b875-7c796b2d4ded" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/ec2f6cf0-8e25-4ab4-a3f9-df87808091db/volumes/kubernetes.io~azure-file/pvc-cf416a30-11c3-4f42-b875-7c796b2d4ded --scope -- mount -t cifs -o dir_mode=0777,file_mode=0777,gid=1000,uid=1000,vers=3.0,actimeo=30,mfsymlinks,<masked> //ff82fd59690354552a80202.file.core.windows.net/kubernetes-dynamic-pvc-cf416a30-11c3-4f42-b875-7c796b2d4ded /var/lib/kubelet/pods/ec2f6cf0-8e25-4ab4-a3f9-df87808091db/volumes/kubernetes.io~azure-file/pvc-cf416a30-11c3-4f42-b875-7c796b2d4ded Output: Running scope as unit: run-rb1e631bebd804a47b40031b569e2641b.scope mount error(2): No such file or directory Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) 16m Warning FailedMount pod/mypod Unable to attach or mount volumes: unmounted volumes=[volume], unattached volumes=[volume default-token-nljrj]: timed out waiting for the condition

    Kubernetes Version:
    Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.11", GitCommit:"c18ff7bd5227b42c617cf371cf315d0423e91d5e", GitTreeState:"clean", BuildDate:"2021-05-25T17:38:04Z", GoVersion:"go1.15.12", Compiler:"gc", Platform:"linux/amd64"}

    Node Size:
    Standard_DS3_v2

    Node Image:
    AKSUbuntu-1804gen2containerd-2021.05.19
    core: 5.4.0-1047-azure

    0 comments
  3. Reza Manouchehri 21 Reputation points
    2021-08-06T13:57:53.187+00:00

    By upgrading node images my issue was solved.

    I was wrong.
    The issue was because of Firewall.
    You need to open port 445 if you have a firewall in place.

  4. Mateusz Pawlik 6 Reputation points
    2021-08-09T11:34:26.54+00:00

    It seems in our case it's also a connectivity issue. For the first time, I had to badly check connectivity to our storage account. When I check the second time it appears that port 445 is blocked by the security group. I verified the connection issues this way: 

    kubectl debug node/<NODE_NAME> -it --image=mcr.microsoft.com/aks/fundamental/base-ubuntu:v0.0.11
# inside shell
if timeout 5 bash -c '</dev/tcp/<STORAGE_ACCOUNT_NAME>.file.core.windows.net/445 &>/dev/null'; then   echo "Port is open"; else   echo "Port is closed"; fi
    0 comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.