Getting Permission denied error when copy files from mysql pod using azure file

Question

Getting Permission denied error when copy files from mysql pod using azure file

Prashanth Kumar 1

Hi,
I have a MySQL pod deployed using the default azurefile storage class and the pod comes up fine with this. Now i logged in to the node where the pod was running and tried to copy the data from the mounted path, gives me Permission Denied error as follows

root@aks-nodepool1-34339388-vmss000000:/host/var/lib/kubelet/pods/b28f3252-1444-4eba-aeb5-055f4374404a/volumes/kubernetes.io~csi
/pvc-a1458ad2-3536-46e4-b30b-cf822e616b1c/mount# ls -lrth
total 109M
drwxrwxrwx 2 root root   0 Jan 18 08:29 mysql
-rwxrwxrwx 1 root root 48M Jan 18 08:29 ib_logfile1
drwxrwxrwx 2 root root   0 Jan 18 08:29 performance_schema
-rwxrwxrwx 1 root root  56 Jan 18 08:30 auto.cnf
-rwxrwxrwx 1 root root 12M Jan 18 08:30 ibdata1
-rwxrwxrwx 1 root root 48M Jan 18 08:30 ib_logfile0

root@aks-nodepool1-34339388-vmss000000:/host/var/lib/kubelet/pods/b28f3252-1444-4eba-aeb5-055f4374404a/volumes/kubernetes.io~csi
/pvc-a1458ad2-3536-46e4-b30b-cf822e616b1c/mount# cp ib_logfile0 ..
cp: error reading 'ib_logfile0': Permission denied

I did Google and found a few solutions related to Azure File share which I am not using but I am assuming this is not really needed.
I am wondering how did the MySql Pod come up fine but accessing the same file using the cp command is failing. Am i missing something?

-Prashanth

Prashanth Kumar 1 Reputation point

2022-01-19T12:02:51.457+00:00

Hi,
Any help on this?

-Prashanth
Sumarigo-MSFT 47,466 Reputation points Microsoft Employee Moderator

2022-01-19T13:28:15.383+00:00

@Prashanth Kumar Thanks for raising this question! Firstly, apologies for the delay in responding here and any inconvenience this issue may have caused

You are getting this error code only while uploading files to azure File storage(File Share)? Can you try download and also try different Azure file storage account and let me know the status
Can you try to provide azure Storage account contributor access role and check for the status
Prashanth Kumar 1 Reputation point

2022-01-19T13:48:32.26+00:00
Hi
Thanks for the reply. In my case, I am not using any azure file share.
If you see the following path, I have logged in to the MySQL pod and switched to the path where the PVC is mounted.

root@aks-nodepool1-34339388-vmss000000:/host/var/lib/kubelet/pods/b28f3252-1444-4eba-aeb5-055f4374404a/volumes/kubernetes.io~csi
/pvc-a1458ad2-3536-46e4-b30b-cf822e616b1c/mount# ls -lrth
total 109M
drwxrwxrwx 2 root root 0 Jan 18 08:29 mysql
-rwxrwxrwx 1 root root 48M Jan 18 08:29 ib_logfile1
drwxrwxrwx 2 root root 0 Jan 18 08:29 performance_schema
-rwxrwxrwx 1 root root 56 Jan 18 08:30 auto.cnf
-rwxrwxrwx 1 root root 12M Jan 18 08:30 ibdata1
-rwxrwxrwx 1 root root 48M Jan 18 08:30 ib_logfile0 -

Here I am just trying to copy one of the files say for eg ib_logfile0 to its parent folder and it says permission denied for cp command. Indeed I can't create any file or execute any os level file operation on this folder.

An additional point is when I deploy the same MySQL app using block storage all work fine.

I would like to know if i need to provide any additional permissions or set permissions somewhere so that i can access the files

As I understand, storage account keys roles are needed if I am using an Azure file share. Correct me if I am wrong here.

-Prashanth
Prashanth Kumar 1 Reputation point

2022-01-20T02:10:03.073+00:00

Small correction.
Below path is logging into the AKS node where MySql Pod is running

root@aks-nodepool1-34339388-vmss000000:/host/var/lib/kubelet/pods/b28f3252-1444-4eba-aeb5-055f4374404a/volumes/kubernetes.io~csi
/pvc-a1458ad2-3536-46e4-b30b-cf822e616b1c/mount# ls -lrth

My end goal is to back up these files using the opensource backup tool kopia which will run as a Kubernetes Job pod on the node where MySql is running. As part of this job, will mount the above host path into the pod and copy the files to the external objectstore.

So the basic issue is even without this tool when we try to run any of the OS level commands for copying a file it fails.

-Prashanth
Sumarigo-MSFT 47,466 Reputation points Microsoft Employee Moderator

2022-01-20T13:32:18.047+00:00

@Prashanth Kumar Thanks for providing the detailed information of the issue, This forum is mainly concentrated on Azure File Issue

May I know how have you logged into the node (best guess, how Azure documentation guides them to). There are a few caveats with that which might be causing this. Instead ask him to try logging in using something like mohatb/kubectl-exec: kubectl-exec is a shell script getting a shell into your kuberntes nodes. (github.com)

If this issue still persist, please check for group permissions on the PVC mount point in the AKS node.

If the query is not rectified I would recommend to post your query in azure-database-mysql These forum community members could provide their expert guidance on your scenario based on their experience. Thanks.

----------

Please do not forget to and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.
Prashanth Kumar 1 Reputation point

2022-01-20T14:24:27.41+00:00

Hi,
As the MySql app is using azure file storage class, I felt this might be the right forum to post the same.
I followed the following doc to login to the Azure node

https://learn.microsoft.com/en-us/azure/aks/ssh

here is the file persmission
stat ib_logfile0
File: ib_logfile0
Size: 50331648 Blocks: 98304 IO Block: 1048576 regular file
Device: e1h/225d Inode: 17582123313998594048 Links: 1
Access: (0777/-rwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2021-12-15 03:48:11.126855400 +0000
Modify: 2021-12-15 03:48:11.126855400 +0000
Change: 2021-12-15 03:48:11.126855400 +0000

As here even the basic OS commands are failing on this node, not sure if this is still related to DB
-Prashanth

1 answer

Your answer

Prashanth Kumar 1 Reputation point

2022-01-19T12:02:51.457+00:00

Hi,
Any help on this?

-Prashanth
Sumarigo-MSFT 47,466 Reputation points Microsoft Employee Moderator

2022-01-19T13:28:15.383+00:00

@Prashanth Kumar Thanks for raising this question! Firstly, apologies for the delay in responding here and any inconvenience this issue may have caused

You are getting this error code only while uploading files to azure File storage(File Share)? Can you try download and also try different Azure file storage account and let me know the status
Can you try to provide azure Storage account contributor access role and check for the status
Prashanth Kumar 1 Reputation point

2022-01-19T13:48:32.26+00:00

Hi
Thanks for the reply. In my case, I am not using any azure file share.
If you see the following path, I have logged in to the MySQL pod and switched to the path where the PVC is mounted.

root@aks-nodepool1-34339388-vmss000000:/host/var/lib/kubelet/pods/b28f3252-1444-4eba-aeb5-055f4374404a/volumes/kubernetes.io~csi
/pvc-a1458ad2-3536-46e4-b30b-cf822e616b1c/mount# ls -lrth
total 109M
drwxrwxrwx 2 root root 0 Jan 18 08:29 mysql
-rwxrwxrwx 1 root root 48M Jan 18 08:29 ib_logfile1
drwxrwxrwx 2 root root 0 Jan 18 08:29 performance_schema
-rwxrwxrwx 1 root root 56 Jan 18 08:30 auto.cnf
-rwxrwxrwx 1 root root 12M Jan 18 08:30 ibdata1
-rwxrwxrwx 1 root root 48M Jan 18 08:30 ib_logfile0 -

Here I am just trying to copy one of the files say for eg ib_logfile0 to its parent folder and it says permission denied for cp command. Indeed I can't create any file or execute any os level file operation on this folder.

An additional point is when I deploy the same MySQL app using block storage all work fine.

I would like to know if i need to provide any additional permissions or set permissions somewhere so that i can access the files

As I understand, storage account keys roles are needed if I am using an Azure file share. Correct me if I am wrong here.

-Prashanth
Prashanth Kumar 1 Reputation point

2022-01-20T02:10:03.073+00:00

Small correction.
Below path is logging into the AKS node where MySql Pod is running

root@aks-nodepool1-34339388-vmss000000:/host/var/lib/kubelet/pods/b28f3252-1444-4eba-aeb5-055f4374404a/volumes/kubernetes.io~csi
/pvc-a1458ad2-3536-46e4-b30b-cf822e616b1c/mount# ls -lrth

My end goal is to back up these files using the opensource backup tool kopia which will run as a Kubernetes Job pod on the node where MySql is running. As part of this job, will mount the above host path into the pod and copy the files to the external objectstore.

So the basic issue is even without this tool when we try to run any of the OS level commands for copying a file it fails.

-Prashanth
Sumarigo-MSFT 47,466 Reputation points Microsoft Employee Moderator

2022-01-20T13:32:18.047+00:00

@Prashanth Kumar Thanks for providing the detailed information of the issue, This forum is mainly concentrated on Azure File Issue

May I know how have you logged into the node (best guess, how Azure documentation guides them to). There are a few caveats with that which might be causing this. Instead ask him to try logging in using something like mohatb/kubectl-exec: kubectl-exec is a shell script getting a shell into your kuberntes nodes. (github.com)

If this issue still persist, please check for group permissions on the PVC mount point in the AKS node.

If the query is not rectified I would recommend to post your query in azure-database-mysql These forum community members could provide their expert guidance on your scenario based on their experience. Thanks.

----------

Please do not forget to and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.
Prashanth Kumar 1 Reputation point

2022-01-20T14:24:27.41+00:00

Hi,
As the MySql app is using azure file storage class, I felt this might be the right forum to post the same.
I followed the following doc to login to the Azure node

https://learn.microsoft.com/en-us/azure/aks/ssh

here is the file persmission
stat ib_logfile0
File: ib_logfile0
Size: 50331648 Blocks: 98304 IO Block: 1048576 regular file
Device: e1h/225d Inode: 17582123313998594048 Links: 1
Access: (0777/-rwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2021-12-15 03:48:11.126855400 +0000
Modify: 2021-12-15 03:48:11.126855400 +0000
Change: 2021-12-15 03:48:11.126855400 +0000

As here even the basic OS commands are failing on this node, not sure if this is still related to DB
-Prashanth

Answer 1

Prashanth Kumar 1

Hi,
Any update on this?

-prashanth

Prashanth Kumar 1 Reputation point

2022-01-26T04:21:38.553+00:00

Hi,
I had seen a few earlier comments on this ticket, now I see none of them are existing. Can someone pls let me know if those comments/answers are gone?

-Prashanth

Share via

Getting Permission denied error when copy files from mysql pod using azure file

1 answer

Your answer