Cannot perform basic file disc check on azure / kubernetes

Question

Cannot perform basic file disc check on azure / kubernetes

Koen de Jaeger 0

Using a pod derived from https://hub.docker.com/r/fcrepo/fcrepo .

I mounted /usr/local/tomcat/fcrepo-home to a btrfs persistant volume (ssd standard, fstype btrfs).

Now I wanted to do a file system check; after installing btrfs-progs. But I get some strange errors and even dmesg is not allowed?

root@ibron--surf-acc-ibron-2-758b588878-dnp76:/usr/local/tomcat# btrfs device stats 
/usr/local/tomcat/fcrepo-home ERROR: getting device info for /usr/local/tomcat/fcrepo-home failed: Operation not permitted 

root@ibron--surf-acc-ibron-2-758b588878-dnp76:/usr/local/tomcat# btrfs check /usr/local/tomcat/fcrepo-home
Opening filesystem to check... ERROR: not a regular file or block device: /usr/local/tomcat/fcrepo-home ERROR: cannot open file system 

root@ibron--surf-acc-ibron-2-758b588878-dnp76:/usr/local/tomcat# btrfs scrub start -Bd /usr/local/tomcat/fcrepo-home 
ERROR: getting dev info for scrub failed: Operation not permitted 

root@ibron--surf-acc-ibron-2-758b588878-dnp76:/usr/local/tomcat# dmesg 
dmesg: read kernel buffer failed: Operation not permitted

Tried customer support but they said, after checking out everything, 'Please check the filesystem that configured via btrfs if there are any issues.'. Wonderfull ... .

srbhatta-MSFT 8,586 Reputation points Microsoft Employee

2023-01-31T03:58:57.7366667+00:00

Hello @Koen de Jaeger , could you please provide the support request service Id?
Goncalo Correia 351 Reputation points Microsoft Employee

2023-01-31T11:35:40.1+00:00

Hi @Koen de Jaeger

Are you trying to fscheck from inside the pod with the fs mounted?
You can run dmesg from the node where the pod is running to see the information about the disk mounted in the pod FS,

However, when the pod is created in the node the pvc mount point is mounted in the pod FS and you should not unmounted it from the node/pod shell manually. As soon as the pod is deleted the PVC is detached from the node.
Can you attach the disk a separate VM and with the FS unmounted run the fscheck?

4 answers

Your answer

srbhatta-MSFT 8,586 Reputation points Microsoft Employee

2023-01-31T03:58:57.7366667+00:00

Hello @Koen de Jaeger , could you please provide the support request service Id?
Goncalo Correia 351 Reputation points Microsoft Employee

2023-01-31T11:35:40.1+00:00

Hi @Koen de Jaeger

Are you trying to fscheck from inside the pod with the fs mounted?
You can run dmesg from the node where the pod is running to see the information about the disk mounted in the pod FS,

However, when the pod is created in the node the pvc mount point is mounted in the pod FS and you should not unmounted it from the node/pod shell manually. As soon as the pod is deleted the PVC is detached from the node.
Can you attach the disk a separate VM and with the FS unmounted run the fscheck?

Answer 1

Koen de Jaeger 0

Looked into kubernetes and sysctl whitelisting, but couldn't get 'kernel.dmesg_restrict' whitelisted to be used in podsecurity. Azure only permits to whitelist: kernel.shm*, kernel.msg*, kernel.sem, fs.mqueue., net.. And so 'dmesg' is not possible to allow? As well as a disc check, which gives the same error??

Answer 2

Koen de Jaeger 0

@Goncalo Correia the above commands were run inside the pod where the PVC is done, where the SSD is mounted onto.

'dmesg' onto the node gives me the same error msg. (tried with https://learn.microsoft.com/en-us/azure/aks/node-access)

Goncalo Correia 351 Reputation points Microsoft Employee

2023-02-02T13:20:48.6933333+00:00

With debug you might not be able to run every command in the host,
Try it with this script:

https://github.com/mohatb/kubectl-exec

You should be able to run dmesg into the node namespace

Answer 3

Koen de Jaeger 0

I posted the solution to my problem on reddit: https://www.reddit.com/r/kubernetes/comments/10pt92e/getting_around_sysctl_restrictions_on_azure/

Answer 4

Koen de Jaeger 0

@srbhatta-MSFT 2301220050000366

Share via

Cannot perform basic file disc check on azure / kubernetes

4 answers

Your answer