Azure data disk mounting as dr--r--r--, cannot change or write due to Operation not permitted

Question

Azure data disk mounting as dr--r--r--, cannot change or write due to Operation not permitted

Nick 0

As of yesterday, the data disk on my Linux VM has gone read-only. Nothing shows in the Azure VM or Disk settings about this, nor in dmesg. However, you cannot write to it, and you cannot change that

The disk appears to mount find in read-write mode, no errors from umount /models && mount -o rw /models

However, the permissions after mounting are very strange and read-only
ls -ld /models
dr--r--r-- 11 nick nick 4096 Jun 11 18:02 /models

Even as root, you can't change that, nor write to it
root@vm:/# chmod 775 /models
chmod: changing permissions of '/models': Operation not permitted

root@vm:/# mkdir /models/test
mkdir: cannot create directory ‘/models/test’: Operation not permitted

I know it's the right disk, as it has all my data on it!

Shutting the VM down and starting it up again doesn't help. Shutting the machine down, doing a az vm disk detach then az vm disk attach doesn't make any difference. I've tried all 3 Host caching options, no difference.

It all feels like Azure thinks there's something wrong with the disk, and is marking it as read-only to protect me. However I can't find anything that shows any issues. How do I get Azure to let me write to my Data Disk again?

deherman-MSFT 38,021 Reputation points Microsoft Employee Moderator

2023-06-12T16:46:22.7533333+00:00

@Nick

It is possible that there is some issue with the filesystem that is causing it to be read only. Can you check dmesg for any errors? It should also show the disk being mounted as read only. If there are errors you can try running fsck on the disk. Before running this please have a snapshot or backup available.
Nick 0 Reputation points

2023-06-12T17:18:16.1866667+00:00

Linux thinks the disk is mounted read-write

root@vm:/# grep models /etc/mtab
/dev/sda1 /models ext4 rw,relatime 0 0

No errors from a fsck

root@vm:/# umount models
root@vm:/# e2fsck -f -C 0 /dev/sda1
e2fsck 1.46.5 (30-Dec-2021)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Models: 1557/67108864 files (0.1% non-contiguous), 123088473/268435200 blocks

But it still then mounts secretly read-only
root@vm:/# mount /models
root@vm:/# ls -ld /models
dr--r--r-- 11 nick nick 4096 Jun 11 18:02 /models
root@vm:/# chmod 775 /models
chmod: changing permissions of '/models': Operation not permitted

No obvious errors in dmesg

[ 64.674587] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.

It's all really strange!
deherman-MSFT 38,021 Reputation points Microsoft Employee Moderator

2023-06-12T20:31:02.3466667+00:00
@Nick I wouldn't say that the disk is read only since mtab shows it is mounted as rw. Perhaps check to make sure the directory wasn't made immutable?

lsattr /models
Nick 0 Reputation points

2023-06-13T18:17:57.9066667+00:00

Ah ha! That was it! And a new bit of ext2+ learning for me...

root@vm:/# lsattr -d /models
----i---------e------- /models

After a chattr and chmod, everything is back to normal. Still no idea how that came to be enabled between shutting the VM down one day and powering it back on the next, but at least it's all sorted now. Thanks for all your help
deherman-MSFT 38,021 Reputation points Microsoft Employee Moderator

2023-06-15T19:59:21.4066667+00:00

@Nick

If the answer has been helpful, we appreciate hearing from you and would love to help others who may have the same question. Accepting answers helps increase visibility of this question for other members of the Microsoft Q&A community.

Thank you for helping to improve Microsoft Q&A!

1 answer

Your answer

deherman-MSFT 38,021 Reputation points Microsoft Employee Moderator

2023-06-12T16:46:22.7533333+00:00

@Nick

It is possible that there is some issue with the filesystem that is causing it to be read only. Can you check dmesg for any errors? It should also show the disk being mounted as read only. If there are errors you can try running fsck on the disk. Before running this please have a snapshot or backup available.
Nick 0 Reputation points

2023-06-12T17:18:16.1866667+00:00

Linux thinks the disk is mounted read-write

root@vm:/# grep models /etc/mtab
/dev/sda1 /models ext4 rw,relatime 0 0

No errors from a fsck

root@vm:/# umount models
root@vm:/# e2fsck -f -C 0 /dev/sda1
e2fsck 1.46.5 (30-Dec-2021)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Models: 1557/67108864 files (0.1% non-contiguous), 123088473/268435200 blocks

But it still then mounts secretly read-only
root@vm:/# mount /models
root@vm:/# ls -ld /models
dr--r--r-- 11 nick nick 4096 Jun 11 18:02 /models
root@vm:/# chmod 775 /models
chmod: changing permissions of '/models': Operation not permitted

No obvious errors in dmesg

[ 64.674587] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.

It's all really strange!
deherman-MSFT 38,021 Reputation points Microsoft Employee Moderator

2023-06-12T20:31:02.3466667+00:00

@Nick I wouldn't say that the disk is read only since mtab shows it is mounted as rw. Perhaps check to make sure the directory wasn't made immutable?

lsattr /models
Nick 0 Reputation points

2023-06-13T18:17:57.9066667+00:00

Ah ha! That was it! And a new bit of ext2+ learning for me...

root@vm:/# lsattr -d /models
----i---------e------- /models

After a chattr and chmod, everything is back to normal. Still no idea how that came to be enabled between shutting the VM down one day and powering it back on the next, but at least it's all sorted now. Thanks for all your help
deherman-MSFT 38,021 Reputation points Microsoft Employee Moderator

2023-06-15T19:59:21.4066667+00:00

@Nick

If the answer has been helpful, we appreciate hearing from you and would love to help others who may have the same question. Accepting answers helps increase visibility of this question for other members of the Microsoft Q&A community.

Thank you for helping to improve Microsoft Q&A!

Answer 1

@Nick

I'm glad that you were able to resolve your issue! I'll repost your solution in case you'd like to "Accept " the answer.

Issue:

You were unable to write or edit permissions for your disk. The disk was showing mounted as read/write, but still root could not make changes.

Solution:
The directory was set to immutable. You discovered this by running:

root@vm:/# lsattr -d /models
----i---------e------- /models

Running chattr -i and chmod you were able to resolve this.

If you still have questions, please let us know in the "comments" and we would be happy to help you. Comment is the fastest way of notifying the experts.

If the answer has been helpful, we appreciate hearing from you and would love to help others who may have the same question. Accepting answers helps increase visibility of this question for other members of the Microsoft Q&A community.

Thank you for helping to improve Microsoft Q&A!

User's image

Share via

Azure data disk mounting as dr--r--r--, cannot change or write due to Operation not permitted

1 answer

Your answer