VM crash after LRS device error

Question

We have our corporate web server on on an Azure Ubuntu VM

The data is stored on a Premium SSD LRS - caching is switched off

This morning the server hung or crashed - it not shown in the console as running.

I restarted and it is now running OK.

In the kernel log its shows superblock errors and disk remounted as read-only.

Then nothing until I rebooted.

I did an fsck and no errors were found.

My questions are:

Is LRS providing redundancy ?

How can I prevent this happening again ?

More details below

Any input appreciated - let me know if you need more info

Thanks

Peter

Kernel log:

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.023320] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#148 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.023364] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#149 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.031393] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#148 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.031415] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#149 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.039196] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#148 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.039224] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#149 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.047244] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#148 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.047268] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#149 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.055379] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#148 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.055405] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#149 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.063228] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#148 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.063256] sd 1:0:0:0: [sdc] tag#148 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=185s

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.063266] sd 1:0:0:0: [sdc] tag#148 Sense Key : Medium Error [current]

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.063273] sd 1:0:0:0: [sdc] tag#148 Add. Sense: No additional sense information

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.063282] sd 1:0:0:0: [sdc] tag#148 CDB: Write(10) 2a 00 00 01 6e b0 00 00 08 00

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.063289] I/O error, dev sdc, sector 93872 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.068527] EXT4-fs warning (device sdc1): ext4_end_bio:343: I/O error 10 writing to inode 3538979 starting block 11734)

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.068538] Buffer I/O error on device sdc1, logical block 11478

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.072749] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#149 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.072759] sd 1:0:0:0: [sdc] tag#149 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=179s

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.072761] sd 1:0:0:0: [sdc] tag#149 Sense Key : Medium Error [current]

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.072763] sd 1:0:0:0: [sdc] tag#149 Add. Sense: No additional sense information

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.072766] sd 1:0:0:0: [sdc] tag#149 CDB: Write(10) 2a 00 03 c4 26 c8 00 00 40 00

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.072768] I/O error, dev sdc, sector 63186632 op 0x1:(WRITE) flags 0x9800 phys_seg 8 prio class 2

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.086090] Aborting journal on device sdc1-8.

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.092733] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#150 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.103450] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#150 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.112369] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#150 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.121027] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#150 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www systemd[1]: mysql.service: Main process exited, code=exited, status=2/INVALIDARGUMENT

Apr 1 08:43:16 rc-corporate-www systemd[1]: mysql.service: Failed with result 'exit-code'.

Apr 1 08:43:16 rc-corporate-www systemd[1]: mysql.service: Consumed 7h 49min 25.121s CPU time.

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.130200] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#150 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.138609] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#150 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.138621] sd 1:0:0:0: [sdc] tag#150 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.138626] sd 1:0:0:0: [sdc] tag#150 Sense Key : Medium Error [current]

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.138628] sd 1:0:0:0: [sdc] tag#150 Add. Sense: No additional sense information

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.138631] sd 1:0:0:0: [sdc] tag#150 CDB: Write(10) 2a 00 03 c4 08 00 00 00 08 00

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.138633] I/O error, dev sdc, sector 63178752 op 0x1:(WRITE) flags 0x9800 phys_seg 1 prio class 2

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.145205] Buffer I/O error on dev sdc1, logical block 7897088, lost sync page write

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.150353] JBD2: I/O error when updating journal superblock for sdc1-8.

Apr 1 08:43:16 rc-corporate-www systemd[1]: mysql.service: Scheduled restart job, restart counter is at 1.

Apr 1 08:43:16 rc-corporate-www systemd[1]: Stopped MySQL Community Server.

Apr 1 08:43:16 rc-corporate-www systemd[1]: mysql.service: Consumed 7h 49min 25.121s CPU time.

Apr 1 08:43:16 rc-corporate-www systemd[1]: Starting MySQL Community Server...

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.572913] EXT4-fs error (device sdc1): ext4_journal_check_start:84: comm mysqld: Detected aborted journal

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.580429] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#151 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.593193] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#151 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.602251] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#151 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.610236] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#151 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.618521] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#151 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.626215] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#151 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.626243] sd 1:0:0:0: [sdc] tag#151 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.626253] sd 1:0:0:0: [sdc] tag#151 Sense Key : Medium Error [current]

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.626260] sd 1:0:0:0: [sdc] tag#151 Add. Sense: No additional sense information

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.626269] sd 1:0:0:0: [sdc] tag#151 CDB: Write(10) 2a 00 00 00 08 00 00 00 08 00

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.626275] I/O error, dev sdc, sector 2048 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.631390] Buffer I/O error on dev sdc1, logical block 0, lost sync page write

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.636726] EXT4-fs (sdc1): I/O error while writing superblock

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.640473] EXT4-fs (sdc1): Remounting filesystem read-only

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.654029] kauditd_printk_skb: 11 callbacks suppressed

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.654033] audit: type=1400 audit(1743496996.980:92): apparmor="DENIED" operation="open" class="file" profile="/usr/sbin/mysqld" name="/etc

/aadpasswd" pid=1841729 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

Apr 1 08:43:17 rc-corporate-www systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE

Apr 1 08:43:17 rc-corporate-www systemd[1]: mysql.service: Failed with result 'exit-code'.

Apr 1 08:43:17 rc-corporate-www systemd[1]: Failed to start MySQL Community Server.

Apr 1 08:43:17 rc-corporate-www systemd[1]: mysql.service: Scheduled restart job, restart counter is at 2.

Apr 1 08:43:17 rc-corporate-www systemd[1]: Stopped MySQL Community Server.

www-admin@rc-corporate-www:/var/log$ more kern.log

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.023320] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#148 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.023364] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#149 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.031393] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#148 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.031415] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#149 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.039196] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#148 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.039224] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#149 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.047244] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#148 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.047268] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#149 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.055379] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#148 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.055405] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#149 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.063228] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#148 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.063256] sd 1:0:0:0: [sdc] tag#148 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=185s

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.063266] sd 1:0:0:0: [sdc] tag#148 Sense Key : Medium Error [current]

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.063273] sd 1:0:0:0: [sdc] tag#148 Add. Sense: No additional sense information

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.063282] sd 1:0:0:0: [sdc] tag#148 CDB: Write(10) 2a 00 00 01 6e b0 00 00 08 00

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.063289] I/O error, dev sdc, sector 93872 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.068527] EXT4-fs warning (device sdc1): ext4_end_bio:343: I/O error 10 writing to inode 3538979 starting block 11734)

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.068538] Buffer I/O error on device sdc1, logical block 11478

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.072749] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#149 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.072759] sd 1:0:0:0: [sdc] tag#149 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=179s

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.072761] sd 1:0:0:0: [sdc] tag#149 Sense Key : Medium Error [current]

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.072763] sd 1:0:0:0: [sdc] tag#149 Add. Sense: No additional sense information

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.072766] sd 1:0:0:0: [sdc] tag#149 CDB: Write(10) 2a 00 03 c4 26 c8 00 00 40 00

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.072768] I/O error, dev sdc, sector 63186632 op 0x1:(WRITE) flags 0x9800 phys_seg 8 prio class 2

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.086090] Aborting journal on device sdc1-8.

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.092733] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#150 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.103450] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#150 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.112369] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#150 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.121027] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#150 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.130200] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#150 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.138609] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#150 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.138621] sd 1:0:0:0: [sdc] tag#150 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.138626] sd 1:0:0:0: [sdc] tag#150 Sense Key : Medium Error [current]

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.138628] sd 1:0:0:0: [sdc] tag#150 Add. Sense: No additional sense information

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.138631] sd 1:0:0:0: [sdc] tag#150 CDB: Write(10) 2a 00 03 c4 08 00 00 00 08 00

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.138633] I/O error, dev sdc, sector 63178752 op 0x1:(WRITE) flags 0x9800 phys_seg 1 prio class 2

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.145205] Buffer I/O error on dev sdc1, logical block 7897088, lost sync page write

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.150353] JBD2: I/O error when updating journal superblock for sdc1-8.

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.572913] EXT4-fs error (device sdc1): ext4_journal_check_start:84: comm mysqld: Detected aborted journal

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.580429] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#151 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.593193] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#151 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.602251] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#151 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.610236] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#151 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.618521] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#151 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.626215] hv_storvsc f8b3781b-1e82-4818-a1c3-63d806ec15bb: tag#151 cmd 0x2a status: scsi 0x2 srb 0x84 hv 0xc0000001

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.626243] sd 1:0:0:0: [sdc] tag#151 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.626253] sd 1:0:0:0: [sdc] tag#151 Sense Key : Medium Error [current]

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.626260] sd 1:0:0:0: [sdc] tag#151 Add. Sense: No additional sense information

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.626269] sd 1:0:0:0: [sdc] tag#151 CDB: Write(10) 2a 00 00 00 08 00 00 00 08 00

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.626275] I/O error, dev sdc, sector 2048 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.631390] Buffer I/O error on dev sdc1, logical block 0, lost sync page write

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.636726] EXT4-fs (sdc1): I/O error while writing superblock

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.640473] EXT4-fs (sdc1): Remounting filesystem read-only

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.654029] kauditd_printk_skb: 11 callbacks suppressed

Apr 1 08:43:16 rc-corporate-www kernel: [3529682.654033] audit: type=1400 audit(1743496996.980:92): apparmor="DENIED" operation="open" class="file" profile="/usr/sbin/mysqld" name="/etc

/aadpasswd" pid=1841729 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

Apr 1 08:43:17 rc-corporate-www kernel: [3529683.241887] audit: type=1400 audit(1743496997.568:93): apparmor="DENIED" operation="open" class="file" profile="/usr/sbin/mysqld" name="/etc

/aadpasswd" pid=1841738 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

Apr 1 08:43:18 rc-corporate-www kernel: [3529683.994786] audit: type=1400 audit(1743496998.321:94): apparmor="DENIED" operation="open" class="file" profile="/usr/sbin/mysqld" name="/etc

/aadpasswd" pid=1841747 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

Apr 1 08:43:19 rc-corporate-www kernel: [3529684.739847] audit: type=1400 audit(1743496999.066:95): apparmor="DENIED" operation="open" class="file" profile="/usr/sbin/mysqld" name="/etc

/aadpasswd" pid=1841757 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

Apr 1 08:43:19 rc-corporate-www kernel: [3529685.495898] audit: type=1400 audit(1743496999.822:96): apparmor="DENIED" operation="open" class="file" profile="/usr/sbin/mysqld" name="/etc

/aadpasswd" pid=1841766 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

Apr 1 09:45:25 rc-corporate-www kernel: [ 0.000000] Linux version 6.8.0-1021-azure (buildd@lcy02-amd64-045) (x86_64-linux-gnu-gcc-11 (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, GNU ld (GN

U Binutils for Ubuntu) 2.38) #25~22.04.1-Ubuntu SMP Thu Jan 16 21:37:09 UTC 2025 (Ubuntu 6.8.0-1021.25~22.04.1-azure 6.8.12)

Answer 1

Hi Peter Young,

Yes, Locally Redundant Storage (LRS) provides redundancy by replicating your data three times within a single data center in the selected Azure region. This setup protects your data against server rack and drive failures. However, LRS doesn't safeguard against data center-wide incidents, such as fires or floods. Although there is no option to choose your preferred availability zone, Azure may move or expand LRS accounts across zones to improve load balancing. LRS provides at least 99.999999999% durability of objects over a given year.

Please check this document for detailed information: Azure Storage redundancy

Your Azure Ubuntu VM experienced a storage subsystem failure, the filesystem protected itself by going read-only when it detected unrecoverable write errors.

Confirm your backups are complete and test restoration, consider taking a new disk snapshot immediately.
Upgrade to ZRS (Zone Redundant Storage): It provides replication across availability zones, for better protection against localized hardware failures.
Set up Azure Monitor Alerts for Disk errors and Disk latency.

If you have any further queries, please do let us know. If the answer is helpful, please click "Accept Answer" and "Upvote it"

Answer 2

I have followed your advice and created a user runbook.

I have now scheduled a runbook to run once an hour and it is working

The script is as follows

Connect-AzAccount -Identity

$vmName = "rc-corporate-www"
$resourceGroupName = "Risk_Control_Corporate"

$status=(Get-AzVM -Name $vmName -ResourceGroupName $resourceGroupName -status).Statuses[1].DisplayStatus
echo "$(Get-Date) status: $status"
if ($status -ne "VM Running") {
    echo "Start VM"
    Start-AzVM -ResourceGroupName $resourceGroupName -Name $vmName
    Start-Sleep -Seconds 10
    $status=(Get-AzVM -Name $vmName -ResourceGroupName $resourceGroupName -status).Statuses[1].DisplayStatus
    echo "$(Get-Date) status: $status"
}

When the VM is powered down the alert is being triggered due to VM-availability dropping below 1 and I receive the email:

Microsoft Azure

Your Azure Monitor alert was triggered

Azure monitor alert rule test-metricAlertRule was triggered for test-storageAccount at April 9, 2025 15:43 UTC.

Rule ID /subscriptions/11111111-1111-1111-1111-111111111111/providers/Microsoft.AlertsManagement/alerts/bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb

View Rule >

Resource ID /subscriptions/11111111-1111-1111-1111-111111111111/resourcegroups/test-RG/providers/microsoft.storage/storageaccounts/test-storageAccount

View Resource >

However the runbook is not having any effect when initiated this way.

Can you think what is missing ?

Thanks

Peter