Virtual machine shutting down spontaneously without prior notice

Question

Virtual machine shutting down spontaneously without prior notice

Teltec - Hugo 1

good afternoon everyone

I would like to know if azure can turn off virtual machines without prior notice?

we have a linux server that restarted for no apparent reason and without prior notice, in the integrity resource we have the information that the machine was restarted by azure due to a failure found in the connection with the disk.(attached)

Is there any setting to prevent shutdown without prior notice?

kobulloc-MSFT 26,811 Reputation points Microsoft Employee Moderator

2022-03-03T19:38:25.83+00:00

Error message:

The azure identification monitoring and diagnostics systems that come with Server3 became unavailable in 2022-02-01, and availability was restored in 2022-02-01. During this time, RDP and SSH connections to the VM or requests to other services running inside the VM may have failed.

Root cause
This unexpected occurrence was caused by an Azure-initiated VM shutdown, triggered by the detection of temporary I/O transaction timeouts between the physical host node where the VM was running and the Azure Storage service where the VHDs ( Virtual hard disks) are located.

The Azure Platform continuously monitors the reads and writes (I/O transactions) of VMs in Azure Storage. If transactions are not completed successfully within 120 seconds (including retries), connectivity will be considered lost and VM shutdown will be initiated to preserve data integrity and prevent VM corruption. After the platform detects that storage service connectivity has been restored, the VM is automatically restarted.

Resolution
The VM was restored after the host node restarted.

1 answer

Your answer

kobulloc-MSFT 26,811 Reputation points Microsoft Employee Moderator

2022-03-03T19:38:25.83+00:00

Error message:

The azure identification monitoring and diagnostics systems that come with Server3 became unavailable in 2022-02-01, and availability was restored in 2022-02-01. During this time, RDP and SSH connections to the VM or requests to other services running inside the VM may have failed.

Root cause
This unexpected occurrence was caused by an Azure-initiated VM shutdown, triggered by the detection of temporary I/O transaction timeouts between the physical host node where the VM was running and the Azure Storage service where the VHDs ( Virtual hard disks) are located.

The Azure Platform continuously monitors the reads and writes (I/O transactions) of VMs in Azure Storage. If transactions are not completed successfully within 120 seconds (including retries), connectivity will be considered lost and VM shutdown will be initiated to preserve data integrity and prevent VM corruption. After the platform detects that storage service connectivity has been restored, the VM is automatically restarted.

Resolution
The VM was restored after the host node restarted.

Answer 1

Hello, @Teltec - Hugo !

How do you prevent an unexpected VM shutdown when there is an attached VHD?
The short answer is that while you can't prevent the VM shutdown when IOPS have been exceeded on VHDs in a single storage account, you can place your VHDs in multiple storage accounts to make sure that this limit isn't exceeded. That in turn will prevent the unexpected shutdown from happening.

When does Azure turn off a VM?
There are several maintenance scenarios that you have control over, many of which do not require a reboot. There's a lot to go through here which I'll include for others who may be reading this however your issue seems to be VHD related which I'll address further down below.

Unexpected VM Restarts and Attached VHDs
Generally speaking, there are a number of unexpected restart scenarios however VHDs have special considerations. If a virtual machine has a large number of attached VHDs that are in the same storage account, it's possible to exceed the scalability targets for the individual storage account which will cause the VM to reboot unexpectedly.

You can tune how many VHDs you have in a single storage account based on the estimated IOPS for your environment specific to your application. Microsoft recommends having 40 or fewer disks in a single storage account. If you are exceeding the scalability targets for your storage account, place your VHDs in multiple storage accounts to reduce the activity in each individual account.

Read more:
https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/unexpected-reboots-attached-vhds

Share via

Virtual machine shutting down spontaneously without prior notice

1 answer

Your answer