Unexpected reboot causing major outage.

Question

Unexpected reboot causing major outage.

Metev, Bogdan 26

Hello,

Last night one of our Azure servers had an unexpected reboot which caused an outage of our SAP Archive solution. I'd like to find out more information about why a restart occurred (without notification) and how we can be informed about the cause in the future.

The process C:\windows\system32\svchost.exe (E2R2S49IAPP0001) has initiated the shutdown of computer server_name on behalf of user NT AUTHORITY\SYSTEM for the following reason: Other (Planned)
Reason Code: 0x80000000
Shutdown Type: shutdown
Comment: Calling CleanShutdown by wvchelper
Reason for shutdown: Stop call

prmanhas-MSFT 17,946 Reputation points Microsoft Employee Moderator

2020-11-10T14:47:52.697+00:00

@Metev, Bogdan Any update on the issue?

If the suggested response helped you resolve your issue, do click on "Mark as Answer" and "Up-Vote" for the answer that helped you for benefit of the community.

Thanks.
Andre 1 Reputation point

2020-11-27T07:10:09.807+00:00

I had the same exact issue. Did you find the cause?
Metev, Bogdan 26 Reputation points

2020-11-27T07:17:47.627+00:00

Hi, Andre,
Yes, we did.

This unexpected reboot was caused by an Azure initiated auto-recovery action. The auto-recovery action was triggered by a hardware issue on the physical node where the virtual machine was hosted. As designed, the VM was automatically moved to a different and healthy physical node to avoid further impact.
Microsoft recommended grouping two or more virtual machines in an availability set to ensure an increased level of protection and redundancy for the application in Azure
Andre 1 Reputation point

2020-11-27T10:23:40.373+00:00

Thanks for the quick reply. I wonder if this is a common occurrence as it caused a major issue for me as well. I was under the impression that hardware issues should not cause outages on Azure.

For reference this occurred on 2020-11-26 at Canada Central using D64s_v3 Windows VM and I was down for 3 hours.

Accepted answer

0 additional answers

Your answer

prmanhas-MSFT 17,946 Reputation points Microsoft Employee Moderator

2020-11-10T14:47:52.697+00:00

@Metev, Bogdan Any update on the issue?

If the suggested response helped you resolve your issue, do click on "Mark as Answer" and "Up-Vote" for the answer that helped you for benefit of the community.

Thanks.
Andre 1 Reputation point

2020-11-27T07:10:09.807+00:00

I had the same exact issue. Did you find the cause?
Metev, Bogdan 26 Reputation points

2020-11-27T07:17:47.627+00:00

Hi, Andre,
Yes, we did.

This unexpected reboot was caused by an Azure initiated auto-recovery action. The auto-recovery action was triggered by a hardware issue on the physical node where the virtual machine was hosted. As designed, the VM was automatically moved to a different and healthy physical node to avoid further impact.
Microsoft recommended grouping two or more virtual machines in an availability set to ensure an increased level of protection and redundancy for the application in Azure
Andre 1 Reputation point

2020-11-27T10:23:40.373+00:00

Thanks for the quick reply. I wonder if this is a common occurrence as it caused a major issue for me as well. I was under the impression that hardware issues should not cause outages on Azure.

For reference this occurred on 2020-11-26 at Canada Central using D64s_v3 Windows VM and I was down for 3 hours.

Answer 1

@Metev, Bogdan Thank you for your query!!!

As per the error code it refers that the shutdown was planned. The system generates a System State Data (SSD) file. This file contains system state information such as the processes, threads, memory usage, and configuration.
If this flag is not present, the shutdown was unplanned. Notification and reporting options are controlled by a set of policies. For example, after logging in, the system displays a dialog box reporting the unplanned shutdown if the policy has been enabled. An SSD file is created only if the SSD policy is enabled on the system. The administrator can use Windows Error Reporting to send the SSD data to a central location, or to Microsoft.

Same has been documented here.

Error is usually associated with an operation coming from platform ( when you have svchost with an error of 80000000 and the user is NT AUTHORITY/SYSTEN)

Event ID 1074 for Portal/PowerShell/API restarts and shutdowns look exactly the same - same Reason Code, same Shutdown Type of shutdown - this is because Hyper-V doesn't have a graceful restart VM operation. The Reset option for a VM in Hyper-V Manager is a hard reset that results in Event ID 41 "rebooted without cleanly shutting down" error in the VM. The Restart-VM Hyper-V cmdlet does the same thing - hard reset resulting in Event 41 in the guest. That is why portal restarts show up as Shutdown Type shutdown, because Hyper-V doesn't have a restart, only (hard) reset, so a graceful restart is achieved by sending graceful Shutdown + Start.

As this issues need more investigation and live troubleshooting for quicker resolution I would recommend you to contact azure support. If you have a support plan, requesting you to file a support ticket, else please do let us know, we will try and help you get a one-time free technical support.

Hope it helps!!!

Please 'Accept as answer' if it helped, so that it can help others in the community looking for help on similar topics

Share via

Unexpected reboot causing major outage.

0 additional answers

Your answer