Compute has full hard disk and is on hanging status: Restarting

Question

Compute has full hard disk and is on hanging status: Restarting

matsuo_basho 45

I inadvertently filled up my VM instance hard disk space on AzureML to full. I subsequently deleted files from /mnt/tmp and then did sudo reboot as per the recommendation.

The VM has been in Status: Restarting for 10+ hours. I'm no longer able to open the terminal. Is there anything I can do or is this VM a goner?

v-vvellanki-MSFT 4,920 Reputation points Microsoft External Staff

2023-09-26T06:31:44.0066667+00:00

Hi @matsuo_basho ,
Thanks for contacting Q&A platform.

Below are some steps you can take to try and recover the VM or at least gather more information about the issue:

1.      Check Azure Portal Status: First, go to the Azure Portal and check the status of your VM. If it's still stuck in the "Restarting" state, you should try to stop the VM from the portal and then start it again.

2.      Azure Diagnostics: Azure provides diagnostics and monitoring capabilities that can help you understand what's happening with your VM. You can access metrics and logs from the Azure Portal. Look for any unusual error messages or indications of resource exhaustion.

3.      Azure Resource Health: Check the Resource Health status for your VM in the Azure Portal. It can provide insights into whether there are any platform-level issues affecting your VM.

4.      Contact Azure Support: If you've tried the above steps and the VM is still unresponsive, it's a good idea to contact Azure Support for assistance. They can investigate the issue further and may have additional tools and options to recover or diagnose the VM.

Note: You can find Support &Troubleshooting option at the bottom of left pan in your azure portal.

5.      Data Recovery: If it turns out that the VM cannot be recovered, you may need to consider data recovery options. If you have valuable data on the VM, you can attempt to attach the VM's OS disk to another VM as a data disk to recover your files.
matsuo_basho 45 Reputation points

2023-09-26T14:30:05.8766667+00:00

@v-vvellanki-MSFT thank you for the detailed response. So when I access the Azure portal, I don't see any of my AzureML VMs. Consequently, not sure how to access the 'Diagnose and solve problems' setting you suggested.
v-vvellanki-MSFT 4,920 Reputation points Microsoft External Staff

2023-09-27T15:49:12.97+00:00
Hi @matsuo_basho ,

My apologies, I initially believed you were referring to a Virtual Machine, but it seems you're encountering an issue with an Azure ML compute instance.

Below are some steps you can take to try:

Log into the Azure Portal: Go to https://portal.azure.com and sign in with your Azure credentials.

Navigate to Your Azure Machine Learning Workspace: Use the search bar or browse through your resources to find and select your Azure Machine Learning workspace

Access Compute Instances: In the left sidebar, under "Settings," click on "Compute."

Select the Stuck Compute Instance: You will see a list of your compute instances. Find the one that is stuck in the "Restarting" status.

Restart the Compute Instance: Select the problematic compute instance by clicking on its name. In the top menu, you should see options like "Start," "Stop," and "Restart." Choose the "Restart" option to initiate a restart for the selected compute instance.

Monitor the Restart Process: After initiating the restart, monitor the status of the compute instance to see if it successfully restarts. It may take a few minutes for the restart to complete.

Consider a Hard Restart (If Necessary): If the instance remains stuck, you might need to consider a hard restart by stopping and starting it forcibly.
matsuo_basho 45 Reputation points

2023-09-27T16:48:35.44+00:00

@v-vvellanki-MSFT , so of course I considered your recommended approach but as you can see from the screenshot below, all the options (except refresh, delete and diganose) are greyed out.

You mentioned a Hard Restart - maybe that's still possible somehow? Otherwise, it's ok if the answer is 'nothing to be done but to delete the machine'. Thanks
vipullag-MSFT 26,537 Reputation points Moderator

2023-09-29T04:24:42.49+00:00

Hello matsuo_basho

Thanks for responding back on this.

This issue needs deeper investigation. Support team will be able to check and help on this. I would recommend you to open a azure support case.

If you don't have a support plan enabled on your subscription, I request you to send an email to AzCommunity@Microsoft.com with Subject as "Attn:Vikas" referencing this thread along with your subscription ID. I can then enable your subscription with a one-time free technical support.

Once the issue is resolved, request you to post the resolution here for the benefit of community.