Azure Machine Learning Compute Instance Stuck in Starting State

Question

Azure Machine Learning Compute Instance Stuck in Starting State

Niket Kumar Singh 840

I'm encountering an issue with an Azure Machine Learning compute instance that's stuck in the "Starting" state. After some time, it transitions to a "Stopped" state without successfully starting.

Error Message: An internal error has occurred while setting up the node.

Details:

Virtual Machine Size: Standard_D16s_v3 (16 cores, 64 GB RAM, 128 GB disk)
Region: Central India

I would appreciate any insights or suggestions from the Microsoft forum community on how to resolve this issue and successfully start my Azure Machine Learning compute instance.

santoshkc 15,615 Reputation points Microsoft External Staff Moderator

2024-05-14T08:48:51.1166667+00:00
Hi @Niket Kumar Singh,

Thank you for reaching out to Microsoft Q&A forum!

Based on the error message you provided, it seems like there was an issue setting up the virtual machine for your Azure Machine Learning compute instance. Here are some steps you can take to troubleshoot the issue:

Verify that you have the necessary permissions to create and manage compute instances.

Check the resource group for resource limits.

Check the logs to see if there are any errors or warnings.

Try creating a new compute instance with the same virtual machine size and region. And also try creating a smaller virtual machine size.

Try clearing cookies, changing the browser, and logging out and logging back in can sometimes help resolving the issue.

I hope this helps! Thank you.
Niket Kumar Singh 840 Reputation points

2024-05-14T09:43:04.43+00:00

Hi santoshkc,

Not on setting up, it was working fine till yesterday in the morning when we try to start then we start facing issue.
santoshkc 15,615 Reputation points Microsoft External Staff Moderator

2024-05-15T12:03:30.4733333+00:00

Hi @Niket Kumar Singh,

Sorry for the trouble you are facing. Could you please share the error logs. We would recommend sending feedback from ml.azure.com feedback icon to debug further on this.

In the meantime, if you have any resolution, please do share that same with the community as it can be helpful to others. Thank you.
Niket Kumar Singh 840 Reputation points

2024-05-15T12:18:28.91+00:00

Hi santoshkc,

We have temporarily cretetd new compute and assigned to user, we have raised support request with MS team their product team is looking on it, once i have the input will share here.

1 answer

Your answer

santoshkc 15,615 Reputation points Microsoft External Staff Moderator

2024-05-14T08:48:51.1166667+00:00

Hi @Niket Kumar Singh,

Thank you for reaching out to Microsoft Q&A forum!

Based on the error message you provided, it seems like there was an issue setting up the virtual machine for your Azure Machine Learning compute instance. Here are some steps you can take to troubleshoot the issue:

Verify that you have the necessary permissions to create and manage compute instances.

Check the resource group for resource limits.

Check the logs to see if there are any errors or warnings.

Try creating a new compute instance with the same virtual machine size and region. And also try creating a smaller virtual machine size.

Try clearing cookies, changing the browser, and logging out and logging back in can sometimes help resolving the issue.

I hope this helps! Thank you.
Niket Kumar Singh 840 Reputation points

2024-05-14T09:43:04.43+00:00

Hi santoshkc,

Not on setting up, it was working fine till yesterday in the morning when we try to start then we start facing issue.
santoshkc 15,615 Reputation points Microsoft External Staff Moderator

2024-05-15T12:03:30.4733333+00:00

Hi @Niket Kumar Singh,

Sorry for the trouble you are facing. Could you please share the error logs. We would recommend sending feedback from ml.azure.com feedback icon to debug further on this.

In the meantime, if you have any resolution, please do share that same with the community as it can be helpful to others. Thank you.
Niket Kumar Singh 840 Reputation points

2024-05-15T12:18:28.91+00:00

Hi santoshkc,

We have temporarily cretetd new compute and assigned to user, we have raised support request with MS team their product team is looking on it, once i have the input will share here.

Answer 1

Gowtham CP 7,960 Volunteer Moderator

Hello @Niket Kumar Singh

Thanks for reaching out in the Microsoft Q&A!

First, check for any known issues on the Azure Machine Learning known issues and review resource logs in the Azure portal to identify any error messages. Ensure that the chosen virtual machine size is available in your region and investigate resource usage for any signs of insufficient disk space. If the issue persists, retry starting the instance and consider redeploying it if necessary. Additionally, enable SSH access for deeper troubleshooting, and if all else fails, reach out to Azure Support for further assistance. By addressing these potential causes, you should be able to resolve the issue and successfully start your Azure Machine Learning compute instance.

If you found this solution helpful, don't forget to upvote and accept as answer. Thank you!

Niket Kumar Singh 840 Reputation points

2024-05-14T13:01:07.9433333+00:00
hi Gowtham C,

Thank you for ypour response.

We have checked there is no issue with the storage.

Also checked the Azure Machine Learning known issues but it is not listed.

We ahve also checked the quota for the it is sufficent.

whenevr i start it is in loop for 30 min for start then throw error "Compute "xxx" start failed"

Error "An internal error has occurred while setting up the node"

Any insight will be helpful.

Share via

Azure Machine Learning Compute Instance Stuck in Starting State

1 answer

Your answer