VMs in Azure Lab Services are not starting (4/21/2022) !!!

Joe Harrison 6

It would appear that there is something going on with Azure Labs. Was able to start one VM in one lab, but none of the others would start. Tried to start the machine using the PowerShell API, and keep getting this error:

"GatewayTimeout","message":"The gateway did not receive a response from 'Microsoft.LabServices' within the specified time period."

In the lab portal, when I try to start a machine, the button label says "Starting...." and then after a few seconds reverts to "Stopped".

The Azure status dashboard is showing that everything is green. Azure Lab Services is clearly not green.

kobulloc-MSFT 23,336 Reputation points Microsoft Employee

2022-04-21T16:14:42.417+00:00
Hello, @Joe Harrison !

Thank you for including the troubleshooting steps that you've taken:

One VM is working, but not others

Multiple methods used (portal, PowerShell)

Azure status dashboard checked

I would agree, that doesn't seem to match what the Azure status dashboard is reporting.

Are you still running into this? My best guess is that there is an intermittent issue but I'm going to do some more research on my end to see what we can do to get you back up and running.
Joe Harrison 6 Reputation points

2022-04-21T16:33:03.537+00:00

Yes the issue is still there.
kobulloc-MSFT 23,336 Reputation points Microsoft Employee

2022-04-21T16:52:06.077+00:00

@Joe Harrison , I just sent a direct message for some additional information so we can look into this on our end.
TheDataXpert 6 Reputation points

2022-04-21T18:27:51.073+00:00

My Azure Lab Services environments are experiencing the exact issue that JoeHarrison-5397. This issue has been going on since approximately 0700a EST today.
TheDataXpert 6 Reputation points

2022-04-21T18:54:24.31+00:00

My Azure Lab Services environments are now starting up successfully!!!

1 answer

kobulloc-MSFT 23,336 Reputation points Microsoft Employee

2022-04-21T21:43:02.743+00:00

Hello again, @Joe Harrison !

There was an issue with one of the dependent services that was causing both the Starting.../Stopped issue in the portal and resulting in the "GatewayTimeout","message":"The gateway did not receive a response from 'Microsoft.LabServices' within the specified time period." error when using PowerShell.

This has since been resolved and we have performed a root cause analysis to ensure that it doesn't happen again. Thank you very much for bringing this to our attention!
Please sign in to rate this answer.
CG 30 Reputation points

2024-01-03T05:35:15.39+00:00

I am replying to a 2 year old post, but I've been experiencing this problem recently (January 2024) with azure labs.

I have 1 VM, which is currently in stopped state. On hitting the toggle switch, the machine tries to start for about 2 minutes, at which point the toggle comes back to the stopped state.

kobulloc-MSFT 23,336 Reputation points Microsoft Employee

2024-01-04T01:26:02.7266667+00:00

Hello, @CG!

Let's start a new thread with a reference to this thread for tracking. It's possible that this is a related issue or it may be a new issue which requires further investigation.

kobulloc-MSFT 23,336 Reputation points Microsoft Employee

2024-01-04T17:09:13.07+00:00

Hello, @CG! Would these happen to be a GPU lab? There is currently an outage that is being worked on. Updates are being tracked here:

https://techcommunity.microsoft.com/t5/azure-lab-services-blog/outage-gpu-labs-that-use-lab-accounts/ba-p/4020895

CG 30 Reputation points

2024-01-04T18:45:08.9966667+00:00

Yes these are GPU-enabled VMs
Sign in to comment