Virtual machines going to FAILED state and then we can't do anything with related resources

Question

Hi! In the subscription

9977b1c2-...

we are running our virtual machines. When we try to delete them, for some reason, they go onto FAILED state and we can't do anything with them. We have 4-9 disks attached to them and all the disks are in the "detaching" phase which lasts forever. We can't delete any of the disks, since they have an ongoing conflicting operation (detachment). We can't stop or restart the VM, nor can we swap the OS disk.

This is causing downtime for us and we need it to be solved ASAP. There are no errors or alerts displayed and we really don't have any idea what's going on. From our code, we issue a request to delete a VM, the request itself doesn't throw any errors, while the machine goes into the FAILED state.

Please, help us resolve this. If possible, let us know where we can see what the issue is, since we've tried many things. Also, if you can see it from your side, that would be great!

Looking forward to collaborating with you and trying to resolve this.

Answer 1

Mirko hi, wow, that sounds like a real headache )) virtual machines stuck in 'failed' state with disks hanging in 'detaching' limbo... yeah, that's no fun...........

u might wanna try forcing the vm deletion using azure cli or powershell. sometimes the portal gets stuck but command line pushes things through. just make sure u grab the vm id first.

check if there are any locks on the resources. azure resource locks can block deletions even if everything else seems fine. u can list them with 'az lock list' and remove if needed microsoft docs on resource locks.

as well check this... if the disks are stuck detaching, u might need to manually break the lease. azure storage has this thing called 'break lease' which can force-release a disk microsoft docs on breaking leases.

when resources get stuck, retry the operation after a few mins. sometimes the backend just needs a breather )) also, look at activity logs or audit trails, they often hide clues about what went wrong. worth looking into, even if u don't see errors upfront.

aha, and one more thing... if u're automating deletions, add some retry logic. api calls can fail silently, so code should handle that. this might help in other tools too, not just azure.

hope this unsticks things for u! if not, hit me up :)) good luck

Best regards,

Alex

and "yes" if you would follow me at Q&A - personaly thx.
P.S. If my answer help to you, please Accept my answer
PPS That is my Answer and not a Comment

https://ctrlaltdel.blog/

Answer 2

Hello @Mirko Krajcer,

Product Team identified an internal issue within a specific partition of the Disk service, caused by a leaked transaction that led to prolonged datastore contention. Our Product and Platform teams have implemented repair actions to enhance telemetry and introduce auto-healing mechanisms, ensuring improved resilience and prevention of recurrence. 

This was a purely internal matter, and no action is required from your side. We regret the inconvenience caused and are committed to resolving this issue and improving the resilience of the service. We appreciate your understanding as we continue to strengthen the reliability of our service.

Hope it helps!