Auto restart the evicted VM

Question

We have a spot virtual machine that gets deallocated or switched off when the eviction policy is triggered. I would like to automate the restart of the evicted virtual machine based on the trigger i receive . Is there any way i can acheive this automation

Answer 1

Hi Anugrah Gaikwad,

Thanks for reaching out to Microsoft Q&A.

The simple way for automating the restart of an evicted spot virtual machine (VM) in Azure can be achieved using Azure Monitor alerts and Azure Functions.

Use Scheduled Events to Monitor for Eviction:

The key is to monitor for the Preempt signal that Azure sends to the VM before it is evicted. You can use the Azure Instance Metadata Service to query for scheduled events, including the Preempt signal, which is sent at least 30 seconds before the actual eviction.You can use the following Python code to poll the Instance Metadata Service and detect the Preempt signal.

import requests
import platform
import os
def stop_service():
    # Add your logic to stop the service here
    print("Stopping service before eviction")
def monitor_eviction():
    while True:
        try:
            response = requests.get("http://169.254.169.254/metadata/scheduledevents?api-version=2019-08-01", headers={"Metadata":"true"}, timeout=1)
            if response.status_code == 200:
                events = response.json()["Events"]
                if events:
                    for event in events:
                        if event["EventType"] == "Preempt":
                            print(f"Eviction event detected: {event}")
                            stop_service()
                            break
        except requests.exceptions.RequestException:
            pass
if __name__ == "__main__":
    # Check if the script is running on an Azure VM
    if "azure" in os.uname().release.lower():
        monitor_eviction()
    else:
        print("This script is not running on an Azure VM.")

The above script continuously polls the Instance Metadata Service and checks for the Preempt signal. When the signal is detected, it calls the stop_service() function, which you can modify to stop your application and perform any necessary cleanup before the VM is evicted.

Use Azure Functions to Automate the Restart:

Once you have detected the eviction event, you can use an Azure Function to automatically restart the evicted VM.

import os
import azure.functions as func
from azure.mgmt.compute import ComputeManagementClient
from azure.identity import DefaultAzureCredential
def main(mytimer: func.TimerRequest) -> None:
    credential = DefaultAzureCredential()
    compute_client = ComputeManagementClient(credential, os.environ["AZURE_SUBSCRIPTION_ID"])
    # Replace with the name of your evicted VM and its resource group
    vm_name = "your-evicted-vm"
    resource_group_name = "your-resource-group"
    try:
        # Start the evicted VM
        compute_client.virtual_machines.begin_start(resource_group_name, vm_name).result()
        print(f"Restarted the evicted VM: {vm_name}")
    except Exception as e:
        print(f"Error restarting the evicted VM: {e}")

The above code is for the Azure Function which is triggered by a timer. You can configure this to run at a regular interval (every minute) to check for and restart the evicted VM. Replace the vm_name and resource_group_name variables with the appropriate values for your evicted VM. This Azure Function is triggered by a timer, which you can configure to run at a regular interval (every minute) to check for and restart the evicted VM. Replace the vm_name and resource_group_name variables with the appropriate values for your evicted VM.

Try this and let me know.

Please 'Upvote'(Thumbs-up) and 'Accept' as an answer if the reply was helpful. This will benefit other community members who face the same issue.