Azure Automation powershell runbook issue

Question

Hello,

I am creating an Azure Automation account having a powershell runbook. The script is simply looping over each VM in a resource group and deleting all files under a particular folder.

$ResourceGroup = '<<myResourceGroup>>'
$TargetDir = '<targetDirectoryPath/.'
$TargetCommand = 'rm -rfv ' + $TargetDir

try
{
	$message = Connect-AzAccount -Identity 	
	
	$myAzureVMs = Get-AzVM -ResourceGroupName $ResourceGroup -status | Where-Object {$_.PowerState -eq "VM running" -and $_.StorageProfile.OSDisk.OSType -eq "Linux"}
	
	if($myAzureVMs.Name.Count -gt 1)
	{
		For($val = 0; $val -le $myAzureVMs.Name.Count-1; $val++)
		{ 
           # the script hangs here
            $message = Invoke-AzVMRunCommand -ResourceGroupName $ResourceGroup -Name $myAzureVMs.Name[$val] -CommandId 'RunShellScript' -ScriptString $TargetCommand 
		}	 
	}

	Write-Output "DONE !!"
}
catch
{
	Write-Error -Message $_.Exception
    throw $_.Exception
}

The script is working fine but the issue is if a VM is not ok or has some connectivity problem then the script hangs just there. No error or exception is thrown. It just keeps waiting and eventually times out after hours.

My question is there any way to check if the VM can be connected or not (or any better solution) and then only the script should execute. I have also tried putting the Invoke-AzVMRunCommand inside a try-catch block to eat up the exception and to move to the next VM but in vain. It's simply hangs on to that VM and no exception thrown.

Please advise.

Answer 1

Atanu Gupta, I understand that you are trying to iterate through all the VMs in a resource and run command using Invoke-AzVMRunCommand - which seems to be getting stuck occasionally. I tested a couple of scenarios and here are some suggestions that should help you with this scenario. I have also included some updates to the script's logic for smoother operation.

1. The Run Command utilizes the "VM Agent" installed on the Azure VM for running the script. A check to ensure that this is in "Ready" state should be included in the script. This is obtained by running "Get-AzVm" and get the instance view (note that you will have to run the Get-AzVm for the particular VM with -Status flag to get the VMAgent status. See this link). The script can be modified like below for this to be included:

   try {
       # Get all VMs in running state
       $myAzureVMs = Get-AzVM -ResourceGroupName $ResourceGroup -status | Where-Object { $_.PowerState -eq "VM running" -and $_.StorageProfile.OSDisk.OSType -eq "Linux" }
   	
       foreach ($VM in $myAzureVMs) 
       {
           # additional check at instance level to ensure that the VMAgent is in running state. 
               
           #get the status of VMAgent which is required to run command using Invoke-AzVMRunCommand
           $VMAgent_Status = (Get-AzVm -ResourceGroupName $ResourceGroup -Name $VM.Name `
                                                                   -Status).VMAgent.Statuses.DisplayStatus
               
           if ($VMAgent_Status -eq 'Ready') 
           {  <call Invoke-AzVMRunCommand> }

2. Another suggestion would be to run the Invoke-AzVMRunCommand in Job mode (using -AsJob flag). See this link for details. This ensures that each command runs as a background job and does not cause blocking for further calls. You can read about jobs in PowerShell here - about_Job_Details

The complete script with this suggestion would look something like

# Ensures you do not inherit an AzContext in your runbook
Disable-AzContextAutosave -Scope Process
# Connect to Azure with system-assigned managed identity
$AzureContext = (Connect-AzAccount -Identity).context
# Set and store context
$AzureContext = Set-AzContext -SubscriptionName $AzureContext.Subscription -DefaultProfile $AzureContext
$ResourceGroup = <resourceGroupName>
$TargetCommand = <command to run>
try {	
    # Get all VMs in running state
    $myAzureVMs = Get-AzVM -ResourceGroupName $ResourceGroup -status | Where-Object { $_.PowerState -eq "VM running" -and $_.StorageProfile.OSDisk.OSType -eq "Linux" }
    foreach ($VM in $myAzureVMs) 
    {
    
        # additional check at instance level to ensure that the VMAgent is in running state.         
        # get the status of VMAgent which is required to run command using Invoke-AzVMRunCommand
        $VMAgent_Status = (Get-AzVm -ResourceGroupName $ResourceGroup -Name $VM.Name -Status).VMAgent.Statuses.DisplayStatus
            
        if ($VMAgent_Status -eq 'Ready') 
        {
            # run the command as background job
            $job = Invoke-AzVMRunCommand -ResourceGroupName $ResourceGroup -Name $VM.Name -CommandId 'RunShellScript' -ScriptString $TargetCommand -AsJob
        }
        else 
        {
            Write-Error -Message "$($VM.Name) : Operation failed as VMAgent is not in Ready state"
        }
    }	 
    Write-Output "Sleeping for 10 seconds" # to ensure that all background process get enough time to initiate the command
    Start-sleep -seconds 10
    Write-Output "Woke-up"
    $job = Get-Job
    Write-output "Job count - $($job.count)"
    
    #Add additional logic, as required to check status of all job and stop them after certain time (similar to timeout)
}
catch 
{
    Write-Error -Message $_.Exception
    throw $_.Exception
}

Hope the suggestions above help. Please let me know if you have any questions.