James Tabor’s blog

Pause Idle Azure SQL Data Warehouse Instances

James R Tabor — Mon, 20 Nov 2017 12:11:07 GMT

Azure SQL Data Warehouse (SQL DW) instances are quite expensive to keep running, so it makes sense to Pause them if they are not being used. This script will NOT resume them if connection attempts are made to a paused SQL DW instance. The easiest way to check if it has been awhile since the SQL DW has 'done anything' is to query a system view called 'dm_pdw_exec_requests'. The script (found here Pause-Idle-SQL-DW-Instances ) will

Get all the SQL Servers defined in a subscription
Loop through these SQL Servers and get all the online SQL DW instances
Connect to the SQL DW and query the dm_pdw_exec_requests system view to get the last time a query ended
If a specified threshold of minutes has elapsed, then pause the SQL DW instance.

You will need to add SQL usernames and passwords into a KeyVault as secrets in order to connect to the SQL DW instances. The KeyVault Secret Name for UserName and Password is in the format

UserName: '{SQL Server Name}{SQL DW Instance Name}UserName'
Password: '{SQL Server Name}{SQL DW Instance Name}Password'

Resize Azure VMs in bulk

James R Tabor — Mon, 30 Oct 2017 19:33:31 GMT

If you need to do a bulk update of VM sizes from one type to another to take advantage of new technology being released, this script should help you out. This script will work for Resource Manager VMs, and NOT Classic VMs. It targets VMs that are deallocated or have failed deployments only. VMs in a Failed deployment state will automatically start when re-sized. I've added a select grid at the end to let you toggle the state of the VMs that have been changed. i.e Selected VMs will be Started if Deallocated and Stopped if in any other state I've preloaded the SizeMapping table with the mapping to move from Dv2 Promo sizes to their equivilent Dv3 and Ev3 sizes. You can change this as to your requirements, just set the Old and New elements for each record. Make sure only select ONE subscription from the Select Grid. [sourcecode gutter="false" title="Resize VMs" language="powershell"] #If you want to stop running VMs, Resize and restart again, set this to $true $ForceStop = $false #Prompt to confirm shutdown for each VM $StopConfirm = $true #Filter on VM name to restrict the list of VMs returned $VMFilter = "*" $AvailSubscriptions = Get-AzureRmSubscription $ErrorActionPreference = "Stop" $SelectedSubscription = $AvailSubscriptions | select Name, Id | Out-GridView -Title "Select ONE (only) Subscription" -PassThru $SubscriptionGUID = $SelectedSubscription.Id Select-AzureRmSubscription -Subscription $SubscriptionGUID #Update this table if you have different mapping size requirements from old to new $SizeMapping = @() $SizeMapping += @{Old='Standard_D2_v2_Promo';New='Standard_D2_v3'} $SizeMapping += @{Old='Standard_D3_v2_Promo';New='Standard_D4_v3'} $SizeMapping += @{Old='Standard_D4_v2_Promo';New='Standard_D8_v3'} $SizeMapping += @{Old='Standard_D5_v2_Promo';New='Standard_D16_v3'} $SizeMapping += @{Old='Standard_D11_v2_Promo';New='Standard_E2_v3'} $SizeMapping += @{Old='Standard_D12_v2_Promo';New='Standard_E4_v3'} $SizeMapping += @{Old='Standard_D13_v2_Promo';New='Standard_E8_v3'} $SizeMapping += @{Old='Standard_D14_v2_Promo';New='Standard_E16_v3'} $SizeMapping += @{Old='Standard_DS2_v2_Promo';New='Standard_D2s_v3'} $SizeMapping += @{Old='Standard_DS3_v2_Promo';New='Standard_D4s_v3'} $SizeMapping += @{Old='Standard_DS4_v2_Promo';New='Standard_D8s_v3'} $SizeMapping += @{Old='Standard_DS5_v2_Promo';New='Standard_D16s_v3'} $SizeMapping += @{Old='Standard_DS11_v2_Promo';New='Standard_E2s_v3'} $SizeMapping += @{Old='Standard_DS12_v2_Promo';New='Standard_E4s_v3'} $SizeMapping += @{Old='Standard_DS13_v2_Promo';New='Standard_E8s_v3'} $SizeMapping += @{Old='Standard_DS14_v2_Promo';New='Standard_E16s_v3'} #You can use a $VMFilter, set above, to limit the script to only add filtered VM names to the array $VMs = Get-AzureRmVM | Where-Object {$_.Name -like $VMFilter} $AlteredVMs = @() foreach ($VM in $VMs) { if ($VM.HardwareProfile.VmSize -in $SizeMapping.Old) { $Status = Get-AzureRmVM -ResourceGroupName $VM.ResourceGroupName -Name $VM.Name -Status $NewSize = $SizeMapping.New[$SizeMapping.Old.IndexOf($VM.HardwareProfile.VmSize)] #If the VM is deallocated, just resize it. if ($Status.Statuses[1].DisplayStatus -eq "VM deallocated") { write-output "Updating VM: $($VM.Name) from $($VM.HardwareProfile.VmSize) to $($NewSize)" $VM.HardwareProfile.VmSize = $NewSize Update-AzureRmVM -VM $VM -ResourceGroupName $VM.ResourceGroupName -Verbose $AlteredVM = new-object psobject -Property @{ Name = $VM.Name ResourceGroup = $VM.ResourceGroupName Location = $VM.Location ResourceID = $VM.Id SubscriptionID = $SubscriptionGUID OldSize = OldProvisioningStatus = $Status.Statuses[0].DisplayStatus OldRunStatus = $Status.Statuses[1].DisplayStatus NewProvisioningStatus = $null NewRunStatus = $null } $AlteredVMs += $AlteredVM } #If the VM has failed provisioning, try and stop the VM firstso it clears the allocation error and doesn't start automatically after changing size if ($Status.Statuses[0].DisplayStatus -eq "Provisioning failed") { $StopAction = $false $TryCount = 0 $ErrorActionPreference = "Stop" do { try { $TryCount += 1 Write-Output "Stopping VM: $($VM.Name)" Stop-AzureRmVM -Name $VM.Name -ResourceGroupName $VM.ResourceGroupName -Force $StopAction=$true } catch { sleep -10 $StopAction = $false } } until ($StopAction -eq $true -or $TryCount -ge 5) $ErrorActionPreference = "Continue" write-output "Updating VM: $($VM.Name) from $($VM.HardwareProfile.VmSize) to $($NewSize)" $VM.HardwareProfile.VmSize = $NewSize Update-AzureRmVM -VM $VM -ResourceGroupName $VM.ResourceGroupName -Verbose $AlteredVM = new-object psobject -Property @{ Name = $VM.Name ResourceGroup = $VM.ResourceGroupName Location = $VM.Location ResourceID = $VM.Id SubscriptionID = $SubscriptionGUID OldProvisioningStatus = $Status.Statuses[0].DisplayStatus OldRunStatus = $Status.Statuses[1].DisplayStatus NewProvisioningStatus = $null NewRunStatus = $null } $AlteredVMs += $AlteredVM } #If you have ForceStop set to $true and the VM is running if ($Status.Statuses[1].DisplayStatus -eq "VM running" -and $ForceStop) { if ($StopConfirm) { $ConfirmStop = Read-Host -Prompt "Please type 'YES' to confirm you want to stop VM $($VM.Name)" if ($ConfirmStop -eq "YES") { $StopAction = $false $TryCount = 0 $ErrorActionPreference = "Stop" do { try { $TryCount += 1 Write-Output "Stopping VM: $($VM.Name)" Stop-AzureRmVM -Name $VM.Name -ResourceGroupName $VM.ResourceGroupName -Force $StopAction=$true } catch { sleep -10 $StopAction = $false } } until ($StopAction -eq $true -or $TryCount -ge 5) $ErrorActionPreference = "Continue" write-output "Updating VM: $($VM.Name) from $($VM.HardwareProfile.VmSize) to $($NewSize)" $VM.HardwareProfile.VmSize = $NewSize Update-AzureRmVM -VM $VM -ResourceGroupName $VM.ResourceGroupName -Verbose Write-Output "Starting VM: $($VM.Name) after resize" Start-AzureRmVM -Name $VM.Name -ResourceGroupName $VM.ResourceGroupName $AlteredVM = new-object psobject -Property @{ Name = $VM.Name ResourceGroup = $VM.ResourceGroupName Location = $VM.Location ResourceID = $VM.Id SubscriptionID = $SubscriptionGUID OldProvisioningStatus = $Status.Statuses[0].DisplayStatus OldRunStatus = $Status.Statuses[1].DisplayStatus NewProvisioningStatus = $null NewRunStatus = $null } $AlteredVMs += $AlteredVM } } } } } $CurrentVMState = @() foreach ($AlteredVM in $AlteredVMs) { $Status = Get-AzureRmVM -ResourceGroupName $AlteredVM.ResourceGroup -Name $AlteredVM.Name -Status $AlteredVM.NewProvisioningStatus = $Status.Statuses[0].DisplayStatus $AlteredVM.NewRunStatus = $Status.Statuses[1].DisplayStatus $CurrentVMState += $AlteredVM } $ToggleStates = $null if ($CurrentVMState -ne $null) { $ToggleStates = $CurrentVMState | select Name, ResourceGroup, NewRunStatus | Out-GridView -Title "Select VMs to toggle state (Running to Deallocated; Deallocated to Running); Click Cancel to change none" -PassThru foreach ($ToggleState in $ToggleStates) { if ($ToggleState.NewRunStatus -eq "VM deallocated") { Start-AzureRmVM -ResourceGroupName $ToggleState.ResourceGroup -Name $ToggleState.Name } else { Stop-AzureRmVM -ResourceGroupName $ToggleState.ResourceGroup -Name $ToggleState.Name -Force } } } [/sourcecode]

Azure VMs - Active Directory members and getting time sync (ntp) right

James R Tabor — Sat, 28 Oct 2017 13:52:14 GMT

I've recently had this question asked of me multiple times recently. "If a VM in Azure is a member of a Domain, where does it get its time from, Azure (time.microsoft.com) or the NTP source set by AD?" The answer is both, unless you make some changes. Here are the specific time setting recomendations, buried inside this post https://docs.microsoft.com/en-us/windows-server/identity/ad-ds/get-started/windows-time-service/accurate-time Azure and Windows IaaS considerations

Azure Virtual Machine: Active Directory Domain Services If the Azure VM running Active Directory Domain Services is part of an existing on-premises Active Directory Forest, then TimeSync(VMIC), should be disabled. This is to allow all DCs in the Forest, both physical and virtual, to use a single time sync hierarchy. Refer to the best practice whitepaper “Running Domain Controllers in Hyper-V”
Azure Virtual Machine: Domain-joined machine If you are hosting a machine which is domain joined to an existing Active Directory Forest, virtual or physical, the best practice is to disable TimeSync for the guest and ensure W32Time is configured to synchronize with its Domain Controller via configuring time for Type=NTP5
Azure Virtual Machine: Standalone workgroup machine If the Azure VM is not joined to a domain, nor is it a Domain Controller, the recommendation is to keep the default time configuration and have the VM synchronize with the host.

So in short, for Active Directory VMs in Azure, update the registry path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\TimeProviders\VMICTimeProvider and set the value for 'Enabled' to 0 To do this via Group Policy the setting looks something like this -

Using OMS and Runbooks to update VMs when they're down

James R Tabor — Tue, 25 Jul 2017 16:47:45 GMT

The aim of this post is to enable a powershell Azure Automation runbook to be fired when a VM is deallocated (Stopped from the Azure management fabric), that will perform given tasks. This is useful if you have non-urgent infrastructural changes that need to be done, but you want it done with the least amount of user impact. When the user, or a schedule stops the VM, the runbook is trigger. You will need an OMS workspace set-up and linked to an Azure Automation account. In the linked Automation Account, create a new Runbook called "DeallocationAction". This will accept the OMS log alert results and allow the required actions to be performed Here is one I've written to give you the idea and get you started. This runbook checks for VMs in East US, created by System and resizes them based on a mapping array. [sourcecode gutter="false" title="Runbook - DeallocationAction" language="powershell"] Param ( [parameter(Mandatory=$false)] [object] $WebhookData ) # Collect properties of WebhookData $WebhookName = $WebhookData.WebhookName $WebhookHeaders = $WebhookData.RequestHeader $WebhookBody = $WebhookData.RequestBody # Collect results. Information converted from JSON. $SearchResults = (ConvertFrom-Json $WebhookBody).SearchResults.value #Retrive the RunAs Connection details $Conn = Get-AutomationConnection -Name 'AzureRunAsConnection' #Login with the RunAs Account Add-AzureRMAccount -ServicePrincipal -Tenant $Conn.TenantID ` -ApplicationId $Conn.ApplicationID -CertificateThumbprint $Conn.CertificateThumbprint $SizeMap = @() $SizeMap += @{Old='Standard_F1';New='Standard_D1'} $SizeMap += @{Old='Standard_F2';New='Standard_D2'} $SizeMap += @{Old='Standard_F4';New='Standard_D3'} $SizeMap += @{Old='Standard_F8';New='Standard_D4'} $CurrSub = $null $CurrSub = (Get-AzureRmContext).Subscription.Id foreach ($Result in $SearchResults) { Write-Output "Result" Write-Output $Result #Make sure it's the final succeeded action, and not Started or Accepted if ($Result.ActivityStatus -eq "Succeeded") { #Change the Subscription if required if ($CurrSub -ne $Result.SubscriptionId) { Select-AzureRmSubscription -SubscriptionId $Result.SubscriptionId $CurrSub = $Result.SubscriptionId } #Get the VM details $ResourceGroup = $Result.ResourceGroup $VMName = $Result.Resource $VM = Get-AzureRmVM -ResourceGroupName $ResourceGroup -Name $VMName $CurrSize = $VM.HardwareProfile.VmSize Write-Output "$($VMName) - Current VM Size: $($CurrSize)" #Check the machine is still deallocated $VMStatus = Get-AzureRmVM -ResourceGroupName $ResourceGroup -Name $VMName -Status if ($VMStatus.Statuses[1].Code -eq "PowerState/deallocated") { #Put any additional filters in here for continuing to change VM size #These are negatives, start with the VM being included then remove based on filters $Enabled = $true if ($VM.Location.ToLower() -ne "eastus") { $Enabled = $false } if ($VM.Tags.CreatedBy -ne "System") { $Enabled = $false } #Continue if passed through previous filters if ($Enabled) { #Make the required changes foreach ($Size in $SizeMap) { if ($CurrSize -eq $Size.Old) { $NewSize = $Size.New } } if ($CurrSize -ne $NewSize) { Write-Output "$($VMName) - New VM Size: $($NewSize)" #Update the VM Size $VM.HardwareProfile.VmSize = $NewSize #Apply the change $UpdateVM = Update-AzureRmVM -VM $VM -ResourceGroupName $ResourceGroup #If the update succeeds, update the SizeChange Tag if ($UpdateVM.IsSuccessStatusCode) { Write-Output "$($VMName) - Change Succeeded" $SizeChange = @{Old=$($CurrSize);UTC=(Get-Date (Get-Date).ToUniversalTime() -Format "dd/MM/yy HH:mm")} $SizeJSON = $SizeChange | ConvertTo-Json -Compress $Tags = $null $Tags = (Get-AzureRmResource -ResourceId $VM.Id).Tags if ($Tags -eq $null) { $Tags = @{ SizeChanged = $SizeJSON } } else { if ($Tags.ContainsKey("SizeChanged")) { #Make sure the current JSON is configured as an array $a = [regex]"\[*\]" if ($a.Match($Tags.SizeChanged).Success) { $CurrSizeChange = (ConvertFrom-Json $Tags.SizeChanged) } else { $CurrSizeChange = (ConvertFrom-Json "[`n$($Tags.SizeChanged)`n]") } $CurrSizeChange += $($SizeChange) $SizeJSON = $CurrSizeChange | ConvertTo-Json -Compress #Make sure Tag value is less than 256, keep dropping first record off until under 256 in length $RecSkip = 0 do { $RecSkip $SizeJSON = $CurrSizeChange | Select-Object -Skip $RecSkip | ConvertTo-Json -Compress $RecSkip += 1 } until ($SizeJSON.Length -lt 256) #Set Tag value $Tags.SizeChanged = $SizeJSON } else { $Tags += @{SizeChanged=$SizeJSON} } } Set-AzureRmResource -Tag $Tags -ResourceId $VM.Id -Force } else { Write-Output "$($VMName) - Change Failed" } } } } } } [/sourcecode] Next, create an alert on the following OMS log query [sourcecode gutter="false" title="OMS Log Query" language="text"] Type=AzureActivity AND OperationName = "Microsoft.Compute/virtualMachines/deallocate/action" AND ActivityStatus="Succeeded" [/sourcecode] Set the other attributes for the alert as follows Time window: 5 minutes Frequency: 5 minutes Number of results greater than 0 Runbook: YES Select the runbook created above Now whenever a VM Deallocation event occurs and is logged through from Azure Activity to OMS (this can take up to 15 minutes), the Runbook you created is triggered and sent the VM information (within 5 minutes) so the required actions can be performed.

Make practical use of NSG Flow Logs for troubleshooting

James R Tabor — Fri, 07 Jul 2017 13:05:27 GMT

So, you've found the new Azure Network Watcher features. Cool, now I can start getting some real information about what's going on in my Azure Network. The NSG flow logs section of the Network Watcher blade, in the Azure Portal, lets you specify a Storage Account for each NSG (Network Security Group) to output detailed information on all the traffic hitting the NSG. Note: The Storage Account must be in the same region as the NSG Awesome right! Then you try and use that information and find there are a heap of sub-folders created for each Resource Group, NSG all with separate files for each hour of logging. Also, they are all in JSON format and really hard to find anything specific. I don't know about you, but I found it really hard to troubleshoot a particular network communication issue with a specific VM and/or Subnet. So, I've written the following Powershell script that pulls all these files from the Storage Account down to a local folder and then builds an array of these 'Events'. The properties built out into the array are - ResourceGroup, NSG, Ref, SrcIP, DstIP, SrcPort, DstPort, Protocol 'T(TCP)/U(UDP)', Direction 'O(Outbound)/I(Inbound)', Type 'A(Accept)/D(Denied)', Time, RuleType, and RuleName. Version 1 : Version 2: This is then presented in a GridView that can be sorted and filtered to find what you are looking for. I've also added an option at the end to dump it out to a CSV if you need to send the output to someone else. Note: Due to the folder nesting for the NSG logging you will need to create a 'short' local root folder to cache locally. Generally the $env:TEMP location will be too many folders deep to allow for the required folder nesting for NSG logs. I've used C:\Temp\AzureNSGFlowLogs here, you will need to either create this folder or update that variable set line in the script. Update [July 22nd 2017]: Allow for user selection of Storage Account and cater for multiple NSGs, in multiple locations, logging to different Storage Accounts. Update [Nov 7th 2018]: Update Find-AzureRmResource to Get-AzureRmResource due to deprecated command (thanks to Victor Dantas Mehmeri for finding) Update [Nov 7th 2018]: Added Version 2 new fields - State (B-Begin, C- Continue, & E-End), Send and Recv Packets and Bytes. [sourcecode gutter="false" title="NSGFlowLogAnalysis" language="powershell"] #Prompt user to select storage account(s) $StorageAccountNames = (Get-AzureRmStorageAccount | Select-Object Location, StorageAccountName | Out-GridView -Title "Select Storage Account(s)" -PassThru).StorageAccountName #$LocalFolder = "$($env:TEMP)\AzureNSGFlowLogs" $LocalFolder = "C:\Temp\AzureNSGFlowLogs" #If you want to override where the local copy of the NSG Flow Log files go to, un-comment the following line and update $TimeZoneStr = (Get-TimeZone).Id #If you want to override the Timezone to something other than the locally detected one, un-comment the following line and update #$TimeZoneStr = "AUS Eastern Standard Time" $StorageDetails = @() foreach ($StorageAccountName in $StorageAccountNames) { $StorageResource = $null $ResourceGroupName = $null $StoKeys = $null $StorageAccountKey = $null $StorageAccountName $StorageResource = Get-AzureRmResource -ResourceType Microsoft.Storage/storageAccounts -Name $StorageAccountName $ResourceGroupName = $StorageResource.ResourceGroupName $ResourceGroupName $StoKeys = Get-AzureRmStorageAccountKey -ResourceGroupName $ResourceGroupName -Name $StorageAccountName $StorageAccountKey = $StoKeys[0].Value $StorageContext = New-AzureStorageContext -StorageAccountName $StorageAccountName -StorageAccountKey $StorageAccountKey $StorageDetail = New-Object psobject -Property @{ StorageAccountName = $StorageAccountName StorageResource = $StorageResource ResourceGroupName = $ResourceGroupName StorageAccountKey = $StorageAccountKey StorageContext = $StorageContext } $StorageDetails += $StorageDetail } $SubscriptionID = (Get-AzureRmContext).Subscription.Id #Function to create the sub folders as required function MakeFolders ( [object]$Path, [string]$root ) { if (!(Test-Path -Path $root)) { $rootsplit = $root.Split("\") $tPath = '' foreach ($tPath in $rootsplit) { $BuildPath = "$($BuildPath)$($tPath)\" $BuildPath if (!(Test-Path -Path $BuildPath)) { mkdir $BuildPath } } } $Build = "$($root)" foreach ($fld in $path) { $Build = "$($Build)\$($fld)" #$Build mkdir "$($build)" -ErrorAction SilentlyContinue } } #End function MakeFolders #Function to get the Timezone information function GetTZ ( [string]$TZ_string ) { $r = [regex] "\[([^\[]*)\]" $match = $r.match($($TZ_string)) # If there is a successful match for a Timezone ID if ($match.Success) { $TZId = $match.Groups[1].Value # Try and get a valid TimeZone entry for the matched TimeZone Id try { $TZ = [System.TimeZoneInfo]::FindSystemTimeZoneById($TZId) } # Otherwise assume UTC catch { $TZ = [System.TimeZoneInfo]::FindSystemTimeZoneById("UTC") } } else { try { $TZ = [System.TimeZoneInfo]::FindSystemTimeZoneById($TZ_string) } catch { $TZ = [System.TimeZoneInfo]::FindSystemTimeZoneById("UTC") } } return $TZ } #end function GetTZ #Set the $TZ variable up with the required TimeZone information $TZ = GetTZ -TZ_string $TimeZoneStr #Get all the blobs in the insights-logs-networksecuritygroupflowevent container from the specified Storage Account $Blobs = @() foreach ($StorageDetail in $StorageDetails) { #$StorageContext $Blob = Get-AzureStorageBlob -Container "insights-logs-networksecuritygroupflowevent" -Context $StorageDetail.StorageContext $Blobs += $Blob } #Build an array of the available selection criteria $AvailSelection = @() foreach ($blobpath in $blobs.name) { $PathSplit = $blobpath.Split("/") $datestr = Get-Date -Year $PathSplit[9].Substring(2) -Month $PathSplit[10].Substring(2) -Day $PathSplit[11].Substring(2) -Hour $PathSplit[12].Substring(2) -Minute $PathSplit[13].Substring(2) -Second 0 -Millisecond 0 -Format "yyyy-MM-ddTHH:mm:ssZ"#2017-07-06T02:53:02.3410000Z $blobdate = (get-date $datestr).ToUniversalTime() #Write-Output "RG: $($PathSplit[4])`tNSG:$($PathSplit[8])`tDate:$($blobdate)" $SelectData = New-Object psobject -Property @{ ResourceGroup = $PathSplit[4] NSG = $PathSplit[8] Date = $blobdate } $AvailSelection += $SelectData } #Prompt user to select the required ResourceGroup(s), NSG(s), and hourly sectioned files $SelectedResourceGroups = ($AvailSelection | Select-Object -Property "ResourceGroup" -Unique | Out-GridView -Title "Select required Resource Group(s)" -PassThru).ResourceGroup $selectedNSGs = ($AvailSelection | Where-Object {$_.ResourceGroup -in $SelectedResourceGroups} | Select-Object -Property "NSG" -Unique | Out-GridView -Title "Select required NSG(s)" -PassThru).NSG $SelectedDates = ($AvailSelection | Where-Object {$_.ResourceGroup -in $SelectedResourceGroups -and $_.NSG -in $selectedNSGs} | Select-Object -Property @{n="System DateTime";e={$_.Date}},@{n="Local DateTime";e={[System.TimeZoneInfo]::ConvertTimeFromUtc((Get-Date $_.Date),$TZ)}} -Unique | Sort-Object 'System DateTime' -Descending| Out-GridView -Title "Select required times (1 hour blocks)" -PassThru).'System DateTime' #Loop though blobs and download any that meet the specified selection and are newer than those already downloaded foreach ($blob in $blobs) { $PathSplit = $blob.Name.Split("/") $blobdate = get-date -Year $PathSplit[9].Substring(2) -Month $PathSplit[10].Substring(2) -Day $PathSplit[11].Substring(2) -Hour $PathSplit[12].Substring(2) -Minute $PathSplit[13].Substring(2) -Second 0 -Millisecond 0 if ($PathSplit[4] -in $SelectedResourceGroups -and $PathSplit[8] -in $selectedNSGs -and $blobdate -in $SelectedDates) { $fld = $blob.name.Replace("/","\") $flds = $fld.Split("\") | Where-Object {$_ -ne "PT1H.json"} $lcl = "$($LocalFolder)\$($fld)" MakeFolders -Path $flds -root $LocalFolder if (Test-Path -Path $lcl) { $lclfile = Get-ItemProperty -Path $lcl $lcldate = (get-date $lclfile.LastWriteTimeUtc) } else { $lcldate = get-date "1 Jan 1970" } $blobdate = $blob.LastModified if ($blobdate -gt $lcldate) { Write-Output "Copied`t$($blob.Name)" Get-AzureStorageBlobContent -Container "insights-logs-networksecuritygroupflowevent" -Context $blob.Context -Blob $blob.Name -Destination $lcl -Force } else { Write-Output "Leave`t$($blob.Name)" } } } #Get a list of all the files in the specified local directory $Files = dir -Path "$($LocalFolder)\resourceId=\SUBSCRIPTIONS\$($SubscriptionID)\RESOURCEGROUPS\" -Filter "PT1H.json" -Recurse #Loop through local files and build $Events array up with files that meet the selected criteria. $Events=@() foreach ($file in $Files) { $PathSplit = ($file.DirectoryName.Replace("$($LocalFolder)\",'')).split("\") $blobdate = get-date -Year $PathSplit[9].Substring(2) -Month $PathSplit[10].Substring(2) -Day $PathSplit[11].Substring(2) -Hour $PathSplit[12].Substring(2) -Minute $PathSplit[13].Substring(2) -Second 0 -Millisecond 0 $blobResourceGroup = $PathSplit[4] $blobNSG = $PathSplit[8] if ($blobResourceGroup -in $SelectedResourceGroups -and $blobNSG -in $selectedNSGs -and $blobdate -in $SelectedDates) { $TestFile = $file.FullName $json = Get-Content -Raw -Path $TestFile | ConvertFrom-Json foreach ($entry in $json.records) { $time = (get-date $entry.time).ToUniversalTime() $time = [System.TimeZoneInfo]::ConvertTimeFromUtc($time, $TZ) $FlowVer = $entry.properties.version foreach ($flow in $entry.properties.flows) { $rules = $flow.rule.split("_") $RuleType = $rules[0] $RuleName = $rules[1] foreach ($f in $flow.flows) { if ($FlowVer -eq 2) { $Header = "Ref","SrcIP","DstIP","SrcPort","DstPort","Protocol","Direction","Type","State","SentPackets","SentBytes","RecvPackets","RecvBytes" $o = $f.flowTuples | ConvertFrom-Csv -Delimiter "," -Header $Header $o = $o | select @{n='ResourceGroup';e={$blobResourceGroup}},@{n='NSG';e={$blobNSG}},Ref,SrcIP,DstIP,SrcPort,DstPort,Protocol,Direction,Type,State,SentPackets,SentBytes,RecvPackets,RecvBytes,@{n='Time';e={$time}},@{n="RuleType";e={$RuleType}},@{n="RuleName";e={$RuleName}} $Events += $o } else { $Header = "Ref","SrcIP","DstIP","SrcPort","DstPort","Protocol","Direction","Type" $o = $f.flowTuples | ConvertFrom-Csv -Delimiter "," -Header $Header $o = $o | select @{n='ResourceGroup';e={$blobResourceGroup}},@{n='NSG';e={$blobNSG}},Ref,SrcIP,DstIP,SrcPort,DstPort,Protocol,Direction,Type,@{n='Time';e={$time}},@{n="RuleType";e={$RuleType}},@{n="RuleName";e={$RuleName}} $Events += $o } } } } } } #Open $Events in GridView for user to see and filter as required $Events | Sort-Object Time -Descending | Out-GridView #Prompt to export to CSV. $CSVExport = read-host "Do you want to export to excel? (Y/N)" #If CSV required, export and open CSV file if ($CSVExport.ToUpper() -eq "Y") { $FileName = "$($LocalFolder)\NSGFlowLogs-$(Get-Date -Format "yyyyMMddHHmm").csv" $Events | Sort-Object Time -Descending | export-csv -Path $FileName -NoTypeInformation Start-Process "Explorer" -ArgumentList $FileName } [/sourcecode] Important Note: The Azure Network Watcher team has provided a release note (https://azure.microsoft.com/en-us/updates/azure-network-watcher-network-security-group-flow-logs-blob-storage-path-update/) detailing that the path of the Flow Logs in the storage account will change on July 31st 2017. Once this is released, I will update my script, if required, to cater for the change.

Get Azure PaaS Endpoint IPs

James R Tabor — Mon, 03 Jul 2017 14:30:16 GMT

I've had a few situations where people are trying to find out the public IP Addresses for their own resources in Azure to use for Network Security Groups or User Defined Routes. Unfortunately there is nothing generic out there that can return this information for every Resource Type, so specific code needs to be developed for each type. I've made a start on this for Storage Accounts and Azure SQL Servers, being two of the most popular ones. You can use the function(s) below to return an array containing the IP, Location and Resource Type for all Storage Accounts and SQL Servers in a given subscription. If you have another Resource Type you need to resolve public IPs for, add the required code under the 'switch ($Res.ResourceTypes)' section. Please feel free to drop me a line if you need help working out how to get the Public IP for different resource types. [Updated: 13 Nov 2017]: Updated to include KeyVault and added function parameter switches to filter which Resource Types will be returned. [sourcecode language="powershell"] function Get-FQDNIP ( [string]$FQDN, [object]$Resource ) { $IPDetails = [System.Net.Dns]::GetHostAddresses($FQDN) $IP = New-Object PSObject -Property @{ IP = $IPDetails.IPAddressToString Locn = $Resource.Location Type = $Resource.ResourceType } Write-Verbose "`t$($IP)" return $IP } function Get-PaaSIPs ( [string]$SubscriptionID, [switch]$StorageAccounts, [switch]$SQLServers, [switch]$KeyVaults ) { Select-AzureRmSubscription -SubscriptionId $SubscriptionID | Out-Null $AllRes = Get-AzureRmResource $IPs = @() $CurrType = "" $AllRes = $AllRes | Sort-Object ResourceType if (!$StorageAccounts -and !$SQLServers -and !$KeyVaults) { $FilteredRes = $AllRes } if ($StorageAccounts) { $FilteredRes = $AllRes | Where-Object {$_.ResourceType -eq "Microsoft.Storage/storageAccounts"} } if ($SQLServers) { $FilteredRes += $AllRes | Where-Object {$_.ResourceType -eq "Microsoft.Sql/servers"} } if ($KeyVaults) { $FilteredRes += $AllRes | Where-Object {$_.ResourceType -eq "Microsoft.KeyVault/vaults"} } ForEach ($Res in $FilteredRes) { if ($CurrType -ne $Res.ResourceType) { $CurrType = $Res.ResourceType Write-Verbose "$($Res.ResourceType)" } switch ($Res.ResourceType) { "Microsoft.Storage/storageAccounts" { $ResDet = Get-AzureRmStorageAccount -Name $Res.ResourceName -ResourceGroupName $Res.ResourceGroupName $Blob = $null; $Blob = $ResDet.PrimaryEndpoints.Blob $File = $null; $File = $ResDet.PrimaryEndpoints.File $Table = $null; $Table = $ResDet.PrimaryEndpoints.Table $Queue = $null; $Queue = $ResDet.PrimaryEndpoints.Queue if ($Blob -ne $null) { $Blob = $Blob.Substring(8,$Blob.Length-9) $IP = Get-FQDNIP -FQDN $Blob -Resource $Res if ($IP.IP -notin $IPs.IP) { $IPs += $IP } } if ($File -ne $null) { $File = $File.Substring(8,$File.Length-9) $IP = Get-FQDNIP -FQDN $File -Resource $Res if ($IP.IP -notin $IPs.IP) { $IPs += $IP } } if ($Table -ne $null) { $Table = $Table.Substring(8,$Table.Length-9) $IP = Get-FQDNIP -FQDN $Table -Resource $Res if ($IP.IP -notin $IPs.IP) { $IPs += $IP } } if ($Queue -ne $null) { $Queue = $Queue.Substring(8,$Queue.Length-9) $IP = Get-FQDNIP -FQDN $Queue -Resource $Res if ($IP.IP -notin $IPs.IP) { $IPs += $IP } } } "Microsoft.Sql/servers" { $IP = Get-FQDNIP -FQDN "$($Res.ResourceName).database.windows.net" -Resource $Res if ($IP.IP -notin $IPs.IP) { $IPs += $IP } } "Microsoft.KeyVault/vaults" { #Write-Output "`t$($Res.Name)" $Vault = Get-AzureRmKeyVault -VaultName $Res.Name -ResourceGroupName $Res.ResourceGroupName $VaultFQDN = $Vault.VaultUri.Substring(8,$Vault.VaultUri.Length-9) $IP = Get-FQDNIP -FQDN $VaultFQDN -Resource $Res if ($IP.IP -notin $IPs.IP) { $IPs += $IP } } default { #Write-Verbose "`t$($Res.Name)" #Write-Verbose "`t`t$($Res.ResourceType)" } } } return $IPs | Sort-Object Locn, IP, Type } [/sourcecode]

Log results from Azure Automation Runbooks directly to OMS

James R Tabor — Wed, 21 Jun 2017 00:18:23 GMT

An easy way to keep track of Azure Automation runbook operations is to create log entries to OMS during execution. https://docs.microsoft.com/en-us/azure/log-analytics/log-analytics-overview There is a Rest API service as part of OMS that let's you post log entries directly into your OMS Workspace, allowing for easy searching and alerting of anything you're runbooks do. There are some code samples provided, in the link above, that allow you to build your own functions to use this API, or you can use the hard work someone else has done and import the OMSIngestionAPI module, which is what I have done. Before you start, you will need to have your OMS Workspace Name, ID, and key available. You can get these from the OMS Workspace settings page. I recommend using (and will use in this post) Azure KeyVault to hold your OMS Workspace Key securely and allow for any key rotation or changes in a programmatic way. Have a look at Getting Started - Azure KeyVault to get one set up if you haven't got one already. Here's the code I used to get things setup and ready for use. This creates Automation variables for OMSWorkspaceName, OMSWorkspaceID and KeyVaultName. Also created is a KeyVault secret for the OMS Workspace Key, granting the Automation RunAs account rights to get secrets from your KeyVault. [sourcecode language="powershell"] $AutomationAccountName = "{My Automation Account Name}" $ResourceGroupName = "{My Automation Account Resource Group Name}" $KeyVaultName = "{My KeyVault Name}" $KeyVaultResourceGroupName = "{My KeyVault Resource Group Name}" $OMSWorkspaceName = "{My OMS Workspace Name}" $OMSWorkspaceID = "{My OMS Workspace ID}" $OMSWorkspaceKey_Raw = "{My OMS Workspace Key}" $OMSWorkspaceKey = convertto-securestring ($OMSWorkspaceKey_Raw) -asplaintext -force New-AzureRmAutomationVariable -Name "OMSWorkspaceName" -Value $OMSWorkspaceName -AutomationAccountName $AutomationAccountName -ResourceGroupName $ResourceGroupName -Encrypted $false New-AzureRmAutomationVariable -Name "OMSWorkspaceID" -Value $OMSWorkspaceID -AutomationAccountName $AutomationAccountName -ResourceGroupName $ResourceGroupName -Encrypted $false New-AzureRmAutomationVariable -Name "KeyVaultName" -Value $KeyVaultName -AutomationAccountName $AutomationAccountName -ResourceGroupName $ResourceGroupName -Encrypted $false Set-AzureKeyVaultSecret -VaultName $KeyVaultName -Name "OMSWorkspaceKey-$($OMSWorkspaceName)" -SecretValue $OMSWorkspaceKey $RunAsConn = Get-AzureRmAutomationConnection -Name 'AzureRunAsConnection' -ResourceGroupName $ResourceGroupName -AutomationAccountName $AutomationAccountName Set-AzureRmKeyVaultAccessPolicy -VaultName $KeyVaultName -ServicePrincipalName $RunAsConn.FieldDefinitionValues.ApplicationId -ResourceGroupName $KeyVaultResourceGroupName -PermissionsToSecrets get [/sourcecode] Now, to log information to OMS you need to pass a json payload and a LogType. I have some script that I add to the top of all my scripts to do this. You will notice I've include various data types so as to ensure the log data is formatted correctly, I've borrowed this from the Rest API reference, so please look the 'Record type and properties' section for all the details. For example, if it's a number the use _d (double) at the end of the parameter so no quotes are placed around the value. [code language="powershell"] #Retrive the RunAs Connection details $Conn = Get-AutomationConnection -Name 'AzureRunAsConnection' #Login with the RunAs Account Add-AzureRMAccount -ServicePrincipal -Tenant $Conn.TenantID ` -ApplicationId $Conn.ApplicationID ` -CertificateThumbprint $Conn.CertificateThumbprint ` -SubscriptionId $Conn.SubscriptionId #Retrieve KeyVault Name to get secure variables (Namely, the OMS Workspace Key) $KeyVaultName = Get-AutomationVariable -Name "KeyVaultName" #Get OMS Workspace details $OMSWorkspaceID = $null $OMSWorkspaceName = Get-AutomationVariable -Name "OMSWorkspaceName" $OMSWorkspaceID = Get-AutomationVariable -Name "OMSWorkspaceID" if ($OMSWorkspaceID -ne $null) { $OMSKey = (Get-AzureKeyVaultSecret -VaultName $KeyVaultName -Name "OMSWorkspaceKey-$($OMSWorkspaceName)").SecretValueText $LogType = "Runbook_AutoshutdownFinal" } else { $OMSKey = $null $LogType = $null } #Set the LogType to use when posting to OMS $LogType = "OMSLogTest" function AddToLog ([Object]$OMSLogData) { if ($OMSWorkspaceID -ne $null) { $LogData = '' foreach ($Key in $OMSLogData.Keys) { switch ($Key.Substring($Key.Length-2)) { '_s' {$sep = '"';$trim=$Key.Length-2} '_t' {$sep = '"';$trim=$Key.Length-2} '_b' {$sep = '';$trim=$Key.Length-2} '_d' {$sep = '';$trim=$Key.Length-2} '_g' {$sep = '"';$trim=$Key.Length-2} default {$sep = '"';$trim=$Key.Length} } $LogData = $LogData + '"' + $Key.Substring(0,$trim) + '":' + $sep + $OMSLogData.Item($Key) + $sep + ',' } $TimeStamp = Get-Date (Get-Date).ToUniversalTime() $LogData = $LogData + '"TimeStamp":"' + $timestamp + '"' Write-Verbose "LogData: $($LogData)" $json = "{$($LogData)}" $PostResult = Send-OMSAPIIngestionFile -customerId $OMSWorkspaceID -sharedKey $OMSKey -body $json -logType $LogType -TimeStampField "TimeStamp" Write-Verbose "PostResult: $($PostResult)" if ($PostResult -ne "Accepted") { Write-Error "Error posting to OMS - $PostResult" } } } [/code] So whenever you want to log anything you would build a hashtable of the data you want to log and pass that to the AddToLog function. For example, if you were to log information from a runbook that starts and stops VMs you would use something like this - [code language="powershell"] $VMName = "MyVM1" $ResourceGroupName = "MyResourceGroup" $SubscriptionID = "7d94a8a1-12c4-1234-aaaa-xxxxxxxxxxxx" $VMState = Get-AzureRmVM -Name $VMName -ResourceGroupName $ResourceGroupName -Status if ($VMState.Statuses[1].DisplayStatus -eq "VM running") { Stop-AzureRmVM -Name $VMName -ResourceGroupName $ResourceGroupName -Force $Action = "Stop VM" } else { $Action = "VM Already Stopped" } $LogData = @{ Trigger_s = "Scheduled" VMName_s = $VMName SubscriptionGUID_g = $SubscriptionId ResourceGroup_s = $ResourceGroupName TargetState_s = "VM dealloacted" CurrentSatet_s = $($VMState.Statuses[1].DisplayStatus) Action_s = $Action } AddToLog -OMSLogData $LogData [/code] Which will look something like this in the OMS logs

Transition Azure Automation schedules from one Runbook to another

James R Tabor — Wed, 14 Jun 2017 23:40:10 GMT

So you've spent a great deal of time setting up all the schedules for an Azure Automation Runbook but now you have a new version that you want to test against all the parameters for the schedules of the existing released version. You've already created and published the new Runbook but what a pain if you've got more than a few different schedules with different parameters for each one. Here is a script that can be used to run tests for each of the existing schedules of the existing Runbook with their parameters and then link the schedules and their parameters to the new one. The test results get displayed in a notepad file for you to review. Tests don't work? Leave the old Runbook schedules as they are and keep tweaking! https://gallery.technet.microsoft.com/scriptcenter/Transition-Azure-f3a38c9a

Export Available Azure VM Sizes for all locations

James R Tabor — Tue, 16 May 2017 15:14:15 GMT

I wanted to share a quick post with a script I seem to use all the time to get a spreadsheet with all the available VM sizes available for use in each region Grab it from here --> Download script

Using Azure Automation and SharePoint Online lists to schedule VM start-up and shutdown

James R Tabor — Mon, 15 May 2017 17:37:14 GMT

To make the most efficient use of Public cloud you need to ensure your servers are only running when they need to be. Using SharePoint online as the source of virtual machine start-up and shutdown schedules is an option that allows for an easy, consolidated view of all your VMs and their schedules plus allows for the granular allocation of permissions for your staff, without having to grant them any Azure fabric access. I'm going to assume some familiarity with SharePoint Online custom lists and Azure Automation principles for this blog post. If you need some help getting started with these, try the following links. How to create a Custom List in SharePoint Online Getting Started with Azure Automation To integrate SharePoint Online with Azure Automation (and Powershell scripts), I've used a fantastic module written by Tao Yang, SharepointSDK. The details can be found here. To get set up for using SharePoint Online and Azure Automation, please refer to my Blog post here --> https://blogs.msdn.microsoft.com/jrt/2017/04/07/setting-it-up-sharepoint-online-lists-with-azure-automation/

Create the SharePoint Custom list for your VMS The columns you'll need are
- VMName {Renamed Title column} - 'Single line of text' This will be the Name of your VM in Azure
- SubscriptionGUID - 'Single line of text' The Azure Subscription GUID where the VM is situated
- ResourceGroup - 'Single line of text' The Resource Group Name of the VM
- TimeZone - 'Choice' A drop down list of the available Timezones for the shutdown times to be run against {See note below}
- ShutdownTimeRange - 'Single line of text' or 'Choice' This is a comma separated list of standard date formatted values using a ~ between times i.e. '6AM~6PM,Saturday,Sunday' or '25 April, 25 December, 1 January'
- OverRide-Start - 'Yes/No' A checkbox indicating to force VM to be Started
- OverRide-Stop - 'Yes/No' A checkbox indicating to force VM to be Stopped {takes precedence over 'OverRide-Start'}
Note: To create the list of available Timezones I used the following snippet of Powershell script -
ForEach ($TZ in ([System.TimeZoneInfo]::GetSystemTimeZones())) { Write-Output "$($TZ.DisplayName) [$($TZ.Id)]"}
(Optional) Create the SharePoint Custom list for logging Automation Actions. The columns you'll need are
- VMName {Renamed Title column} - 'Single line of text'
- SubscriptionGUID - 'Single line of text'
- ResourceGroup - 'Single line of text'
- AutoActionLog - 'Multiple lines of text'
Create a script or Azure Automation runbook to populate the SharePoint List with your VMs

The best way to populate your SharePoint list is to either run a Powershell script or create an Azure Automation Runbook to do that. I've posted a script to do this here --> Load Azure VMs into a SharePoint Online List and an Automation Runbook that can be imported here --> Load Azure VMs into a SharePoint Online List - Automation Runbook

Alternatively you can just enter the required Virtual Machine details directly into the SharePoint List.

Lastly, the actual automation script that does all the work.

Download Link from TechNet.

The most complex piece to this is the TimeZone adjustments. Azure Automation runs on UTC time so you need to make sure when you specify to shutdown a VM at 6pm on a Friday, you don't end up shutting down your VM at 8am on Friday, which is what happens using AST (Sydney, Australia) time!

I got myself lost when trying to make a small change that I had to stop all code development and actually write a Flow Chart of how script works. I strongly suggest if you are trying to work out how this script does what it does, that you work with this Flowchart open alongside. AutoShutdown-Flowchart (Note: Please ping me if you want the Visio version of this to update any changes of your own)

To have this run regularly to check all VMs for their required state simply add a schedule for the Runbook. I would suggest running it at 5 minutes past the hour to ensure it picks up any machines correctly that need to be stopped or started. The frequency of the schedule will depend on how many different timeframes you need to use across all your VMs.