about operations management ...

Analyzing Backup Activity with PowerBI

MarcoDias — Fri, 24 Nov 2017 16:27:45 GMT

This time it’s all about Power BI. The challenge was to visualize in some kind of calendar style the daily activity of backups. Starting with a detailed activity log, adding a couple of calculated columns and measures, we´ve ended with a pretty slick looks that helps to understand bubbles of failed backups or empty slots that can be used for schedules. In this particular example, the log comes from an export of the backup solution and the structure looked something like this: Not bad, at least there’s no need to major data transforms or aggregations. So now is just a simple import into Power BI and start creating some calculated columns and measures like such: Client Details = BackupActivity[Client Name] & " - " & BackupActivity[subclient] & " -" & BackupActivity[Backup Level] Number Schedules = INT(RIGHT(BackupActivity[Result],find("/",BackupActivity[Result],1,2)-1)) Number Completed = INT(LEFT(BackupActivity[Result],find("/",BackupActivity[Result],1,2)-1)) Number Not Completed = BackupActivity[Number Schedules] - BackupActivity[Number Completed] Percent Completed = SUM(BackupActivity[Number Completed])/sum(BackupActivity[Number Schedules]) And after playing around with conditional formatting of 'Percent Completed' in a matrix We´ll end up with a sort of heat map calendar structure that helps to discover some insights of backup activity. Refer to this repo ( https://github.com/madiasOnGit/PowerBI/tree/master/BackupActivity ) for the complete report. Marco

Mail Flow Round Trip test using Azure Automation and Azure Log Analytics

MarcoDias — Tue, 12 Sep 2017 15:44:09 GMT

Every once in a while a customer asks if we can create a mail flow between different email systems and measure its roundtrip time. So, this reference highlights one solution based on a PowerShell script that connects to one mailbox hosted in Exchange, sends a message to another mailbox hosted in Gmail, waits for its return and tracks the roundtrip time. We can definitely run such a script from SCOM, but for this example I wanted to use as much of the new world ( Azure Management Services ) as I could.

The test will run in Azure Automation ( required to run the script )
Trigger in an interval by an Azure Logic App ( Although we can use a schedule in Azure Automation, with logic Apps we can have an interval as low as 1 minute )
The result will be sent to a Log Analytics Workspace ( required to store the test results )
Use Power BI report to have appealing visuals of the solution ( not required but cool )

Let´s start !! Configure the test accounts: I´ve just use my Microsoft account @outlook.com and a new Gmail test mailbox that I´ve configure to forward all incoming emails back to my account. Configure script dependencies: The script makes use of Microsoft Exchange Web Services Managed API so we need to make this available in the environment where the script will run. Because we´ll be running in Azure Automation and Bartek Bielawski was so kind in providing a wrapper around Microsoft.Exchange.WebServices library, we only need to add the PowerShell module to an existing Azure Automation account. Then, import the module and we can use simple commands such as Connect-EWSService or New-EWSMessage to connect to the mailbox and send or retrieve messages. Finally, add the credential and the Log Analytics Workspace ID and Key variables to the automation account assets library so that they can be securely retrieved during runtime: $cred = Get-AutomationPSCredential -Name "" $customerId = Get-AutomationVariable -Name "" $SharedKey = Get-AutomationVariable -Name "" Run the flow test script: Create a new Azure automation PowerShell Runbook, paste the script and adapt the variables to your own environment: The script simple creates a new guid to be sent as the message subject and tracks the round trip time with a Diagnostics.Stopwatch If the message is sent, we then wait for it to come back by searching the inbox with the previous created guid as a filter. If we don't time out, then a "Result:Success" log will be created in the Log Analytics service: Create a schedule in Azure Logic Apps If you haven't play around with Logic Apps then you don't know what you're missing :) There’s already a huge amount of triggers and actions available in the service that can help to integrate different systems and create simple “playbooks”. One of those triggers is a recurrence that we can configure to trigger an action of type Create Job in Azure automation: Querying the output: After a couple a minutes, we'll be able to query entries of MailRoundTrip_CL Type ( assuming that the log type wasn’t changed in the script ). This example plots the roundtrip time as a timechart: MailRoundTrip_CL | where TimeGenerated > ago(6h) | extend FromLocation = case(Location_s == "", "Not Available", Location_s), Domains = split(To_s, "@") | project FromLocation , RoundTripToDomain = Domains[1], TimeGenerated, Result_s, RoundTripTime_d | extend test= strcat(FromLocation, " <-> ", RoundTripToDomain) | summarize avg(RoundTripTime_d) by test, bin( TimeGenerated, 10m) | render timechart And this one will get the last test result for by location: MailRoundTrip_CL | where TimeGenerated > ago(4h) | extend FromLocation = case(Location_s == "", "Not Available", Location_s), Domains = split(To_s, "@") | project FromLocation , RoundTripToDomain = Domains[1], TimeGenerated, Result_s, RoundTripTime_d | summarize TestTime = arg_max(TimeGenerated, Result = Result_s) by FromLocation, tostring(RoundTripToDomain) Creating the Power BI Dashboard: Now that the data is being created, we just need to click the Power BI query in the export options: Then, open Power BI desktop, click Get Data, choose blank query, click advanced editor and paste the contents from the download file. After authenticating, we should be able to get a dataset that can be used to create appealing visuals: Refer to this repo ( https://github.com/madiasOnGit/MailFlow ) for the complete script. Happy Dashboarding, Marco

Azure Resource Manager Template for an Azure Dashboard with Log Analytics Tiles

MarcoDias — Mon, 11 Sep 2017 22:30:49 GMT

In this example I´ll be creating an ARM template with a couple of Azure Log Analytics tiles. I wanted to create a Dashboard with some performance metrics that could be reused in different environments. So, create a new dashboard, pin the intended tiles, "export" to an ARM template and reuse, right ? Well, the export part is a bit tricky. First, open up the new Log Analytics query editor and pin the tiles to a new dashboard ( that should already exists as a saved dashboard in a resource group ) . In this example I´m querying the Average CPU usage for all my agents in the last 24 hours to render as a time chart: Perf | where CounterName == "% Processor Time" and TimeGenerated > ago(24h) | summarize AVGCPU = avg(CounterValue) by bin(TimeGenerated, 1h), Computer | render timechart Then, when you´re happy with the looks, the fun part begins… We need to open the dashboard definition and resource explorer can give us a hand: Navigate to the created dashboard and grab the definition to a text/ARM editor of choice. Create a new ARM template definition with parameters for requesting the dashboard Name, Log Analytics Workspace Name and location, a variable for generating a unique name and chose "2015-08-01-preview" as the API version. Paste the previous grabbed dashboard from resource explorer into the resources tag and modify static values like so:

{

"name": "ComponentId",

"value": {

"SubscriptionId": "[subscription().subscriptionId]",

"ResourceGroup": "[parameters('logAnalyticsWorkspaceResourceGroup')]",

"Name": "[parameters('logAnalyticsWorkspaceName')]"

}

...

{

"name": "DashboardId",

"value": "[concat('/subscriptions/',subscription().subscriptionId,'/resourceGroups/',resourceGroup().Name,'/providers/Microsoft.Portal/dashboards/',variables('dashboardID'))]"

...

{

"name": "PartId",

"value": "[concat(uniquestring(variables('dashboardID')), 'Part1')]"

And that’s it!

Refer to this repo ( https://github.com/madiasOnGit/Azure.ARM.Dashboards/tree/master/KustoDashboardExample ) for the complete template.

Happy Dashboarding, Marco