Microsoft IT's Cloud Management Services Playbook

Monitoring Runbooks / Azure Automation + Log Analytics

Matthew Raffaele — Fri, 05 Aug 2016 09:45:13 GMT

A while ago, I wrote a couple of posts showing how we are collecting and storing Azure Automation runbook job history for monitoring and support using a custom runbook and Azure SQL database. Since then I have learned that Log Analytics for Azure Automation is in available in preview. After setting it up and taking it for a quick spin around the block I've found it to be a fantastic resource for proactive monitoring of the workflows we have running in Azure Automation. We have been using OMS alerting for a couple of weeks now to alert us when certain workflows have failed with great success. Operations teams are notified when there are problems and fix them before customers have to escalate. In addition to alerting, Log Analytics also allows an alternative for getting counts and amounts of job history - it's really pretty neat. Below are details on setup - it is pretty simple, the tricky part is learning to create the queries you will need to use for monitoring and alert rules. Below is an example of a view on the OMS portal which shows counts of failures for a couple of important runbooks we have made available to customers to enable self-service jobs we call: Snooze (shutdown and deallocate an Azure VM or bring it back online) and Resize (Change the SKU of an Azure VM.): [caption id="attachment_185" align="alignnone" width="612"] Failures by runbook[/caption] Setting up Log Analytics Setup is pretty straight forward, below is an example of how we got everything up and running: # Connect to Azure Login-AzureRmAccount # Configure storage account for logs and set up diagnostics on your Automation Account # assumes ARM storage account in the same regions as the automation account $AA = Find-AzureRmResource -ResourceNameContains "MyAutomationAccount" -ResourceType Microsoft.Automation/AutomationAccounts $SA = Find-AzureRmResource -ResourceNameContains "MyStorageAccount" Set-AzureRmDiagnosticSetting -ResourceId $AA.ResourceId -StorageAccountId $SA.ResourceId # Enable logs in Log Analytics following these instructions # https://azure.microsoft.com/en-us/documentation/articles/log-analytics-powershell-azure-diagnostics-json # storage account that logs will be written to... $storageAccountId = "/subscriptions/1aba1abc-1234-5678-acb1-234567def1234/resourceGroups/automation_analytics/providers/Microsoft.Storage/storageAccounts/MyStorageAccount" $supportedResourceTypes = ("Microsoft.Automation/AutomationAccounts") # location to match your storage account location $resources = (Get-AzureRmResource).Where({$_.ResourceType -in $supportedResourceTypes -and $_.Location -eq "eastus2"}) foreach ($resource in $resources) { Set-AzureRmDiagnosticSetting -ResourceId $resource.ResourceId -StorageAccountId $storageAccountId -Enabled $true -RetentionEnabled $true -RetentionInDays 1 } # Import the log analytics module Install-Module -Name AzureDiagnosticsAndLogAnalytics # Run the UI configuration script # -guides you through configuring Log Analytics to collect logs from Azure diagnostics storage Add-AzureDiagnosticsToLogAnalyticsUI # Open your OMS workspace and search Log Analytics for Type=AutomationAccounts # https://mysuite.portal.mms.microsoft.com/

SQL Server Database migration to Azure SQL DB – using SSMS. Comparison of SSMS, SSDT, SAMW, SQL Server 2016 Upgrade Advisor.

joe.mazzotta — Tue, 02 Aug 2016 23:46:42 GMT

If you are planning to migrate an on-prem SQL Server to Azure, there are a few key considerations:

Understand the Azure SQL Database General Limitations and Guidelines
Review the Azure SQL Database Transact-SQL differences
Review the Azure SQL Database resource limits

Migration will involve both migration of the schema and data from your source SQL DB to Azure SQL DB (PaaS). There may be some re-engineering required if your workload is not compatible with Azure SQL DB. See guidelines and limitations above. Below are details on use of SQL Server Management Studio 2016 Migration Wizard. This approach is best used for SQL DB migration where the DB is less than 1TB in size and where there is persistent (good) connectivity between your source and Azure SQL DB. Alternate approaches such as BCP, SQL Server replication, manually migrating SQL DB or any other means will be covered in future posts. Pre-reqs:

SSMS 2016
If needed, ensure SQL Server DB has SQL auth enabled (including create SQL login to enable migration of DBs)
Open inbound/outbound ports 1433 to SQL Server DB

Migration Steps:

In my example, I’ll migrate my Azure IaaS VM SQL Server Database to Azure SQL Database. The same steps apply if you migrate an on-prem SQL Server DB to Azure SQL DB.
I’ll need to prepare my IaaS VM SQL Server for migration
Ensure inbound and outbound network firewall rules enable port 1433. You need to enable this so that SSMS can access and process migration.

3a. Go to Azure Portal and select the IaaS VM running SQL Server you wish to migrate * (only required if you are migrating an IaaS VM running SQL Server DB; if its an on-prem VM, log in to the Windows Server and open firewall rules enabling inbound/outbound ports 1433) 3b. From settings, click on Network Interfaces 3c. Next click on Network Security Groups 3d. Select the security group that applied to the IaaS VM with SQL Server to be migrated 3e. Set both the Inbound and Outbound security rules to enable port 1433 4. Next we’ll log in to SQL Server to begin migration process through SSMS 2016: 5. To begin the migration process, Select the DB you wish to migrate to Azure SQL DB by right clicking then selecting Tasks -> Deploy Database to Microsoft Azure SQL Database. Note you can also migrate an on-prem SQL Server to an IaaS VM SQL Server however consider benefits of Azure SQL vs managing your own VM. 6. This will open the Deploy Database wizard. The first step of the migration wizard will be to connect to your Azure subscription (Azure SQL DB) destination where you’re migrating your on-prem SQL DB to 6a. SSMS deployment database wizard will then validate and perform both the schema and data migration from your source and migrate your database to Azure SQL SB.

You are notified once the operation is complete.

Here we have completed the migration of an SQL Server 2008 R2 database from an VM to Azure SQL DB. SQL Server Management Studio is a very easy tool to perform simple migrations of SQL Server DB to Azure SQL DB if you do not require schema changes and your source DB is compatible with Azure SQL DB. In summary some tools allow edit of schema to correct any incompatibilities, tools vary between developer audience vs SA audience, and finally some tools provide compatibility analysis vs some tools just to BCP/migration of schema and/or data. cheers Joe

WinRM | It's kind of a like / hate thing...

Matthew Raffaele — Thu, 16 Jun 2016 15:46:59 GMT

Sure, I like WinRM. It enables remote management of Windows over HTTP and it is at the core of how we perform configuration and change automation here in MSIT. When configured properly it makes execution of tasks on remote machines insanely simple and without it I don't think we'd be able to perform half of the activities we have automated in SMA and Azure Automation Workflow runbooks. The problem is that it has to be enabled and properly configured - out of the box you aren't going to find it very useful unless you've already configured via custom images, group policy or the like. This is where the side of the equation opposite to like comes into play. Even if you are enabling WinRM at build time, at data center scale you are bound to come across machines that are not going to play nice and will break your automation and bring dishonor to your family. So lets look at what to configure and how we can test for and remediate problem at runtime. Enable WinRM and configure it to allow remote access: 1.) I'm sure we all know this already: On the remote machine, execute the following from an elevated cmd prompt: 'winrm quickconfig /force'. This will enable the service and create an http listener endpoint for you. ^ But what if you cannot run the command on the machine w/o if WinRM isn't enabled? No need to hate - there's a simple trick we use in our automation that includes psexec and a new scheduled task that I'll write up in a separate post real soon. Need to access network resources from within your WinRM session? Lets say that in your SMA or Azure Automation runbooks you need to install or execute software from the network within a remote session or PowerShell Workflow InlineScript. You'll need to enable CredSSP on your runbook workers so that the remote machine can act as a credential delegate and pass specified creds along to anything on the network you need to authenticate against. To allow the use of CredSSP perform the following on the client (the machine you are remoting from:) 1.) Execute the following from an elevated PS cmd prompt: 'Enable-WSManCredSSP -DelegateComputer localhost -Role client -Force' 2.) Use gpedit.msc and look at the following policy: Computer Configuration -> Administrative Templates -> System -> Credentials Delegation -> Allow Delegating Fresh Credentials. Verify that it is enabled and configured with a 'wsman/*' SPN. [caption id="attachment_135" align="alignnone" width="672"] Delegate Fresh Creds[/caption] Cool, now you'll be able to authenticate against and access resources on the network over a 'third hop' from the machine you are remoting into. To force the use of CredSSP in your connections, simply use the authentication and credential parameters with your InlineScript block or new PoSh session: [caption id="attachment_145" align="alignnone" width="654"] Demo InlineScript with CredSSP[/caption] or.... [caption id="attachment_155" align="alignnone" width="884"] PS Session with CredSSP[/caption]

Monitoring a Busy Azure Automation Infrastructure (Part 2)

Matthew Raffaele — Mon, 06 Jun 2016 21:34:38 GMT

Keeping Operations and our customers happy with telemetry OK, so as we established in part one of this subject we’ve got a ton of Azure Automation (AA) runbook job history stored in Azure, and we need to begin collecting it and storing somewhere that is quick and easy to access and can serve as a feed for an AA jobs dashboard. Here in part two, I'll provide an overview of how I went about getting this done. The first order of business for me was to provision a Azure SQL DB instance. This was a bit of a no-brainer decision. It is super simple to setup and aside from managing logins, there is little to no maintenance overhead to bother us in the future. If you aren’t familiar with the process of setting up Azure SQL, there is a ton of helpful information available online including a video hosted here to guide you through the process to ‘create DBs in seconds.’ One detail I will mention here is that once your DB is provisioned, you can access the connection string from the portal which will be needed in just a bit. [caption id="attachment_45" align="alignnone" width="1158"] Connection String[/caption] OK, great! So now we’ve got an Azure SQL DB stood up named runbookjobs. We’ll connect to it through the SQL Server Management Studio using the login specified when we provisioned it and create a single table called jobs with columns for each of the data points we want to store about runbook job history for future reference. I’m a huge fan of simplicity so (for now at least) this simple schema will suffice. [caption id="attachment_55" align="alignnone" width="373"] Jobs Table[/caption] Right on, now we’ve got a home for our job history data. Next we’ve got to get it populated. What better way to do that then to create a PowerShell runbook in AA to do the work? What I did was to create a runbook with the hugely imaginative name of Get-RunbookJobHistory and started writing the script to pull runbook job history stored in Azure and stuff it into our shiny, new runbookjobs DB. Getting a the runbook job history is not too difficult, especially if you’ve previously done it in SMA to monitor child runbook status from a parent runbook or other process. The concept is pretty much the same, except that only a summary of the job output is available in the output stream of you are not referencing them directly with a JobId. To get the complete job output, you need find your job using the Get-AzureRmAutomationJob and then reference the job by JobId with the Get-AzureRmAutomationJobOutput cmdlet. So, let’s go ahead and get the Jobs we care about along with the properties of each we can get now for storing in SQL. After logging into Azure using a credential asset stored in AA we can iterate through our list of jobs. Here, I’m going to collect all of the jobs that have been created since the last time my history collection ran: [caption id="attachment_75" align="alignnone" width="743"] Get Jobs[/caption] You can see above that we are simply walking the list of jobs run in the Azure Automation Account we are interested in and then for each, if it is newer than the last one we collected during the last cycle, we use the JobId we found for it to reference it again directly so we will have access to the full job output stream. For each job then, getting 10 of the 11 values we want for each job is readily available and we pluck them right out and stuff them into a custom object. For the all important output stream, there is one more step we need to take. We'll stuff the following snippet into the foreach loop to get the output details: [caption id="attachment_85" align="alignnone" width="709"] Get Job Output[/caption] Here we have to make a 3^rd call to Azure using the previously mentioned Get-AzureRmAutomationJobOutput cmdlet to get the complete job output. Then, we pipe that to the Get-AzureRmAutomationJobOutputRecord cmdlet which finally gets us access to the full output stream. You may also notice above that as I’m grabbing the output, I am replacing single quotes with double quotes so that when we insert the data into SQL, we don’t goof up our query string. Having to make so many calls out to Azure slows down the how fast we can walk through all of our jobs, but hey, that’s one of the points of using AA in the first place - let it handle those tedious, long running tasks for you silently in the background while you are getting real work done, right? I have this runbook running on a 4 hour schedule with a Webhook enabled to kick it off with a couple of lines of PoSh anytime the Operations team needs closer to real time data. So this is pretty much it – all we need to do now is store the data in SQL and we are off to the races! Below is the entire contents of the PowerShell runbook as it exists today, it isn’t terribly complex, but I hope it helps someone out. FWIW, I desperately want to be able to place code here that you could copy and paste, but it is near impossible with the WordPress tools available for me here. If you want the actual PS1 file, hit me up at mattraff@msn.com and I'll get it to you. [caption id="attachment_95" align="alignnone" width="998"] Full Runbook[/caption]

Monitoring a Busy Azure Automation Infrastructure (Part 1)

Matthew Raffaele — Mon, 06 Jun 2016 11:55:43 GMT

*Update - Here is a better, supported way to monitor Automation... Since this blog was written, Microsoft has released Log Analytics for Azure Automation. Keeping Operations and our customers happy with telemetry After finally making forward progress in the push to migrate our automation workflows from SMA to Azure Automation (AA) we’ve got several thousand jobs running every week in AA including our heavy hitter, an MSIT tool called Snooze. Although there are a lot of big advantages of AA over SMA such as the ability to run native PowerShell, integration with DSC and the new availability of Webhooks, there are two problems that existed in SMA and are still around in AA: 1.) Detailed runbook telemetry and 2.) Ops support for monitoring and troubleshooting busy systems. First, telemetry – we got a bit of an upgrade with SMA, we now have the ability to filter through job history by state such as Success, Failed, Suspended, etc. The second bonus is that the high level dashboard view of job history comes in a pie chart which looks much prettier than that ugly old line graph we had in SMA. The problem though, is that getting any meaningful data such as counts and amounts per runbook or the ability to search job history by runbook input parameters or their actual output is still missing. Using real data from our production environment as an example I see that there are 98 failed jobs of the 5,433 total. Not perfect, but that’s still less than a 2% failure rate. However, these 98 failures may have come from a particular workflow which was only run 100 times - then we’d have a 98% failure rate of a particular automation operation – not so great :/ I can easily filter my job history in the portal to focus on the failures. When I do that, I see that the failures were almost all from the Snooze runbook and they happened prior to our production switchover from SMA to AA – these failures occurred in test, whew! That is nice to know, but we aren’t completely out of the woods yet… You see, these runbooks are called from a service sitting behind a web front-end and we do our best to make sure that they always return a result back to that front-end (FE) to let our customers know what the status of their request is. In order to make the best attempt to get these details to the FE, we need to ensure that any errors encountered are caught and handled properly before they can result in a state that puts the runbook in a Failed or Suspended state. We want to make sure that we never let the runbook die even if we hit an error that ultimately results in the failure of the end user’s request. The actual customer request results are formatted as json object and piped through the runbook job output where they can be retrieved by the FE. This behavior hides the actual success/fail rates from AA so we cannot get a dashboard view of the customer’s experience. If we ran the service like this for any length of time, we’d only have customer escalations (or hopefully a lack there of) to gauge the health of our automation system. This certainly isn’t the way to drive customer satisfaction in a positive direction. We also need to keep Operations happy and efficient for when the (hopefully trickle) of escalations come in, they are able to quickly identify the problems so they can be resolved. AA provides no mechanism to search historical job data for input parameters and job output. Let’s say that one of the 5,400 jobs submitted was for a server named \\guacamole on Monday evening and it failed. If Operations just started working on the issue today four days later on a Friday, they would have to manually flip through the hundreds of jobs that ran Monday evening looking through the job input and output properties of each until they found the one that referenced \\guacamole. In the Azure Portal, this is a mind-blowingly tedious process and chore that could take 30 minutes or more – just to find one failure. It’s the needle in a haystack of suck! We simply could not run and manage a service this way, so we created a simple way to collect and store runbook job history in Microsoft SQL Azure. From this store of job history, engineers and Operations have quick access to details which would otherwise been hidden away and all but inaccessible – these are details which let us see the communication between our FE applications and AA such as runbook input parameters and output streams. We also collect start and stop dates and times, end runbook status, the credentials of who started each job and more. Now we are getting somewhere – we have a single location of data upon which we can enable Operations to work efficiently and upon which can build a great dashboard for truly accurate monitoring. Next, I’ll go into details about how the solution to build the AA runbook job history data store was created in PowerShell and runs in AA itself with a Webbook also available to trigger a collection cycle to get up-to the minute information as needed. See Part 2 here: Monitoring a Busy Azure Automation Infrastructure (Part 2)

Already using Azure and curious which services are enabled in Resource Manager mode

joe.mazzotta — Tue, 26 Apr 2016 09:34:13 GMT

I work with many IT teams and often get get questions from external customers at EBC on what is ARM, why leverage ARM, and whether all existing Azure services are ARM enabled?

Azure has provided documentation ARM:

Azure Resource Manager mode provide many advantages to enterprise IT including:

1. ability to tag resources which aid in classification for undiscoverable data (eg billing individual resources to team, type of environment, type of IT services dependent upon Azure resource, etc).

2. ability to set granual role access through RBAC - roles based access controls

3. ability to group resources for lifecycle management

4. decoupling network model which allows separation of compute from networking - this enables enterprise IT to gain better scale of networking in Azure to IaaS VMs

During Q4 FY16, MSIT is driving requests for new internal subscriptions towards ARM and enabling ARM templates to achieve benefits described above.

While Azure continues to actively enable all existing Azure services on ARM, below is a list of Azure services/resources which currently are not enabled for ARM or enabled in a limited way:

Virtual Machines (classic)
Remote Apps
Cloud Service (classic)
StorSimple
Machine Learning
Data Catalog
Media Services
Backup
Site Recovery

In the near-term, continue to monitor the Azure documentation on ARM for updates.

For teams who are interested in proceeding and determining ability to co-exist with classic Azure resources and ARM enabled services and resources, we recommend you contact MSIT ARM Champions who can help define architectures in Azure which may enable your specific ARM scenario(s).

cheers

Joe