1. On Hyper-v or whatever hypervisor solution you use, install the OS from an ISO
2. Perform your customizations
3. Run SYSPREP (for Windows) or waagent -deprovision+user (for Linux)
4. Upload the VHD to a Storage Account by using Azure Storage Explorer, AzCopy, Add-AzureRmVHD, etc.
5. Create a managed Image of this VHD in order to be able to easily deploy VMs from that golden image

1. Azure Active Directory
2. Azure Storage Accounts
3. Azure Storage Tables
4. Azure Storage Queues
5. Azure Automation

• Tier 0 Storage Account – Storage account that contains the configuration table, the queues for VHD copy process and Managed Image creation process. Finally it is the initial point where the VHD is uploaded first.
• Tier 1 Blobs – Multiple copies of the original VHD are made inside of the Tier 0 Storage Account to allow multiple parallel copies (this is defined during setup).
• Tier 0 Subscription – Subscription where all automation components are created.
• Tier 2 Subscriptions – Subscriptions that holds tier 2 Storage Accounts and will get the Managed Image created.
• Tier 2 Storage Accounts – Final destination of the VHDs, they can be in same or different subscription, different regions, etc.

1. After building your VHD, use UploadVHD.ps1 script to upload it to a tier 0 storage account
2. UploadVHD.ps1 script will upload the VHD to Tier 0 Storage Account and will trigger Start-ImageManagementTier1Distribution runbook
3. Start-ImageManagementTier1Distribution will create multiple copies of the VHD in order to allow parallel copies (when using Start-AzureStorageBlobCopy, the source blob is locked until the copy is in progress, meaning that no other copy operation can start)
4. After all copies are done, a message is placed in the copy-process-queue stating that there is a VHD to distribute to Tier 2 Storage Accounts (with other information that will be consumed later)
5. Runbook Start-ImageManagementTier2Distribution is scheduled to run every hour (minimum value) and check the copy-process-queue, if there is a message there, it will trigger one runbook called Start-ImageManagementVhdCopyTier2 that resides in a different Automation Account per destination Tier 2 Storage Account. Note that this is in another Automation Account due to the 200 runbook jobs running at the same time, for a complete list of limitations, please refer to https://docs.microsoft.com/en-us/azure/azure-subscription-service-limits#automation-limits)
6. After each individual runbooks completes its execution, a message per copied VHD is placed in the image-creation-process-queue
7. Another scheduled runbook called Start-ImageManagementImageCreation will monitor that queue (hourly again) and trigger one New-ImageManagementImage runbook (located in another Azure Automation Account due to same reasons already explained, it also can be in multiple Automation Accounts, all depends on how you did the initial setup) per VHD and perform the image creation.
8. The end result is that the managed images will start showing up at the locations you defined.

Setup Guide

Details on SetupInfo.json file

• general
• requiredModulesToInstall
• storage
• automationAccount

General section

RequiredModulesToInstall section

Storage section

tier0StorageAccount subsection

tier2StorageAccounts subsection

Automation Account section

• Main Automation Account, this is the one that contains the Start-ImageManagementImageCreation, Start-ImageManagementTier1Distribution and Start-ImageManagementTier2Distribution runbooks.
• Copy Process Automation Account(s), one or more automation accounts to execute the tier 2 copy process, the number of this type of Automation Accounts will depend on number of subscriptions, regions and images, as it was previously explained, a single Automation Account can execute a maximum of 200 runbook jobs (we cap at a default of 150 in this solution) so if you have, lets say 300 subscriptions, 2 images (1 Windows and 1 Linux) and need to include two regions, you will need at least 8 copy dedicated Automation Accounts in order to be as parallel as possible.
• Image Creation Process Automation(s), same case of the previous description, this automation account is dedicated to create the Managed Images.

Runbooks subsection

Element Name	Description	Modification Required?
name	Runbook Name.	No
scriptPath	Path to the script to be imported into the PowerShell runbook. This path can be a http path as well, when you download the script directly from GitHub raw path for example.	No
scheduleName	Name of the schedule to create and assign to the runbook, the type is hourly schedule only.	No
startTimeOffset	This value is used to set the start time of the schedule, in minutes, it tells how many minutes to get started with the schedule during the schedule creation, good for staggering between tier 2 copy runbook and the image creation runbook.	No
executeBeforeMoveForward	Forces the execution of the runbook as soon as it gets imported and wait for it to finish. In our case we add the update runbook from GitHub and execute it in order to update all existing modules in the Automation Account since they are outdated at the moment of account creation, if we don’t execute it all other runbook imports fails.	No
parameters	This element define an array of key and value pairs, this defines all parameters that the runbook needs when scheduling its execution. For example:	No
requiredModules	An array of modules to import in this automation account if not imported yet. The module list item needs to come from the requiredModulesToInstall section.	No

Hello Everyone, in this blog post I would like to introduce you to my latest PowerShell module, similarly to the AzureRmStorageTables module that I developed a while ago, this module works with Storage Queues, enabling IT Pros to rely on this great service that Azure Storage provides.

A fully function example of how IT Pros could leverage tables and queues is presented in this GitHub repository where I published an Azure Automation based solution that helps customers upload custom virtual hard drives to Azure and have it distributed amongst different regions/subscriptions and generating managed images. This solution will have a separate blog post shortly.

Introduction

Azure Storage Queues is one of the four Microsoft Azure Storage abstractions available (Blobs, Tables and Azure Files are the other ones) at the time that this blog was written. It provides cloud messaging between application components. In designing applications for scale, application components are often decoupled, so that they can scale independently. Queue storage delivers asynchronous messaging for communication between application components, whether they are running in the cloud, on the desktop, on an on-premises server, or on a mobile device. Azure also provides a more full featured alternative called Azure Service Bus which is similar in concept but provides other features.

For more information about Service Bus, please refer to Service Bus messaging: flexible data delivery in the cloud.

For more information about Azure Storage Queue, please refer to Queue storage and Get started with Azure Queue storage using .NET.

If you want to compare Service Bus against Storage Queue, please refer to Storage queues and Service Bus queues - compared and contrasted.

Since up to today there are no official cmdlets to support placing/retrieving/deleting/modifying messages on Storage Queues from Azure PowerShell module, I decided to create this simple module to help IT Pros to leverage this service without having knowledge of .NET framework through some simple cmdlets as follows:

There are a number of use cases for using it with PowerShell, specially during automation scenarios where you want to scale your automation to the next level. One example is the before mentioned solution I posted in GitHub. I would love to get more examples from the community over time in the comments section of this post.

Requirements

• PowerShell 4.0 or greater
• This module requires Azure PowerShell module installed, which can be downloaded from https://aka.ms/webpi-azps or the following minimum modules installed via PSGet module from PowerShell Gallery:
  • AzureRM.Profile
  • AzureRM.Storage
  • AzureRM.Resources
  • Azure.Storage

Installation/Source Code

Since this module is published on PowerShell Gallery, you can install this module directly from PowerShell 5.0 and Windows 10 / Windows Server 2016 by executing the following cmdlet in an elevated PowerShell command prompt window:

Install-Module AzureRmStorageQueue

Note that if you want to use the PowerShellGet module in Windows 2012 R2 with PowerShell 4, please look at the https://www.powershellgallery.com/ initial page on how to download and install it. The process is not that straight forward to enable it on Windows 2012 R2 and because of that I placed a summary of the steps below:

1. Download PowerShellGet source code from GitHub (https://github.com/PowerShell/PowerShellGet/releases)
2. Right-click the ZIP flie and click “Unblock”
3. Extract it anywhere
4. Copy the folder <your extraction folder>\PowerShellHGet-XYZ\PowerShellGet to C:\Program Files\WindowsPowerShell\Modules
5. Download and install PackageManagement PowerShell Modules Preview from https://www.microsoft.com/en-us/download/details.aspx?id=51451
6. Open an elevated PowerShell command prompt and execute
   
   

   Install-PackageProvider Nuget –Force

7. Close the command prompt
8. Open a new elevated PowerShell command prompt and you’re ready to install the Azure Rm Storage Queue module
   

As an alternative, you can manually download the module from my GitHub repository (here) and extract to C:\Program Files\WindowsPowerShell\Modules and rename the folder to AzureRmStorageQueue. Please remember to unblock the zip file before extracting it.

Working with Azure Rm Storage Queue PowerShell Module

The following steps will walk you through loading the module and perform one or more example tasks of the basic operations offered in this module.

Before you start working with it, you need to authenticate to Azure and select the correct subscription if you have multiple subscriptions:

Add-AzureRmAccount

Select-AzureRmSubscription -SubscriptionName <your subscription>

Next, lets import AzureRmTableQueue PowerShell module and list the functions available on it:

Import-Module AzureRmStorageQueue

Get-Command -Module AzureRmStorageQueue -CommandType Function

You should see the following cmdlets

For the sake of simplicity, we need to define some variables at this point to make our examples a little bit more clean, please, make sure you have a general purpose storage account to use and that you change the values of the variables below to reflect your environment. Notice that one of the variables is called $partitionKey, in this example we are using only one partition, please refer to the documentation at the end of this blog in order to get a better understanding on benefits of partitions.

$resourceGroup = "imageprocess-rg"

$storageAccountName = "pmcstorage07tier0"

$queueName = "my-test-queue"

In this example I’m using the storage account named pmcstorage77tier0 and as we can see below I don’t have any queues created yet, this screen shot snippet was taken from Azure Storage Explorer, that can be installed from here.

Getting/Creating a queue

$queue = Get-AzureRmStorageQueueQueue -resourceGroup $resourceGroup -storageAccountName $storageAccountName -queueName $queueName

This is the result of the cmdlet below, it created a new queue since I didn’t had yet one named my-test-queue, notice that queue names cannot contain capital letters and spaces.

If we look at the https://portal.azure.com you can see that now I do have a new queue in my storage account:

Adding a message to the queue

In order to add messages to the queue we need to define a hash table, it can be in a separate object or directly defined in the cmdlet as you can see:

Example 1

Add-AzureRmStorageQueueMessage -queue $queue -message @{"type"="copy";"vhdname"="newvhd.vhd";"sourceStorageAccount"="pmcstorage05";"subscription"="mysubscription"}

Example 2

$message = @{"type"="copy";

             "vhdname"="newvhd01.vhd";

             "sourceStorageAccount"="pmcstorage05";

             "subscription"="mysubscription"}

Add-AzureRmStorageQueueMessage -queue $queue -message $message

The effect behind the scenes is that this hash table will become a json string in the queue so you can easily consume it in whatever application you need.

Details of message 1

Peeking messages from the queue

Peek process in a queue allows you to get the message without de-queuing it, useful when you need to just inspect values inside of it, we do have two cmdles for this process, one that peeks the first available message (Invoke-AzureRmStorageQueuePeekMessage) or one to get the all messages in the queue (Invoke-AzureRmStorageQueuePeekMessageList).

Example 1 – Peeking a single message

$message = Invoke-AzureRmStorageQueuePeekMessage -queue $queue

Result

Example 2 – Peeking all messages

$messageList = Invoke-AzureRmStorageQueuePeekMessageList -queue $queue

Result

De-queuing messages from the queue

This process is a little different because this is the regular queue operation, it hides the message as soon as it is de-queued for processing for example.

$message =  Invoke-AzureRmStorageQueueGetMessage -queue $queue

Result

Please notice the DequeueCount property of the message, stating that it was de-queued one time. Now if we perform a peek to get all messages we just see the second message added because it was not de-queued. You can verify because the message ID is different as well.

Clearing queue from all messages

Clear-AzureRmStorageQueue -queue $queue

Result

if you don’t want to be asked for confirmation you can just add –Confirm:$false to the end of the line.

Clear-AzureRmStorageQueue -queue $queue -Confirm:$false

Removing messages from the queue

In this situation we have two ways to preform this, one is by passing the CloudQueueMessage object which is obtained by using the Invoke-AzureRmStorageQueueGetMessage cmdlet or if you know the message id and the popReceipt values you can use them directly. Notice that you can only delete a message if you de-queue it first.

Example 1 – Passing CloudQueueMessage object as argument

Adding a test message to queue

Add-AzureRmStorageQueueMessage -queue $queue -message @{"type"="copy";"vhdname"="vhd01.vhd";"sourceStorageAccount"="pmcstorage05";"subscription"="mysubscription"}

De-queuing de message

$message =  Invoke-AzureRmStorageQueueGetMessage -queue $queue

Removing the message based on the CloudQueueMessage object

Remove-AzureRmStorageQueueMessage -queue $queue -message $message

Notice that after deleting it I tried to de-queue the message but nothing came out because I deleted the message.

Example 2 – Passing message ID and popReceipt as argument

Adding another test message in the queue

Add-AzureRmStorageQueueMessage -queue $queue -message @{"type"="copy";"vhdname"="vhd02.vhd";"sourceStorageAccount"="pmcstorage05";"subscription"="mysubscription"}

De-queuing message

$message =  Invoke-AzureRmStorageQueueGetMessage -queue $queue

Displaying message contents

Those red highlighted properties are the ones we will pass in the next cmdlet to delete the message.

Remove-AzureRmStorageQueueMessage -queue $queue -messageId "969f7550-982d-49d5-bff8-26324adcf587" -popReceipt "AgAAAAMAAAAAAAAAgi9WalsU0wE="

Of course I could use the $message object I already as $message.id and $message.popReceipt but my idea was to show how the actual values looks like.

As a side note, message only gets the popReceipt when you de-queue it.

Updating a message and placing it back to the queue

Sometimes all you need is to update the message content, this operation exemplifies how to perform this operation.

Adding a test message to the queue

Add-AzureRmStorageQueueMessage -queue $queue -message @{"type"="copy";"vhdname"="vhd03.vhd";"sourceStorageAccount"="pmcstorage05";"subscription"="mysubscription"}

De-queuing the message

$msg = Invoke-AzureRmStorageQueueGetMessage -queue $queue

In this example we are changing the VHD name property of the JSON string with a random name, but first we need convert this JSON string into a PowerShell object to make things easier.

$msgObj = $msg.AsString | ConvertFrom-Json

Changing the property content

$msgObj.vhdname = [string]::format("{0}.vhd",[guid]::NewGuid().guid)

Updating the message object

$msg.SetMessageContent(($msgObj | ConvertTo-Json -Depth 100))

Updating the message in the queue

Update-AzureRmStorageQueueMessage -queue $queue -message $msg

Making the message visible again in the queue and peeking the queue to check the changes made, notice that the popReceipt is null meaning the message is ready to get de-queued again.

Update-AzureRmStorageQueueMessage -queue $queue -message $msg -visibilityTimeout 0

Invoke-AzureRmStorageQueuePeekMessage -queue $queue

Well, that’s it for this post, I hope you enjoy this new PowerShell module and that it helps you build great PowerShell scripts using Azure Storage Queues.

Regards

Paulo

UPDATE (08/19/2017): The PowerShell Module described below now supports both Managed and  Unmanaged Disks, so there is no need anymore to use this manual process unless you want to understand what happens behind the scenes of the module.

This blog post walks you through the manual process of adding an Azure virtual machine that uses managed disk to an availability set using a manual process purely inside the Azure portal (https://portal.azure.com). Note that if you have virtual machines using unmanaged disks that needs to be included in an availability set, you can use a PowerShell module posted at the TechNet Gallery using this link.

This blog assumes you have a great understanding of what is an availability set, if you don’t know what this is, please refer to Azure availability sets guidelines for Windows VMs and Manage the availability of Windows virtual machines in Azure. It also assume that you have the concepts of managed disks in Azure, if not, please refer to Azure Managed Disks Overview.

Finally we need the following items already deployed in Azure:

• Availability set aligned to be used with managed disks
• The virtual machine using managed disks that is not yet associated with an availability set

This example also assumes that you don’t have any data disk attached to your virtual machine, if that’s the case you need to change the dataDisks in a similar way I do with the osDisk property.

Steps

1. Sign in at https://portal.azure.com
   
2. Export the current template definition of you live resource (virtual machine in this case)
   
   1. At the top navigation bar of Azure Portal, click in the search field and type “resource explorer”
   2. Click on “resource explorer”
      
      
      
      
   3. Easiest way to navigate over your resources is expanding Subscriptions->Subscription Name->Providers->Microsoft.Compute->virtualMachines->VM Name
      
      
      
      
   4. On the details pane, where the template shows up, copy all its template contents and save it locally with notepad or any tool you may find appropriate.
   5. Close “Resource Explorer”
      
3. Delete your virtual machine (yes, as of the day this article was written you can delete the virtual machine in Azure and the disks gets preserved)
   
4. Create a new deployment
   
   1. Click on “+ New”, at the search field, type “template deployment” and press enter
      
      
      
      
   2. Select “Template Deployment” from the “Everything” blade and click “Create”
      
      
      
      
      
   3. At the “Custom deployment” blade, click “Edit template”. This will bring a bring a blank template structure.
      
      
      
      
      
   4. At line 5, between the square brackets “[]”, paste the template you saved in the file.
      
      Blank template
      
      
      
      Template after content was pasted
      
      
      
      
      
   5. Delete the line that contains “vmId” property, in this guide this is line 7
      
   6. Replace the contents of imageReference:
      
      1. From
         
         

         "imageReference": {

             "publisher": "OpenLogic",

             "offer": "CentOS",

             "sku": "7.3",

             "version": "latest"

         },

         
         
      2. To
         

         "imageReference": null,

         
         
         
   7. Replace the contents of osDisk:
      
      1. From
         
         

         "osDisk": {

             "osType": "Linux",

             "name": "centos02",

             "createOption": "FromImage",

             "caching": "ReadWrite",

             "diskSizeGB": 31

         },

         
         
      2. To
         
         

         "osDisk": {

           "createOption": "Attach",

           "osType": "Linux",

           "managedDisk": {

             "id": "<Resource ID of your OS Disk>"

           }

         },

         Where <Resource ID of your OS Disk> needs to point to the resource ID of the currently existing OS Disk, which can be obtained by searching for the disk in the search field of the top navigation pane and copying the Resource ID property (note that you need to open a new tab on your internet browser)
         
         
         
         
         
         
         
   8. Replace the content of “osProfile”
      
      1. From
         
         

         "osProfile": {

             "computerName": "centos02",

             "adminUsername": "pmcadmin",

             "linuxConfiguration": {

                 "disablePasswordAuthentication": false

             },

             "secrets": []

         },

         
         
      2. To
         
         

         "osProfile": null,

         
         
         
   9. Insert the new element for the Availability Set. For that, go to the line where it starts with “provisioningState”, Insert a line before “provisioningState” and paste the following element (including the extra comma)
      
      

      "availabilitySet": {

        "id": "<Resource ID of the Availability Set>"

      },

      
      To obtain the resource Id:
      1. At the top navigation pane, click the search icon
      2. Type the availability set name at the search field
      3. Select the availability set
      4. Click “Properties”
      5. Copy the Resource Id by clicking at the copy icon ()
         
         
   10. Notice that if your VM have any installed extension, you need to remove them from this template otherwise the deployment may fail, you can always add the extensions back later, dealing with extensions is out of scope of this blog post.
       
   11. Just after the very last property of your template, in my case “name”, which is the virtual machine name, add a comma at the end of the “name” line and add the following line (you must have a comma before this new “apiVersion”, otherwise validation will fail.
       
       

       "apiVersion": "2016-04-30-preview"

       
       
   12. Your template code should be similar to this
       
       
       

       {

         "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",

         "contentVersion": "1.0.0.0",

         "parameters": {},

         "resources": [

           {

             "properties": {

               "hardwareProfile": {

                 "vmSize": "Standard_A1"

               },

               "storageProfile": {

                 "imageReference": null,

                 "osDisk": {

                   "createOption": "Attach",

                   "osType": "Linux",

                   "managedDisk": {

                     "id": "/subscriptions/<GUID>/resourceGroups/TEST03-RG/providers/Microsoft.Compute/disks/centos02"

                   }

                 },

                 "dataDisks": []

               },

               "osProfile": null,

               "networkProfile": {

                 "networkInterfaces": [

                   {

                     "id": "/subscriptions/<GUID>/resourceGroups/test03-rg/providers/Microsoft.Network/networkInterfaces/centos0287"

                   }

                 ]

               },

               "availabilitySet": {

                 "id": "<Resource ID of the Availability Set>"

               },

               "provisioningState": "Succeeded"

             },

             "type": "Microsoft.Compute/virtualMachines",

             "location": "westus",

             "id": "/subscriptions/<GUID>/resourceGroups/test03-rg/providers/Microsoft.Compute/virtualMachines/centos02",

             "name": "centos02",

             "apiVersion": "2016-04-30-preview"

           }

         ]

       }

       
       
   13. Click “Save”
       
       
   14. Select the same exact resource group where your virtual machine existed at the “Resource group” field and same location if available to choose, then click “I agree to the terms and conditions stated above” and click “Purchase”
       
       
       
       
       

If any errors come up, you can always click in the notification icon at the top navigation pane and select the failed deployment:

This will bring the failed deployment blade

Then just click on “Redeploy” button so you can edit the template and fix any issues.

Again, if you have virtual machines using unmanaged disks that needs to be included in an availability set, you can use a PowerShell module that I developed and poste at the TechNet Gallery using this link.


That’s it for this blog and I hope this helps.

Basically this bash script with Azure CLI 2.0 deploys the following components:

• 1 x Resource Group
• 1 x Storage Account
• 2 x Virtual Networks
• 1 x Network Security Group
• 2 x Linux Virtual Machines, one in each virtual network

This script requires JQ and Azure CLI 2.0 installed in your Linux/Bash on Ubuntu on Windows.

If you want to use Bash on Ubuntu on Windows 10, please refer to this documentation in order to enable it.

Installing Azure CLI 2.0 on Bash on Ubuntu on Windows

Azure CLI 2.0 is a new version of Azure Xplat CLI, and it is used to manage Azure (Resource Manager mode only) through command line on Windows, Linux and OS X.

• Updating Python to the latest version (minimum supported is 2.7.6)
  • To check installed version: python –version,
  • Execute the following steps to update to the latest version if needed:

  sudo add-apt-repository ppa:fkrull/deadsnakes-python2.7

  sudo apt-get update 

  sudo apt-get install python2.7

• Install other dependencies
  
  

  sudo apt-get update && sudo apt-get install -y libssl-dev libffi-dev python-dev

• Install Azure CLI 2.0
  
  

  sudo su

  curl -L https://aka.ms/InstallAzureCli | bash

  
  When asked type the following answers:
  
  

  /usr/lib/azure-cli

  /usr/bin

  y

  /etc/bash.bashrc

• Exit bash and open it again

Installing Azure CLI 2.0 on Ubuntu 16.10

• These steps assumes you installed a VM based on Ubuntu 16.10 from Azure marketplace, perform upgrade on existing packages
  
  

  sudo apt-get update

  sudo apt-get upgrade –y

• Install dependencies
  
  

  sudo apt-get update && sudo apt-get install -y libssl-dev libffi-dev python-dev build-essential

  sudo su

  curl -L https://aka.ms/InstallAzureCli | bash

  
  

• When asked type the following answers:
  
  

  /usr/lib/azure-cli    

  /usr/bin

  y

  /etc/bash.bashrc

  
  

• Logoff of this session and open session again

For more information about Azure CLI 2.0 please refer to https://docs.microsoft.com/en-us/cli/azure/overview.

Installing JQ

jq is a json command-line json processor which helps querying (it also transforms) json strings to extract information.

• On Bash on Ubuntu on Windows 10 prompt execute the following steps:

  • Install dependencies
    
    

    sudo apt-get install build-essential libtool autoconf gcc –y

    
    

  • Download and extract oniguruma and jq sources
    
    

    curl -L https://github.com/kkos/oniguruma/releases/download/v5.9.6_p1/onig-5.9.6_p1.tar.gz -o ./onig-5.9.6_p1.tar.gz

    curl -L https://github.com/stedolan/jq/releases/download/jq-1.5/jq-1.5.tar.gz -o jq-1.5.tar.gz

    tar -xvzf ./onig-5.9.6_p1.tar.gz

    tar -xvzf ./jq-1.5.tar.gz

    
    

  • Install oniguruma
    
    

    cd onig-5.9.6

    ./configure && make && sudo make install

    
    

  • Install jq
    
    

    cd jq-1.5

    ./configure && make && sudo make install

    
    

• To install on full Ubuntu 16.10 execute the following:
  
  

  sudo apt-get install jq -y

  
  

For more information on jq, please refer to https://stedolan.github.io/jq/.

Running the script

The script can be manually downloaded from https://github.com/paulomarquesc/AzureCLI2.0BashDeployment or you can download directly using curl:

curl -O https://raw.githubusercontent.com/paulomarquesc/AzureCLI2.0BashDeployment/master/azuredeploy.sh

Change the file properties to be able to be executed as a script:

chmod 700 ./azuredeploy.sh

Authenticate to azure with az login (follow the steps to sign on using a browser and entering your Azure AD or MSA account that you use to manage Azure resources):

az login

Basic Script syntax:

./azuredeploy.sh -s <subscription name> -r <resource group name> -l <location>

Hope that this blog post gives you a good visibility on how to use Azure CLI 2.0 and bash. This script was tested on Ubuntu 16.10 and Bash on Ubuntu on Windows 10 environments.

Regards

Paulo

Just to close the discussion started on Creating an Ubuntu Server 16.10 generalized image on Azure and Deploying a new VM based on this image using Azure CLI 2.0, where we deployed an Ubuntu VM from the Azure Marketplace, then customized its image and deployed other VM based on this generalized image by using Azure CLI 2.0, here is a quick follow up with a complement option, which is deploying a VM from from that same image but now from the portal.

This portal option is easier than the process described in my previous article, please just follow these steps in order to perform this deployment:

1. From the Azure Portal (https://portal.azure.com), click on “More Services”
   
   
   
   
2. Type images in the search textbox and click “Images” result
   
   
   
   
   
3. If you followed the steps I outlined in the mentioned blog post, you will have at least one image listed at the Images blade, click on the image name
   
   
   
   
   
4. Click on “+ Create VM”
   
   
   
   
   

Follow the regular steps to complete the VM deployment similar to the steps you take with a regular Azure Marketplace image.

That’s it for this quick and simple post, I just think that this is a really cool feature that I had to blog about it .

Regards

Paulo

This blog post shows an end-to-end scenario for executing a bash script remotely on an Azure Linux VM. I put this together because out there we have documentation for the commands individually but no end-to-end example, here I show how to obtain the storage account key, how to check if a container exists and create one if it doesn’t and execute the script securely without public access turned on.

The following steps will guide through running a bash script through the Custom Script Extension for Linux.

It is also assumed that you have Azure CLI 2.0 installed in a management computer by following the steps outlined here to perform the installation, all command line instructions below assumes you are executing them on Linux bash, I’m using Bash on Ubuntu on Windows 10 to write this post.

Steps

1. We need to obtain the storage account key which is our authentication method to a storage account since we need to upload the script and tell the Custom Script Extension to download the script from that location when it gets executed inside the virtual machine. This key later is used for other operations.
   

key=$(az storage account keys list --account-name pmcstorage100 --resource-group Azure-Work-Week-RG --query '[0].value' --output tsv)



2. The script needs to be located in a container within the storage account, so we need to check if a container exists
   
   

   az storage container show --name scripts --account-key ${key} --account-name pmcstorage100

   
   This is the output when the tested exists:
   
   
   
   
   If that container does not exists a message “The specified container does not exist” will be displayed:
   
   
   
   
   If you need to test this in a script, follows a bash code snippet that exemplifies one of the ways to achieve it:
   
   

   RESULT=$(az storage container show --name scripts1 --account-key ${key} --account-name pmcstorage100)

   if [ "$RESULT"  == "" ]

   then

       echo "Container does not exist"

   else

       echo "Container exists"

   fi

   
   
3. If the tested container does not exist, the following command line will create it (notice that access is private)
   
   

   az storage container create --name "scripts01" --account-name pmcstorage100 --account-key ${key} --public-access off

   
   
4. Next step is to upload the file, notice that we keep reusing $key variable that contains the storage account key that is required
   
   

   az storage blob upload --container-name scripts01 --file ./script01.sh --name script01.sh --account-key ${key} --account-name pmcstorage100

   
   
5. Another important step is to build the Protected Settings JSON string, which will be used by the Custom Script Extension to download the file or files from the storage account
   
   

   protectedSettings='{"storageAccountName":"pmcstorage100","storageAccountKey":"'${key}'"}'

   
   
6. Finally, we execute the Custom Script Extension, passing several arguments, one of then is the setting JSON string (which can be built in other ways) where we indicate the URL(s) of the file(s) to download and the one that will be executed, plus the protected settings which contains the storage account key
   

   az vm extension set --publisher Microsoft.Azure.Extensions --name CustomScript --version 2.0 --settings '{"fileUris": ["https://pmcstorage100.blob.core.windows.net/scripts01/script01.sh"], "commandToExecute":"./script01.sh"}' --protected-settings $protectedSettings --vm-name Ubuntu-VM-04 --resource-group linux-test-rg

   
   Output
   
   
   
   

Side note

If you’re like me that is just learning how to work with Linux and Bash, here is an example on how to obtain a shared access signature token:

# Getting next day date and time in UTC for expiry date of the SAS Token

dateVar=$(TZ=UTC date +"%Y-%m-%dT%H:%MZ" -d "+1 days")

# Obtaining the SAS token

sas=$(az storage container generate-sas --name scripts --expiry ${dateVar} --permissions rl --account-name pmcstorage100 --account-key ${key} --output tsv)

This is not directly related to what we did in this post but may be useful one day if you need to generate AdHoc SAS tokens. For more information about these tokens, please refer to Using Shared Access Signatures (SAS).

That is it for this blog, thanks for stopping by.

Regards

Paulo

This step by step guide will show you how to prepare a generalized image from an Ubuntu 16.10 image from the marketplace, using the newest managed disks that were released on 02/08/2017.

Generalized virtual machines are nothing more than a virtual machine installed from the Azure Marketplace then manually customized and prepared to become an image, if it is based on Windows, that means executing Sysprep as a final step, if it is Linux-based, that means de-provisioning the Linux agent (the Azure agent) as a final step.

In order to follow this guide, please deploy a new Ubuntu 16.10 virtual machine using managed disks instead of unmanaged disks (the old way with storage accounts), through the portal or any other deployment method, for this example, I customized the image by installing XRDP on it, by following the steps outlined in this document.

It is also assumed that you have Azure CLI 2.0 installed in a management computer by following the steps outlined here to perform the installation, all command line instructions below assumes you are executing them on Linux bash only. I’m using Bash on Ubuntu on Windows 10 to write this post.

Creating the generalized image from specialized image

1. After you have your master Ubuntu virtual machine configured the way you need, please execute the following commands in order to make it available to be used as an image, this needs to be accomplished inside your virtual machine.
   
   

   sudo waagent -force -deprovision

   export HISTSIZE=0

   
   
2. From a management computer (where Azure CLI 2.0 is installed) shutdown your specialized virtual machine
   
   

   az vm deallocate --resource-group linux-test-rg --name ubuntu-vm-01

   
   
3. Generalize the virtual machine from Azure perspective
   
   

   az vm generalize --resource-group linux-test-rg --name Ubuntu-VM-01

   
   
4. Still from the management computer, create an image from the generalized virtual machine, must be in the same resource group as your master virtual machine.
   
   

   az image create --resource-group linux-test-rg --name UbuntuMasterImage01 --os-type Linux --source Ubuntu-VM-01 --location eastus

   
   

Deploying a new VM from image using managed disks

1. Obtain the image id
   
   

   az image list --query '[].{name: name, id: id}' 

   
   Result
   
   
   
   

2. Create the VM based on the generalized image
   
   

   az vm create \

       --image /subscriptions/<GUID>/resourceGroups/LINUX-TEST-RG/providers/Microsoft.Compute/images/UbuntuMasterImage01 \

       --admin-username azureuser \

       --admin-password Test@2017-123! \

       --public-ip-address-dns-name tstubuntudemo01 \

       --resource-group linux-test-rg \

       --location eastus \

       --name Ubuntu-VM-03 \

       --vnet-name vnet \

       --subnet subnet01 \

       --size standard_a1_v2 \

       --os-type linux \

       --authentication-type password \

       --os-disk-name Ubuntu-VM-03-osdisk

   
   

Adjusting the automatically created Network Security Group

The customization made to this image was enabling RDP service on Ubuntu, since the az vm create command also creates a network security group by default, we need to change it to allow port 3389 which is the one that RDP uses.

The following steps will guide you through adding a new security rule to the existing NSG:

1. To obtain the NSG associated with this new virtual machine, first let's get the ID of the network adapter
   
   

   az vm show --resource-group linux-test-rg --name Ubuntu-VM-03 --query 'networkProfile.networkInterfaces[0].id' --output tsv

   
   Result
   
   
   
   
2. Getting the network security group id
   
   

   az network nic show --ids /subscriptions/<GUID>/resourceGroups/linux-test-rg/providers/Microsoft.Network/networkInterfaces/Ubuntu-VM-03VMNic --query 'networkSecurityGroup.id' --output tsv

   
   Result
   
   
   
   
3. Getting the network security group name
   
   

   az network nsg show --ids /subscriptions/<GUID>/resourceGroups/linux-test-rg/providers/Microsoft.Network/networkSecurityGroups/Ubuntu-VM-03NSG --query 'name' --output tsv

   
   Result
   
   
   
   
   
4. Adding the network security rule to the existing NSG
   
   

   az network nsg rule create --resource-group linux-test-rg \

       --nsg-name Ubuntu-VM-03NSG \

       --name allow-rdp \

       --protocol tcp \

       --direction inbound \

       --priority 300 \

       --source-address-prefix "*" \

       --source-port-range "*" \

       --destination-address-prefix "*" \

       --destination-port-range 3389 \

       --access allow

   
   

Since XRDP does not come installed by default, this demonstrates that the new virtual machine came from the specialized image that you just created.

If you want to expand this scenario and execute a post deployment script on this newly deployed virtual machine, please refer to the post Executing Custom Script Extension using Azure CLI 2.0 on an Azure Linux Virtual Machine.

I hope you have enjoyed this short post and have fun creating your own custom images in Azure.

For more information, please refer to the following documents:

Azure Managed Disks Overview

Create a Linux VM using the Azure CLI 2.0 Preview (az.py)

Add a disk to a Linux VM

Create a copy of a VHD stored as an Azure Managed Disk by using Managed Snapshots

Upload and create a Linux VM from custom disk by using the Azure CLI 2.0 (Preview)

See you in the next post!

Paulo

In this quick article I show how we can use custom template deployment to domain join a Windows virtual machine on Azure to an existing domain that the virtual machine has access without the need to logon to the VM.

For this template to work we do have some prerequisites that must be met:

1. Virtual machine to join to a domain must exist beforehand
2. An Active Directory Forest must exist and a Domain Controller must be accessible by the virtual machine either on-premises or in Azure
3. The user that is required in this template must have the necessary rights to join computers to an Active Directory Domain
4. Domain DNS Name must be resolved by the virtual machine

Details about some of the parameters that will be presented to us when using this template:

1. location - current location of the exising virtual machine. E.g. West US.
2. dominJoinUserName - this parameter must be in domain\username notation, where domain is the NetBios name of the domain. E.g. contoso\myAdmin
3. ouPath - This is an optional parameter that allows you to join this virtual machine into a specific OU instead of the default Computers container. E.g. OU=MyCorpComputers,DC=Contoso,DC=com

This deployment uses the “Deploy to Azure” button of the Azure Quickstart Templates that brings the custom template deployment blade on Azure Portal (https://portal.azure.com).

In order to domain join a virtual machine, please follow these steps:

1. Go to the following Azure quick start template (201-vm-domain-join-existing)
   https://azure.microsoft.com/en-us/documentation/templates/201-vm-domain-join-existing/
2. Click “Deploy to Azure” button
3. Type the virtual machine name in “VM (string)” field on “Parameters blade”
4. Type the location name (e.g. “East US”) in “LOCATION (string)” field
5. Type “domain\username” in “DOMAINJOINUSERNAME (string)”, this is a user that has rights to domain join a computer on Active Directory domain, make sure you match the domain name and username with a valid user in the domain that has rights to perform computer domain join operation.
6. Type the user’s password in “DOMAINJOINUSERPASSWOD (securestring)”
7. Type domain FQDN in “DOMAINFQDN (string)”
8. Click “OK”
9. Back to “Custom deployment” blade, make sure you select “Use existing” at Resource group section and select the resource group where the virtual machines is located in the combo box
10. Click “Review legal terms” on “Purchase” blade
11. Click “Purchase”
12. Click “Create” on “Custom deployment”

That’s it for this post, simple and fast, we don’t even need to logon to the computers in order to domain join them .

Regards

Paulo

Hi Everyone,

As you may know, Azure Virtual Machines relies on Virtual Machine Extensions to perform some critical functionality, PowerShell DSC is one of them and it allows us to execute post installation tasks or to keep existing virtual machines in a state that we require them to be. This blog post shows some quick steps to help testing your DSC configuration locally, just before you add it to the Azure Resource Manager template or in a PowerShell deployment script, which saves time and troubleshooting gets a little bit easier. For more information about available extensions, please refer to this article.

This example shows the steps to execute locally a configuration file that installs an MSI file (Azure PowerShell Module).

1. In a computer with Powershell 4.0 or greater, copy you DSC configuration file (xyz.PS1) to a folder, e.g. C:\DscTest
2. Copy any executable or MSI files to the same folder if you’re using it (in this case I’m installing Azure-Powershell.1.5.1.msi, all available module versions can be downloaded from this GitHub link.)
3. Rename your <xyz>.PS1 file to localhost.ps1
4. Install any necessary DSC Modules to C:\Program Files\WindowsPowerShell\Modules
5. Edit the localhost.ps1 file with Powershell ISE and add the configuration name to the end of the file (must be after the closing curly bracket), like the example below
   
   
   
   
6. Execute this file from ISE
7. You will notice that a MOF file will be created in a folder with the same name as the configuration file and the MOF file will be named localhost.mof, for example:
   
   
   
   
8. To execute the DSC just run this cmdlet
   
   

   Start-DscConfiguration -Path C:\DscTest\InstallAzurePowershell -ComputerName localhost -Force -wait -Verbose

   
   Note: the –path argument is the path where your localhost.ps1 resides plus the configuration folder name
   
   
9. This is the resultant output that you can use to identify issues and make necessary corrections, in this example everything is fine and we are good to implement this configuration to our deployment scripts or Azure Resource Manager template.
   
   
   
   
10. If you identify any issue with your DSC configuration, just fix your PS1 file and execute steps 6 through 8 again to test it, so it get a new MOF file with the updated configuration.

Troubleshooting

When executing DSC on step 8 you get the following error message

The client cannot connect to the destination specified in the request. Verify that the service on the destination is running and is accepting requests. Consult the logs and documentation for the WS-Management service running on the
destination, most commonly IIS or WinRM. If the destination is the WinRM service, run the following command on the destination to analyze and configure the WinRM service: "winrm quickconfig".
    + CategoryInfo          : ConnectionError: (root/Microsoft/...gurationManager:String) [], CimException
    + FullyQualifiedErrorId : HRESULT 0x80338012
    + PSComputerName        : localhost

Enable WinRM with the following cmdlet:

Enable-PSRemoting

That’s it for this quick article.