Azure Image Management Automation

 

Hello, in this blog post, I will talk about the Azure Image Management Automation which is a set Azure Automation Runbooks that allows us to alleviate some of the overhead work that comes with the adoption of custom “golden” images built on-premises and used in Azure.

This solution is a team effort between John Knightly (which is also a PFE) and me, where he could propose an initial solution to his customer and after some discussions between us we could design, develop  and test this solution.

Overview

To give a little perspective of how does this solution may help you, lets think about the manual process of having to build a custom golden image (steps 1-3 will exist anyways):

1. On Hyper-v or whatever hypervisor solution you use, install the OS from an ISO
2. Perform your customizations
3. Run SYSPREP (for Windows) or waagent -deprovision+user (for Linux)
4. Upload the VHD to a Storage Account by using Azure Storage Explorer, AzCopy, Add-AzureRmVHD, etc.
5. Create a managed Image of this VHD in order to be able to easily deploy VMs from that golden image

 

These steps are actually very simple but that simplicity stops when you need to make that image available to a second region, in this above example I uploaded the VHD in West US region and into a single subscription, now, what if you need to start deploying VMs from the same golden image throughout one more region? Simple, copy the VHD from the Storage Account in West US to West US 2 and then create a new managed image in that region. Why is that? Because managed images can be only created from VHDs located in the same region. Now, let’s add more, what if you have a DEV, QA and PROD subscriptions? Well, multiply the two steps by 3, giving you 6 operations. Expanding a little bit more, add a Windows and Linux golden image. This math will go to 12 operations you need to manually execute and monitor. Do that each month if you have a monthly golden image release cycle. Already too much work for a small shop correct?

To close this thought, imagine you’re a big global company with lots of business units and each one with its own requirements on Azure, leading you to have 300 subscriptions, be present in at least 10 Azure regions and need to maintain at least two golden images, one for Windows and one for Linux? Well, that will result in 6,000 manual operations a month. This would result in lots o man/hours of work and be error prone.

This solution helps you to simplify and automate this process, what you need to do is to continue producing your golden images and just use a single script to upload the VHD and wait the images being created in each subscription and region.

 

It uses the following Azure components:

1. Azure Active Directory
2. Azure Storage Accounts
3. Azure Storage Tables
4. Azure Storage Queues
5. Azure Automation

 

We are also relying on two PowerShell modules I built in order to interact with Storage Tables and Storage Queues. It also depends on a module created specifically for this solution to avoid code duplication as much as possible.

This diagram summarizes the process:

 

The following diagram shows how the process works under the hood, from uploading the VHD to producing the managed Image.

 

 

Before I continue to more details, let me define some terms:

• Tier 0 Storage Account – Storage account that contains the configuration table, the queues for VHD copy process and Managed Image creation process. Finally it is the initial point where the VHD is uploaded first.
• Tier 1 Blobs – Multiple copies of the original VHD are made inside of the Tier 0 Storage Account to allow multiple parallel copies (this is defined during setup).
• Tier 0 Subscription – Subscription where all automation components are created.
• Tier 2 Subscriptions – Subscriptions that holds tier 2 Storage Accounts and will get the Managed Image created.
• Tier 2 Storage Accounts – Final destination of the VHDs, they can be in same or different subscription, different regions, etc.

 

Step by step explanation:

1. After building your VHD, use UploadVHD.ps1 script to upload it to a tier 0 storage account
2. UploadVHD.ps1 script will upload the VHD to Tier 0 Storage Account and will trigger Start-ImageManagementTier1Distribution runbook
3. Start-ImageManagementTier1Distribution will create multiple copies of the VHD in order to allow parallel copies (when using Start-AzureStorageBlobCopy, the source blob is locked until the copy is in progress, meaning that no other copy operation can start)
4. After all copies are done, a message is placed in the copy-process-queue stating that there is a VHD to distribute to Tier 2 Storage Accounts (with other information that will be consumed later)
5. Runbook Start-ImageManagementTier2Distribution is scheduled to run every hour (minimum value) and check the copy-process-queue, if there is a message there, it will trigger one runbook called Start-ImageManagementVhdCopyTier2 that resides in a different Automation Account per destination Tier 2 Storage Account. Note that this is in another Automation Account due to the 200 runbook jobs running at the same time, for a complete list of limitations, please refer to https://docs.microsoft.com/en-us/azure/azure-subscription-service-limits#automation-limits)
6. After each individual runbooks completes its execution, a message per copied VHD is placed in the image-creation-process-queue
7. Another scheduled runbook called Start-ImageManagementImageCreation will monitor that queue (hourly again) and trigger one New-ImageManagementImage runbook (located in another Azure Automation Account due to same reasons already explained, it also can be in multiple Automation Accounts, all depends on how you did the initial setup) per VHD and perform the image creation.
8. The end result is that the managed images will start showing up at the locations you defined.

 

Setup Guide

To quick install the solution, please refer to the Setup Guide.

For operations and troubleshooting, please refer to the Operations Guide.

 

Details on SetupInfo.json file

In order to provide an easy to use solution, I provided a setup script that will automate the installation process for you. But just before we go into this process, we need to review the SetupInfo.json as follows:

This JSON file contains the information needed by the setup script to perform its installation, it is basically split into four main sections as follows:

• general
• requiredModulesToInstall
• storage
• automationAccount

 

Following items describes in detail all sections and what is required to be changed in order to set this up in your own environment.

 

General section

This section mainly requires the Azure Active Directory tenant name and defines the queue names.

Element Name	Description	Modification Required?
tenantName	That’s the Azure Active Directory tenant name. e.g. test.onmicrosoft.com	Yes
copyProcessQueueName	Name of the queue monitored by the copy process	No
imageCreationQueueName	Name of the queue monitored by the image creation process	No
jobTableName	Name of the table that holds information about an image distribution job	No
jobLogTableName	Name of the log table, it holds all log information for all distribution steps	No
imagesResourceGroup	Resource Group where the managed images will be created in all configured subscriptions	No

 

Section example

 

RequiredModulesToInstall section

This sections needs to remain unchanged unless you want to have additional modules installed in your Automation Account as soon as it gets created, This is consumed by the setup script to download the required module from PowerShell Gallery and make it available in the Tier 0 storage account for later Azure Automation Account deploment.

This is a simple array with the module names that you want to install.

Section example

 

Storage section

This section defines the Tier 0 Storage Account information, number of Tier 1 blob copies and other important information, it also defines each Tier 2 Storage Account which is each individual Storage Accounts that will receive a copy of the VHD, can be in the same or different region/subscription.

This section is split between tier0StorageAccount and tier2StorageAccount as follows:

tier0StorageAccount subsection

Defines information about Tier 0 Storage Account. This SA is the one that contains the configuration table, queues and the blobs to be copied.

Element Name	Description	Modification Required?
storageAccountName	Full name of the tier 0 storage account, make sure this name is unique and can be tested with the cmdlet: Get-AzureRmStorageAccountNameAvailability -Name <sa name> Example: mytierstorageaccount01	Yes
resourceGroup	Name of the resource group that you want this storage account to be created on.Example: <Company Initials>-OS-Images-Solution-rg	Yes
location	A valid Azure location, to obtain the Azure location list you can execute Get-AzureRmLocation to get a list of all locations, use location value.	Yes
subscriptionId	This is guid of the subscription that will have the Tier 0 Storage Account created on. Use Get-AzureRmSubscription cmdlet to list all subscriptions you have access.	Yes
container	This is the name of the container that will have the VHD copied to plus the tier 1 copies. Default is “goldenvhds”.	No
modulesContainer	This is a container at the storage account that will get all necessary modules uploaded to, later on, the setup process will use this as the source for the Azure Automation modules setup.	No
configurationTableName	This is the table that has all configurations used by this solution, manual edit of this table can be done later if you need to change something. Defaults to imageManagementConfiguration.	No
tier1Copies	This is the number of copies inside of Tier 0 Storage Account the VHD blob will have, this number will help you to have more copies done in parallel, so the higher the number, the higher the number of concurrent copies. This dictates how fast step 4 described earlier will be but will put some strand on step 3 since will will have multiple copy operations waiting for the source blob to complete the previous copy so a good number will be based on your own environment. This defaults to 300 copies.	Yes, adjust it accordingly to how many copy process you will have, maximum supported value is 999.
imagesResourceGroup	Resource Group name where the managed images will be created in the tier 0 subscription. Usually this name is the same as the imagesResourceGroup setting of the General section.	Yes

 

tier2StorageAccounts subsection

This is an array that defines information about each Tier 2 Storage Account. This SA is the one that will receive the final VHD blob and will be used by the managed image creation process. Initially we asked to provide these values inside the SetupInfo.json file, now there is a PowerShell script (Scripts\GenerateTier2StorageJson.ps1) that fills this section based on a text file with the list of subscriptions or single subscription provided from the command line, described later in this document.

Element Name	Description	Modification Required?
id	Id that uniquely identifies this storage account within this section, this is used to perform a test if the storage account exists during setup phase, it looks for this Id in the configuration table, if it is there, it means that the storage account was already deployed and no action is needed. Generally it is obtained using a guid, the GenerateTier2StorageJson.ps1 already adds this value.	Yes
enabled	Indicates if this storage account is enabled or not to receive the VHDs and finally get the image created in the subscription and region where it was deployed, this is enabled by default.	No
imagesResourceGroup	Resource Group name where the managed images will be created in the tier 2 subscription. Usually this name is the same as the imagesResourceGroup setting of the General section.	Yes
storageAccountName	This is the full tier 2 storage account name, the provided example shows how to use a PowerShell 5 class in order to generate the storage account name during the setup phase, the first parameter of this method is the name prefix which is the important change that needs to be made. The provided script (GenerateTier2StorageJson.ps1) asks for the prefix as one of its parameters and build that value.Example: "^([StorageAccountName]::new(\" <sa name prefix> \",[storageAccountTier]::tier2)).GetSaName($true)"	Yes, just the <sa name prefix> part.
resourceGroup	Name of the resource group that you want this storage account to be created on.	Yes
location	A valid Azure location, to obtain the Azure location list you can execute Get-AzureRmLocation to get a list of all locations, use location value.	Yes
subscriptionId	This is guid of the subscription that will have the Tier 2 Storage Account created on. Use Get-AzureRmSubscription cmdlet to list all subscriptions you have access.	Yes
container	This is the name of the container that will have the VHD copied to plus the tier 1 copies. Default is “goldenvhds”.	No

 

You will have as many Tier 2 Storage Accounts listed in this section as you need, they need to be added as an extra element of this array. Notice that if in the future you need to expand your solution to have more/new Tier 2 Storage Accounts, make sure you use the same JSON file with the additions and run the setup script again. The script will create the new storage accounts or any other extra items you place here, like new number of Automation Accounts as you will see further in this article.

 

Section example

 

You will notice in the sample section that we used a different notation for some of the values, We are adding the ^ sign in the beginning of the string, which means that the setup script will execute a PowerShell expression evaluation of the content of the string. In this case what we are referencing is a value inside of the same JSON file. $config is the object we use to load the content of this file, so you can use the paths that we are showing and reference existing values throughout the file so if you have values that repeats, you don’t need to keep copying all the time, but this is complete optional and if you think that adding the actual values will be more educational for documentation purposes feel free to use the actual values. Intention here is just to offer a way to avoid repeating content.

 

Automation Account section

This section defines your Automation Account settings, we will have here some general settings for the Automation Accounts plus specific sections for three Automation Account types (this is solution wise and does not mean that Azure contains different types of Automation Accounts) as follows:

• Main Automation Account, this is the one that contains the Start-ImageManagementImageCreation, Start-ImageManagementTier1Distribution and Start-ImageManagementTier2Distribution runbooks.
• Copy Process Automation Account(s) , one or more automation accounts to execute the tier 2 copy process, the number of this type of Automation Accounts will depend on number of subscriptions, regions and images, as it was previously explained, a single Automation Account can execute a maximum of 200 runbook jobs (we cap at a default of 150 in this solution) so if you have, lets say 300 subscriptions, 2 images (1 Windows and 1 Linux) and need to include two regions, you will need at least 8 copy dedicated Automation Accounts in order to be as parallel as possible.
• Image Creation Process Automation(s) , same case of the previous description, this automation account is dedicated to create the Managed Images.

 

The formula to easily determine the number of worker automation accounts (copy and image types described above) is :

 

(Number of Subscriptions X Number of Images X Number of Regions ) / 150 = Value to be used on workerAutomationAccountsCount value.

 

This section contains the following items:

Element Name	Description	Modification Required?
automationAccountPrefix	This string is used to compose the Automation Account name, this name will be used as the main Automation Account, whereas copy Automation accounts will have this prefix appended with –Copy999 where 999 is an incremental counter and image creation Automation Accounts will have –Img999 appended in the same way. Since these account names cannot be greater than 50, this prefix cannot be more than 42 characters.	Yes
resourceGroup	Name of the resource group that you want the Automation Accounts to be created on.	Yes
location	A valid Azure location, to obtain the Azure location list you can execute Get-AzureRmLocation to get a list of all locations, use location value.	Yes
subscriptionId	This is guid of the subscription that will have the Automation Accounts created on. Use Get-AzureRmSubscription cmdlet to list all subscriptions you have access.	Yes
applicationDisplayNamePrefix	This is the prefix that will be used to create the service principals in Azure Active Directory, these names will end up being with <prefix>-Copy999 and <prefix>-Img999 the same way the Automation Account Prefix.	Yes
workerAutomationAccountsCount	This element will dictate how many Automation Accounts will be created for Copy and Image Creation processes, by default it will create two of them, allowing this solution to serve up to 300 subscriptions with one tier 2 storage account and in one location at the same time, increase the number if you want to have more parallel distributions. For example, if we have 100 subscriptions with 3 locations each and maintain one Linux and one Windows image we will end up with (100x3x2) = 600 copy and image creation operations, and in this case we can divide this number by 150 runbook executions and we will have our new number which will be 4.You can start with a lower number, like the default and increase it to as much as needed and execute the setup gain, setup will add the extra new automation accounts. Note that if you decide to decrease the number this will not remove the extra Automation Accounts, therefore nothing will happen in that setup execution.	Yes (depending of how may copy and image creation operations needed for 100% parallel operations)
maxDedicatedCopyJobs	Maximum number of copy jobs, this number is a control number, used in a way that the solution knows the maximum and perform corrections if needed.	No
maxDedicatedImageCreationJobs	Similar case of maxDedicatedCopyJobssetting but dedicated to image creation jobs.	No
connectionName	This is the name of the connection to be created (runas account) during setup.	No

 

 

Runbooks subsection

This section defines all necessary runbooks to setup in each Automation Account type.

It is split between mainAutomationAccount, copyProcessAutomationAccount and imageCreationProcessAutomationAccount and for the sake of trying to keep this post as small as possible (which is being hard to ) I’m describing one element that is used to define a runbook in any of the sections.

 

Element Name	Description	Modification Required?
name	Runbook Name.	No
scriptPath	Path to the script to be imported into the PowerShell runbook. This path can be a http path as well, when you download the script directly from GitHub raw path for example.	No
scheduleName	Name of the schedule to create and assign to the runbook, the type is hourly schedule only.	No
startTimeOffset	This value is used to set the start time of the schedule, in minutes, it tells how many minutes to get started with the schedule during the schedule creation, good for staggering between tier 2 copy runbook and the image creation runbook.	No
executeBeforeMoveForward	Forces the execution of the runbook as soon as it gets imported and wait for it to finish. In our case we add the update runbook from GitHub and execute it in order to update all existing modules in the Automation Account since they are outdated at the moment of account creation, if we don’t execute it all other runbook imports fails.	No
parameters	This element define an array of key and value pairs, this defines all parameters that the runbook needs when scheduling its execution. For example:	No
requiredModules	An array of modules to import in this automation account if not imported yet. The module list item needs to come from the requiredModulesToInstall section.	No

 

Section example

This section is highly sensitive to what runbooks to install, order and modules so if changing it be sure to don’t change the already existing items.

 

 

That’s it for this blog post and I hope that this can help.

 

Regards

Paulo

Comments

• Anonymous
  August 16, 2017
  Great work Paulo, I appreciate the name drop, but this is your code! :-)
• Anonymous
  August 20, 2017
  Thanks Paulo. This will absolutely help me expand on a managed disks and image automation script I developed. Awesome work.
  • Anonymous
    August 24, 2017
    Thanks Charles, glad to know this is helping you.
• Anonymous
  August 20, 2017
  Awesome stuff Paulo. This will totally help me expand on a managed disk and image automation script I developed. Thanks man.
• Anonymous
  August 30, 2017
  I totally appreciate your work, but isn't this overcomplicating this a lot?Why not use packer to build your image on Azure? It's even now an available task on VSTS and has been around for a long while to build images on many platforms.
  • Anonymous
    September 08, 2017
    Hi David,Thanks for your comments. It depends on the scenario, this solution was built to a customer that requires their images to be built on-premises due to some security compliance, also, the solution proposed was built to scale, the initial scenario of that customer have more than 300 subscriptions using at least 12 regions each, (although not tested) this scales to any number of subscriptions and regions to cover, we do have some customers with up to 1900 subscriptions, so using a single script to upload the VHD and have it taken care for you automatically in these 1900 subscriptions seems a great deal :-) .RegardsPaulo