Export / Import Variables to Azure Automation

Anthony Watherston — Tue, 05 Jun 2018 09:58:48 GMT

For unencrypted variables in an Azure Automation account - the script linked here will export or import them. Usage is Manage-Variables.ps1 [-AutomationAccountName] [-ResourceGroupName] [-Path] [-Mode]

Azure Service Principal in Jenkins

Anthony Watherston — Wed, 18 Apr 2018 09:45:14 GMT

A quick update to a previous post - below is how to add an Azure Service Principal to a Jenkins job - then use it to authenticate to Azure using PowerShell Core.

Ensure that the Azure AD Plugin is installed in the Jenkins instance - it requires a restart of the service.
Add a new credential - select Microsoft Azure Service Principal
Fill in the details from your application.
Click Verify Service Principal. If it is working you should get a success message.

When using the service principal in a job.

1. Ensure that the Use secret text or file(s) is ticked and select the service principal.
2. In the ps1 file - add the following function (or create a module on the Jenkins node) - the you can just call it to authenticate to Azure.


function Connect-AzureJenkins {
$tenantId = $env:AZURE_TENANT_ID
$username = $env:AZURE_CLIENT_ID
$password = ConvertTo-SecureString -String $env:AZURE_CLIENT_SECRET -AsPlainText -Force
$credential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $username, $password
Add-AzureRmAccount -ServicePrincipal -Credential $credential -TenantId $tenantId
}

PowerShell Core and Jenkins

Anthony Watherston — Tue, 23 Jan 2018 17:25:41 GMT

I have been working with a customer who wants to get some usage data from Azure around virtual networks and available IP addresses. Specifically how many IP addresses are available in a subnet within an Azure virtual network. This is a simple enough script and I can use the Get-AzureRMVirtualNetworkUsageList cmdlet to get the exact information. However the customer's environment has some challenges.

It has to be non-interactive (eventually this script will run as part of their build process)
They are using Jenkins to run the script
Jenkins is on a Linux box and I can't add a Windows slave
Azure CLI is installed but won't get me the information I need

This post explains how I overcame these hurdles using the new cross platform capability of PowerShell Core. Non-Interactive Azure Login To be able to authenticate with Azure I need to use a token - this is so I don't have to run Login-AzureRMAccount and the process can work unattended. For this I have to create a new application in Azure so I can authenticate. In Azure Active Directory - I create a new application. Click on New application registration. Fill in some details about the application and click Create. I need to generate a client secret for use with my login - select the application and click on Settings -> Keys. Enter a description for the key and click Save. A secret will be generated - you must copy and save it somewhere because when you close the blade it will be unaccessible again. This secret will be used in the script run by Jenkins. Now that the application is created we can update the script with the application ID. Script to calculate available IP addresses. The script I use is located at https://github.com/anwather/SubnetCount and performs the following steps.

Logs in to Azure using the application created above
For each virtual network it iterates through the subnets and calculates how many IP addresses are available.

Very simple - but it makes for a quick example. Provision and Configure Jenkins I can create a Jenkins instance using the Azure marketplace image which will do all the work of setting up a virtual machine and installing the product for me. In Azure I search for Jenkins in the marketplace. Select it and click Create. Step through the wizard and complete the details required - Azure will complain if you use any reserved names. You can also select instance size. I have screenshots below of the changes I made - for anything not listed I just used the default settings. When complete click Create - the deployment will begin and the portal will notify you when it is complete. Configuring Jenkins and PowerShell Core To configure Jenkins you can follow the steps at https://docs.microsoft.com/en-us/azure/jenkins/install-jenkins-solution-template. First we have to connect to the Jenkins instance running in Azure by browsing to its URL. In the virtual machine you have created it will be listed in the overview page under DNS Name. Browse to this and when the Jenkins page appears click Log in. Because we don't want anyone to be able to log in to your instance we disable the public IP on it - you have to use ssl to connect and forward the port to your local machine. I use the Bash shell on Windows 10 and just follow the instructions replacing the username with my username. You will have to enter Yes to accept the host thumbprint and also enter your password for the Jenkins instance. When complete you should be presented with a login as below. Let's install PowerShell Core on the Jenkins box. I follow the instructions at https://github.com/PowerShell/PowerShell/blob/master/docs/installation/linux.md#ubuntu-1604 and simply copy and paste the commands in the section below into the command prompt. The full output from this will look similar to the image below. At the bottom of the image is where I launch PowerShell (pwsh) Now I can just install the modules for AzureRM Networking,Profile and Resources by running the command below. Install-Module AzureRM.Profile.NetCore,AzureRM.Network.NetCore -Force Don't close the terminal window as we can now configure the job in Jenkins. Creating and Running the Jenkins Build Open the web browser to http://localhost:8080. The Jenkins page should appear and you can click on Login. You can use the username "admin" - the initial password can be retrieved by running sudo cat /var/lib/jenkins/secrets/initialAdminPassword at the terminal prompt. Once you have logged in click New Item and select Freestyle Project. Click ok. In the section marked Source Code Management select git and enter the repository information as in the image. In the section Build click Add Build Step -> Execute Shell. Enter the command as below replacing the sections with the information from your Azure Service Principal you created earlier. pwsh -File "IPUsage.ps1" -TenantID <> -ClientID <> -ClientSecret <> Click on Save and then Click Build Now - you will see the build start... Click on the number of the build and select console output. You should see the network details from your Azure subscription. So this is just an example that when faced with the possibility of writing a script in Azure CLI or Shell, now we have the option of using PowerShell Core instead.

Continuous Deployment to Azure Automation DSC Part 4

Anthony Watherston — Fri, 21 Jul 2017 11:10:26 GMT

Picking this series up after a break. Last time I mentioned I wanted to be able to add multiple configurations and ensure they compiled as well. What I did to my source directory was simply clone the sample configuration I've made. As Azure Automation DSC treats them as different configurations I just kept all the settings for the configuration exactly the same. I copied the configuration and commit it to the dev branch to begin the build. I'll use a pull request to pull the contents of my configtesting branch into the dev branch. In VSTS go to the code section and ensure the correct branch is selected. Then I just click on Create a Pull Request. Make sure the correct details are filled in and then click Create (I could add reviewers if I wanted someone to check the code or link a work item to the request) I can then go and complete the pull request. The build gets automatically triggered (by a change to the dev branch) and completes successfully without me having to change anything. If I check the artifact which is created I now have my SecondConfigurationDEV.ps1 file in there ready for publishing to the Azure Automation service. I can go and manually trigger the release for the DEV configurations. And when it is complete I can see the new DEV configuration in Azure Automation DSC. Not too hard? Well it is about to get hard.... I still need to work out how to actually test the configurations before releasing them - they can compile which is fine but will they actually work in my environment and I still need to handle credentials. I can do that using Azure Automation DSC but I'm concerned about the time to actually register the node and test the configuration. In the next post I'll try and use an on premises build agent to actually run and test the configurations can apply.

Continuous Deployment to Azure Automation DSC Part 3

Anthony Watherston — Wed, 12 Apr 2017 14:07:33 GMT

Following on from the previous post - this time I create a separation between development and production. I've created a build script to be used in the dev environment - it scans each of the script files and if the first line matches "Configuration" then it appends a DEV suffix to the configuration name and creates a new file name <>DEV.ps1. This allows me to create a new version of an existing configuration so I don't affect the production configurations. I had to make a change to the dev build task so it would run my new PowerShell script so I added in a new task to the build as below. I also updated the Copy Publish Artifact step to include all files. What this has allowed me to do is create an artifact with my dev configuration - my release script has also had some smarts added to it so it grabs all the *DEV.ps1 configurations and publishes and compiles them. When I transfer these changes to the production release (triggered by a merge to master) all I've had to do is update my release production script to publish and compile any configurations not including DEV in the name. So far so good - next step will be to introduce multiple configurations and make sure I can process them.

Continuous Deployment to Azure Automation DSC Part 2

Anthony Watherston — Wed, 12 Apr 2017 11:43:37 GMT

This is a continuation from the post here where I created a build and release pipeline for my DSC configurations. This post looks at attaching some process around when a build and release should be completed. I've used my dev branch in Git for a development build and the master branch for a production release. This means that when someone changes the dev branch a build will be automatically triggered and same for the master branch. To do this I had to perform a couple of steps for the build process. Aside from renaming to make more sense - I basically cloned the existing build process and modified it for the master branch. Each branch has a trigger set on it to perform a build when the corresponding branch is updated... I also create a production artifact now - this allows me to trigger a release when a new artifact is created. I created a new script to be added to this artifact for the production release. It does exactly the same as the dev one however uses a different path (different artifact path generated). I want to later on be able to update the development process to be able to release a dev DSC configuration which won't affect my production machines. Doing that takes care of the build process - now for the release. Aside from some cosmetic changes it is exactly the same as the development release process however I updated the script to be the ReleasePROD.ps1 file. All tested and working after a couple of minor changes - but now I have a process as follows, VSTS allows me to lock down my branches in source control so I can control who can merge.

User creates a new branch and adds/updates a configuration.
They submit a pull request to add their changes to the dev branch (they don't have permissions to modify dev)
I approve the pull request and the build to dev is triggered.
If it passes I can then trigger a dev release which uploads and compiles the configuration. I still need to find a way to mark a configuration as a dev configuration so it doesn't change production nodes.
When this is all done and tested - I can submit a pull request to pull the dev branch into master.
Once approved it will trigger a build for production - the generated artifact then triggers a release into production.

So my process is coming together - next steps are to refine and separate the configurations between development and production. Of course this still for configurations that have no external resources or credentials - that is going to add a new layer of complexity.

Continuous Deployment to Azure Automation DSC Part 1

Anthony Watherston — Tue, 11 Apr 2017 21:23:28 GMT

Update: Part 2 is located here and Part 3 is here I've been recently involved in creating some DSC configurations and using Azure Automation DSC to host them. However what I wanted was a way to continuously deploy the configurations and have them updated on the servers as they are altered in source control. There are so many moving parts involved in this that I thought I should detail what I have done. I started by defining what I want to happen (or think should happen) and then working the automation around it.

I create a new configuration or update an existing one
Something recognises that a new configuration has been created (or updated)
That configuration is copied to the Azure Automation DSC service and compiled

It seems simple enough but I want the whole process integrated with Visual Studio Team Services - this enables me to break the tasks down into smaller parts but also introduces new layers of complexity.

I know I can use VSTS to build the configuration file and run tests etc.
When the build is completed I can use release management in VSTS to deploy and compile the configuration.
I need to use source control to trigger the build / release process.

Following along? Lets start with the basics - creating a new project in VSTS - I can fill in all the other parts later. I already have a VSTS account so I can just create a new project. Let me prefix this by saying it is my approach and may not be correct :) Created the project... Initialized the repository and added a readme file... This will allow me to clone the repository to my local machine - which I've done below. The plan is to be able to trigger a build when dev branch is updated, and trigger a release when the master branch is updated but that looks a long way off at the moment. Let's get some simple stuff out of the way and make sure that the basics work. First I create a super basic server configuration - no parameters, no configuration data required - so no issues yet. Couldn't get much simple than this - it is fairly guaranteed that if my node has a C drive then it can create this one folder. So let's work through the steps to get it into an Azure Automation account - I have one created with nothing in it as yet..... That is going to give me a place to host my configurations and compile them. With that in place - in the most simplest of simple ways I have everything I need to begin deploying my configurations to Azure Automation. Forgoing tests, script analysis and probably a few other requirements in the CI/CD pipeline I can create a simple Build task which simply outputs my configuration. I only require a single task which outputs a build artifact - this is required for me release later on, I will fix and refine the process later on. Run my build..... You can scroll through the output and see what is done - basically it clones my repository to the hosted agent (here is where I can run a build script and maybe some tests) then copies my script to location as a build artifact. At this point I kind of realised that I don't actually know what a build artifact is or where it goes - I had to play around to work out what it had done. But my build completed and I have a published artifact. Now I can create a release task - there is a built in task which will allow me to run something inside Azure - I can use that to add and compile my configuration. I've created a super basic script to do just that (I had to experiment to find out the directory where the configuration is located - there is probably a better way to do it but..... I created a basic script which uploads the configuration to my automation account and compile it. I can then call this script in the release task. The cool thing about the Azure PowerShell script task is that it picked up my subscription details and did all the authorization for me. Now all that is left to do is to run it..... Success!! And to verify I can check my automation account. So there it is - my first compiled DSC configuration in Azure Automation DSC released via VSTS. It's a long way from finished - there are some other parts which I will update once complete:-

Triggering the builds and releases from pull requests
Ensuring there are no syntax errors in the configuration
Using configuration data in the compilation - so I can use credentials in my configurations.
Handle multiple configuration files.

Change in focus...

Anthony Watherston — Wed, 22 Mar 2017 11:58:22 GMT

I forgot I haven't touched this in a while - but will not be posting anything about Service Manager any more. I'm now changing focus to all Automation related technologies - specifically Azure Automation and PowerShell Desired State Configuration.

Using DSC to Setup the New SCSM Portal Server

Anthony Watherston — Mon, 15 Feb 2016 23:41:38 GMT

Hooray the new portal is out – you can get it from https://www.microsoft.com/en-us/download/details.aspx?id=49556 The portal server requires some configuration of roles before the setup can be run. Below is a DSC configuration block which you can run to configure the web server pre-requisites.

Configuration PortalWebServer
{

Node localhost
{

WindowsFeature WebServer
{
Ensure="Present"
Name="Web-Server"
}

WindowsFeature NETASP
{
Ensure="Present"
Name="NET-Framework-45-ASPNET"
DependsOn = "[WindowsFeature]WebServer"
}

WindowsFeature WebAuth
{
Ensure="Present"
Name="Web-Windows-Auth"
DependsOn = "[WindowsFeature]WebServer"
}
WindowsFeature BasicAuth
{
Ensure="Present"
Name="Web-Basic-Auth"
DependsOn = "[WindowsFeature]WebServer"
}
WindowsFeature ASP
{
Ensure="Present"
Name="Web-ASP"
DependsOn = "[WindowsFeature]WebServer"
}
WindowsFeature ASPNET
{
Ensure="Present"
Name="Web-ASP-Net"
DependsOn = "[WindowsFeature]WebServer"
}
WindowsFeature ASPNET45
{
Ensure="Present"
Name="Web-ASP-Net45"
DependsOn = "[WindowsFeature]WebServer"
}
WindowsFeature HTTPAct
{
Ensure="Present"
Name="AS-HTTP-Activation"
DependsOn = "[WindowsFeature]WebServer"
}
}
}

PortalWebServer
Start-DscConfiguration -Path .\PortalWebServer -Verbose -Wait

Domain Join Using Azure Automation DSC

Anthony Watherston — Mon, 15 Feb 2016 23:23:22 GMT

Now we have the ability to store DSC configurations in the cloud as part of an Azure Automation account. There are a few caveats to doing though – in this blog I will take you through joining a computer to a domain using Azure Automation DSC. First things first I suggest you read https://azure.microsoft.com/en-us/documentation/articles/automation-dsc-overview/ and watch the video as it explains how the service works. One of the items highlighted not in the actual blog but in the comments is that the configurations do not support PSCredential objects – as we have no way to secure them or use configuration data. So the resource needs to be modified to be able to accept string input else it will not work. Below are some of the steps I went through to get this to work. As this feature is still in preview you need to run the below commands once you have connected to your Azure account to register for Azure Automation DSC The commands for AA DSC are all part of Azure Resource Manager – so generally we are using that mode to run PowerShell against the service. We also need to use the preview portal in Azure to manage the features. Ensure you have an automation account registered in the preview portal. You can create a new account by using the new account button in the portal or the command below: Using Powershell below I do a couple of things. I create a new automation account and link it to my resource group (Line 3). Then I retrieve that account and store it in a variable (Line 5). Finally I run the Get-AzureAutomationRegistrationInfo cmdlet against my account and store the result in the $registrationinfo variable. I will use fields from this variable to register my node to AA DSC. Copying directly from the link above we need to put settings for the Local Configuration Manager (LCM) into a JSON variable. We then pass this when the extension is installed. You can modify some of the variable to suit your needs if required. Now we have enough information to register a node in AA DSC. I put the VM I want to register into a variable using Get-AzureVM – then use the Set-AzureVMExtension to add the DSC extension. After this you must update the VM to apply the configuration. You just have to wait for the extension installation to complete – but when you check your automation account in the portal it will show the node as registered under the DSC Nodes blade. If there are issues you can check on the VM in C:\WindowsAzure\Logs\Plugins\Microsoft.Powershell.DSC\2.0.0.0 – there are log files which details what is happening as the extension is installed. Now for modifying the resource – I am doing a domain join so I will use the xComputerManagement DSC resource and modify it. The original schema looks like below – it contains a couple of Credential objects. As credentials object are not yet supported in AA DSC I need to modify the resource to accept string. First I go through all the folders and rename the ‘x’ prefix to ‘c’ to indicate I am changing the module. This has to be done everywhere including the schema mof file. I then altered some of the properties inside the schema file to accept different values i.e. string for a password. Here is my new schema. As this was a quick play around I have forgone the unjoin credential and just added a password field. I also added a username value. In my DSC functions I will create a credential object using a combination of the domainname \ username. I also modified the module file for the resource. I changed the parameters for each of the functions to match the schema: - The original MOF file has a credential parameter – I added in some code to create a credential object using the value I supply in my $password parameter. This was added to each of the functions. One last thing – at this time AA DSC doesn't support module versions – I need to rename the folder version number to cComputerManagement. My overall structure for the resource looks like below. Once this has all been modified I can zip up the resource and load it into Azure Automation using the preview portal. To ensure the module has loaded correctly I can use the Get-AzureAutomationModule cmdlet and check the status of my new module – it can take a few minutes before it is provisioned. Now that this is complete I can write the DSC configuration for my server – as yet AA DSC doesn’t support configuration data so I have to accept any parameters into my configuration script and process them within. The node name I use does not have to reflect a server name – however it will appear as the configuration I supply to my node. If your module is loaded correctly on your system you can use Ctrl-Space inside the resource to prompt you for the appropriate fields. I save my file and we are ready to compile it inside Azure. The Azure module contains cmdlets to process your configuration – these are done via compilation jobs. You supply the parameters for you configuration as a hash table to the compilation cmdlet. The script below loads the configuration into Azure and then starts a compilation job with the parameters I supply. The compilation job takes a few minutes to run – you can query the status by using the Get-AzureAutomationDSCCompilationJob cmdlet – if the jobs status returns as suspended you can supply the ID of the job to the above cmdlet and it will show you the errors genererated. Once this is complete our configuration is ready to apply to the server using the cmdlets below. The cmdlet below will show me my configuration is available. I can then apply it my server. After the configuration has been applied and when the refresh frequency has been given time to apply the configuration your server will now be joined to domain. To force the configuration you can use the Update-DSCConfiguration cmdlet so it will contact the Azure DSC pull server.

AutoMagically