Simple challenge

We have been tasked to find out the monthly consumption cost of Azure services that we would be consuming as part of our solution. Although our production example had approximately a dozen services in it, to keep this example simple, in this blog we’ll be looking at only a couple of azure services. These are

• 2 instances of Standard_D1 Cloud services
• 800 GB of Geo Redundant storage, Standard IO – Page Blob/Disk

Everything in financial calculations start and end in Excel. But before we transfer these Azure service components into an Excel spreadsheet, we need to finalise some additional information. Azure service prices are dependent on the geo and regions from where they are consumed and also on the currency they are transacted in. For our calculations, we’ve assumed the following

• Cloud Services will be instantiated from the ‘US East’ region
• Storage will also be in the same region, which in Azure terms gets classified as ‘Zone 1’
• The currency of transaction will be USD – ‘$’.

Plan

We will use Microsoft Excel for our calculation. This file will have two worksheets ‘Azure Rate Card’ and ‘Bill of Material’. We will copy the Azure RateCard information into the Azure Rate Card worksheet and the computation for monthly consumption will be carried out in the ‘Bill of Material’ tab.

Accessing and Downloading Azure RateCard data

We will use Azure PowerShell script to download the Azure RateCard data applicable to our azure subscription.

What is Azure RateCard API? Azure RateCard API is part of Azure Billing API service, which also includes Usage API. The RateCard API returns the list of the resources available at an Azure offer level (Pay-as-you-go, MSDN,…) and this is not specific to a customer’s subscription, so what you would get back here is the list of ALL the resources that are available for billing under an offer.  More information on Azure RateCard API is available here and here.

Step 1. Configuring Authentication using the Management Portal

In order for our PowerShell client application to access the billing API data, the Azure subscription needs to be configured to allow access to this API from client applications. The process of doing this is clearly and succinctly described here – you’ll need to execute only the Steps 1 and Step 2 . Whilst creating the application, ensure that you specify https://localhost/ as the re-direct URI. You will need to substitute the in the PowerShell script. Once the application has been created, make a note of the Client-ID – the GUID number – that is assigned to the application that you added. This GUID will be used in the PowerShell script.

Whilst you are logged onto the Azure management portal, make a note of the following items

• Subscription ID and Azure Offer-ID. Navigate to ‘- https://account.windowsazure.com/Subscriptions’ and select the subscription under which the client access configuration was added. Make a note of subscription ID and offer-ID values.
• The primary tenant that you are authenticated against in Azure. This is the primary directory that you login to and in the old portal (manage.windowsazure.com), can be found under Settings à Subscriptions à Make a note of the Directory record next to your subscription ID

Step 2. Load and configure the PowerShell script

The complete PowerShell script is coped here. Substitute the following values in the script from the previous step

Step 3. Execute the PowerShell script.

Save a copy of the script and run the script inside the ‘Windows Powershell ISE’ desktop application. After successful execution of the script, you should have the following data file under the folder ‘C:\Users\<username>\AppData\Local\Temp’

• Ratecardoutput.txt

Step 4. Load the output data file inside the Excel

• Data file – Ratecardoutput.txt – is a character delimited file and the delimiting character is chosen to be the exclamation mark (!).
• Open Excel and create two worksheets inside it. One called ‘Bill of Material’ and another called ‘Azure Rate Card’.
• Select the Azure Rate Card Tab in the Excel spread-sheet. Select Data Menu and click on Get-External Data – “From Text”
• Open File dialog box is shown. Navigate to “C:\Users\<username>\AppData\Local\Temp” and open the file ‘ratecardoutput.txt’
• You’ll be shown the Text import Wizard – Step 1 of 3 screen. - Select Delimited option and also ‘My data has headers’
• In Step 2, uncheck the ‘Tab’ delimiter and enter ‘!’ in the ‘Other’ box. Click ‘Finish’.
• Data dialog box shown, ensure that the data is inserted at the top left hand row – Enter ‘='Azure Rate Card'!$A$1’ value in the field.
• If this step is incorrectly executed, all the rest of calculations will be incorrect

Step 5. Adding computational logic

At this point, we’ve loaded the RateCard data in our Excel. We now need to add the computational logic into the ‘Bill of Material’ worksheet.

Excel implementation details

As per the requirement, we create two rows. One containing the details about Storage and other about the virtual machine required. To facilitate this look-up, we have manually substituted the relevant ‘Meter Guid’ from the previous downloads of Azure Ratecard. This is a one-time process. In order to calculate the monthly bill, Azure services from the ‘Bill of Material’ worksheet will reference the relevant record from ‘Azure Rate Card’ worksheet.

=IFERROR(VLOOKUP(E6, 'Azure Rate Card'!$A$2:$I$2000,4,FALSE), )

Explanation: ‘E’ column contains the Meter Guid. We take the value of Meter Guid and match it against the corresponding Rate in the ‘Azure Rate Card’ worksheet.  For more information on VLOOKUP, please see this article

The formula used in the column J – consumption column is

=ROUND(IF(ISNUMBER(I6),H6*I6,CalculateTieredRate(H6, I6)), 2) or =ROUND(IF(ISNUMBER(I7),H7*I7*744,CalculateTieredRate(H7, I7)), 0)

The rate value obtained from Azure Ratecard API can either be in numeric or in string format. When it’s in string format, the rates are expressed as a key-value pairs. For e.g. the rate value obtained for storage is like – “0,0.095;1024,0.08;51200,0.07;512000,0.065;1024000,0.06;5120000,0.055”.  This value is not suitable to calculate the consumption. Therefore, in the Consumption Cost – Monthly column, we check the value of Rate column. If its numeric, we multiply it with the Quantity and monthly hours to come up with monthly consumption cost. Otherwise we use a separate VBA formula - CalculateTieredRate - to calculate the tiered metered cost. This VBA function is copied here

We use two flavours of formula, depending upon the unit of measure. If the rate is provided as hourly charge, then we multiply the consumption with 744 (= number of hours in a month) to obtain the monthly consumption figure.

Sample excel file, with all the mentioned functionality, can be downloaded from here

Result:

Now you should have a fully functioning Azure monthly estimator which takes the Azure rates applicable to your offer-ID and also takes into consideration tiered meter usage.

PowerShell Script

# Activate Azure Powershell module

Import-Module Azure

# TODO - Replace the following values

# =================================================

$adTenant = "microsoft.onmicrosoft.com"

# Set well-known client ID for Azure PowerShell

$clientId = "XXXXXXXX-XXXX-4d50-937a-96e123b13015"

# subscription guid

$SubscriptionId = 'XXXXXXXX-XXXX-4802-a5e6-d9c5a43c72a0'

# Set redirect URI for Azure PowerShell

$redirectUri = New-Object System.Uri('https://localhost/')

# Azure on Internal subscription

$OfferDurableId = 'MS-AZR-0121p' # Azure on Internal subscription

# =================================================

# Set Resource URI to Azure Service Management API

$resourceAppIdURI = "https://management.azure.com/"

# Set Authority to Azure AD Tenant

$authority = "https://login.microsoftonline.com/$adTenant"

# Create Authentication Context tied to Azure AD Tenant

$authContext = New-Object Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext($authority)

# Acquire token

$authResult = $authContext.AcquireToken($resourceAppIdURI, $clientId, $redirectUri, "Auto")

$ResHeaders = @{'authorization' = $authResult.CreateAuthorizationHeader()}

$ApiVersion = '2015-06-01-preview'

$Currency = 'USD'

$Locale = 'en-US'

$RegionInfo = 'US'

$File = $env:TEMP + '\resourcecard.json'

$OutputFilename = $env:TEMP + '\ratecardoutput.txt' # This is usually C:\Users\<username>\AppData\Local\Temp

$ResourceCard = "https://management.azure.com/subscriptions/{5}/providers/Microsoft.Commerce/RateCard?api-version={0}&`$filter=OfferDurableId eq '{1}' and Currency eq '{2}' and Locale eq '{3}' and RegionInfo eq '{4}'" -f $ApiVersion, $OfferDurableId, $Currency, $Locale, $RegionInfo, $SubscriptionId

Invoke-RestMethod -Uri $ResourceCard -Headers $ResHeaders -ContentType 'application/json' -OutFile $File

$Resources = Get-Content -Raw -Path $File -Encoding UTF8 | ConvertFrom-Json

# Insert the header line

$strHeaderLine = "{0}!{1}!{2}!{3}!{4}!{5}!{6}!{7}!{8}" -f "MeterId", "MeterSubCategory", "MeterRegion", "MeterRates", "MeterCategory", "MeterName", "Unit", "EffectiveDate",  "IncludedQuantity"

# Create the header in the output file

$strHeaderLine | Out-File $OutputFilename

    # We create this loop to take care of tiered meter enteries in the table

    foreach($meterObj in $Resources.Meters)

    {

        # convert the darn PSCustomeObject created by ConvertFrom_Json to our dictionary object

        $meterRates = @{}

         $meterObj.MeterRates | Get-Member -MemberType Properties | SELECT -exp "Name" | % {

                $meterRates[$_] = ($meterObj.MeterRates | SELECT -exp $_)

          }

        $strLine = ""

        if ($meterRates.Count -gt 1)

        {

            $nCount = 0

            foreach($meterRatePair in $meterRates.GetEnumerator() | Sort -Property Value -Descending)

            {

                $strLine += $($meterRatePair.Name) + "," + $($meterRatePair.Value)

                if ($nCount -lt ($meterRates.Count -1))

                {

                    $strLine += ";";  

                }

                $nCount++;

            }

        }

        else

        {

            foreach($meterRatePair in $meterRates.GetEnumerator())

            {

                $strLine = $($meterRatePair.Value)  # this line will execute only once.

            }

        }        

        $strOutputLine = "{0}!{1}!{2}!{3}!{4}!{5}!{6}!{7}!{8}" -f $meterObj.MeterId, $meterObj.MeterSubCategory, $meterObj.MeterRegion, $strLine, $meterObj.MeterCategory, $meterObj.MeterName,   $meterObj.Unit, $meterObj.EffectiveDate,  $meterObj.IncludedQuantity

        $strOutputLine | Out-File $OutputFilename -Append

    }

Remove-Item -Force -Path $File

Description of the PowerShell shell script.

1.       Start of the script gathers all the authentication, subscription, offer-id etc data.

2.       It logs on to the Azure subscription using the credentials provided and obtains an authentication token.

3.       Script formats the RateCard request whilst substituting the values for subscription-id, offer-id, locale, currency, Region-info.

4.       Scripts invokes REST based API call using the previously obtained authentication token and formatted request.

5.       On success, the return values are written to output file – ‘resourcecard.json’ and its copied to location - C:\Users\<username>\AppData\Local\Temp

6.       Script then filters and picks up only the ‘Meters’ data and converts the Json formatted data into a ‘!’ – exclamation mark -  character delimited file. 

a.       We will use this delimiting character to load the file into Excel.

7.       The output of this formatted data is written to file ‘ratecardoutput.txt’ which is copied to the folder - C:\Users\<username>\AppData\Local\Temp

Excel VBA function – Calculate Storage value

Option Explicit

Function CalculateTieredRate(Quantity As Double, MeterRate As String)

    ’The assumption is that the MeterRate that is passed is already sorted in value

    ’MeterRate format is "0, 0.024;1024,0.0236;51200,0.0232;512000,0.0228;1024000,0.0224;5120000,0.055"

    CalculateTieredRate = 0

    Dim MeterRatePair() As String

    Dim MeterValue()  As String

    Dim nNumberOfRates  As Long

    nNumberOfRates = UBound(Split(MeterRate, ";")) ‘ ; is our delimiting characters between records

    ReDim MeterRatePair(nNumberOfRates + 1) ‘ because the number of pairs would be one greater

    MeterRatePair = Split(MeterRate, ";")

    Dim i As Integer

    For i = 0 To nNumberOfRates  ‘ Its a zero bound array

        Dim lowerBound  As Double

        Dim upperBound As Double

        Dim currentRate As Double

        MeterValue = Split(MeterRatePair(i), ",")

        lowerBound = MeterValue(0)

        currentRate = MeterValue(1)

        Dim costAtThisLevel As Double

        costAtThisLevel = 0

        If (i >= nNumberOfRates) Then

            ‘ we are already at the last record. there is no upper bound for us at this point

            costAtThisLevel = (Quantity - lowerBound) * currentRate

            CalculateTieredRate = CalculateTieredRate + costAtThisLevel

        Else

            MeterValue = Split(MeterRatePair(i + 1), ",")

            upperBound = MeterValue(0)

            If (Quantity > (upperBound - lowerBound)) Then

                costAtThisLevel = (upperBound - lowerBound) * currentRate ‘ rate of the full band

                CalculateTieredRate = CalculateTieredRate + costAtThisLevel

            Else

                costAtThisLevel = (Quantity - lowerBound) * currentRate

                CalculateTieredRate = CalculateTieredRate + costAtThisLevel

                Exit For

            End If

        End If

    Next i

End Function

Step-By-Step guide

This article shows how a MPI applications can be setup using “bare metal” Linux nodes on Azure.

It must be emphasized that this article shows how easy it is to setup and configure the Azure environment to execute the MPI application but the configurations shown here are not recommended for a production environment.  It is assumed that the reader has basic knowledge of Azure and medium to advanced knowledge on Linux, especially Ubuntu distros.

The techniques shown here have been extended from this Ubuntu article.

1.      Creating a Cluster

As the first step, we will configure a Linux cluster under Azure. It will consist of four nodes and one of them will be the master node. The cluster will be created inside a VPN. To keep things very simple, we won’t create a DNS server and instead modify the /etc/hosts file.

The script for creating the Linux cluster is shown at the bottom of this article. Couple of things to note in the script are:

-          We create Linux cluster from Ubuntu-14_04_2_LTS. This image is available from the Azure gallery.

-          We will use NFS to created shared storage between the Linux nodes. In order for the NFS to be successful, required ports, 2049 and 111 are configured during the provisioning of the nodes.

After successful execution of creation script, you should see the Linux nodes configured under the VNET in your subscription as shown below

Figure 1: Linux nodes under Azure VPN

After the Linux nodes have been provisioned and up and running, we use PuTTY to establish a SSH connection and login to the node.

NOTE: The process of accessing and logging on to the provisioned Linux nodes is described in detailed here. 

We will use the Linux node, linuxdistro-001, as the master node.

After logging onto the node, we edit the /etc/hosts file and add the nodes TCP endpoints to this file. This step is replicated across all nodes.

linuxuser@linuxdistro-001:~$ sudo vi /etc/hosts

After editing, the content of /etc/hosts file should look like following:

linuxuser@linuxdistro-001:~$ cat /etc/hosts

127.0.0.1 localhost

10.0.0.4 ub0

10.0.0.5 ub1

10.0.0.6 ub2

10.0.0.7 ub3

After repeating the above step across each node, each node can communicate with each other without the DNS being provisioned but please bear in mind that DNS is always a better option.

As we need to execute same commands across the Linux nodes, we will install ssh & pssh utilities on the master node.

Install pssh on the master node

linuxuser@linuxdistro-001:~$ sudo apt-get install ssh

linuxuser@linuxdistro-001:~$ sudo apt-get install pssh

Test that pssh is working correctly

linuxuser@linuxdistro-001:~$ parallel-ssh  -i -A  -H  10.0.0.5  -H 10.0.0.6 -x "-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o  GlobalKnownHostsFile=/dev/null" echo hi

Note: To supress SSH warnings, we add the options x "-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o  GlobalKnownHostsFile=/dev/null" to every pssh  command

2.      Provisioning shared Storage

We will create /mirror folder under the master node and this will be shared across the nodes via NFS

Install nfs-client on all nodes except the master node

linuxuser@linuxdistro-001:~$ parallel-ssh  -i -A  -H  10.0.0.5  -H 10.0.0.6 -H 10.0.0.7 -x "-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o  GlobalKnownHostsFile=/dev/null" sudo apt-get install -y nfs-client

Create same folder structure on all nodes

linuxuser@linuxdistro-001:~$ parallel-ssh  -i -A  -h host_file -x "-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o  GlobalKnownHostsFile=/dev/null" sudo mkdir /mirror

Install the nfs-server on the master node

linuxuser@linuxdistro-001:~$ sudo apt-get install nfs-server

Ensure that the /mirror folder is set to share

linuxuser@linuxdistro-001:~$ echo "/mirror *(rw,sync)" | sudo tee -a /etc/exports

Restart the nfs share on the master node

linuxuser@linuxdistro-001:~$ sudo service nfs-kernel-server restart

Mount the share across the client nodes

linuxuser@linuxdistro-001:~$ parallel-ssh  -i -A  -H  10.0.0.5  -H 10.0.0.6 -H 10.0.0.7 -x "-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o  GlobalKnownHostsFile=/dev/null" sudo mount ub0:/mirror /mirror

Test the share across the client nodes.

linuxuser@linuxdistro-001:~$ sudo cp /etc/hosts /mirror

linuxuser@linuxdistro-001:~$ parallel-ssh  -i -A  -H  10.0.0.5  -H 10.0.0.6 -H 10.0.0.7 -x "-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o  GlobalKnownHostsFile=/dev/null" sudo ls /mirror

3.      Setting up mutual trust for non-admin user

We will create a non-admin user – mpiu. This user needs to be created because Ubuntu does not support root trust. We also assign our shared folder, /mirror, as the home folder for this user. newusers command is used to create this user across all nodes.

Note: We use the newusers command to create the user across the various nodes as it provides the ability to create execute the command in non-interactive mode. The parameters for the user mpiu is specified in a file – /mirror/userfile. 

Creating non-admin user

linuxuser@linuxdistro-001:~$ cd /mirror

linuxuser@linuxdistro-001:/mirror$ vi userfile

linuxuser@linuxdistro-001:/mirror$ cat userfile

mpiu:<password removed>:1002:1000:MPI user:/mirror:/bin/bash

linuxuser@linuxdistro-001:/mirror$ parallel-ssh  -i -A  -h host_file -x "-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o  GlobalKnownHostsFile=/dev/null" sudo newusers /mirror/userfile

Change the owner of shared folder to newly created user.

linuxuser@linuxdistro-001:/mirror$ sudo chown mpiu /mirror

Configuring password-less SSH communication across

1.       Install ssh components on the master node

linuxuser@linuxdistro-001:~$ sudo apt-get install ssh 

2.       Next, Login with our newly created user

linuxuser@linuxdistro-001:/mirror$ su - mpiu 

3.       Generate an RSA key pair for user mpiu

mpiu@linuxdistro-001:~$ ssh-keygen –t rsa 

4.       Add this key to authorized keys file

mpiu@linuxdistro-001:~$ cd .ssh

mpiu@linuxdistro-001:~$  cat id_rsa.pub >> authorized_keys

5.       As the home directory is common across all nodes (/mirror), there is no need to run these commands on all nodes

Test SSH run – it should not ask you for password for connecting to the machine.

mpiu@linuxdistro-001:~$  ssh ub1 hostname

4.      MPI (HPC Application) installation and configuration

1.       Login as the admin user

We logout from our existing mpiu session and we should be back in our admin – linuxuser – session.

mpiu@linuxdistro-001:~$ logout

linuxuser@linuxdistro-001:

2.       Install GCC

We need a compiler to compile all the code. This needs to happen only on the master node.

linuxuser@linuxdistro-001: sudo apt-get install build-essential

3.       Install MPICH2

The MPICH2 structure folder structure needs to be same across all nodes, therefore we execute the following command across all nodes/

linuxuser@linuxdistro-001: parallel-ssh  -i -A  -h /mirror/hosts_file -x "-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o  GlobalKnownHostsFile=/dev/null" sudo apt-get install -y mpich2

Note the content of /mirror/hosts_file is

linuxuser@linuxdistro-001:~$ cat ./host_file

10.0.0.4

10.0.0.5

10.0.0.6

10.0.0.7

4.       Test installation

linuxuser@linuxdistro-001:/mirror$ su – mpiu

mpiu@linuxdistro-001:~$ which mpiexec

mpiu@linuxdistro-001:~$ which mpirun

5.       Setting up machine file

Create a configuration file – nodefile – in mpiu’s home folde. Within the file specify the node names followed by the number of processes to spawn on each node.  We have two cores per node, so we are going to specify 2 or 1 process per node.

mpiu@linuxdistro-001:~$ cat nodefile

ub0

ub1:2

ub2:2

ub3

mpiu@linuxdistro-001:~$

6.       Create a testing MPICH2 C source code and compile.

Dummy mpi_hello.c program.

mpiu@linuxdistro-001:~$ cat mpi_hello.c

#include <stdio.h>

#include <mpi.h>

 int main(int argc, char** argv) {

    int myrank, nprocs;

    MPI_Init(&argc, &argv);

    MPI_Comm_size(MPI_COMM_WORLD, &nprocs);

    MPI_Comm_rank(MPI_COMM_WORLD, &myrank);

    printf("Hello from processor %d of %d\n", myrank, nprocs);

    MPI_Finalize();

    return 0;

}

                Compile the source code into an executable

mpiu@linuxdistro-001:~$ mpicc mpi_hello.c -o mpi_hello

5.      Job Execution

mpiu@linuxdistro-001:~$ mpiexec -n 6 -f nodefile ./mpi_hello

6.      Results capture and analysis

If everything is successful, you should see the following result.

mpiu@linuxdistro-001:~$ mpiexec -n 6 -f nodefile ./mpi_hello                   

Hello from processor 5 of 6

Hello from processor 0 of 6

Hello from processor 3 of 6

Hello from processor 1 of 6

Hello from processor 4 of 6

Hello from processor 2 of 6

mpiu@linuxdistro-001:~$

7.      Clean the environment

One of the biggest advantage of cloud computing is to utilise the pay-as-you-go features of the service. What that means is that once the experiment and exercise is over, the environment could be completely torn-down and it costs nothing to the end-user.

Execute the script shown here to clean the Azure environment.

Appendix

Script - Provisioning the Linux cluster

#CreateLinuxVirtualMachines - In NON AVAILABILITY GROUPS

#azure config mode asm

# The Subscription name that we want to create our environment in

$SubscriptionNameId = "<enter your subscription id here>"

# Storage account name. This should BE CREATED BEFORE EXECUTING THE SCRIPT

$StorageAccountname = "ubuntuimages"

# AffinityGroup name. This should BE CREATED BEFORE EXECUTING THE SCRIPT

$AffinityGroupName = "linuxdistros"

# Network name. This should BE CREATED BEFORE EXECUTING THE SCRIPT

$VnetName = "nktr21"

# Subnetname. This should BE CREATED BEFORE EXECUTING THE SCRIPT

$SubnetName = "linuxcluster"

# Availability Set

$AvailabilitySetName = "linuxdistro"

# Cloud Service name. This service will be created

$CloudServiceName = "linuxcloudsrv"

# Instance size

$InstanceSize = "Medium"

# Linux Admin Username

$password = "<yourpassword>"

# Linux Admin Password

$username = "linuxuser"

# Name Prefix of the VM machine name. Numeric counter number is appended to this to create the final

$LinuxImageNamePrefix = "linuxdistro-00"

# Load the keys for us to login to

$LoadSettings = Import-AzurePublishSettingsFile "NinadKNew.publishsettings"

# Important to specify the CurrentStorageAccountName, especially otherwise you might get Storage not accessible error when creating linux machines.

Set-AzureSubscription -SubscriptionId $SubscriptionNameId -CurrentStorageAccountName $StorageAccountname -ErrorAction Stop

Select-AzureSubscription -SubscriptionId $SubscriptionNameId -ErrorAction Stop

# Get the image from Azure repository that we want to create. Its the Ubuntu 14_04_LTS variant. We can speed up the creation script by caching the name.

# the name thus obtianed is 'b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04_2_LTS-amd64-server-20150309-en-us-30GB'

#$UbuntuImage = Get-AzureVMImage | where {($_.ImageName -match 'Ubuntu-14_04') -and ($_.ImageName -match '2015') -and ($_.ImageName -match 'LTS') -and ($_.Label -eq 'Ubuntu Server 14.04.2.LTS')} | Select ImageName

#Write-Host $UbuntuImage[0].ImageName

# Guid of the image name that we want to create

$ImageNameGuid = "b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04_2_LTS-amd64-server-20150309-en-us-30GB"

# If the service does not exist, we'll create one.

$GetCloudService = Get-AzureService -ServiceName $CloudServiceName -Verbose -ErrorAction Continue

if (!$GetCloudService )

{

    # service does not exist.   

    $CreateCloudService = New-AzureService -ServiceName $CloudServiceName -Label "Created from Windows power shell" -Description '16 June 2015'

    Write-Host ("Return value from creating the cloud service = {0}" -f $CreateCloudService.OperationStatus.ToString())

}

$PortmapPort = 2049

$NfsPort = 111

$Portcounter = 0

$Counter = 1

# Loop to create the Linux machines.

do

{

    # Prepend the VM name

    $LinuxImagename = $LinuxImageNamePrefix + $Counter.ToString()

    # Configure VM by specifying VMName, Instance Size, ImageName,  and specify AzureSubnet

    $VMNew = New-AzureVMConfig -Name $LinuxImagename -InstanceSize $InstanceSize -ImageName $ImageNameGuid

    # Add the username and password to the VM creation configuration

    $VMNew | Add-AzureProvisioningConfig -Linux -LinuxUser $username -Password $password | Set-AzureSubnet $SubnetName

    # Create and start the VM. Remember, it won't be fully provisioned when the call returns.   

    $Result = $VMNew | New-AzureVM -ServiceName $CloudServiceName -AffinityGroup $AffinityGroupName -VNetName $VnetName

    Write-Host ("Created VM Image {0}, return value = {1}" -f $LinuxImagename, $Result.OperationStatus )

    $Counter++

    $Portcounter++;

}

while ($Counter -le 4)

Script - Removing the Linux cluster

# CleanLinuxVirtualMachines

# The Subscription name that we want to create our environment in

$SubscriptionNameId = "<your subscription id>"

# Cloud Service name. This service will be created

$CloudServiceName = "linuxcloudsrv"

# Name Prefix of the VM machine name. Numeric counter number is appended to this to create the final

$LinuxImageNamePrefix = "linuxdistro-00"

# Load the keys for us to login to

Import-AzurePublishSettingsFile "NinadKNew.publishsettings"

Set-AzureSubscription -SubscriptionId $SubscriptionNameId -ErrorAction Stop

Select-AzureSubscription -SubscriptionId $SubscriptionNameId -ErrorAction Stop

$Counter = 1

$AllVmsStopped = 1

$OkToRemoveVM = 0

do

{

    $OkToRemoveVM = 0

    # Prepend the VM name

    $LinuxImagename = $LinuxImageNamePrefix + $Counter.ToString()

    Write-Host ("VM Image name = {0}" -f $LinuxImagename)

    $VMPresent = Get-AzureVM -ServiceName $CloudServiceName -Name $LinuxImageName

    if ($VMPresent)

    {

        if ($VMPresent.Status -eq "StoppedVM" -or $VMPresent.Status -eq "StoppedDeallocated" )

        {

            Write-Host ("{0} VM is either stopped or StoppedDeallocated" -f $LinuxImagename )

            $OkToRemoveVM =1

        }

        else

        {

            Write-Host ("[Stopping] VM {0}" -f $VMPresent.Name)

            $StopVM = Stop-AzureVM -Name $VMPresent.Name -ServiceName $VMPresent.ServiceName -Force

            if ($StopVM.OperationStatus.Equals('Succeeded'))

            {

                $OkToRemoveVM =1

            }

            else

            {

                Write-Host ("Not able to stop virtual machine {0}, cloud service will not be removed" -f $VMPresent.Name)

                $AllVmsStopped = 0  

            }

        }

        if ( $OkToRemoveVM -eq 1)

        {   

            Write-Host ("[Removing] virtual machine {0}" -f $VMPresent.Name)

            Remove-AzureVM -Name $VMPresent.Name -ServiceName $VMPresent.ServiceName -ErrorAction Continue

        }

    }

    else

    {

        Write-Warning ("No Vm found " -f $LinuxImageName)

    }

    $Counter++

}

while ($Counter -le 4)

Converting Azure PowerShell scripts to Linux Scripts.

In one of my recent customer engagements, I was demonstrating the capabilities of Azure HPC and Azure Batch. During the meeting the customer mentioned that as much as they liked the Windows PowerShell environment, they use Linux for undertaking most of their HPC compute requirements and would like to see the same running in the Linux environments.

What it means is it’s time to get the Azure Cross-Platform Command-line interface (Cli) a whirl and with the big weekend break coming up, it was time for a mini-project.

Project objective: ‘Convert and successfully execute Windows PowerShell script in the Linux environment’

Engagement Constraints:

-          Be as native as possible – meaning shell scripts only and utilising no-browser on Linux or on Windows. 

-          No Desktop installation on Linux variants. I personally don’t believe in installing Desktops on Linux servers as it sorts of defeats the purpose.

Migration Script

First let’s take a moment to review the source PowerShell script that needs to be migrated.

At a very high level, the script creates six Linux ‘Ubuntu 2015 LTS’ ‘Medium’ instances inside a Virtual network and under a specific Subnet.

The complete script, is repeated in here

#CreateLinuxVirtualMachines

azure config mode asm

# The Subscription name that we want to create our environment in

$SubscriptionNameId = "<your-subscription-id-guid>"

# Storage account name. This should already be created

$StorageAccountname = "<you-storage-account-name>"

# AffinityGroup name. This should already exist before the script is executed

$AffinityGroup = "linuxdistros"

# Network name. Should already be created

$VnetName = "linuxdistronetwork"

# Subnetname. Should exists before the script is executed.

$SubnetName = "Subnet-1"

# Availability Set

$AvailabilitySetName = "linuxdistro"

# Cloud Service name. This service will be created

$CloudServiceName = "<your-cloud-service-name-here>"

# Instance size

$InstanceSize = "Medium"

# Linux Admin Username

$password = "<your password here>"

# Linux Admin Password

$username = "linuxuser"

# Name Prefix of the VM machine name. Numeric counter number is appended to this to create the final VM name

$LinuxImageNamePrefix = "linuxdistro-0"

# Load the certificate keys for us to login to

$LoadSettings = Import-AzurePublishSettingsFile "NinadKNew.publishsettings"

# Important to specify the CurrentStorageAccountName, especially otherwise you might get Storage not accessible error when creating Linux machines.

Set-AzureSubscription -SubscriptionId $SubscriptionNameId -CurrentStorageAccountName $StorageAccountname -ErrorAction Stop

Select-AzureSubscription -SubscriptionId $SubscriptionNameId -ErrorAction Stop

# Get the image from Azure repository that we want to create. Its the Ubuntu 14_04_LTS variant. We can speed up the creation script by caching the name.

# the name obtianed is 'b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04_2_LTS-amd64-server-20150309-en-us-30GB'

$UbuntuImage = Get-AzureVMImage | where {($_.ImageName -match 'Ubuntu-14_04') -and ($_.ImageName -match '2015') -and ($_.ImageName -match 'LTS') -and ($_.Label -eq 'Ubuntu Server 14.04.2.LTS')} | Select ImageName

Write-Host $UbuntuImage[0].ImageName

# If the service does not exist, we'll create one.

$GetCloudService = Get-AzureService -ServiceName $CloudServiceName -Verbose -ErrorAction Continue

if (!$GetCloudService )

{

    # service does not exist.   

    $CreateCloudService = New-AzureService -ServiceName $CloudServiceName -AffinityGroup $AffinityGroup -Label "Created from Windows power shell" -Description '13 April 2014'

    Write-Host ("Return value from creating the cloud service = {0}" -f $CreateCloudService.OperationStatus.ToString())

}

$Counter = 1

# Loop to create the Linux machines.

do

{

    # Prepend the VM name

    $LinuxImagename = $LinuxImageNamePrefix + $Counter.ToString()

    # Configure VM by specifying VMName, Instance Size, ImageName, AvailabilitySet and specify AzureSubnet

    $VMNew = New-AzureVMConfig -Name $LinuxImagename -InstanceSize $InstanceSize -ImageName $UbuntuImage[0].ImageName -AvailabilitySetName $AvailabilitySetName | Set-AzureSubnet $SubnetName

    # Add the username and password to the VM creation configuration

    $VMNew | Add-AzureProvisioningConfig -Linux -LinuxUser $username -Password $password

    # Create and start the VM. Remember, it won't be fully provisioned when the call returns.   

    $Result = $VMNew | New-AzureVM -ServiceName $CloudServiceName -VNetName $VnetName

    Write-Host ("Created VM Image {0}, return value = {1}" -f $LinuxImagename, $Result.OperationStatus )

    $Counter++

}

while ($Counter -le 6)

Figure 1: Migration Source Script

Note:

Script assumes that following entities already exists within the Azure subscription environment –

§  Valid Azure account: The subscription id of the valid subscription under the account is contained in the variable $SubscriptionNameId 

§   Affinity group: Name of the affinity group is contained in the cmdlet variable $AffinityGroup

§   Network: Azure network should be pre-created with a subnet. The VNet name is contained in the variable $VnetName  and subnet name is contained in the variable $SubnetName

§   Storage account: Valid storage account to host and create Azure VMs should already exist. The name of the storage account is contained in the variable - $StorageAccountname

If the cloud service does not exists, script will create one ($CloudServiceName) and the virtual machines will be created inside the affinity group ($AvailabilitySetName).

·         To connect to my Azure subscription, I use Import-AzurePublishSettingsFile cmdlet. You will need to execute Get-AzurePublishSettingsFile cmdlet first to download the file.

·         The script uses Get-AzureVMImage cmdlet and then filters it to get the name of the VM image that we want to create.  Instead of retrieving the image name, one could instead use the complete name directly instead – ‘'b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04_2_LTS-amd64-server-20150309-en-us-30GB’

·         Script uses New-AzureVMConfig  and AzureProvisioningConfig cmdlets to configure the Linux VMs and finally create a Linux VM using cmdlet New-AzureVM

·         The script to clean the environment is available here

Create Linux Host and establish connection

Now we need to create a Linux host that will be used as a host for running our scripts. We use a simple script to create the VM.

Once the VM has been created and provisioned, we access the provisioned VM by finding out the public port the VM is running and using DNS name to connect. We can find out the public port by using cmdlet Get-AzureEndpoint as shown in the following example.

$VMEndpoints = Get-AzureVM -ServiceName $CloudServiceName -Name $LinuxImagename | Get-AzureEndpoint

if ($VMEndpoints)

{

    foreach($item in $VMEndpoints)

    {

        if ($VMEndpoints.Name.Equals('SSH'))

        {

            Write-host ("{0}.cloudapp.net:{1}" -f $CloudServiceName, $item.Port)

        }

    }

}

Figure 2: Get-AzureEndPoint Script

Note:

Complete script of the above example is copied here.

Outside the virtual network, public DNS name of the VM is made up of <your-cloud-service-name> + ‘cloudapp.net’

The output of the above script shows the DNS name and the public port to connect to the Linux machine.

Now we need an SSH client to connect to the Linux VM. I’ll use PuTTY for our example. You can download PuTTY client from https://aka.ms/putty.  Enter the DNS address and port number and click Open as shown in the screen snapshot below

Figure 3: PuTTY Login screen

When you connect for the first time, you’ll see some security warnings, similar to the ones that get displayed when using a RDP connection. Click continue.

Enter username and password credentials on the PuTTY connection window. These are the same credentials that you used when you created the VM. It is important to note that you’d be logging in as pseudo root user.

If the login is successful, you would be in the Linux world. Welcome.

login as: linuxuser

linuxuser@<Removed>.cloudapp.net's password:

Welcome to Ubuntu 14.04.2 LTS (GNU/Linux 3.16.0-31-generic x86_64)

* Documentation:  https://help.ubuntu.com/

  System information as of Mon Apr 13 21:31:13 UTC 2015

  System load:  0.0               Processes:           235

  Usage of /:   2.7% of 28.80GB   Users logged in:     0

  Memory usage: 3%                IP address for eth0: 100.78.2.22

  Swap usage:   0%

  Graph this data and manage this system at:

    https://landscape.canonical.com/

  Get cloud support with Ubuntu Advantage Cloud Guest:

    https://www.ubuntu.com/business/services/cloud

0 packages can be updated.

0 updates are security updates.

Last login: Mon Apr 13 21:31:14 2015 from <Removed>

linuxuser@ubuntuhost:~

Figure 4: Linux Login Screen

Install Azure Cross-Platform Cli on Linux Machine

All the commands that you would now execute would be at ‘root’ level privileges. To execute commands at elevated privileges, on the command prompt, prefix each command with ‘sudo’ keyword. ‘sudo’ stands for ‘super user do’. To execute a command, you specify the command after this keyword.

First we need to install the Azure cross-platform Command Line Interface (Cli). The instructions for the same can are available from here.

First execute two commands to download the latest updates and upgrade the machine.

linuxuser@ubuntuhost:~$ sudo apt-get update

linuxuser@ubuntuhost:~$ sudo apt-get upgrade

Figure 5: Linux VM Update and Upgrade

Click [Y] for confirmation to upgrade

Next install the Azure cross-platform Cli environment by executing following three commands

linuxuser@ubuntuhost:~$ sudo apt-get install nodejs-legacy

linuxuser@ubuntuhost:~$ sudo apt-get install npm

linuxuser@ubuntuhost:~$ sudo npm install -g azure-cli

Figure 6: Install azure-cli

First command installs the compatible node.js environment. Second command installs the azure client library for node. This library provides a Node.js package that makes it easy to consume and manage Azure services. Last command installs the azure cross-platform Cli using npm.

Now let’s test the installation. Entering ‘azure’ on the command line should display us the cli banner – as shown.

linuxuser@ubuntuhost:~$ azure

info:             _    _____   _ ___ ___

info:            /_\  |_  / | | | _ \ __|

info:      _ ___/ _ \__/ /| |_| |   / _|___ _ _

info:    (___  /_/ \_\/___|\___/|_|_\___| _____)

info:       (_______ _ _)         _ ______ _)_ _

info:              (______________ _ )   (___ _ _)

info:

info:    Microsoft Azure: Microsoft's Cloud Platform

info:

info:    Tool version 0.8.17

help:

help:    Display help for a given command

help:      help [options] [command]

help:

help:    Log in to an Azure subscription using Active Directory. Currently, the user can login only via Microsoft organizational account

help:      login [options]

help:

help:    Log out from Azure subscription using Active Directory. Currently, the user can log out only via Microsoft organizational account

help:      logout [options] [username]

help:

help:    Open the portal in a browser

help:      portal [options]

help:

help:    Commands:

help:      account        Commands to manage your account information and publish settings

help:      config         Commands to manage your local settings

help:      hdinsight      Commands to manage your HDInsight accounts

help:      mobile         Commands to manage your Mobile Services

help:      network        Commands to manage your Networks

help:      sb             Commands to manage your Service Bus configuration

help:      service        Commands to manage your Cloud Services

help:      site           Commands to manage your Web Sites

help:      sql            Commands to manage your SQL Server accounts

help:      storage        Commands to manage your Storage objects

help:      vm             Commands to manage your Virtual Machines

help:

help:    Options:

help:      -h, --help     output usage information

help:      -v, --version  output the application version

linuxuser@ubuntuhost:~$

Figure 7: Test Azure Cli Installation

Connecting to Azure Account from Linux Machine

There are two ways to connect to the Azure account from the cli environment.

·         Account based authentication: using ‘azure login’ commands

·         Certificate based authentication: using ‘azure account download’ commands

linuxuser@ubuntuhost:~$ azure login

info:    Executing command login

warn:    Please note that currently you can login only via Microsoft organizational account or service principal. For instructions on how to set them up, please read https://aka.ms/Dhf67j.

Username: <Details Removed>@microsoft.com

Password: ************

-info:    Added subscription <Details Removed>

info:    Setting subscription <Details Removed> as default

info:    Added subscription <Details Removed> Subscription

info:    Added subscription <Details Removed>

+

info:    login command OK

linuxuser@ubuntuhost:~$ azure logout -u <Details Removed>@microsoft.com

info:    Executing command logout

info:    You have logged out.

info:    logout command OK

linuxuser@ubuntuhost:~$ azure account download

info:    Executing command account download

info:    Launching browser to https://go.microsoft.com/fwlink/?LinkId=254432

help:    Save the downloaded file, then execute the command

help:      account import <file>

info:    account download command OK

linuxuser@ubuntuhost:~$

Figure 8: Login to Azure from Linux Machine

Note: A good comparison between the two approaches of authentication is provided here

Now one challenge that I encountered with running the azure cmdlets on the Linux environment was that although the second command reported success, as there is no browser installed on the machine, the second command does not work. The Linux machine VM does not have Desktop installed, no browser installed and therefore the certificate file cannot be downloaded.

Cunning Plan – Azure File share.

We would download the certificate file on a Windows client machine, and upload it on a file share and download it onto this Linux VM.

We would use Azure Files service to achieve this aim as it provides the ideal – SMB – environment to achieve this.

In order to achieve that we need to carry out following steps

1.      Create an Azure File share

2.      Upload the file to Azure File share

The script to carry out Steps 1 and 2 is copied here. The script creates the Azure File share by utilising New-AzureStorageShare and New-AzureStorageDirectory  cmdlets.

Note:

Azure File is currently in preview and you’ll need to request access from the azure preview page. Azure Files allows Virtual Machines (VMs) in an Azure Data Centre to mount a shared file system using the SMB protocol.

Copying and storing the Azure certificate file in an online storage poses security risk. It is recommended that the share and the uploaded file be closed as soon as the file has been downloaded on to the Linux box.

3.      Mount the shared folder in the Linux machine

4.      Download the file onto Linux machine

linuxuser@ubuntuhost:~$ mkdir Share

linuxuser@ubuntuhost:~$ pwd

/home/linuxuser

linuxuser@ubuntuhost:~$ sudo mount.cifs //<your-file-storage-name>.file.core.windows.net/linuxshare/share /home/linuxuser/Share -o vers=2.1,username=<your-file-storage-name>,password=<your-password-key-ending-with ==>,dir_mode=0777,file_mode=0777

linuxuser@ubuntuhost:~$ cp ./Share/NinadKNew.publishsettings ./Script

linuxuser@ubuntuhost:~$ ls -l ./Script

total 12

-rwxrwxr-x 1 linuxuser linuxuser   76 Apr 14 11:16 createvm.sh

-rwxrwxr-x 1 linuxuser linuxuser 7503 Apr 14 19:32 NinadKNew.publishsettings

linuxuser@ubuntuhost:~$

Figure 9: Linux mount file share and download file commands

In the Linux shell, we first create a folder ‘Share’, next we mount the Azure File shared folder on the ‘Share’ folder using mount.cifs command. After the mount has been successful, we copy the file from the Share to the local scripts folder.

Note: if cifs is not installed on the server, it can be installed using the following command

linuxuser@ubuntuhost:~$ sudo apt-get install cifs-utils

You can un mount the Share using the following ‘umount’ command

sudo umount /home/linuxuser/Share

Create Equivalent UNIX Script

Now we are ready to create our cli script.  First let’s create a folder where the script file will reside and create the skeleton shell file using the famous text editor – ‘VI’.

linuxuser@ubuntuhost:~$ pwd

/home/linuxuser

linuxuser@ubuntuhost:~$ mkdir Script

linuxuser@ubuntuhost:~$ cd Script

linuxuser@ubuntuhost:~/Script$ pwd

/home/linuxuser/Script

linuxuser@ubuntuhost:~/Script$ echo $SHELL

/bin/bash

linuxuser@ubuntuhost:~/Script$ vi createvm.sh

Figure 10: Start Creating UNIX equivalent script -1

Inside the VI editor, insert following line [VI Command àEsc i]

#!/bin/bash

echo 'This script will create Linux VMs in Azure subscription'

Figure 11: UNIX script inside VI

The first line in the script informs the system that it’s a script file and that the shell to be used for this is /bin/bash. This value is contained in the environment variable $SHELL.

Save the script file [VI command àEsc:wq] and change its mode to indicate it’s an executable. Finally execute the file from the command line.

linuxuser@ubuntuhost:~/Script$ vi createvm.sh

linuxuser@ubuntuhost:~/Script$ chmod +x createvm.sh

linuxuser@ubuntuhost:~/Script$ ls -l

total 4

-rwxrwxr-x 1 linuxuser linuxuser 76 Apr 14 11:16 createvm.sh

linuxuser@ubuntuhost:~/Script$ ./createvm.sh

This script will create Linux VMs in Azure subscription

linuxuser@ubuntuhost:~/Script$

Figure 12: Create and Test Linux Script

The Linux script is divided into two parts.

The first parts uses the standard UNIX stream find tools to get the azure image name. The second script creates the six Linux VMs just as the original Windows script did.

In the first script, Azure account based authentication is used as that is the only valid option. In the second script, Azure certificate based authentication is used.

#!/bin/bash

 echo 'This script will output the name of the Linux image file'

#Login using account subscription and not with certificate login

 azure login -u '<REMOVED>@microsoft.com' -p '<REMOVED>'

IMAGENAME =$(azure vm image list | grep 'Ubuntu-14_04_2' | awk '{print $2}')

 echo $IMAGENAME

 azure logout -u '<REMOVED>@microsoft.com'

Figure 13: UNIX Script 1

Script first logins to the Azure account using cmdlet azure login . After a successful login, we get the list of all the Azure images and pipe the result to UNIX grep and awk command. Script display’s the captured image name on the console window and then finally logout (azure logout) from the Azure account.

The output from the first script -

‘b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04_2_LTS-amd64-server-20150309-en-us-30GB’ is the image name of the Ubuntu virtual machine and it’s used in the second script to create all the virtual machines.

#!/bin/bash

echo 'This script will create Linux VMS in Azure subscription'

IMAGENAME='b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04_2_LTS-amd64-server-20150309-en-us-30GB'

SF="/home/linuxuser/Share/NinadKNew.publishsettings"

SUBSID='<your-subscription-id>'

STORAGEACCNAME="<your-storage-account-name>"

AVAILABILITYSET="linuxdistroset"

AFFINITYGROUP="linuxdistros"

CLOUDSERVICENAME="<your-cloud-service-name>"

VNETNAME="linuxdistronetwork"

SUBNETNAME="Subnet-1"

INSTANCESIZE="Medium"

PASSWORD="<your-password>"

USERNAME="linuxuser"

# Load the certificate file to login to azure account

azure account import $SF

# select the appropriate subscription

azure account set $SUBSID

# Let’s select the appropriate storage

azure storage account set $STORAGEACCNAME

FILENAMEPREFIX='linuxdistro-0'

COUNTER=1

while [ $COUNTER -lt 7 ]

do

 TEMPFILENAME=$FILENAMEPREFIX$COUNTER

 # echo $TEMPFILENAME

 # echo current value of counter is $COUNTER

if [ $COUNTER -eq 1 ]

then

        echo Creating first VM

        azure vm create -n $TEMPFILENAME -z $INSTANCESIZE -A $AVAILABILITYSET -w $VNETNAME -a $AFFINITYGROUP -b $SUBNETNAME $CLOUDSERVICENAME $IMAGENAME $USERNAME $PASSWORD

else

        echo Creating next VM

        azure vm create -n $TEMPFILENAME -z $INSTANCESIZE -A $AVAILABILITYSET -w $VNETNAME -b $SUBNETNAME -c $CLOUDSERVICENAME $IMAGENAME $USERNAME $PASSWORD

fi

 COUNTER=$(($COUNTER+1))

done

Figure 14: UNIX Script 2

We use the downloaded certificate file and consume the cmdlet azure account import to login to the Azure subscription. Next we set the appropriate subscription (azure account set) and azure storage (azure storage account)

The above Bash script creates Ubuntu instances using cmdlet azure vm create. Script uses normal Bash keywords and constructs to iterate through and create the require number of VM instance. Whilst creating the first instance we pass extra parameter ($AFFINITYGROUP) because the cloud service is not yet created. 

Conclusion

Microsoft Azure cross-platform cli makes it a real breeze to use the Linux shell environment to execute Azure cmdlets. It does not have all the intellisense and debugging support that Windows PowerShell ISE provides but in the hands of a familiar UNIX person, that should not be a deterrence.

Proof: - and yes I am aware that I am moving away from my goal of staying away from any GUI – but just to confirm the results, we will use the Azure management portal and see that our Linux instances are up and running.

Figure 15: Virtual instance running under service account

Figure 16: Virtual instances running under network configuration

Note: Please make sure you stop and clean your installation otherwise any they will incur cost to your subscription. The script to clean the environment is provided here.

Scripts

Clean-Up Script

# CleanLinuxVirtualMachines

# The Subscription name that we want to create our environment in

$SubscriptionNameId = "<Account subscription id>"

# Cloud Service name. This service will be created

$CloudServiceName = "<cloud-service-name>"

# Name Prefix of the VM machine name. Numeric counter number is appended to this to create the final

$LinuxImageNamePrefix = "linuxdistro-0"

# Load the keys for us to login to

Import-AzurePublishSettingsFile "NinadKNew.publishsettings"

Set-AzureSubscription -SubscriptionId $SubscriptionNameId -ErrorAction Stop

Select-AzureSubscription -SubscriptionId $SubscriptionNameId -ErrorAction Stop

$Counter = 1

$AllVmsStopped = 1

do

{

    # Prepend the VM name

    $LinuxImagename = $LinuxImageNamePrefix + $Counter.ToString()

    Write-Host ("VM Image name = {0}" -f $LinuxImagename)

    $VMPresent = Get-AzureVM -ServiceName $CloudServiceName -Name $LinuxImageName

    if ($VMPresent)

    {

        if ($VMPresent.Status -eq "StoppedVM" -or $VMPresent.Status -eq "StoppedDeallocated" )

        {

            Write-Host ("{0} VM is either stopped or StoppedDeallocated" -f $LinuxImagename )

        }

        else

        {

            Write-Host ("[Stopping] VM {0}" -f $VMPresent.Name)

            $StopVM = Stop-AzureVM -Name $VMPresent.Name -ServiceName $VMPresent.ServiceName -Force

            if ($StopVM.OperationStatus.Equals('Succeeded'))

            {

                Write-Host ("[Removing] virtual machine {0}" -f $VMPresent.Name)

                Remove-AzureVM -Name $VMPresent.Name -ServiceName $VMPresent.ServiceName

            }

            else

            {

                Write-Host ("Not able to stop virtual machine {0}, cloud service will not be removed" -f $VMPresent.Name)

                $AllVmsStopped = 0  

            }

        }

    }

    else

    {

        Write-Warning ("No Vm found " -f $LinuxImageName)

    }

    $Counter++

}

while ($Counter -le 6)

#Remove the cloud service now

# only if we were albe to stop all running ones

if ($AllVmsStopped -eq 1)

{

    # is it present

    $CloudServicesPresent = Get-AzureService -ServiceName $CloudServiceName -ErrorAction Continue

    if ($CloudServicesPresent)

    {

        $ResultRemoveCloudService = Remove-AzureService -ServiceName $CloudServiceName -Force

    }

    else

    {

        Write-Warning ("No {0} cloud service found" -f $CloudServiceName)

    }

}

else

{

    Write-Host ("Not able to stop virtual machine, cloud service will not be removed")

}

Figure 17: Clean-up Script

Create Linux Host VM

#CreateHostLinuxVirtualMachine

# The Subscription name that we want to create our environment in

$SubscriptionNameId = "<your subscriber id here>"

# Storage account name. This should already be created

$StorageAccountname = "portalvhds0mll57sg9hvjd"

# Cloud Service name. This service should exist.

$CloudServiceName = "<insert-name-of-your-cloud-service-here>"

# Instance size

$InstanceSize = "Medium"

# Linux Admin Username

$password = "<insert your password here>"

# Linux Admin Password

$username = "<insert your username here>"

# Linux image name

$LinuxImagename = "ubuntuhost"

# Load the keys for us to login to

$LoadSettings = Import-AzurePublishSettingsFile "NinadKNew.publishsettings"

# Important to specify the CurrentStorageAccountName, especially otherwise you might get Storage not accessible error when creating linux machines.

Set-AzureSubscription -SubscriptionId $SubscriptionNameId -CurrentStorageAccountName $StorageAccountname -ErrorAction Stop

Select-AzureSubscription -SubscriptionId $SubscriptionNameId -ErrorAction Stop

# Get the image from Azure repository that we want to create. Its the Ubuntu 14_04_LTS variant. We can speed up the creation script by caching the name.

# the name obtained is

$UbuntuImage = 'b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04_2_LTS-amd64-server-20150309-en-us-30GB'

# Configure VM by specifying VMName, Instance Size, ImageName, AvailabilitySet and specify AzureSubnet

$VMNew = New-AzureVMConfig -Name $LinuxImagename -InstanceSize $InstanceSize -ImageName $UbuntuImage

# Add the username and password to the VM creation configuration

$VMNew | Add-AzureProvisioningConfig -Linux -LinuxUser $username -Password $password

# Create and start the VM. Remember, it won't be fully provisioned when the call returns.   

$Result = $VMNew | New-AzureVM -ServiceName $CloudServiceName

Write-Host ("Created VM Image {0}, return value = {1}" -f $LinuxImagename, $Result.OperationStatus )

Figure 18: Create Linux Host Script

Get Public Endpoints of a VM

# GetPublicEndPointOfaVM

# The Subscription name that we want to create our environment in

$SubscriptionNameId = "< insert your subscription id here>"

# Cloud Service name. This service should exist.

$CloudServiceName = "< insert cloud-service-name here >"

# Linux image name

$LinuxImagename = "< insert image name here >"

$LoadSettings = Import-AzurePublishSettingsFile "NinadKNew.publishsettings"

Set-AzureSubscription -SubscriptionId $SubscriptionNameId -ErrorAction Stop

Select-AzureSubscription -SubscriptionId $SubscriptionNameId -ErrorAction Stop

$VMEndpoints = Get-AzureVM -ServiceName $CloudServiceName -Name $LinuxImagename | Get-AzureEndpoint

if ($VMEndpoints)

{

    foreach($item in $VMEndpoints)

    {

        if ($VMEndpoints.Name.Equals('SSH'))

        {

            Write-host ("{0}.cloudapp.net:{1}" -f $CloudServiceName, $item.Port)

        }

    }

}

Figure 19: Get Public Endpoints of a VM Script

Creating Azure File Share and uploading file

# creating Azure file share and uploading the certificate file to it.

# CreateAzureFileStorageScript

$SubscriptionName = "<Insert-your-Subscription-name-here>" # just go for one subscription

$LoadSettings = Import-AzurePublishSettingsFile "NinadKNew.publishsettings"

Set-AzureSubscription -SubscriptionName $SubscriptionName -ErrorAction Stop

# Wow this Select-AzureSubscription is more important than anything else.

Select-AzureSubscription -SubscriptionName $SubscriptionName -ErrorAction Stop

$StorageAccountname = "<insert-your-file-storage-name>"

$AffinityGroup = "linuxdistros"

if (!$GetStorageAccount)

{

    # Storage account does not exists, let’s create one

    $GetStorageAccount = New-AzureStorageAccount -StorageAccountName $StorageAccountname -Label "PS Created" -AffinityGroup $AffinityGroup

    Write-host ("Storage account {0} created status = {1}" -f $StorageAccountname, $GetStorageAccount.OperationStatus)

}

#lets get the storage context now

$AzureSampleShare = "linuxshare"

$ShareDirectoryName = "share"

$StorageContext =New-AzureStorageContext -StorageAccountName $StorageAccountname -StorageAccountKey (Get-AzureStorageKey -StorageAccountName $StorageAccountname).Primary -Protocol Https

if ($StorageContext)

{

     $ExistingStorageShare = Get-AzureStorageShare -Name $AzureSampleShare -Context $StorageContext -ErrorAction Continue

     if (!$ExistingStorageShare)

     {

        $ExistingStorageShare = New-AzureStorageShare -Name $AzureSampleShare -Context $StorageContext

        Start-Sleep -s 30

        Write-Host ("New Azure Storage Share created, name = {0}" -f $AzureSampleShare)

        # create a share directory name

        $NewAzureStorageDirectory = New-AzureStorageDirectory -Share $ExistingStorageShare -Path $ShareDirectoryName

     }

     # upload our file

     Set-AzureStorageFileContent -Share $ExistingStorageShare -Source C:\Azure\Linux\NinadKNew.publishsettings -Path $ShareDirectoryName

     # check that the file has been uploaded.

     Get-AzureStorageFile -Share $ExistingStorageShare -Path $ShareDirectoryName

}

Figure 20: Creating Azure File Share and uploading file Script

References

1.       Install and configure Azure Cross-Platform Command-Line Interface

2.       Running a remote desktop on Windows Azure Linux VM

3.       How to use SSH with Linux on Azure

4.       Shared folder on Linux via Azure Files

5.       Best Practices for Managing Azure Subscriptions in Windows Azure PowerShell Cmdlets