Once we have gone through how to authenticate the application with Azure AD through the multi-tenant approach and single tenant approach, in this blog we will look at deploying a VM on a VNet, linked to a storage account. Typically when you are looking at automating workloads, you would like to simplify user input as much as possible.

This application loads a simple form @ https://127.0.0.1:8000/deploy that takes inputs from the user. This is a simplified sample of course, but in the back-end it could be either creating an ARM template and deploying, which is a preferred route for large scale deployments or deploy single components at a time for small scale/adhoc requirements.

We will look at the latter today, and in the coming few days we could look at the template based deployment.

You can download and install the latest version of Azure Python SDK with the pip installer:

pip install azure

This is the link to view the SDK in detail.

https://github.com/Azure/azure-sdk-for-python

Coming back to our application, following is the loading screen, assuming that the user has logged in with their application user name & password.

 

Figure 1: Loading screen

If we take a look at the deployment option module, views.py. The def deployment method loads the admin settings (which is the client_id & secret of the application registered with the AD it in subscription it is hosted in).

When the screen is loaded, we pre-load it with list of regions available today, storage account types and the various VM sizes available. The code has a hard coded else part, which could be replaced by reading from local DB settings.

To call any Python SDK function, we need to send two parameters for all the functsions:

1.       Access token

2.       Subscription IDs

Since this simply retrieving a standard list we will use the application’s own client_id and secret using the following method.

We first extract it from the default Django storage:

if AuthSettings_Admin.objects.filter(user_id="Admin").exists():

            admin_authsettings = AuthSettings_Admin.objects.filter(user_id="Admin")

            for admin_auth in admin_authsettings:

                client_id = admin_auth.client_id

                tenant_id=admin_auth.tenant_id

                subscription_id=admin_auth.subscription_id

                client_secret =admin_auth.client_secret

Using these we get the access token generated calling the method:

token = get_access_token(tenant_id=tenant_id,client_id=client_id,client_secret=client_secret)

The method get_access_token described below is based on raw REST call, we now have an inbuilt ADL library for Python just released, and will update the article to add that as well in some time:

Make sure you urlencode the client secret in the body.

def get_access_token(tenant_id,client_id,client_secret):   

   

    url = "https://login.microsoftonline.com/" + tenant_id + "/oauth2/token"       

       body_data = "&grant_type=client_credentials&resource=https://management.core.windows.net/&

client_id="+ client_id + "&client_secret="+  urllib.quote_plus(client_secret)

    #body_data = urllib.urlencode(body_data_json)

    headers = {"Content-Type":"application/x-www-form-urlencoded"}

    req = Request(method="POST",url=url,data=body_data)

    req_prepped = req.prepare()

    s = Session()

    res = Response()

    res = s.send(req_prepped)

    access_token_det = {}

    if (res.status_code == 200):

        responseJSON = json.loads(res.content)

        access_token_det["details"]= responseJSON["access_token"]

        access_token_det["status"]="1"

        access_token_det["exp_time"]=responseJSON["expires_in"]

        access_token_det["exp_date"]=responseJSON["expires_on"]

        access_token_det["accessDetails"]=responseJSON

    else:

        access_token_det["details"]= str(res.status_code) + str(res.json())

        access_token_det["status"]="0"

    return access_token_det

Once you get the access token, we call the following functions of the SDK to get the list of regions, vm sizes and storage account types:

We have added a layer of abstraction between the SDK functions and the Django application. This is to avoid changing the application if the function’s return types change.

Please refer to the module arm_framework on this link.

loc_list = arm_fw.get_list_regions(token["details"],subscription_id)

pricing_tier = arm_fw.get_list_vm_sizes(token["details"],subscription_id)

stor_acc_types = arm_fw.get_list_storage_types(token["details"],subscription_id)                   

We send all of these as template parameters to the html page. Please look at the template file for details on how it is loaded, it’s typical Django & and javascript code.

Now that we have the form loaded, the user can enter the deployment name and everything gets populated.

Next step is to Get VM Image list. The Azure Resource Manager provides VM list based on location, and in multi-Step:

1.       Get Publisher

2.       Get Offer for the publisher

3.       Get SKUs for the offer, publisher

4.       Get Versions for the offer, publisher and SKUs

The form first loads the list of publisher, it is loading as a raw list which is not a very good way of loading, ideally one must provide a search box and return search results based on that.

We use the following function from SDK to load the list of publishers (getvmimagepublishers):

pub_list = arm_fw.get_list_vm_publishers(access_token=acc_token, subscription_id=subscription_id,region_name=region_name)

Based on the publishers, we call the rest of the functions:

Offers:

offer_list = arm_fw.get_list_vm_offers(access_token=acc_token, subscription_id=subscription_id,region_name=region_name,publisher_name=publisher_name)

SKUs:

skus_list = arm_fw.get_list_vm_skus(access_token=acc_token, subscription_id=subscription_id,region_name=region_name,publisher_name=publisher_name,

offer_name=offer)

Versions:

versions_list = arm_fw.get_list_vm_versions(access_token=acc_token, subscription_id=subscription_id,region_name=region_name,publisher_name=publisher_name,

offer_name=offer,sku_name=skus)

Note: You will notice, that for some publishers the version list returns empty. That means that VM image does not exist. While this application doesn’t handle it, you must ideally cater to it by providing appropriate messages.

Fetch Subscriptions:

The last bit is to fetch the list of subscriptions that the user initially had entered into settings manually or through the OpenID connect methodology explained earlier. This is done through list stored in the local DB.

Submit Deployment

This is the last step of the deployment method. First thing the client determines is whether it is an app-only authentication or user-based authentication. This it does by calling the web method on Django server as shown below in the html template file:

If it’s app-only it is very simple, the client sends all the deployment details in a JSON object directly to the server:

           url: '../authdetails/',

            type: 'POST',

            data: deploymentDetails,

            dataType: 'json',

            headers: { "X-CSRFToken": getCookie('csrftoken') },

            error: function(e){console.log(e.message)},

            success: function (data) {

                deploymentDetails["auth_type"] = data["auth_type"]

                //console.log(data["auth_type"])

                if (data["auth_type"] == "A") {

                    $.ajax({

                        url: '../deploy/',

                        type: 'post',

                        headers: { "X-CSRFToken": getCookie('csrftoken') },

                        data: deploymentDetails,

                        dataType: 'json',

                        success: function (data) {

                            $("#deploy_results").html(JSON.stringify(data))

                            $("#dep_spinner").hide()

                        }

                    })

However, if it’s the user-based authentication,  the client opens a pop-up with the auth_url that the server returns:

var url = data["auth_code_url"]

                    popup = window.open(url, "Login", width=300,height=200);

This is to get the authorization code that will be used to generate the token.

The views.py constructs the auth_code_url using the client_if & tenant_id saved by the user as shown below.

authorize_url = "https://login.microsoftonline.com/" + tenant_id +"/oauth2/authorize?client_id="+client_id+"&response_type=code"

            auth_response["auth_type"] = auth_type

            auth_response["auth_code_url"] = authorize_url           

            auth_response["auth_token"]= None

The popup window loads the user Azure AD login page. Once the user successfully logs in, it returns the control back to the Django Server as provided in the redirect URI and passes the access code as a query string.

This function immediately returns the control back to the template calling it’s get_Access_code function.

def get_access_code(request):

    query_string = request.META["QUERY_STRING"]

    code_det = {"code":query_string}

    htmlstring = "<html><head></title><script type='text/javascript'>window.opener.get_access_code("+json.dumps(code_det)+");window.close();</script></head><body></body></html>"

    return HttpResponse(htmlstring)

The javascript function get_access_code is called as described below:

code_str = code["code"]

        var code_arr = ""

        code_arr = code_str.toString().split('&')

        code_val = code_arr[0].replace("code=", "")

        deploymentDetails["auth_code"] = code_val

        $.ajax({

            url: '../deploy/',

            type: 'post',

            headers: { "X-CSRFToken": getCookie('csrftoken') },

            data:deploymentDetails,

            dataType:'json',

            success: function (data) {

                $("#dep_spinner").hide()

                $("#deploy_results").html(JSON.stringify(data))

            }

The function sends the details to the same Django application function create_deployment() as described below:

The method gets the access token either directly through the app-only route or using the authorization code:

           if auth_type == "U":

                        auth_code = request.POST["auth_code"]

                        token_url = "https://login.microsoftonline.com/" + tenant_id+"/oauth2/token"

        

                        body="grant_type=authorization_code&code="+auth_code+"&redirect_uri="+redirect_uri+"

&client_id="+client_id+"&client_secret="+ urllib.quote_plus(client_secret)+"&resource=https://management.core.windows.net/"

headers ={"Content-Type":"application/x-www-form-urlencoded"}

                        req = Request(method="POST",url=token_url,data=body)

                        req_prepped = req.prepare()

                        s = Session()                       

                        res = Response()

                        res = s.send(req_prepped)

                       

                        responseJSON = json.loads(res.content)

                        token = responseJSON["access_token"]

                        token_result = {"POSTURL":token_url,"body":body,"headers":headers,"Response":responseJSON}

                    else:

                        res = get_access_token(tenant_id=tenant_id,client_id=client_id,client_secret=client_secret)

                        token_result = {"Response":res}

                        if(res["status"]=="1"):

                            token = res["details"]

                        else:

                            token=None

                    deploymentDetails = request.POST

                    subscription_id=deploymentDetails["subscriptionID"]   

               

Create Resource group:

res_grp_created_status = arm_fw.create_resource_group(access_token=token,subscription_id=subscription_id,resource_group_name=

deploymentDetails["resourceGroupName"],region=deploymentDetails["location"])

This sends the details to our abstract layer which in turn calls the Create Vnet function in the Python SDK:

def create_resource_group(access_token, subscription_id,resource_group_name, region):

    cred = SubscriptionCloudCredentials(subscription_id, access_token)

    resource_client = ResourceManagementClient(cred, user_agent='SDKSample/1.0')

    resource_group_params = ResourceGroup(

                location=region,

                tags={

                    'RGID': subscription_id + resource_group_name,

                },

            )

    result_create = resource_client.resource_groups.client.resource_groups.create_or_update(resource_group_name, resource_group_params)

    success=False

    if result_create.status_code == 200:

        success = True

    elif result_create.status_code == 201:

        success = True

    else:

        success = False

    result_json = {"success":success, "resource_created":result_create.resource_group.name}

    return result_json

     

Create Virtual Network

if the resource group gets created successfully, we then create the Virtual Network

if res_grp_created:

                        vnet_created_status = arm_fw.create_virtual_network(access_token=token,subscription_id=subscription_id,

resource_group_name=res_grp_name_created,virtualnetwork_name=deploymentDetails["vnetName"],region=deploymentDetails["location"], subnets=None,addresses=None,dns_servers=None)

As described in the function below, it is creating a single address space, subnet and dns server

def create_virtual_network(access_token, subscription_id,resource_group_name, virtualnetwork_name, region, subnets=None,

                           addresses=None, dns_servers=None):

    cred = SubscriptionCloudCredentials(subscription_id, access_token)

    resource_client = ResourceManagementClient(cred, user_agent='SDKSample/1.0')

    resource_client.providers.register('Microsoft.Network')

    network_client = NetworkResourceProviderClient(cred)

    network_parameters = None

    if (subnets != None and addresses != None and dns_servers != None):

        addr_space = AddressSpace(address_prefixes=addresses)

        dhcp_opts = DhcpOptions(dns_servers=dns_servers)

        subnet_det = subnets

        network_parameters = VirtualNetwork(location=region, virtualnetwork_name=virtualnetwork_name,

                    address_space= addr_space,dhcp_options=dhcp_opts,subnets=subnet_det)               

    else:

        if DefaultNetworkSettings.objects.filter(setting_type_id="default").exists():

            def_settings = DefaultNetworkSettings.objects.filter(setting_type_id="default")

            add_sp=None

            sb_net_prefix=None

            sb_net_name=None

            for ds in def_settings:

                add_sp = ds.default_address_space

                sb_net_prefix = ds.default_address_range

                sb_net_name = ds.default_subnet_name

            addr_space = AddressSpace(address_prefixes=[add_sp])

            subnet_det = [Subnet(name=sb_net_name,address_prefix=sb_net_prefix)]

        network_parameters = VirtualNetwork(location=region, virtualnetwork_name=virtualnetwork_name,address_space=addr_space,dhcp_options=None,

subnets=subnet_det)

    result_create=network_client.virtual_networks.create_or_update(resource_group_name,virtualnetwork_name,network_parameters)

    success=False

    if result_create.status_code == 200:

        success = True

    elif result_create.status_code == 201:

        success = True

    else:

        success = False

    result_json = {"resource_created": virtualnetwork_name, "success":success,"subnet_name":sb_net_name}

    return result_json

 

Create Storage Account

stor_acc_created_status = arm_fw.create_storage_account(access_token=token,subscription_id=subscription_id,

resource_group_name=res_grp_name_created,storage_account_name=deploymentDetails["storageAccountName"],region=deploymentDetails["location"],storage_type=deploymentDetails["storageAccountType"])

Following is the underlying function for creating storage account:

def create_storage_account(access_token, subscription_id,resource_group_name, storage_account_name, region, storage_type):

    cred = SubscriptionCloudCredentials(subscription_id, access_token)

    resource_client = ResourceManagementClient(cred, user_agent='SDKSample/1.0')

    resource_client.providers.register('Microsoft.Storage')

    storage_client = StorageManagementClient(cred, user_agent='SDKSample/1.0')

    storage_params = StorageAccountCreateParameters(

        location=region,

        account_type=storage_type,

        )

    result_create = storage_client.storage_accounts.create(resource_group_name,storage_account_name,storage_params)

    success=False

    if result_create.status_code == 200:

        success = True

    elif result_create.status_code == 201:

        success = True

    else:

        success = False

    result_json = {"success":success, "resource_created":storage_account_name}

    return result_json

Create Virtual Machine

Finally we call the create Virtual Machine sending the created details.

vm_created_status = arm_fw.create_virtual_machine(access_token=token,subscription_id=subscription_id,resource_group_name=res_grp_name_

created,vm_name=deploymentDetails["vmName"],vnet_name=vnet_name_created,subnet_name=subnet_name_created,

vm_size=deploymentDetails["vmPricingTier"],storage_name=stor_acc_created_name,region=deploymentDetails["location"],vm_username=deploymentDetails["vmUserName"],vm_password=deploymentDetails["vmPassword"],publisher=deploymentDetails["vmPublisher"],offer=deploymentDetails["vmOffer"],sku=deploymentDetails["vmSKU"],version=deploymentDetails["vmVersion"])

With underlying function as:

def create_virtual_machine(access_token, subscription_id,resource_group_name, vm_name, vnet_name,

                           subnet_name, vm_size,  storage_name, region, vm_username, vm_password,

                           publisher, offer, sku, version):

    interface_name=vm_name + 'nic'

    cred = SubscriptionCloudCredentials(subscription_id, access_token)

    resource_client = ResourceManagementClient(cred)

    resource_client.providers.register('Microsoft.Compute')

    compute_client = ComputeManagementClient(cred)

    hardware_profile = HardwareProfile(virtual_machine_size=vm_size)

    os_profile = OSProfile(computer_name = vm_name, admin_username = vm_username, admin_password = vm_password)

    image_reference = ImageReference(publisher=publisher, offer=offer, sku=sku, version=version)

    storage_profile = StorageProfile(os_disk=OSDisk(caching=CachingTypes.none, create_option=DiskCreateOptionTypes.from_image,

                                                   name=vm_name, virtual_hard_disk=VirtualHardDisk(

uri='https://{0}.blob.core.windows.net/vhds/{1}.vhd'.format( torage_name,vm_name)

)), image_reference=image_reference   )

    nic_reference = create_network_interface(access_token,subscription_id,resource_group_name,interface_name, vnet_name, subnet_name, region)

    nw_profile = NetworkProfile(

            network_interfaces=[

                                NetworkInterfaceReference(reference_uri=nic_reference)

                                ]

                                )

    vm_parameters = VirtualMachine(location=region, name=vm_name, os_profile=os_profile, hardware_profile= hardware_profile,

                                   network_profile=nw_profile, storage_profile=storage_profile, image_reference=image_reference)

    result_create = compute_client.virtual_machines.create_or_update(resource_group_name, vm_parameters)

    success=False

    if result_create.status_code == 200:

        success = True

    elif result_create.status_code == 201:

        success = True

    else:

        success = False

    result_json = {"resource_created": vm_name, "success":success}

    return result_json

The application right now returns the names of the created resources as is, one could modify this to show a user friendly message.

Refresh Token

Please note, ideally you should be adding a check to ensure the access token does not expire, and to call the refresh token before it expires.

This is the Azure Single Tenant application approach where one deploys the application in their own subscription to manage and automate workloads.

While the steps to register the application remains same, we need to add the Application as a contributor to the subscription. We have seen that using the REST API in the previous blog post, here we will use Azure CLI command line and certificate based authentication.

The application can retrieve the Auth code using the Application secret as well, however that may not be the preferred method as you will be storing the secret in the application code/config most likely.

More details of the client secret based approach is present in the following link in the previous blog post.

Let’s assume you have the application registered with your AD, next step will be to give access to the subscription.

Step 1: Add the application as Service principal to the subscription

Please note you could add this to multiple subscriptions that you have the rights to manage.

On your Linux VM, run the following commands:

Create a service principal for your

azure ad sp create <<client ID of the application>>

This command will return the ObjectID of the application, please make sure you copy this as you will use it for the next step

Assign Contributor role to the subscriptions

azure role assignment create –objectId <object ID>-o Contributor -c /subscriptions/{subscriptionId}/

Step 2: Update the manifest.json file with the certificate for certificate based authentication:

On your Linux VM:

Create the certificate, public and private key:

openssl req -x509 -days 3650 -newkey rsa:2048 -keyout azkey.pem -out azcert.pem

This creates two files, the azkey.pem file with the private key and azcert.pem file with the certificate and public key.

Please run the following command to export the certificate thumbprint and Base64 it.

echo $(openssl x509 -in cert.pem -fingerprint -noout) | sed 's/SHA1 Fingerprint=//g' | sed 's/://g' | xxd -r -ps | base64

Run the following command to export the public key of the certificate.

tail -n+2 cert.pem | head -n-1

On the portal, under your application configure tab, click on the manage manifest option and select download manifest

It will download a JSON file which has basic details of your application registered with AD. We will modify this JSON file to add the thumbprint and the public key of the certificate we will be using to authenticate the application with Azure AD.

In the manifest, update the keyCredentials file with the certificate details (please note , copy paste the values as is from your VM and do not add/update/delete spaces)

keyCredentials": [

       {

              "customKeyIdentifier": "Base 64 thumbprint",

              "keyId": "GUID generated ", (you can use uuidgen –r to generate a GUID)

              "type": "AsymmetricX509Cert",

              "usage": "Verify",

              "value": (Paste the public key value here”

Your JSON file should look like the one below:

 

Save the JSON file and upload the file using the Upload manifest option. Please make sure it uploads successfully. If it gives an error without specifying the field name, it’s probably because there is an additional character that has been added. Please make sure you have copied the thumbprint encoded & the public key value as is. If you have, try using another editor.

Make sure you save the private key in a location where you can reference it in your code.

Following code snippet shows how to obtain the Access token from AD based on certificate:

The client_det dictionary hold the tenant_id, client_id of the application.

                      endpoint = 'https://login.windows.net/' + client_det['tenant_id'] + '/oauth2/token'

                      jwt_token_body = {}

              jwt_token_body['aud']='https://login.windows.net/'+client_det['tenant_id']+'/oauth2/token'

                      jwt_token_body['iss']=client_det['client_id']

                      jwt_token_body['sub']=client_det['client_id']

                      dt_now = int(time.time())

                      jwt_token_body['nbf']= str(dt_now)

                      jwt_token_body['exp']=str(dt_now +600)

                      jwt_token_body['jti']= str(uuid.uuid1())

                      privateKey = open(CERTIFICATE_KEY_PATH,'r').read()

                      headers = {'x5t':'oxvnmTdtpW4H1o64tVooW7NzNXc='}

                      #cert_str = "-----BEGIN CERTIFICATE-----"

                      #cert_obj = load_pem_x509_certificate(cert_str, default_backend())

                      #public_key = cert_obj.public_key()

                      #private_key = cert_obj.private_key()

                      client_assertion_val = jwt.encode(payload=jwt_token_body,key=privateKey,algorithm='RS256',headers=headers)

                      payload = {

                             "grant_type":"client_credentials",

                             "client_id" : client_det['client_id'],

                             "client_assertion_type":"urn:ietf:params:oauth:client-assertion-type:jwt-bearer",

                             "client_assertion":client_assertion_val,

                             "resource":_resource

                             }

                     

              else:

                      endpoint = 'https://login.microsoftonline.com/' + _tenant_id + '/oauth2/token'

                      payload = {

                             'grant_type': 'client_credentials',

                             'client_id': _client_id,

                             'client_secret': _client_secret,

                             'resource': _resource

                      }

             

              response = requests.post(endpoint, data=payload)

              #print('response content ' + response.content)

              json_val = json.loads(response.content)

              access_token_det = {}

              if (response.status_code == 200):

                      access_token_det["access_token"] = json_val["access_token"]

                      access_token_det["status"] = "1"

                      access_token_det["exp_time"] = json_val["expires_in"]

                      access_token_det["exp_date"] = json_val["expires_on"]

                      access_token_det["accessDetails"] = json_val

              else:

                      access_token_det["details"] = str(response.status_code)

                      access_token_det["status"] = "0"

             

              return access_token_det

The application creates a JWT token, which follows the following structure

jwt_token_body['aud']='https://login.windows.net/'+client_det['tenant_id']+'/oauth2/token'

                      jwt_token_body['iss']=client_det['client_id']

                      jwt_token_body['sub']=client_det['client_id']

                      dt_now = int(time.time())

                      jwt_token_body['nbf']= str(dt_now)

                      jwt_token_body['exp']=str(dt_now +600)

                      jwt_token_body['jti']= str(uuid.uuid1())

Make sure the audience claim (aud) is same as the endpoint, and you add the header with the certificate thumbprint.

Encode the token using the jwt.encode() method and pass the private key as the parameter for key.

jwt.encode(payload=jwt_token_body,key=privateKey,algorithm='RS256',headers=headers)

Then create the payload as the following:

                      payload = {

                             "grant_type":"client_credentials",

                             "client_id" : client_det['client_id'],

                             "client_assertion_type":"urn:ietf:params:oauth:client-assertion-type:jwt-bearer",

                             "client_assertion":client_assertion_val,

                             "resource":_resource

                             }

You will notice it is different from the client secret based payload which is in the else part:

                      payload = {

                             'grant_type': 'client_credentials',

                             'client_id': _client_id,

                             'client_secret': _client_secret,

                             'resource': _resource

                      }

Once you get the access token, you can proceed to deploy or manage ARM infrastructure on Azure as explained in the next blog part.

We will look at the user-based authentication along with the deployment description in the next post.

This would be the most likely scenario you want to automate Azure infrastructure. Where you are either a multi-cloud infrastructure management provider or Azure only, you will need to incorporate the AD authentication mechanism for a user.

Azure AD implements OAuth2.0 authentication, following are the steps you can follow to set it up.

Please note at the time of writing of this blog, Azure ADL for Python has just been released. We will modify this blog later to add those details.

Meanwhile, you could refer to the complete code at:

https://github.com/shwetams/arm-samples-py 

Pre-requisite

Step 1: Add application AIM to your subscription

You can add the application through the portal or through command line. We will show the portal route, but you could refer to the link below to do that through command line.

https://azure.microsoft.com/en-us/documentation/articles/active-directory-integrating-applications/

Step 2: Add Azure Service Management & AD app permissions to your application

Under the configure tab, click on Add application under permissions to other applications section:

The popup window loads with default Microsoft Apps, select Windows Azure Service management API and select ok.

Once the application gets added, click on delegated permissions and select the Access Azure Service Management option.

Then save the settings.

Obtain Authorization from user
These set of steps will need to be done each time a user is added or wants to edit their logon settings. Or perhaps add additional subscriptions they want to manage.

All the steps below have been implemented in the aad_integration module.

Step 3: Redirect user to login using OpenID Connect to get a JWT token

If you take a look at the aad_integration module views.py, function authorise_user() creates the Open ID logon url.

The initialize() method called initializes the client_id, secret and other variables of the application generated and stored in steps above.

The method expects a hint on the domain name whether it is a work account or a Microsoft account. As each of them have a separate logon url generated.

def authorise_user(request):     initialise()     PROVIDER_ISSUER_URL = "https://login.windows.net/common/OAuth2/Authorize"     if request.method == "GET":        isMSA = False        noonce_val = str(uuid.uuid4())        if isMSA == True:            PROVIDER_ISSUER_URL = "https://login.windows.net/common/OAuth2/Authorize?client_id="+client_id+"&response_mode=form_post&response_type=code+id_token&nonce="+\ noonce_val + "&redirect_uri=https://127.0.0.1:8000/armaccess/&resource=https://management.azure.com/\ &scope=openid+profile&\ domain_hint=live.com"        else:            PROVIDER_ISSUER_URL ="https://login.windows.net/common/OAuth2/Authorize?client_id="+client_id+"&response_mode=form_post&response_type=code+id_token&nonce="+ \ noonce_val + "&redirect_uri=https://127.0.0.1:8000/armaccess/&resource=https://management.azure.com/& \ scope=openid+profile"     return redirect(PROVIDER_ISSUER_URL)

This redirect URI will take the user to the Azure AD login page and request the user to login as shown in the diagram below:

Figure 1: Azure AD Login page redirect

Once the user logs in, the user will be automatically redirected to a “Allow access” page where the user will be requested explicit access for the application to access Azure Service Management (Preview).

Note: This will work seamlessly with organizational accounts. However, for Microsoft accounts, the account needs to be explicitly added to the AD of your account.

Figure 2: Allow access to application:

Once the user clicks on “Accept”. Azure redirects the response to the Redirect URI provided in the post request. In our case it is https://localhost/armaccess.

The application is added as a Service Principal to the user’s subscription automatically by Azure AD.

Azure returns a JWT token which has user details including tenant ID and an access token.

Step 4: Use JWT to fetch subscriptions the logged in user has authority to manage

The returned Json Web Token has all the details of the user, following is a sample JWT token.

The code sample uses Python library jwt to decode the JSON web token. Please take a look at the armaccess function in views.py of aad_integration module.

We first retrieve the token which returns in the POST request.

id_token = request.POST["id_token"]

       

token_payload = jwt.decode(id_token,verify=False)

Then we call the access_token method to get the access token for next steps.

access_token = get_access_token(tenant_id=str(token_payload["tid"]),client_id=client_id,code=code,client_secret=client_secret, \

redirect_uri=redirect_uri)  

Please do keep in mind that the tenant_id we are passing in this case is not the tenant_id of the application but it is of the customer who has logged in and returned with the JWT token as tid.

With this token that is returned we retrieve the list of subscriptions the user has linked to his/her account:

Let’s take a look at the get_subscription_list function. The url to get the subscription list is:

https://management.azure.com/subscriptions?api-version=2014-04-01-preview

We use the GET method and pass the token obtained in the earlier step in the header.

def get_subscription_list(token_det):

    initialise()

    subs_url = "https://management.azure.com/subscriptions?api-version=2014-04-01-preview"

    headers = {"Authorization":"Bearer " + token_det }

    req = Request(method="GET",url=subs_url,headers=headers)

    req_prepped = req.prepare()

    s = Session()

    res = Response()

    res = s.send(req_prepped)

    subs_list = []

    if (res.status_code == 200):

        resJSON = json.loads(res.content)

        subs_list = resJSON["value"]

    else:

        subs_err = {}

        subs_err["state"]="error"

        subs_err["subscriptionId"]="Error"

        subs_err["displayName"] = res.content + "token " + token_det

        subs_list.append(subs_err)

    return subs_list

The API returns an array of JSON objects containing the SubscriptionID (subscriptionID), Display name (displayName) and if it is enabled.

But before we move to the next step, we should ideally filter out subscriptions that the particular user doesn’t have access rights to manage. This is because we will be adding the application as a service principal on each of these subscriptions using the token retrieved by user details.

Let’s take a look at the following function:

def get_subscription_permissions(token_det, subscription_id):

    initialise()

    url = "https://management.azure.com/subscriptions/"+subscription_id+"/providers/  \

microsoft.authorization/permissions?api-version=2014-07-01-preview"

    headers = {"Authorization":"Bearer " + token_det}

    req = Request(method="GET",url=url,headers=headers)

    req_prepped = req.prepare()

    s = Session()

    res = Response()

    hasPermission = False

    #hasPermission = {}

    res = s.send(req_prepped)

    per_sp = "microsoft.authorization/roleassignments/write"

    per_gen = "microsoft.authorization/*/write"

    hasPermission = res.content

    if (res.status_code == 200):

        resJSON = json.loads(res.content)

        actions_result = resJSON["value"]

        for actions_r in actions_result:

            actions = actions_r["actions"]

            notactions = actions_r["notActions"]

            if per_sp in actions:

                if per_gen not in notactions:

                    hasPermission = True

            else:

                if per_gen in actions:

                    if per_gen not in notactions:

                        hasPermission = True

    return hasPermission

Basically, the function is called for every subscription ID that is returned and then checks for the permissions the user has. If either the permission of role assignment (microsoft.authorization/roleassignments/write) or a generic permission that has all writes (microsoft.authorization/*/write) in Actions and not present in notActions then the function returns True. Which is then included in the subscription list below and sent back as a response as shown below.

Figure 3: List subscriptions

I had created a free trial account with my Office 365 ID. Hence it lists only one subscription in the screenshot.

Step 5: Let the user select from the list of subscriptions they want to manage

As the screenshot above shows the application will return a list of subscriptions that the user can select from. Once the user selects the subscription she/he wants the application to deploy infrastructure on and clicks on Connect. Following is the assignrole(request) function which is invoked on the server side when a user clicks the button.

The html page sends a list of subscription IDs, User ID & Tenant ID in the POST request

def assignrole(request):

    initialise()

    subs_list_assigned = []

    if request.method == "POST":

        subs_list = json.loads(request.POST["subs_list"])

        user_id = request.POST["user_id"]

        t_id = request.POST["tenant_id"]

        token_user = request.POST["token_user"]

We need to again use the tenant_id of the user to generate the token that will be used to get the service principal ID that was created when the user had clicked on Accept.

Please do make a note of the resource URI that we are passing it is different from what we generally call:

get_acacess_token_app(tenant_id=t_id,client_id=client_id,client_secret=client_secret, \

resource="https://graph.windows.net/")

The function get_principal_id(token=access_token["details"],tenant_id=t_id,client_id=client_id)

Is called to get the Principal ID. Following is its body.

def get_principal_id(token,tenant_id,client_id):

    initialise()

   

    get_url = "https://graph.windows.net/" + tenant_id + "/servicePrincipals?api-version=1.5&$filter=appid%20eq%20'"+  client_id +"'"

    headers = {"Authorization": "Bearer " + token}

    req = Request(method="GET",url=get_url,headers=headers)

    req_prepped = req.prepare()

    principalId = None

    s = Session()

    res = Response()

    res = s.send(req_prepped)

    if (res.status_code==200):

        responseJSON = json.loads(res.content)

        value = responseJSON["value"]

        for val in value:

            principalId = val["objectId"]

           

    return principalId

It returns the ObjectId value which is the Principal ID of the application for the user’s account.

Step 6: Add AIM as a contributor to all the subscriptions selected by user

Once we have the Principal ID and the list of subscriptions that the user can manager, we need to Assign the application as a contributor role to all the subscriptions:

Please take a look at the code snippet we create a PUT request which sends the body that contains the principal ID, the access rights the application needs and the Contributor’s role ID which is:

### Predefined role definition ID

role_def_id_reader = "acdd72a7-3385-48ef-bd42-f606fba81ae7"

role_def_id_contributor = "b24988ac-6180-42a0-ab88-20f7382dd24c"

put_url = "https://management.azure.com//subscriptions/" + subs +"/providers/Microsoft.Authorization/roleAssignments/" + role_assgn_id + "?api-version=2014-07-01-preview"

                body = {"properties":{"roleDefinitionId":"/subscriptions/"+ subs +"/providers/Microsoft.Authorization/roleDefinitions/" + role_def_id_contributor ,"principalId":principal_id}}

               

                headers_val = {"Authorization":"Bearer " + token_user, "Content-Type":"application/json"}

                req = Request(method="PUT",headers=headers_val,json=body,url=put_url)

                req_prepped = req.prepare()

                s = Session()

                res = Response()

                res = s.send(req_prepped)

                if res.status_code == 201:

                    subs_list_assigned.append({"subscriptionid":subs,"assigned_status":True})

     

Step 7: Save the subscription IDs and application’s configuration against the user

In this sample application, we are storing the user’s configuration in the default SQLLite storage.

The application stores a JSON array of subscription IDs, the client_id, client_secret (same as the application’s registered one) and tenant_id.

When a user deploys resources through the application, the application will use these details to generate an Auth token and submit the ARM requests.

auth_settings = {}

    auth_settings['auth_type']="A"

    auth_settings['client_id']=client_id

    auth_settings['tenant_id']=t_id

    auth_settings['redirect_uri']=redirect_uri

    auth_settings['client_secret']=client_secret

    auth_settings['subscription_id']=json.dumps(subscription_list)

    auth_settings['user'] = user_id

    user = User.objects.get(pk=auth_settings['user'])

    if AuthSettings.objects.filter(user_id_id=auth_settings['user']).exists():

         u = AuthSettings.objects.get(user_id_id=auth_settings['user'])

         u.delete()       

    settings = AuthSettings(user_id=user,auth_type=auth_settings['auth_type'],client_id=auth_settings['client_id'],tenant_id=auth_settings['tenant_id'],client_secret=auth_settings['client_secret'],redirect_uri=auth_settings['redirect_uri'],subscription_id=auth_settings['subscription_id'])       

    settings.save() 

Once this is done, the application returns the user a confirmation as shown above (it’s a basic message, you could modify it to make it more pretty! The user is not ready to deploy infrastructure on his/her set of subscriptions.

 

Through this two part series we will look at using Python to setup and create Linux virtual machines on Azure and customing the machines through Custom Script Extensions (Part 1) and Chef VM Extensions (Part 2).

This is part 2 series where we will use the same script (available on github) and deploy a MySQL Percona cluster using the Linux Chef VM Extension.

In part 1, the Python script creates a VM, and installs a Custom script extension. This extension downloads the custom Linux scripts from a blob storage and executes them. Logging the execution details. We created a script that downloads and installs the Chef client and then runs a MySQL Percona runbook.

The Linux Chef extension is a very specific extension, which automatically installs the chef client and runs a single or multiple run books for provisioning the server.

Please follow all the steps explained in Part 1, as everything remains the same except the script “python percona_cluster_setup.py” . Please use the “percona_cluster_setup_linuxchefclient.py” script instead.

The script is largely the same except that the Get_Resource_Extension() function returns a LinuxChefClient resource extension details instead of Linux VM Extension explained in Part 1.

 

Following is the function.

def Get_Res_Ext_List():

    chef_server_url = "https://api.opscode.com/organizations/paigeinc"

    validation_client_name = "paigeinc-validator"

    run_list = "role[percona]"

    # For multiple cookbooks : run_list =  "recipe[cookbook1::recipe1],role[test_role1],recipe[cookbook2::recipe2],role[test_role2]
    ##Note- update following path with your local

    validation_key_file_path = "validation.pem"

 

    res_ext_ref = []

    res = ResourceExtensionReference()

    res.reference_name="LinuxChefClient"

    res.name = "LinuxChefClient"

    res.publisher = "Chef.Bootstrap.WindowsAzure"

    res.version = "11.16"

    res.label = "LinuxChefClient"

 

    # Sets the public configuration for extension

    res_chef_client_rb = ResourceExtensionParameterValue()

    res_chef_client_rb.key = "PublicParams"

    pub_config = {}

    pub_config['client_rb'] = "chef_server_url \t %s\nvalidation_client_name\t%s" % (json.dumps(chef_server_url), json.dumps(validation_client_name))

    pub_config['runlist'] = json.dumps(run_list)

    res_chef_client_rb.value = _encode_base64(json.dumps(pub_config))

    res_chef_client_rb.type = "Public"

 

    # Sets the Private configuration for extension

    res_chef_client_validation = ResourceExtensionParameterValue()

    res_chef_client_validation.key = "PrivateParams"

    validation_key_file = open(validation_key_file_path, "r+")

    pri_config = {}

    pri_config['validation_key'] = validation_key_file.read()

    res_chef_client_validation.value = _encode_base64(json.dumps(pri_config))

    res_chef_client_validation.type = "Private"

 

    res.resource_extension_parameter_values.resource_extension_parameter_values.append(res_chef_client_rb)

    res.resource_extension_parameter_values.resource_extension_parameter_values.append(res_chef_client_validation)

 

    res_ext_ref.append(res)

    return res_ext_ref

 

First we set the LinuxChefClient resource extension details, you can find this information if you use the Get-AzureVMExtension  command on powershell/Azure CLI or the REST API. If you just need to get details of a specific extension you could use this link.  

Back to our Chef Client extension, following is the latest version at the time of writing of this blog.

    res.reference_name="LinuxChefClient"

    res.name = "LinuxChefClient"

    res.publisher = "Chef.Bootstrap.WindowsAzure"

    res.version = "11.16"

    res.label = "LinuxChefClient"

 

Since this is a Chef extension, all we need is to provide the rb & .pem files and the list of runbooks that need to be run.  

The client.rb file and the runbook list goes as “public” parameters, and the validation key goes as private. The parameters are passed as JSON in the format shown below:

 

pub_config = {}

    pub_config['client_rb'] = "chef_server_url \t %s\nvalidation_client_name\t%s" % (json.dumps(chef_server_url), json.dumps(validation_client_name))

    pub_config['runlist'] = json.dumps(run_list)

    res_chef_client_rb.value = _encode_base64(json.dumps(pub_config))

    res_chef_client_rb.type = "Public"

pri_config = {}

    pri_config['validation_key'] = validation_key_file.read()

    res_chef_client_validation.value = _encode_base64(json.dumps(pri_config))

    res_chef_client_validation.type = "Private"

 

Please ensure that the validation key file contents are base64 encoded. I have used the Azure Python SDK’s existing function _encode_bas64() , it could be a custom function that you would like to write.

Once, you run the script as explained in Part 1, give it some time till the VM’s provisioning is done. You will notice the “Linux Chef Client” extension installed in the extensions installed section of the portal’s dashboard.

 

Troubleshooting

While the vm_getstatus.py script can be used to get status of the request submission, sometimes the startup script might fail to run during the provisioning state. You can monitor the VM creation progress on the portal too.

Once the VM reaches the running status, you can logon to the VM and view the following logs in case of any errors in VM provisioning.

Following log file logs all the VM provisioning activity by Azure VM agent.

/var/log/waagent.log

 

Following log logs running of the extension script.

 

 /var/log/azure/Microsoft.OSTCExtensions.LinuxChefClient /1.1/extension.log