Please check Ashish's post for more interesting details

 https://blogs.msdn.com/b/ashish/archive/2015/07/12/milliman-going-big-in-the-cloud-with-microsoft.aspx

Milliman is a great example of a company leveraging Azure extensively and in very innovative ways. In this post I have asked Paul Maher, CTO - Integrate & MG-ALFA, to share some insights on the work his team are doing with Integrate™ (their flagship Cloud solution delivered to the Life Insurance Market).

Quoting Paul Maher:

"Integrate is an end-to-end, revolutionary cloud-based system for the life insurance market. Built on the Microsoft Azure platform, it reimagines the relationships between people, process and technology.

Integrate clients are able to run large actuarial jobs on-demand year round, benefiting from the Azure platform’s scale and extensibility in a pay-as-you-go business model.

By using Integrate with Azure, Milliman was able to unlock the potential of human capital to drive business, not simply manage it. Integrate, offered as a SaaS solution, and eliminates the need for costly on-premises infrastructure. It also benefits from cloud platform capabilities such as compute power and storage, as well as feature-rich services such as Azure Data Factory and HDInsight.

Benefits of Azure

• Speeds time-to-market: By taking advantage of Azure and its associated services, we have been able to accelerate the development and time-to-market of the Integrate solution. The Azure platform has been a familiar and intuitive environment for our development team to build the Integrate solution.
• Provides massive scalability on demand: We fulfill our client’s large computational requirements by taking advantage of Azure’s ability to scale up or down as needed, in a pay-as-you-go pricing model.
• Reduces IT costs and management: With the Integrate cloud-based solution, Milliman clients pay only for the processing and storage capabilities they need, without having to deploy, manage, and upgrade hardware. With Azure, companies avoid large capital expenses, because the Integrate solution can access high-performance computing resources on demand at a small fraction of what it would cost to own and operate those resources.
• Supports data management and reporting: Azure Data Factory and HDInsight has provided the capabilities needed to support data management, data manipulation, and reporting requirements for our clients.
• Delivers security and reliability: Security and reliability are major concerns for financial services companies. That is why Azure, which is run in Microsoft data centers, is a good choice for an industry solution. Azure is an enterprise-grade platform with attendant security and reliability. And, because Microsoft has datacenters worldwide, information can be hosted in jurisdictions

Azure was instrumental in helping us deliver a robust, dynamic risk management solution that stretches the boundaries of what is possible in enterprise computing. With the Microsoft cloud, we have the capabilities and resources needed to rapidly bring an innovative, industry-changing solution to market. As a result, insurance companies worldwide can benefit from sophisticated financial modeling and big data analytics at nominal cost”.

For further information, please use the following resources:

• Milliman website
• Integrate website
• Integrate Customer Story

Paul Maher began at a small actuarial firm in the U.K., building pension software, and transitioned to IT roles and leadership positions at Microsoft, KPMG, and others prior to joining Milliman in 2013. He has a passion for building software solutions that help solve challenging business problems. Contact Paul at LinkedIn.

This template deploys a 2-10-node cluster using the ARM "copy" feature. Each node has 2 network cards: 

• The first one is on a "public" subnet.
• The second one is on a "private" subnet.

The public subnet is fronted by a load-balancer with one dynamically assigned public IP address and as many NAT ports as there are nodes, starting from port 50000. The template also asks for a dns name and a location for the ip address. Thus, if you want to connect to node 0, you'll type:

• ssh <user>@<name>.<location>.cloudapp.azure.com -p 50000

The private network is intended for cluster communications only.

The template also configures:

• A storage account where all the virtual hard disks are stored.
• A virtual network where the private and public subnet reside.

The template invokes a custom bash script that configures the second network card on CentOS nodes. By default the second card is recognized but the network stack is not set up.

The limit on the number of nodes is artificial. Alas, ARM does not support arithmetic operators yet, so one has to list all the possible NAT ports for the load-balancer configuration. I listed 10. Feel free to add more and you'll be able to increase the number of nodes. Also, if not all nodes require external connections, you can have as many nodes as the current subnet limit.

In part 2 I will add:

- Shared storage for all the nodes using azure file shares.

- A script to set up opengrid and mpi.

Here is a summary of the steps required to do so:

1. Create a vnet in each desired datacenter, ideally with at least 2 subnets (1 for the domain controllers, another for the file servers) + 1 for the gateway
2. Establish cross-premise connectivity in Azure, for instance as explained in https://msdn.microsoft.com/en-us/library/azure/dn690122.asp
3. Deploy domain controllers on the relevant subnets in both datacenters. Make sure that the dns option in the vnet configurations points to both domain controllers in the appropriate order (dc1, dc2 on vnet1 - dc2, dc1 on vnet2).
   1. It is recommended that you assigned a fixed ip to the domain controllers / dns servers. You will need Powershell to do that as explained in https://msdn.microsoft.com/en-us/library/azure/dn630228.aspx.
4. Deploy servers on the relevant subnets and join them to the domain.
5. Add at least a data volume each to the servers to host the file shares. If you are planning to replicate its content using dfs-r, it may make sense to put the volume on a locally redundant storage account.
6. Log into each server and, in “configure local server”, add the DFS namespace and DFS replication features.
   1. Alternatively, you could use Powershell desired configuration to deploy servers with those features enabled. See https://blogs.msdn.com/b/powershell/archive/2014/08/07/introducing-the-azure-powershell-dsc-desired-state-configuration-extension.aspx 
7. In the dfs management console on server 1, create a namespace.
   1. Select “edit settings” and assign full access to administrator, r/w to other users
   2. Make sure that it is a domain-based namespace and “2008 mode” is enabled. Information about namespace roots is then replicated on namespace servers in the domain.

8. Add server 2 as a namespace server to the namespace you've just created for availability.
9. Select “new folder” in the namespace.
10. Create a new file share hosted on the local data volume (e.g.\\server1\share1). Assign permissions as desired (e.g. admins full access and rest r/w).
11. Click on the share you've just created and select “replicate folder”.
12. Add a target for the replication on server2 e.g. \\server2\share1
13. Create replication group as required, by following prompts.
14. You may also want to verify that replication works both ways by creating a share on server2 that is replicated to a target on server1.

• Given a certain hardware configuration, for software package A with an input size of X, on average you'll need Y cores.

On Azure, the hardware configuration is known and you can deploy as many instances as your subscription allows. It would be useful to capture that knowledge in order to optimize our resource utilization. For instance, we can pass that estimated size to our node manager / autoscaler (e.g. in GeRes, HPC Pack or similar batch scheduler service), let it deploy the cluster and then automatically resize as required if the job turns our to be larger or smaller than estimated.

Azure Machine Learning can be used to provide that initial estimate on the basis of historical utilization data. We'll build an Azure ML service, submit a cue to it (e.g. the type of job to run) and get an estimate in return. The process to set it up can be summarized as follows:

1. Collect a representative data sample and save it in CSV format, e.g. {Type of Job, Input Size, N. of Cores required}.
2. Build a workflow in Azure ML to train and score an appropriate regression model (several are on offer, e.g. linear, Bayesian, neural networks).
3. Evaluate the regression models.
4. Publish one (or more) as Azure ML service.
5. Use the ML API to pass the cue and retrieve the estimate.
6. Use the estimate to deploy the n. of nodes suggested, then let the autoscaler correct as required until the job ends
7. Collect the usage data and feed back into the training workflow at the next opportunity.

Here is an example of such workflow in ML Studio:

It starts with a simple CSV file containing {Type of Job, Size of Input, N. of Cores Required}. 

The "project columns" and "metadata editor" blocks are used to select the columns of interest (or remove empty ones), then change the data description. The job type in our case is a numeric value, but we really want to treat it as a category not to be computed, hence the change in metadata.

We split the data into two parts: one will be used to train the regression algorithm, another to score it, i.e. evaluate the efficacy of its predictions against a set of observed inputs / outputs.

We then initialize two regression models: a linear one and a neural network. This is not strictly necessary in this case. It is just a way to illustrate how you can test several models and then choose the one that provides better predictions. In order to train the models, we supply them with the same portion of the data.

The next step is to score the models, i.e. use them to predict the required n. of cores for the remaining input set, then compare the results against the sampled data.

Here is an example of the scored results for linear regression:

... and for neural networks:

Just by looking at the scatter plot we can see that the neural network is more accurate than simple linear regression.

The evaluation module confirms our finding (the first row in the table is for linear regression, the second one for neural networks):

Now that we have established which technique fits the problem better, we'll save the trained model into our library and publish it as a service.

Click on the "Train Model" block output and select "Save as Trained Model":

With our trained model, we build a simple new workflow.

The "Score Model" block is the one that will provide us with estimates given previous history, so we publish its input and output ports as i/o ports for the prediction service.

Finally we run the model again, check its results and if satisfied we publish it as a service. 

In the process of publishing, an API key will be generated that we'll use to access the Azure ML REST API. 

Here's some simple Python code to submit a cue and receive an estimate - in our case we submit a job type and receive an estimated number of cores:

It is a simple batch job manager written in C# (or Python for the older version).

Geres provides:

• Command line utilities (e.g. qsub, qlist, jobcancel, joblist) to queue tasks for computation, check on their status, cancel them.
• 3 task queues (highq, mediumq, lowq) in order of priority.
• An agent to install on Azure VMs. It will pick tasks off the queues in that order and spawn the required processes. 
• A simple notifier application that monitors the status of tasks as reported by the agents.
• An autoscaler, running as an extra-small PaaS worker role, which will deploy or remove worker VMs of the desired size based on waiting time for jobs in the queues.

The agent will run as many tasks on a node as there are cores. If a task is marked "exclusive" at submission, it will be the only one to run. This is useful for those applications that consume most of the VM resources.

The agent is also responsible to update the status of the spawned processes: running, failed, completed or cancelled. When a node is idle for longer than a pre-configured time, the agent will queue it for removal.

The autoscaler only deploys or removes worker nodes - it does not keep track of task status.

Note that this distributed architecture has the advantage of being highly resilient when compared to a traditional "head node" running the scheduler and apportioning work.

The nodes can fail at any time and the incomplete jobs will pop back into the queues for other nodes to pick up.

The autoscaler can fail at any time and simply be restarted without affecting computations in process.

Azure storage tables and queues are used to handle and keep track of tasks. Service bus topics are used for notifications and commands. Geres benefits from their built-in redundancy and resilience.

For further details on the architecture, please read the release notes on Codeplex.

This short video will show you a typical usage scenario.

[View:https://www.youtube.com/watch?v=7OlpcS3A_OM&feature=youtube_gdata:0:0] 

If you want to create a virtual machine, you need to:

1. Create a cloud service of which the VM will be an instance
2. Select a vhd image for the machine, either from Microsoft catalog or one you created.
3. Indicate which blob in which storage account will store the vhd file of the VM
4. Create a configuration set containing machine metadata
5. Finally, deploy the VM

Note that you'll need a management certificate to interface with the Azure Service Management Service. The documentation suggests using openssl to create one, which is fine on Linux and Mac, but not on Windows. On Windows, use makecert instead:

makecert -sky exchange -r -n "CN=<CertificateName>" -pe -a sha1 -len 2048 -ss My "<CertificateName>.cer"

This will put the certificate into the current user's personal store and generate a .cer file for you to upload to Azure.

Also note that there is a bug in the Python SDK which affects the creation of endpoints. See https://github.com/WindowsAzure/azure-sdk-for-python/pull/83 for a description.

You will need to edit the init.py file as described on github.

You can do the rest in python:

from azure import *
from azure.servicemanagement import *

subscription_id = '<enter your subscription id>'
# The certificate comes from the local store, not from a .pem file on Windows

certificate_path ="CURRENT_USER\\my\\<enter your certificate name>"

# instantiate a service management service
sms = ServiceManagementService(subscription_id, certificate_path)

#provide a service name and location

name = '<name of service you want>'
location = 'West US'

# You can either set the location or an affinity_group
sms.create_hosted_service(service_name=name, label=name,location=location)

# Name of an os image as returned by list_os_images (catalog and yours)
image_name = '<your image name>.vhd'

# Destination url://storage account/container/blob where the VM disk will be created
media_link = 'https://<account name>.blob.core.windows.net/vhds/'+name+'.vhd'

# The documentation shows a Linux VM. Windows is more complicated.
# WindowsConfigurationSet contains metadata for a Windows VM
windows_config = WindowsConfigurationSet(<machine name>, '<admin password>')
# by default the api will look for domain credentials. if you want no domain:

windows_config.domain_join = None

# Here's the hard disk for the os

os_hd = OSVirtualHardDisk(image_name, media_link)

# Unless you specify endpoints, you won't be able to connect to the VM.

# The documentation is unclear on the matter.

endpoint_config = ConfigurationSet()
endpoint_config.configuration_set_type = 'NetworkConfiguration'

endpoint1 = ConfigurationSetInputEndpoint(name = 'rdp', protocol = 'tcp', port = '33890', local_port = '3389', load_balanced_endpoint_set_name = None, enable_direct_server_return = False)
endpoint2 = ConfigurationSetInputEndpoint(name = 'web', protocol = 'tcp', port = '8080', local_port = '80', load_balanced_endpoint_set_name = None, enable_direct_server_return = False)

#endpoints must be specified as elements in a list

endpoint_config.input_endpoints.input_endpoints.append(endpoint1)
endpoint_config.input_endpoints.input_endpoints.append(endpoint2)

#Finally you can deploy the VM

sms.create_virtual_machine_deployment(service_name=name,
deployment_name=name,
deployment_slot='production',
label=name,
role_name=name,
system_config=windows_config,
network_config = endpoint_config,
os_virtual_hard_disk=os_hd,
role_size='Small')

Powered by Qumana

Keep in mind that in Microsoft's HPC scheduler implementation, jobs are just containers. They contain one or more tasks to be executed and have properties associated to them (e.g. credentials, priority).

Such a container can be created empty, then filled with tasks and finally submitted for execution. If you have a look at the Create-InstallJob.ps1 script that comes with Cloudblitz, you'll see an example of that.

Let us suppose that you want to write a simple job submission script:

1. Import the Cloudblitz.Powershell module and get the hpc object types:

Import-MyModule -Name "CloudBlitz.Powershell" -Path $cmdletsPath
$hpcTypesPath = [System.IO.Path]::GetDirectoryName($cmdletsPath) + "\HpcSchedulerManagement.dll"
Add-Type -Path $hpcTypesPath -ErrorAction Stop

2. Provide the credentials for the jobs to be submitted

# Fill in hpc scheduler credentials
 $hpcCredentials = New-Object CloudBlitz.Powershell.Cmdlets.HpcSchedulerCredentials
 $hpcCredentials.ClusterName = $ClusterName
 $hpcCredentials.AdminLogin = $AdminLogin
 $hpcCredentials.AdminPassword = $AdminPassword
 $hpcCredentials.CertificateThumbprint = $mgmtcert.Thumbprint

Note that you will need the thumbprint of the certificate associated with the service at deployment. That is required because the powershell commandlets use the hpc scheduler REST API to communicate with the service. All the parameters can be passed on the command line or retrieved from the ConfigurationVariables.ps1 file that you edited before deployment. 

3. Create an empty job container

# Create the job on the scheduler
 $Job = New-Object HpcSchedulerManagement.DataContracts.SchedulerJob
 $Job.FailOnTaskFailure = $True
 $Job.Name = "Job"+$(get-random 1000000)
 $Job.Id = Add-SchedulerJob -Credentials $hpcCredentials -Job $Job -ErrorAction Stop

The first line creates an empty container, then we set some properties (e.g. random name) and finally we create the job on the hpc scheduler and retrieve its id.

4. Add tasks into the job container

# Create the task
 $Task = New-Object HpcSchedulerManagement.DataContracts.SchedulerTask
 $Task.Name = "Task"+$(Get-Random 1000000)
 $Task.CommandLine = $CommandLine
 $Task.Id = Add-SchedulerTask -Credentials $hpcCredentials -JobId $Job.Id -Task $Task

As with the job, we create an empty object, set its properties, then add it to the job container whose id we retrieved before. Note that one of those properties is $Task.CommandLine, which takes a string. The string is an arbitrary command to be executed on the remote nodes. If you want to start a MPI job, it will look like

mpiexec <mpi parameters> <your mpi>.exe <app parameters>

5. Submit the job for execution

# Submit the job
  Write-Host "Submitting the job"
  Submit-SchedulerJob -Credentials $hpcCredentials -JobId $Job.Id

# Wait for the job to complete
  Write-Host "Waiting for job $($Job.Id) to complete"
  $finalState = Wait-SchedulerJob -Credentials $hpcCredentials -JobId $Job.Id

  Write-Host "Job $finalState"

Note that you need not wait for the job to complete. Those lines just show a way to retrieve and display the final job state. Normally, you will submit, then query for status later with something like:

Get-schedulerjob -Credentials $hpcCredentials -JobId $JobId

That line will retrieve the job object and also display all its properties (not just those set in these samples).

You can retrieve the whole list of jobs in the scheduler with:

Get-schedulerjobList -Credentials $hpcCredentials

The job properties are those supported by Microsoft's HPC scheduler (https://msdn.microsoft.com/en-us/library/microsoft.hpc.scheduler.ischedulerjob_properties(v=vs.85).aspx).

Let's see an example of those, retrieved using Get-SchedulerJob:d                  : 8

Name                : Job641676

UserName            : topguy

Project             :

RuntimeSeconds      : 0

MinCores            : 1

MaxCores            : 1

MinSockets          : 1

MaxSockets          : 1

MinNodes            : 1

MaxNodes            : 1

UnitType            : Core

RequestedNodes      :

IsExclusive         : False

RunUntilCanceled    : False

NodeGroups          :

FailOnTaskFailure   : True

AutoCalculateMax    : True

AutoCalculateMin    : True

Preemptable         : True

MinMemory           : 0

MaxMemory           : 0

MinCoresPerNode     : 0

MaxCoresPerNode     : 0

SoftwareLicense     :

OrderBy             :

ClientSource        : AzureRestServiceAgent

Progress            : 100

ProgressMessage     :

TargetResourceCount : 0

ExpandedPriority    : 2000

ServiceName         :

NotifyOnStart       : False

NotifyOnCompletion  : False

EmailAddress        :

Priority            : Normal

Password            :

JobTemplate         : Default

JobType             :

RequeueCount        : 0

AutoRequeueCount    :

RequestCancel       :

Owner               : topguy

SubmitTime          : 5/15/2013 11:20:47 PM

CreateTime          : 5/15/2013 11:20:46 PM

StartTime           : 5/15/2013 11:20:48 PM

EndTime             : 5/15/2013 11:20:50 PM

State               : Finished

HoldUntil           :

HasRuntime          : False

CanGrow             : True

CanShrink           : True

PreviousState       : Running

ChangeTime          : 5/15/2013 11:20:50 PM

ExcludedNodes       :

AllocatedNodes      : 

You will typically want to set at job submission MinCores and MaxCores (min. and max n. of cores required, default is 1). Note that properties are inherited by the tasks, unless overridden by those set at the task level (https://msdn.microsoft.com/en-us/library/microsoft.hpc.scheduler.ischedulertask_properties(v=vs.85).aspx). For MPI jobs the number of cores is fixed at submission, so MinCores=MaxCores.

You can find it at https://cloudblitz.codeplex.com

The tool builds on the HPC scheduler for Azure and the Powershell commandlets for Azure to solve a common problem, i.e. provide a scriptable way to create and interact with an Azure PaaS cluster. I refer you to the Codeplex page for the complete description and documentation. In this blog post, I'll cover how to build it and use it.

1. Make sure that you download and install all the dependencies listed on the codeplex page.

2. Download and extract the source code in its own directory.

3. Open a Visual Studio 2012 command prompt as administrator (dear old cmd.exe)

4. Go to the "BuildScripts" directory and run BuildCloudBlitz.cmd. This will build the required modules and copy the results in a "Packages" directory.

5. Open a Windows Azure Powershell prompt as administrator. Go to the CloudBlitz.ClusterTemplate directory

6. Edit the ServiceDefinition.csdef file (it is just an XML file). In particular note:

<WorkerRole name="ComputeNode" vmsize="Small">

...

<WebRole name="HeadNode" vmsize="Small">

Choose your VM size and type it in. In general, a small head node is sufficient (the head node does NOT run SQL, rather it uses SQL Azure, which reduces its memory requirements). If you're planning to run thousands of tasks at a time, go for a large instance.

7. Go to the CloudBlitz.Powershell\Scripts directory

8. Edit the ConfigurationVariables.ps1 file. In particular note:

$subscriptionName = "<your subscription name>"
$subscriptionId = "<your subscription id>"
$administratorName = "<admin name for the hpc cluster>"
$systemPassword = "<admin password for hpc cluster>"
$certificateFileNameRoot="<name of certificate>"
$deploymentName="<name of service>"
$deployLocation="<data center location>"
$HeadNodeInstanceCount = 1
$ComputeNodeInstanceCount = <n. of desired compute nodes>
$removePreviousDeploymentAndService = $true

9. Run InstallAndExportCertificate.ps1 This will create a .cer and a .pfx self-signed certificate file, install it in the local user certificate store.

10. Via the Azure portal, upload the .cer file as a management certificate.

11. Run the DeployDeployerRole.ps1 script. It will upload the packages, create an azure worker role that deploys a hpc cluster as per configuration, then remove the deployer role. Such a temporary role is necessary to avoid any firewall and connectivity issues. Most people will have ports 80 and 443 open for http and https. The Azure HPC scheduler deployment also requires a connection to SQL Azure on port 1433, which is normally closed. By letting the temporary role manage the process, such connections are all internal to Azure.

You can now connect to your cluster.cloudapp.net via RDP and start submitting jobs in the "traditional" way via the job manager or the command line.

Alternatively, you can use the set of powershell commandlets we developed from your workstation. They will be the subject of the next blog post :-)