This blog post is out-of-date as it is now possible to use custom images and VNETs using a regular 'Batch service' Batch account; a 'user subscription' Batch account is no longer required.  There is documentation available for how to use custom images and VNETs.

• Azure Batch will create virtual machine pool VMs directly in the subscription associated with the Batch account; one or more Virtual Machine Scale Set’s (VMSSs) and associated storage accounts will be created and visible in the Azure portal, PowerShell, etc.
• Virtual machine core quotas will be applied, not the Batch core quota.
• The Batch client API must use Azure Active Directory authentication, which is now supported in addition to the existing account name and key authentication. Azure Batch support for Azure Active Directory is documented here.
• It will not be possible to create Cloud Service pools.

Creating “User Subscription” Batch Accounts

Subscription Access Control

• Select "Subscriptions" from the left-hand menu, then select the subscription which you will use to create the "User Subscription"Batch account.
• Select the "Access Control (IAM)" menu item for the subscription.
• Select the "Add" button, select "Contributor" for the role, select either "MicrosoftAzureBatch" or "Microsoft Azure Batch" as the member, then Save the change.

Create a Key Vault

• Select "Key vaults" from the main portal menu, then select "Add".
• Enter a name.
• Pick an existing resource group or create a new one; note down the resource group.
• Ensure you pick the same Location and the new Batch account.
• Leave all other properties at their default values.

Create the “User Subscription” Batch Account

• If creating a new Key Vault, then set the Resource Group to be the same as you will use for the Batch account.
• Keep all other properties as default; they will be configured as part of the "New Batch account" process.

API Authentication for "User Subscription" Batch Accounts

Creating Batch Pools using the "User Subscription" Batch Account

Quotas

• One storage account (Quota = “Storage Accounts”, Provider = “Microsoft.Storage”)
• One public IP (Quota = “Public IP Addresses”, Provider = “Microsoft.Network”)
• One virtual network (Quota = “Virtual Networks”, Provider = “Microsoft.Network”)
• One network security group (Quota = “Network Security Groups”, Provider = “Microsoft.Network”)
• One virtual machine scale set (Quota = “Virtual Machine Scale Sets”, Provider = “Microsoft.Compute”)
• One load balancer (Quota = “Load Balancers”, Provider = “Microsoft.Network”)

• Quota = “Total Regional Cores”, Provider = “Microsoft.Compute”
• Quota = “… Family Cores”, Provider = “Microsoft.Compute”

Pool Creation

• For "Image Type", the value of "Cloud Services (Windows only)" cannot be selected; for standard images "Markeplace (Linux/Windows)" must be selected.
• A new "Image Type" value of "Custom Image (Linux/Windows)" is available.

Create a Pool using a Custom Image

• One unique custom image VHD blob can support up to 40 Linux VM instances or 20 Windows VM instances. You will need to create copies of the VHD blob to create pools with more VMs.  For example, a pool with 200 Windows VMs needs 10 unique VHD blobs to be supplied in the 'osDisk' property.
• The storage accounts holding the custom image VHD blobs need to be in the same subscription as the Batch account.
• The specified storage accounts should be in the same region as the 'User Subscription' Batch account.
• Only Standard storage is currently supported; Premium storage will be supported in the future.
• You can specify one storage account with multiple custom VHD blobs or multiple storage accounts each having a single blob. We recommend you to use multiple storage accounts to get a better performance.
• You need to select the appropriate nodeAgentSkuId depending on the OS of the base image of your VHD. You can get mapping of available nodeAgentSkuId to the OS Image reference by performing the List supported NodeAgentSkuId request.

• Open blade for a batch account, select Pools menu item, on the Pools blade select the "Add" command; the "Add pool" blade will be displayed.
• Select “Custom Image (Linux/Windows)” from Image Type dropdown, the “Custom Image” picker will be displayed, open the picker and pick one or multiple VHDs from the same container and click Select button.  The support for multiple VHDs from different storage accounts and different containers will be added soon.
• Pick the correct Publisher/Offer/Sku for your custom VHDs and select the desired Caching mode, then fill all the other parameters for the pool.
• To check if a pool is based on custom image see the "Operating System" property in resource summary section of Pool details blade - it should have a value of "Custom VM image".
• All the custom VHDs associated with a pool are displayed on the pool properties blade.

Include Pool VMs in a Virtual Network

• The specified virtual network (VNET) must be in the same Azure region as the ‘User Subscription’ Batch Account.
• The specified VNET must be in the same subscription as the ‘User Subscription’ Batch Account.
• The specified VNET must be an Azure Resource Manager (ARM) based VNet.  Classic VNets created via Azure Service Management are not supported.
• The specified subnet should have enough free IP addresses to accommodate the number of VMs that will be in the pool. If the subnet doesn't have enough free IP addresses, the pool will partially allocate compute nodes, and a resize error will occur.
• The specified subnet must allow communication from the Azure Batch service to be able to schedule tasks on the compute nodes. This can be verified by checking if the specified VNET has any associated Network Security Groups (NSG). If communication to the compute nodes in the specified subnet is denied by an NSG, then the Batch service will set the state of the compute nodes to unusable.
• If the specified VNET has any associated Network Security Groups (NSG) then inbound communication must be enabled; ports 29876, 29877, and 22 for a Linux pool; 3389 for a Windows pool.

• High Availability HPC Cluster in Azure – Enterprise customers of HPC Pack cluster take the HA of workloads running in cluster with very high priority. Starting from HPC Pack 2016, users can easily deploy a new HPC cluster with or without HA with Azure ARM templates. The multiple active HNs will not just make SLA for customer but also provide load balance to the services across the cluster. For more details, please go to this page for guidance.
• Use AAD to manage HPC Pack cluster in Azure – with the AAD integration with HPC Pack 2016, customers can now directly leverage the AAD authentication to manage the cluster and job schedule. This also gives customers the potential to connect the on-premises AD to AAD to make authentication across the hybrid environment much simple. Please go to this Azure content page for more details.
• Support N-series GPU VMs in Azure – with the announcement of the N-series of GPU VMs in Azure, HPC Pack will also support to add these GPU capable VMs into the HPC Pack cluster. For how to use HPC Pack cluster to manage the GPU nodes and jobs, please go to the help doc here.
• Support auto grow/shrink for ARM cluster – in HPC Pack 2016, if customers deploy the cluster with Azure ARM resources, the built-in auto grow/shrink service will be supported. It works the same way as we did in classic Azure resources. For more details, please see the helps doc here.
• Support new Windows/SQL platforms – we are going to support the latest version of major platforms: Windows Server 2016 and SQL Server 2016.

Step-by-step: OpenFOAM MPI toolbox with CentOS-HPC on Batch using IntelMPI

Prerequisites

• Azure account: If you don't already have an Azure subscription, create a free Azure account.
• Batch account: Once you have an Azure subscription, create an Azure Batch account.
• Storage account: See Create a storage account in About Azure storage accounts.

OpenFOAM input data preparation

Prepare the coordination command script

Create CentOS-HPC pool (and prepare nodes for OpenFOAM)

Create a Batch job

Submit MPI task for OpenFOAM

Download the output data

OpenFOAM solution end-to-end sample code

Run HelloWorld MPI application with Ubuntu on Batch using OpenMPI

Create Ubuntu pool (and install OpenMPI)

Submit MPI task on Ubuntu pool

Run HelloWorld MPI application with CentOS on Batch using OpenMPI

Create CentOS pool (and install OpenMPI)

Submit MPI Task on CentOS pool

• Prerequisite
• Using Linux on Batch
• Command Line Support (Azure xplat CLI)
• Troubleshooting - Using SSH
• Linux Support Matrix
• Pricing Information
• Reference

• Dot Net SDK can be downloaded from Nuget.
• Python SDK is part of the Azure Python SDK on PyPi.
• Node.js SDK can be found on npm.
• Java support will come shortly in the future.

creds = SharedKeyCredentials(account, key) config = BatchServiceConfiguration(creds, base_url = batch_url) client = BatchService(config)

new_pool = PoolAddParameter(id = "<pool name>", vm_size = "e.g. STANDARD_A4")
new_pool.target_dedicated = 4
start_task = StartTask()
start_task.run_elevated = True
start_task.command_line = '<pool prep task cli>'
new_pool.start_task = start_task
ir = ImageReference(
    publisher = "Canonical",
    offer = "UbuntuServer",
    sku =  "14.04.2-LTS",
    version = "latest")

vmc = VirtualMachineConfiguration(
    image_reference = ir,
    node_agent_sku_id = "Batch.Node.Ubuntu 14.04")

new_pool.virtual_machine_configuration = vmc

pool_info = PoolInformation(pool_id = pool_id)
job = JobAddParameter(id=jobId, pool_info = pool_info)

job_preparation_task = JobPreparationTask()
job_preparation_task.command_line = "<start up cli> "
job_preparation_task.run_elevated = True # If specified, task will run as su
job.job_preparation_task = job_preparation_task
client.job.add(job)

taskId = '<task name>'
taskCmdLine = '<task cli>'

task = TaskAddParameter(id = taskId, command_line=taskCmdLine)
task.run_elevated = True # If specified, task will run as su

client.task.add(bound_job.id, task)

nodes = client.compute_node.list(pool_id)
for node in nodes:
    login = client.compute_node.get_remote_login_settings(pool_id,
                                                          node.id)
    print("{0} {1} {2} {3}".format(node.id,
                                   node.state,
                                   login.remote_login_ip_address,
                                   login.remote_login_port)

tvm-3436469628_1-20160320t055249z ComputeNodeState.idle 52.160.94.74 50002

tvm-3436469628_2-20160320t055249z ComputeNodeState.idle 52.160.94.74 50003

tvm-3436469628_3-20160320t055249z ComputeNodeState.idle 52.160.94.74 50000

tvm-3436469628_4-20160320t055249z ComputeNodeState.idle 52.160.94.74 50001

import getpass

• Multi-instance task support. Batch service now support MPI type of workloads. Only Windows and MS-MPI combination is fully supported but we are working hard on other OS and MPI implementations. For detail, check this blog post.

• Take node offline without deleting it. Our customers have been asking for this feature for a long time. With the Dec., 2015 service update, it's possible to disable a node from scheduling any new tasks without deleting it. This feature can be accessed from Microsoft.Azure.Batch.PoolOperations.DisableComputeNodeScheduling and Microsoft.Azure.Batch.PoolOperations.EnableComputeNodeScheduling.

• Change autoscale formula evaluation interval. By default, the Batch service adjusts a pool's size according to its autoscale formula every 15 minutes. This interval is configurable now through the property of CloudPool.AutoScaleEvaluationInterval. The minimum interval is 5 minutes, and the maximum is 168 hours. If an interval outside this range is specified, the Batch service will return a Bad Request (400) error.

All features mentioned above are available across all regions immediately. To access those features, the REST API version should be "2015-11-01.2.1" or using Azure Batch Client Library 3.0.0. All earlier version of REST and .NET API are still available. We will not obsolete them for now.

We are happy to announce today the release of MPI support on Azure Batch services.

We have released a service update recently with several feature enhancements, MPI support being the most important of them. MPI support is provided through a new feature in Batch called multi-instance task. Azure customer can now submit and run MPI tasks on their Batch pools globally. For now, only MS-MPI on Windows platform has been fully tested but we are working on support of other platform and MPI libraries.

Furthermore, by simply using MS-MPI on A8/A9 pool, Batch MPI tasks can fully leverage the high speed low latency RDMA network on those Azure VMs - no extra work needed.

A detail article on how to build, compile, and run a MPI program on Batch will be published soon.

We will also talk about other feature enhancement in a series of blogs so stay tuned to our blog.

Before the HPC Pack 2012 R2 Update 3 released, it’s quite difficult to use HPC Pack to fully utilize the GPU resources in HPC Pack. It’s also hard to manage the nodes with GPU and monitor their GPU performance count easily. I would like to show some details of Update 3 that resolve these issues.

Deployment

First things first, we need to deploy the compute nodes with GPU resources in HPC Pack cluster. Are there any extra steps we need to do? What if I install a new GPU card to an existing HPC compute node, are there any other steps we need to follow? 

HPC Pack provides a quite straightforward and easy way to deploy GPU nodes WITHOUT any extra steps. As long as your compute nodes installs a supported GPU card, we will recognize and handle it automatically. The only one requirement here is the GPU card should be in our supported module list. In update 3, we only support NVIDIA CUDA GPUs. 

Management and Monitoring

Before the Update 3 of HPC Pack 2012 R2, HPC admins have to manually identify and mark which nodes are GPU capable. It’s also hard for them to easily monitor some key GPU performance counters, like GPU time, to show which GPUs are in use and how they are used.  Here are some key features we introduced to resolve these management and monitor problems.

In Update 3, as long as HPC Pack recognizes the GPU node, it will automatically create new node group “GpuNodes”. All the recognized GPU capable nodes will be listed there.

It’s also straight forward to find out more GPU specific details in the node properties under the “GPU Information” tab.  

Head maps provide an easy way to find out how the nodes and system run. We are going to leverage this to show the GPUs’ performance counters as well. You can customize your own view of the heat maps to monitor GPU usage:

And if there are multiple GPUs in one compute node, multiple metrics value will be shown as well. For example, the following screenshot shows that 8 GPUs in 2 nodes are fully used when a GPU job runs.

GPU Job Scheduling

For GPU jobs, it’s natural to schedule based on GPU resources. Before Update 3 of HPC Pack 2012 R2, we cannot see any GPUs in compute nodes and it’s also hard to monitor the overall GPU resources and dispatch to each node for execution. This has been improved dramatically in Update 3.

When you create a new job, you will find out that there is a new schedule unit of “GPU”. This is the choice for HPC jobs that want to be scheduled based on GPU resources.

Once you choose the GPU for the job scheduling, you need to make sure the job resources you choose have GPUs deployed. If HPC job scheduler failed to find any GPU available in the job resources selection results, the job will fail with corresponding error messages.

We also support scheduling GPU jobs via HPC PowerShell and APIs. There are also more information and suggestions about how to schedule GPU jobs easily and CUDA coding sample. Please find more detailed info at the end of this blog.

Roadmaps

Microsoft Azure announced the plan to support N-Series VMs in the near future. HPC Pack is also planning to support the new type of VMs with GPU once they’re available.

This is the first step that HPC Pack is taking to introduce GPU support introduced the GPU supports. We are looking forward to hearing your feedbacks when you use it and see how we can improve it in future.  You can reach out to us via email hpcpack@microsoft.com

For more detailed information about the GPU supports in HPC Pack 2012 R2 Update 3, please visit the TechNet article here.

We know that GPGPU in HPC is now getting very popular not just in academic research but also in enterprise engineering works. HPC pack customers were asking for it for quite a while. In this update, we introduced the support of the NVidia Tesla GPU So HPC Pack users can manage and monitor the supported HPC resources and schedule HPC jobs to fully leverage GPU resources.  For more details, please go to see the follow up blog post of How to fully use GPU resources to run GPGPU jobs in Microsoft HPC Pack and detailed GPU supports introduction article in TechNet.

About 6 months ago, we introduced Azure Linux VM support for HPC Pack in Update 2. Now with Update 3, HPC Pack starts to support Linux for on premise compute node. Customers running high-performance computing clusters on Linux can now use HPC Pack deployment, management and scheduling capabilities, and the user experience is very similar to the Windows nodes. For more details, please go to see the description of On-premises Linux support in TechNet.

Microsoft introduced Azure Batch Service months ago, it provides high scalable job scheduling and compute management service. From this update, HPC Pack is able to deploy Azure Batch pools from the Head Node and treat the compute resource pool as a “fat” node in the system and thus batch jobs and tasks can be scheduled on those pool nodes. For more information about Azure Batch support with HPC Pack, please visit burst azure batch from Microsoft HPC Pack

There are also great improvements introduced, for example:

SOA improvements – this release includes a better mapping for broker worker logs with sessions, Support Excel running under console session by default and provide SOA job view in web portal.

Auto Grow/Shrink supports SOA workloads - Now the service can grow nodes based on the outstanding calls in a SOA job instead of task numbers.

Per instance heat maps - Originally we only have aggregated instance heat maps but now you can view per instance heat maps through the “Overlaying view”

Customizable Idle Detection Logic - Originally the workstation nodes or the unmanaged server nodes are treated as idle through Keyboard/Mouse detection or CPU usage for process other than HPC ones. In This update we provide several customizable ways to detect whether these machines are idle.

Remove the limitation on the number of parametric sweep task in a job. HPC Pack used to have a limitation of 100 tasks in a parametric sweep job, now the limitation is removed

Microsoft MPI 7.0 included.

To learn more about the detailed description and how to get the new package, please visit the what’s new and release notes. As always, we look forward to hearing from you. You can get in touch with us via the Windows HPC Forum, or you can email us directly.

MS-MPI v7 Beta is the pre-release version for the successor to MS-MPI v6 (v6.0.12436, released on 5/12/2015).

You can download a copy of MS-MPI v7 Beta here

This Beta version of MS-MPI v7 includes the following features, improvements, and fixes:

• Support for the following nonblocking collective operations: MPI_Iallreduce, MPI_Iscatter, MPI_Iallgather, MPI_Iscatterv, MPI_Igatherv (in addition to the already supported MPI_Ibcast, MPI_Ireduce, MPI_Igather, and MPI_Ibarrier). For more details on these APIs please refer to the MPI 3.1 Standard document here

• Support for configuring port range used for Network Direct connections using the MSMPI_ND_PORT_RANGE environment variable

• Revamped process management (mpiexec/smpd) that will offer better reliability and performance

• Performance improvements for collective operations

This Beta version of MS-MPI v7 will also introduce the MS-MPI Launch Service, which allows the launching of MPI processes for local and remote machines with user provided credentials. The MS-MPI redistributable installer will install the MS-MPI launch service under “Manual” mode. To start the service, run the following command:

sc start MSMPILaunchSvc [options].

There are two new options to mpiexec (-pwd and -savecreds) that will allow you to provide the necessary credentials for launching processes using the launch service.  The -pwd option allows specifying the password for the submitting users in non-interactive scenarios (e.g. using a script or scheduled task).  Note that in this case the password is provided in clear text.  The -savecreds option causes the provided credentials (if specified with -pwd) to be securely stored on all the hosts specified in the mpiexec command line.

For example, the following command will prompt for the user's password and ask the user if he or she wants to store it on the specified machines "host1" and "host2", then launch three processes, one on the machine named “host1” and two on the machine named “host2”:

mpiexec –hosts 2 host1 1 host2 2 –wdir C:\MpiTests mpiapp.exe [parameters]

As another example, the following command will authenticate the user using the provided password, store it on the specified machines "host1" and "host2", then launch three processes, one on the machine named “host1” and two on the machine named “host2”:

mpiexec –hosts 2 host1 1 host2 2 –wdir C:\MpiTests -pwd <password> -savecreds mpiapp.exe [parameters]

Once you run mpiexec with the -savecreds option on a set of nodes, you will not have to provide the password on subsequent runs for those nodes unless the password is changed. If the user is running in interactive mode, mpiexec will prompt for the password if the launch service is running and the password has not been provided or previously saved with the -savecreds option.

These are the options you can pass to the launch service at service start:

-p | -port <port> Change the port that launch service is listening on.

-g | -group <group name> Only allow members of the specified group to run MPI applications.  If not specified, the default is authenticated users.

Important: This software is a pre-release version for evaluation purposes only, and it should not be used in a production environment. Features and behavior may change before the final release.

Note: The SDK components for MS-MPI (headers and libraries) ship separately from the redistributable package binary files. The SDK components are also available in this Beta release.

To learn more about MS-MPI and ways to contact us, see Microsoft MPI on MSDN