Azure 4 Research

Cloud computing guide for researchers – Automated testing using Drone

Dr Kenji Takeda — Tue, 06 Mar 2018 15:35:47 GMT

This is the first in a series of blog posts by the Research Software Engineering team at Imperial College London describing activities funded by their RSE Cloud Computing Award. The team is exploring the use of selected Microsoft Azure services to accelerate the delivery of RSE projects via a cloud-first approach. Mark Woodbridge, Research Software Engineering Team Lead, Imperial College London (Reproduced with permission from https://wwwf.imperial.ac.uk/blog/research-software-engineering/2018/02/13/cloud-first-simple-automated-testing-using-drone/) A great way to explore an unfamiliar cloud platform is to deploy a familiar tool and compare the process with that used for an on-premise installation. In this case we'll set up an open source continuous delivery system (Drone) to carry out automated testing of a simple Python project hosted on GitHub. Drone is not as capable or flexible as alternatives like Jenkins (which we'll consider in a subsequent post) but it's a lot simpler and a suitable example of a self-contained webapp for our purposes of getting started with Azure. We'll be automatically testing this repository, containing a trivial Python 3 project with a single test which can be run via python -m unittest. We add a single YAML file to the repository to configure Drone accordingly. There are then just three (short!) steps to get Drone testing the repository whenever code is pushed to GitHub. You don't need anything except a web browser and an Azure account: 1: Create an Azure VM where we'll install Drone You can do this via the Azure Portal but we'll use the new Azure Cloud Shell as it's quicker – and easier to document, which is important for reproducibility. Drone is distributed as a Docker image so we'll provision a minimal Container Linux VM to host it. We need to create a resource group, add the VM, give it a public DNS name (you will need to choose your own, instead of my-ci-server) and enable HTTP(S) access:

az group create -l westeurope --name my-rg

az vm create --name my-ci-server --resource-group my-rg --image CoreOS:CoreOS:Stable:1632.2.1 --generate-ssh-keys --size Basic_A0

az network public-ip update --name my-ci-serverPublicIP --resource-group my-rg --dns-name my-ci-server

az network nsg rule create --resource-group my-rg --nsg-name my-ci-serverNSG --name HTTP --destination-port-ranges 80 --priority 1010

az network nsg rule create --resource-group my-rg --nsg-name my-ci-serverNSG --name HTTPS --destination-port-ranges 443 --priority 1020

2: Register a new OAuth application in GitHub In order to provide Drone with access to the repository (or repositories) we want to test, visit this page and enter the following, replacing the hostname appropriately:

Application name: Drone
Homepage URL: https://my-ci-server.westeurope.cloudapp.azure.com
Authorization callback URL: https://my-ci-server.westeurope.cloudapp.azure.com/authorize

Save the Client ID and Client Secret for the next step 3: Install and configure Drone Run the following commands back in the Cloud Shell. You again need to replace the hostname, and also provide your GitHub username and the Client ID and Secret from the previous step.

ssh my-ci-server.westeurope.cloudapp.azure.com

sudo docker run -d --name drone-server -e DRONE_HOST=https://my-ci-server.westeurope.cloudapp.azure.com -e DRONE_ADMIN=mwoodbri -e DRONE_GITHUB=true -e DRONE_GITHUB_CLIENT=xxxxxxxxxxxxxxxxxxxx -e DRONE_GITHUB_SECRET=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx -e DRONE_LETS_ENCRYPT=true -v drone:/var/lib/drone/ -p 80:80 -p 443:443 --restart=unless-stopped drone/drone

sudo docker run -d --name drone-agent --link drone-server -e DRONE_SERVER=drone-server:9000 -v /var/run/docker.sock:/var/run/docker.sock --restart=unless-stopped drone/agent

Then visit https://my-ci-server.westeurope.cloudapp.azure.com and toggle the switch next to the name of the relevant repository. Next steps Drone is now monitoring the code for changes, and will run the test suite in response. If we deliberately break our unit test by making this change and pushing the code then Drone will immediately run the code and identify a problem: It will also annotate the commit as bad and provide us with a badge that can be dynamically embedded in our README.md. We can then go onto configure Drone to notify us via email, Slack etc of failures using one of its many plugins. Summary We've seen how various features of the Azure platform, including Virtual Machines, Cloud Shell, and the extensive Marketplace can be combined with GitHub and Drone to rapidly deploy a secure, private CI system entirely from your browser. There exist alternative means of achieving the same result – not least various hosted, subscription based systems – and there are Azure recipes for Jenkins and Drone itself. However, the approach demonstrated here is applicable to any container-based software and therefore provides a flexible and efficient means of at least prototyping new services – via a cloud-first strategy.

Cloud computing guide for researchers – Understanding shipping behaviour with Microsoft Azure

Dr Kenji Takeda — Tue, 12 Dec 2017 10:02:51 GMT

Keeping track of global shipping has previously suffered from a lack of data. Current tracking technology has transformed the problem into one of an overabundance of information, as huge amounts of vessel tracking data are slowly becoming available, mostly due to the Automatic Identification System (AIS). Due to the volume of this data, traditional data mining and machine learning approaches are challenged when called upon to decipher the complexity of these environments. Scientists from the University of the Aegean, Dept. Product & Systems and Design Engineering and MarineTraffic (the leading vessel tracking website), are using data science in the cloud to transform billions of records of spatiotemporal (AIS) data into information for understanding the patterns of global trade by adopting distributed processing approaches. The research team led by Assoc Prof. Dimitris Zissis, is leveraging novel algorithmic approaches and Microsoft Azure, to transform Big Data into actionable information for a multitude of maritime stakeholders. Members of the team include: Giannis Spiliopoulos, Researcher at MarineTraffic; Konstantinos Chatzikokolakis, Senior Researcher at MarineTraffic; Elias Xidias, Senior Researcher at the University of the Aegean.

Why?

The importance of a well-developed understanding of the maritime traffic patterns and trade routes is critical to all seafarers and maritime stakeholders. Unlike roads, shipping lanes are not carved in stone. Their size, boundaries and content vary over space and time, under the influence of external sources. Today we only have a vague understanding of the specific routes vessels follow when travelling between ports, which is an essential metric for calculating any valid maritime statistics and indicators (e.g trade indicators, emissions and others). From a security perspective, it is necessary for understanding areas of high congestion, so that smaller vessels can avoid collisions with bigger ships. Moreover, an understanding of vessel patterns at scale can assist in the identification of anomalous behaviors and help predict the future location of vessels. Additionally, by combining ship routes with a model to estimate the emission of vessels (which depends on travel distance, speed, draught, weather conditions and characteristics of the vessel itself), emissions of e.g. CO2 and NOx can be estimated per ship and per national territory

From data to knowledge

Data is received through the MarineTraffic system and for the purposes of this work we use a dataset of approximately 5 billion messages recorded from January to December 2016. As the amount of available spatiotemporal data grows to massive scales, it is becoming clear that applying traditional techniques to AIS data processing can lead to processing times of several days, if applied to global data sets of considerable size (current datset over 500G). Due to this, for the distributed processing tasks, we rely on a HDInsight Azure Spark (2.1.0 ver-sion) cluster made up by: 6 worker nodes (D4v2 Azure nodes), each one equipped with 8 processing cores and 28 GB RAM; and 2 head nodes (D12 v2 Azure nodes), each one equipped with 4 processing cores and 28 GB RAM, summing up to a total of 56 computing cores and 224 GB RAM. The work is described in detail in Giannis Spiliopoulos, Dimitrios Zissis, Konstantinos Chatzikokolakis, A Big Data Driven Approach to Extracting Global Trade Patterns, Mobility Analytics for Spatio-temporal and Social Data with VLDB 2017, Munich 2017.

Global scale data analysis for researchers

Today, the growing number of distributed sensors and tracking systems are generating overwhelming amounts of high velocity spatiotemporal data. Processing these datasets is a highly complex task. Traditional state of the art techniques and technologies have proven incapable of dealing with such volumes of loosely structured spatiotemporal data. Towards building more efficient systems, methods of distributing processing and storage across a cluster of computers has been proposed. The trend towards more data (big data) is leading to changes in the computing paradigm, and in particular to the notion of computational approaches. A transition is currently taking place from a computing-centric model where data live on disk and are moved to a central processing unit for computational tasks towards a data-centric model, where computation is moved towards the data. Using services from Microsoft's Azure, the project harnesses the power of the cloud to perform complex analytics on these datasets. Thanks to Microsoft Azure the researchers are able to reduce the processing time from several days to just a few hours. Find out more on our Cloud Computing Guide for Researchers.

Need access to Microsoft Azure?

There are several ways you can get access to Microsoft Azure for your research. Your university may already make Azure available to you, so first port of call is to speak to your research computing department. There are also other ways for you to start experimenting with the cloud:

Try out Microsoft Azure for free including:

Cloud computing guide for researchers – Getting started with AI and Machine Learning

Dr Kenji Takeda — Fri, 17 Nov 2017 20:06:35 GMT

We are on the verge of the biggest revolution in computing since Alan Turing formalised the concepts of computation. Today software is hand-made, relying on highly-skilled craftspeople to intricately tell computers how to do useful work. A Cambrian explosion in computing is imminent, as artificial intelligence and machine learning are enabling computers to learn from data. But what is the reality beyond the hype? There is a huge range of techniques, tools, and technologies available now for you to use, without having to have a specialist PhD in AI or machine learning. There are great opportunities to accelerate and improve your research projects with AI across research domains such as chemistry, engineering, environmental and earth sciences, genomics, humanities, physics, and social sciences. The Microsoft AI platform is easy to use and perfect for all researchers - https://microsoft.com/ai. Here we show you how to get started with artifical intelligence (AI) and machine learning (ML), but this is only the tip of the iceberg in what you can achieve.

AI and ML made easy

Microsoft Azure provides powerful support for AI and machine learning for experts and beginners alike, be it TensorFlow or CNTK, Apache Spark or Data Bricks, scaling out on the latest GPUs. As the Microsoft AI platform is so extensive, it can be daunting to know where to start. So here are some suggestions to get you going, and grow your confidence to explore the full power of the Microsoft AI platform.

We love Jupyter notebooks, so try out the Jupyter Notebooks-as-a-Service on Azure. They are free, executable and shareable over the web. Organize your notebooks and datasets in one centralized location. Libraries are saved automatically and can be viewed from any device, anywhere. These are a great way to do research in a reproducible way. You can start from scratch or upload your existing notebooks to Azure at https://notebooks.azure.com Here is one that Matthew Johnson at Microsoft Research uses to teach deep learning at the University of Cambridge - https://aka.ms/jupyterdeeplearning
Azure has a huge range of different virtual machines for you to use in the cloud. There are a number of pre-built images (VMs with software pre-installed), and one of our favourites is the Data Science Virtual Machines, on Linux and Windows. You can create a basic VM (e.g. Ubuntu Linux) and install whatever software you like by following these instructions. We suggest that you start with a small VM (A1) to get started while you are getting to grips with the basics. You can easily rebuild/reboot a bigger VM (many cores and large memory) once you are familiar with the environment. There is even a deep learning virtual machine with Tensorflow, Caffe2, CNTK, Chainer pre-installed, along with Anaconda Python, Jupyter notebooks, and more tools that you can run on the most powerful GPUs.
Explore Azure Machine Learning that is a complete end-to-end, easy to use, web-based system to experiment with your own machine learning algorithms. It makes it easy to test and deploy machine learning models, including with your own Python and R code using standard libraries like Sci-Kit Learn. There are many walkthroughs and tutorials to get you started. See https://studio.azureml.net/

Image classification is a key computer vision technique that has many uses, and there are some excellent tools to help you build a full pipeline to create a dedicated classifier for you project. It is not easy though, and requires a high degree of expertise. The Custom Vision Service makes this much more straightforward, while using a state-of-the-art deep neural network behind the scenes to train a model based on your own input image data. You can build a remarkably accurate classifier in just a few minutes by following this short tutorial - https://docs.microsoft.com/en-us/azure/cognitive-services/custom-vision-service/getting-started-build-a-classifier. (This isjust one useful dataset you can use to play with - https://www.robots.ox.ac.uk/~vgg/data/pets/)

When you do need to build your own machine learning pipeline, training using many machines, then it can take a bit of work to create the infrastructure to do this, even in the cloud. Azure Batch AI provides you with a streamlined way of spinning a GPU cluster of machines for training your model with whatever AI framework you prefer. It makes it easy t:o provision clusters of GPUs or CPUs on demand; installing software in a container or with a script; automatic or manual scaling to manage costs; access to low priority virtual machines for learning and experimentation, and; mounting shared storage volumes for training and output data.

Machine Learning Services

When you are spending hours working on your AI and ML research, you want to have a smooth workflow and tools that make it really easy. The Machine Learning Workbench is a downloadable desktop application and command-line interface for Windows and MacOS that does just that. Built-in data preparation learns your data preparation steps as you perform them. Project management, run history, and notebook integration bolsters your productivity. Take advantage of the best open source frameworks, including TensorFlow, Cognitive Toolkit, Spark ML, and scikit-learn. You can then use the Experimentation Service to easily scale up on virtual machines or scale out using Spark clusters. Proactively manage model performance, identify the best model, and promote it using data-driven insights. Collaborate and share solutions using popular Git repositories. Running and developing models is streamlined, but versioning them, and tracking which ones are being used in production, versus experimentally, can be a real headache. Azure Machine Learning Model Management allows you to do: model versioning; tracking models in production; deploying models to production through AzureML Compute Environment with Azure Container Service and Kubernetes; creating Docker containers with the models and testing them locally, and; automated model retraining. You can also use AI tools for Visual Studio Code (on Mac, Linux, and Windows), to develop across desktop, cloud, and edge devices. This extension to VS Code allows you to build, test, and deploy Deep Learning / AI solutions. It features a seamless integration with Azure Machine Learning, notably a run history view, detailing the performance of previous trainings and custom metrics. It offers a samples explorer view, allowing to browse and bootstrap new project with Microsoft Cognitive Toolkit (previously known as CNTK), Google TensorFlow, and other deep-learning framework. Finally, it provides an explorer for compute targets, which enables you to submit jobs to train models on remote environments like Azure Virtual Machines or Linux servers with GPU.

AI School and LearnAnalytics@Microsoft

This only scratches the surface of what is possible with the Microsoft AI platform. We've created AI School so that you can dig deeper into more of what is possible. So have fun learning more about the art of the possible, and how easy it is at https://aischool.microsoft.com/learning-paths We also have specialist resources for data scientists with LearnAnalytics@MS. This includes webinars, on-demand videos, and catalogue of classroom (in-person) training.

Need access to Microsoft Azure?

Sign-up to a one month free trial here
There are several free Azure services for you to explore, listed here, and including:

Cloud computing guide for researchers – Research data management and collaboration

Dr Kenji Takeda — Thu, 14 Sep 2017 22:24:33 GMT

Research is a collaborative endeavour, but it's not always easy to share data, workflows and software with others in the lab, research group, or around the world. Cloud computing makes this much easier, by being able to host data, workbooks, and computing together in one place. You can share as much data as you like, and it's as easy as using Azure Storage Explorer, Python, or command line tools, as explained in this video, and this detailed walkthrough on Github. Jupyter notebooks are one example of how collaborative open science is developing, with Azure providing executable Notebooks-as-a-Service for free. Azure provides the ideal platform for creating services that make research data and outputs easily available. Data Commons and reproducible research platforms are ideally suited for implementation on the cloud, by bringing data, compute, and tools together for researchers to create results, collaborate, analyse data, share, archive and preserve their outputs.

Making Research Data Work

Making data available requires a combination of scalable infrastructure and tools that are researcher-friendly. Organisations such as the National Biodiversity Network and Ag-Analytics are using Microsoft Azure to provide seamless access to research data and tools. The ability to easily design and deliver end-to-end research data and collaboration tools means that teams can focus on creating the best experience for their users. A wide range of cloud technologies makes this easier than ever for research data management, including:

Azure storage, with a tiered model for optimising cost and efficiencies:
- Azure blob storage for scalable object storage of unstructured data. It includes blob object-level tiering Azure for optimizing costs without moving your data:
  - Hot storage tier frequently accessed data
  - Cool storage tier with same, fast access times as hot storage, at much reduced storage cost
  - Archive storage for low cost, durable, highly available, secure cloud storage for rarely accessed data with flexible latency requirements (on the order of hours).
- Azure Files. Simple, distributed, cross-platform replacement to network file share in the cloud
- StorSimple for automated, hybrid on-premise and Azure storage
Azure Data Lake Store for unstructured, semi-structuers and structured data, with no limit on size. Azure Data Lake Store is secured, massively scalable and built in accordance with the open HDFS standard, allowing you to run massively parallel analytics.

For structured relational databases, Azure offers Database-as-a-Service to allow you to easily deploy and manage systems without having to worry about the underlying infrastructure. This includes support for the most popular database systems, as well as new cloud-scale services.

Azure database Postgres and MySQL. This allows you to spend more time building your research management services by providing automatic database patching, automatic backups, built-in monitoring, security and more.
CosmosDB globally distributed, multi-model database service. It is flexible, with native MongoDB API support, support Gremlin graph APIs, and Spark connector or doing real-time machine learning.
Azure Tables, flexible NoSQL key-value store for semi-structured datasets

Jupyter Notebooks made easy – and free!

Jupyter notebooks are one example of how collaborative open science is developing, with Microsoft providing Notebooks-as-a-Service for free. These Jupyter notebooks on Azure, available as a service, fully executable, and shareable. These are a great way to do science in a reproducible way. You can start from scratch or upload your existing notebooks to Azure at https://notebooks.azure.com These can be used to share your science, such as the LIGO Open Science Centre. They have made data and analysis pipeline available using Azure Notebooks, so that researchers can reproduce the results of the gravitational waves discovery – see https://losc.ligo.org/tutorials/ You can also build more complex Jpyter notebook setups combining free Azure Notebooks, Jupyter hub on Azure virtual machines (including GPUs) and on-premise servers. See the UK Faculty Connection blog for more details.

Research-as-a-Service

Microsoft Azure means that you can not only host data, but build researcher-centric services to make this data work for users. For example, Nicola Bonzanni at ENPICOM uses Azure to build Conbind, phylogenetic fingerprinting Research-as-a-Service for genomics researcher worldwide. Read more about it here. Parker MacCready at the University of Washington has used Azure to complement his university's Linux HPC cluster, by building a cloud system to manage his regional climate simulation output data, and make it available to other as part of the LiveOcean project. Jan Newton, principal oceanographer at the Applied Physics Laboratory and co-director of the Washington Ocean Acidification Project (WOAP), believes it may change how the public sees climate change and ocean chemistry. She explains, "Data portals and models like LiveOcean can really make a bridge [of understanding] because even if people don't understand the chemistry, they'll look at the color-coding and see how this changes with location and season." You can read more about it here.

Learn more

There are more general Azure getting started videos at https://azure.microsoft.com/en-us/get-started/ and a full set of Azure for Research self-pace walkthroughs at https://aka.ms/learncloud We are publishing more in this blog series on more advanced topics for researchers to take advantage of Azure. So stay posted in our cloud computing guide for researchers.

Need access to Microsoft Azure?

Sign-up to a one month free trial here

There are several free Azure services for you to also explore:

Cloud computing guide for researchers – Real HPC with linear scaling on thousands of cores

Dr Kenji Takeda — Tue, 01 Aug 2017 18:18:45 GMT

If you need to run lots of computations, either as a large supercomputer simulation, or hundreds of smaller runs for a parameter sweep calculation, then cloud computing can really help speed up your research. Microsoft Azure Big Compute can scale to thousands of compute cores using Message Passing Interface (MPI) due to our HPC-optimised, InfiniBand-enabled virtual machines (VMs). The availability of essentially limitless compute capabilities, on-demand, means that you can achieve more, in less time. No need to queue for your job to run, have your job priority lowered because you've run too many jobs, or compete with other users on a shared system. You also have full flexibility to install any software you like, without having to wait for someone else to do it for you. You can do real HPC in the public cloud today.

Real HPC in the cloud

High performance computing (HPC) requires low-latency, high bandwidth dedicated networking between compute nodes, and Microsoft Azure has this. Our Azure A-Series (A8 & A9) and H-Series (H16) VMs, with InfiniBand, achieve near bare-metal HPC performance and can run LINPACK at over 90% efficiency. You can even run parallel GPU (Graphic Processing Unit) nodes with our N-Series, which also have low-latency, high-speed networking, e.g. NC24r VMs have 24 CPU cores, 4 x K80 GPUs, 224GB RAM and InfiniBand.It is easy to run tightly-coupled distributed software for applications fluid dynamics, molecular modelling, climate research that need specialized hardware to run efficiently.

Unlike other public clouds, scaling does not stop at a couple of hundred cores, but can scale well to thousands of cores on real-world applications.

We have tested many HPC applications such as OpenFOAM, Fluent, STAR-CCM+, NAMD, PAM-CRASH and other highly-parallel, MPI-based codes can scale well to over a thousand cores. For example, you can see here how Fluent computational fluid dynamics software performs on up to 1024 compute cores. This test case is what Formula One teams were routinely running just a few years ago, in their dedicated CFD facilities, and is now possible for anyone via Microsoft Azure from their laptop. It shows how scientists and engineers can have the same HPC capability from anywhere in the world. It opens a world of possibilities for those who do not have easy access to HPC facilities. It is equally empowering for researchers, universities and HPC Centres, as it provides extra flexibility, capability, and agility to complement existing HPC investments. A hybrid cloud HPC model enables service providers to give their users the best possible experience, and focus energy on accelerating time to insight and scientific impact.

Microsoft Azure Big Compute is flexible in how you want to use it for your HPC applications. How you do this will depend on your individual and organisational situation, experience, and what you are trying to achieve.

HPC cluster in the cloud

Setup your own HPC cluster in Azure as your own personal supercomputer, with whatever software you like on it. Having full administrator access to your cluster gives you full flexibility and control over your own research software stack. There is no need to wait for someone else to install your software and libraries before you can start computing. It is possible to setup different clusters for different users, with custom configurations of CPU, memory, network, and software. You can think differently about how to get research done, scaling the cluster, creating new ones, and tearing them down when you don't need them. You can see how to setup a SLURM cluster here.

Research applications in the cloud

Burst your research application into the cloud. You can easily enable a desktop application, command-line tool, or web-application/service to run at scale without leaving your familiar environment. Azure Batch manages the compute resources in the cloud, so you can transparently run your application from whichever environment is most comfortable for you. This transforms how you can do your research, extending your desktop to literally thousands of machines in the cloud. You can even use Docker containers with Azure Batch, with our open-source Batch Shipyard toolkit here, and a detailed walkthrough here.

Researchers at the University of Newcastle, UK, used Azure Batch to run a parameter study for high-resolution flood modelling across 571 cities in Europe. By running across 40 x A11 Azure VMs they were able to run all of their simulations in two days, instead of two and a half months. Read the details here.

Nicola Bonzanni at ENPICOM uses Azure to build Conbind, phylogenetic fingerprinting Research-as-a-Service for genomics researcher worldwide. Read more about it here.

HPC burst to the cloud

Augment your on-premise HPC cluster with the cloud, to provide additional capacity when you need it. It's a great way to accelerate research even more by providing on-demand extra resources. It is also a way you can move users who may not need specialised on-premise HPC hardware, such as for thousands of single core jobs. This can help put the right users on the right hardware, maximising investment, job throughput, and user satisfaction. You can use Microsoft HPC Pack, or many of the most commonly used HPC cluster tools, such as SLURM, Cycle Computing, Bright Computing, and Altair PBS Pro.

Learn more

You can find out more on our dedicated Azure Big Compute web pages:

Azure Batch and HPC documentation explains different scenarios in detail.
Batch Shipyard is a tool to help provision and execute batch processing and HPC Docker workloads on Azure Batch compute pools. No experience with the Azure Batch SDK is needed; run your Dockerized tasks with easy-to-understand configuration files! There is a detailed hands-on walkthrough for this on Github.
doAzureParallel – a lightweight R package built on top of Azure Batch, that allows you to easily use Azure's flexible compute resources right from your R session.
Azure Container Service makes it easy to orchestrate your work using DC/OS, Docker Swarm, or Kubernetes. There is a detailed hands-on walkthrough for this on Github

You can now use low-priority VMs with Azure Batch for non-time critical jobs, to significantly reduce your costs by up to 80%. Learn more about it here.

There are more general Azure getting started videos at https://azure.microsoft.com/en-us/get-started/ and a full set of Azure for Research self-pace walkthroughs at https://aka.ms/a4rgithub

We are publishing more in this blog series on more advanced topics for researchers to take advantage of Azure. So stay posted in our cloud computing guide for researchers.

Need access to Microsoft Azure?

Sign-up to a one month free trial here
Apply for an Azure for Research award. Microsoft Azure for Research awards offer large allocations of cloud computing for your research project, and already supports hundreds of researchers worldwide across all domains.

There are several free Azure services for you to also explore:

Cloud computing guide for researchers – Learn how to use Microsoft Azure for research and data science

Dr Kenji Takeda — Tue, 20 Jun 2017 10:01:33 GMT

Cloud computing can provide almost every piece of computing and data capability that you may require for your research. Whether it is a virtual machine with tons of memory; large clusters for high-performance computing, Hadoop or Spark; Stream processing for Internet of Things devices, or sharing your research data and methods with others. Microsoft Azure is easy to use, but you will need to spend a little bit of time learning how to use this wide array of technology. This article provides a jumping off point to the mountains of learning resources available online, on Github, and in-person training opportunities.

Are you experienced?

How you get started depends on what your current level of experience is with computing, and the cloud. Whether you're just starting out, are self-taught computational scientist, or experienced using other clouds, there are resources just for you. If you are not sure how Azure might fit your research, then this blog post will help orientate you on different ways the cloud can help you.

Beginners start here

The dedicated Azure for Research Online training portal provides detailed walkthroughs for using the Azure portal, Azure storage, HPC, Docker conatiners, machine learning, Spark/Hadoop HDInsight clusters, streaming analytics, data lake, and serverless computing (Azure Functions). You can use an existing Azure account, or a free trial, to run through this training. Sign-in so that you have the best learning experience. See https://aka.ms/learncloud
- All of the material is available on Github under an Open Source license without the need to sign-in, so please fork, contribute, and help us keep growing this resource for the community.

Confident users jump in here

The Azure for Research online portal is still a good place to start, as you should be able to work through the walkthrough quickly.

There are general Azure getting started videos, including launching virtual machines and handling storage at https://azure.microsoft.com/en-us/get-started/
Microsoft Imagine provides resources for Faculty, researchers and students on Github. There is a whole host of repositories there for you to explore - https://github.com/msftimagine
The Microsoft Azure training and certification website has many videos from our partner Pluralsight. Get started here - https://azure.microsoft.com/en-gb/community/training/
- There are dedicated MOOC courses – typically 16-24 hours – on dozens of topics including Devops, IoT, Identity, networking, and more - https://www.microsoft.com/en-us/learning/azure-skills-training.aspx

Azure for AWS users

If you know how to use AWS, then this guide is just for you - https://docs.microsoft.com/en-us/azure/architecture/aws-professional/
- This shows how AWS features map to Azure – https://docs.microsoft.com/en-us/azure/architecture/aws-professional/services
- Online documentation is here - https://azure.microsoft.com/en-us/documentation/
- In-depth online course here (16-24 hours) - https://azure.microsoft.com/en-us/documentation/
Official Azure documentation - https://azure.microsoft.com/en-us/documentation/

Doing data science

We have specialist resources for data scientists with LearnAnalytics@MS. This includes webinars, on-demand videos, and catalogue of classroom (in-person) training. There are a couple of data science services that are particularly easy to learn:

Try the Jupyter Notebooks-as-a-Service on Azure. They are free, executable and shareable over the web. Organize your notebooks and datasets in one centralized location. Libraries are saved automatically and can be viewed from any device, anywhere. These are a great way to do research in a reproducible way. You can start from scratch or upload your existing notebooks to Azure at https://notebooks.azure.com .
Explore Azure Machine Learning that is a complete end-to-end, easy to use, web-based system to experiment with your own machine learning algorithms. It makes it easy to test and deploy machine learning models, including with your own Python and R code using standard libraries like Sci-Kit Learn. There are many walkthroughs and tutorials to get you started. See https://studio.azureml.net/

Doing your research

We are publishing more in this blog series on more advanced topics for researchers to take advantage of Azure. So stay posted in our cloud computing guide for researchers. This interactive website is an easy way to find out more and navigate the plethora of Azure services available to you.

Get certified

If you would like to get certified with more formal Azure qualifications and training, then there are plenty of options to grow your career. The Microsoft training website has the full portfolio of training options for you. Here are some highlights:

Explore Microsoft certifications, from infrastructure, machine learning and open source here.
The Microsoft IT Pro Career Centre will help you map your cloud career path.
EdX provides many Microsoft and Azure online courses, that you can get certified for. See https://www.edx.org/school/microsoft

Need access to Microsoft Azure?

Several free services for you to explore:
Sign-up to a one month free trial here
Apply for an Azure for Research award. Microsoft Azure for Research awards offer large allocations of cloud computing for your research project, and already supports hundreds of researchers worldwide across all domains.

Cloud computing guide for researchers - Cloud technology for high-energy physics computing

Dr Kenji Takeda — Wed, 07 Jun 2017 02:27:59 GMT

Randall Sobie is the HEP Computing Group Leader at the University of Victoria. He works on ATLAS project, a collaboration of 3,000 international physicists. The project has collected hundreds of petabytes of data and Randall’s team faced the challenge of storing and accessing information on-premise. With the help of Microsoft Azure, Randall created a flexible cloud system for large workloads such as high-energy physics computing. The system has streamlined user workflow, sparking a digital transformation in the research community. "Azure basically gives us a solid, rock solid performance - no questions asked." Rolf Seuster, HEPnet Technical Manager, University of Victoria Play YouTube video here.

Cloud computing guide for researchers – Parameter sweep computation made easy with Azure Batch

Dr Kenji Takeda — Thu, 27 Apr 2017 19:21:37 GMT

When you need to go from running on your laptop or desktop computer, to run dozens, or hundreds, of computations, then cloud computing can literally be like a time machine. Azure Batch makes it easy to create many machines to simultaneously run your jobs, so you can get your results in few hours or days, rather than waiting weeks or months.

Accelerating science from 80 days to just two

Dr Vasilis Glennis and Professsor Chris Kilsby at the University of Newcastle, UK, have developed a highly-detailed flood model for cities. The City Catchment Analysis Tool (CityCAT) system combines topographic maps, building, and green space data, with a sewer and surface water model to be able to simulate how water accumulates when there is intense rainfall. Flood risk mapping and analysis with CityCAT allows: Identification of vulnerable areas; Identification of sources of runoff; Identifying opportunities for alleviation, and; Design and testing of schemes The initial high-resolution modelling of their home town showed good results, which they could check with data from a major flood that took place on 28th June 2012. The team's modelling is used in the European RAMSES research project, that aims to measure the impact of climate change, analysing the cost and benefits of different adaptation measures in cities. Knowing how flood risk changes under climate change, and can be reduced with interventions, is a key part of the project. The problem was that the researchers had to move from simulating one city, to 571 across Europe. Flood map for Newcastle under 150mm/hour rainfall scenario

Going from one to n machines is easy

The CityCat model can take two days to run on a single powerful computer. The team was worried that they would not be able to finish all their simulation in time, as they had to run nine different rainfall scenarios for each of the 571 cities across Europe. They turned to Microsoft Azure to setup a parameter sweep process, using the Azure Batch system. Azure Batch is a service that handles the provisioning, scheduling, and scaling of a pool of (virtual) machines, to complete a suite of jobs. The steps we used for the setup of the parameter sweep jobs were simple:

Uploading the data and the application to Azure storage;
Creating jobs and tasks for the pool of VMs; and
Calling a script for each task, with a set of parameters to configure and run the application.

Once a task is completed the results are uploaded to the cloud and the VMs terminated after the completion of the jobs. To perform the simulations for the 571 cities, a pool of 40 A11 virtual machines was created, each with 112 GB of RAM and 16 cores. By running the tasks in parallel on each virtual machine the total simulation run-time reduced from 80 days on a single machine, to approximately 2 days using Microsoft Azure.

Delivering results through Research-as-a-Service

To make this large set of results available the team setup an interactive Azure web site that shows simulation results for every European city at https://ceg-research.ncl.ac.uk/ramses/ By hosting the simulation results data in the cloud, it was easy to then create a compelling website to allow people to easily explore this research. You can read the full paper, entitled "Pluvial Flooding in European Cities—A Continental Approach to Urban Flood Modelling", published in the journal Water here.

Ramping up your calculations using Microsoft Azure

There are several other ways you can scale your workloads onto multiple machines, depending on what you are used to doing:

doAzureParallel – a lightweight R package built on top of Azure Batch, that allows you to easily use Azure's flexible compute resources right from your R session.
Batch Shipyard is a tool to help provision and execute batch processing and HPC Docker workloads on Azure Batch compute pools. No experience with the Azure Batch SDK is needed; run your Dockerized tasks with easy-to-understand configuration files! There is a detailed hands-on walkthrough for this on Github.
Azure Container Service makes it easy to orchestrate your work using DC/OS, Docker Swarm, or Kubernetes. There is a detailed hands-on walkthrough for this on Github

Need access to Microsoft Azure?

Sign-up to a one month free trial here
Apply for an Azure for Research award. Microsoft Azure for Research awards offer large allocations of cloud computing for your research project, and already supports hundreds of researchers worldwide across all domains.

There are several free Azure services for you to also explore:

Learn more:

Cloud computing guide for researchers – eBird big data analysis using Microsoft Azure

Dr Kenji Takeda — Mon, 06 Feb 2017 22:10:32 GMT

Citizen scientists all over the world have been helping the Cornell Lab of Ornithology to collect bird observations for 15 years using eBird. This real-time, online checklist program enables birders to record birds they see, keep track of lists, explore maps and graphs, share with the eBird community and contribute to ornithological science and bird conservation. For the researchers, creating high-resolution models using this data is very computationally-intensive. Even a single species model would take around three weeks to run on the mid-sized supercomputer systems available to the team. The researchers needed a more scalable computing platform to be able to generate results for the 700 bird species in North America, to be able to release to conservation staff. The team turned to the Microsoft Azure for Research program that provided them with the cloud computing resources they needed to move their existing R and Linux workloads to the cloud. They have built a truly scalable, future-proofed system, taking advantage of Microsoft Azure Storage, Microsoft Azure HDInsight Service, Microsoft R Server, Linux Ubuntu, and Apache Hadoop MapReduce and Spark. The team can now run models that used to take three weeks in just three hours. They no longer have to wait in a queue on their high performance computing cluster, but can use Microsoft Azure on-demand, any time of day or night. It is transforming how they do their research, and now enables them make their research results available to bird conservationists to help protect and improve bird populations. You can read the full story with technical details by Rashim Gupta here.

Cloud computing guide for researchers – Azure IoT for Croke Park Smart Stadium

Dr Kenji Takeda — Tue, 31 Jan 2017 12:13:22 GMT

Using sensors in a busy environment, such as a sports stadium, can unveil deep insights to provide a better experience for visitors to a game, the local community, and tourists wanting to enjoy the venue. Deploying sensors, collecting data, processing it, making sense of the information, and visualising it are key challenges that researchers at Dublin City University were tasked with solving at Croke Park Stadium in Dublin, Ireland, working with Intel and Microsoft’s technical team. The team used Azure IoT Suite to easily build an end-to-end solution gathering data from sound sensors, and a weather station, and providing analysed information into dashboards via Power BI. The system can be used to provide fans with fun experiences and encouraging the home team with louder cheering! It also helps to monitor for sound pollution, improving relationships with the local community. The weather station improves health and safety for the fantastic Etihad Skyline tour. You can read the full story by Niall Moran here, which goes into the technical architecture and implementation details.