Azure Batch - Using Python with MPI

AJG 436

Does anyone know what is the best way to set things up to run parallel Python programs that use MPI on Azure Batch?

There are obviously a number of things to consider - such as the mix of packages, how to deploy in a simple way, etc

It looks like this technology combination is used in Azure ML - but I dont know if this uses customized libraries - rather than a standard toolset.

are there any examples that show the best way to do this?

1 answer

Winston 2,761 Reputation points

2020-09-01T18:10:34.403+00:00

Hi @AJG ,

I'd advise you to take a look at this sample here. This tutorial is a python application that uses Azure Batch to run a simple python program that submits a job to the pool, and then submits a task which performs a simple MPI pingpong command. If this is a good example for you please mark this answer as accepted to help other community members. Thanks!
Please sign in to rate this answer.
AJG 436 Reputation points

2020-09-02T02:11:52.84+00:00

Hello,

all that example does is create a bunch of tasks that run ffmpeg independently on the command line - this is only 'parallel processing' in the very weakest and trivial sense.

I specifically referenced Python and MPI on Azure - because I am interested in running python scripts that use MPI on azure.

this is different from using python as the control/orchestration language from which to control the azure batch environment - for my purposes the language used for that is more or less irrelevant.

any non trivial distributed analytics require inter-process communications which mediates distributed workflow - MPI is the most common technology used to achieve that - across a number of technology stacks.

so, what I am interested is a case where python code is running on the compute nodes, and this code uses MPI - this is not over complicated to do in an environment that you have total control over - the question is how to do it in azure.

Winston 2,761 Reputation points

2020-09-02T16:55:21.97+00:00

Hi @AJG , I updated my original answer with a link to a more fitting example. Does that example help you more? You can find more Azure Batch Python Examples here.
Sign in to comment