Previous

Next

Exercise - Create an Azure Batch job in the CLI to run your parallel task

Completed

Azure CLI lets you create and deploy Azure Batch resources from the command line.

As the solution architect for the non-profit, you want to start creating a Batch solution for the OCR scenario on Azure. As a first step, you're going to test creating a Batch account, a pool of worker nodes, a job, and tasks, all using the Azure CLI.

The Azure CLI is a cross-platform command line for creating and managing Azure resources. It uses a consistent az noun verb --properties syntax. CLI commands to manage Azure Batch follow this convention, and so all commands take the form of az batch verb --properties.

Important

The exercises in this module are optional. To complete the exercises, you'll need your own Azure subscription.

Create the Azure Batch resources

First, you'll create the Azure Batch account and pool of worker nodes for your test scenario. You'll then create a Batch job and tasks to run in the pool.

Create a new resource group

1. Sign into the Azure portal.

2. Select Resource groups in the left menu.

3. Select + Create.

4. Complete the form to create a new resource group for the exercise and note the name.

Create the Azure Batch account

1. In portal, open the Cloud Shell by selecting the Cloud Shell icon (>_) in the top menu. Use the drop-down menu at the top of the terminal window to make sure you're in Bash.

2. Create environment variables for your resource group and batch resource.

   RESOURCE_GROUP=<your resource group>
   BATCH_ACCOUNT=batchaccount$RANDOM
   

3. Run the following command in the Cloud Shell to create an Azure Batch account. Choose from one of the following locations:

   westeurope,eastus,eastus2,westus,northcentralus,brazilsouth,northeurope,centralus,eastasia,japaneast,australiasoutheast,japanwest,koreasouth,koreacentral,southeastasia,southcentralus,australiaeast,southindia,centralindia,westindia,canadacentral,canadaeast,uksouth,ukwest,westcentralus,westus2,francecentral,southafricanorth
   

   az batch account create \
    --name $BATCH_ACCOUNT \
    --resource-group $RESOURCE_GROUP \
    --location <choose a location from the preceding list>
   

4. Run the following command to sign in to your Azure Batch account:

   az batch account login \
    --name $BATCH_ACCOUNT \
    --resource-group $RESOURCE_GROUP \
    --shared-key-auth
   

5. Create a pool of three Standard A1 v2 VMs running Ubuntu 18.04:

   az batch pool create \
    --id mypool --vm-size Standard_A1_v2 \
    --target-dedicated-nodes 3 \
    --image canonical:ubuntuserver:18.04-LTS \
    --node-agent-sku-id "batch.node.ubuntu 18.04"  
   

6. Run the following command to periodically determine whether the nodes are ready:

   az batch pool show --pool-id mypool \
    --query "allocationState"
   

   When this query shows that the nodes are "steady," they're ready to work. While they show as "resizing," they're actually being provisioned. You can use the up arrow key to repeat the most recent command in the Cloud Shell if you want to query the status of the nodes more than once.

7. Run the following command to create a Batch job, which is a logical grouping for all the tasks that will run on your nodes:

   az batch job create \
    --id myjob \
    --pool-id mypool
   

8. Run the following command to create Batch tasks:

   for i in {1..10}
   do
      az batch task create \
       --task-id mytask$i \
       --job-id myjob \
       --command-line "/bin/bash -c 'echo \$(printenv | grep \AZ_BATCH_TASK_ID) processed by; echo \$(printenv | grep \AZ_BATCH_NODE_ID)'"
   done
   

   This loop creates a series of numbered Batch tasks, each of which outputs information about the task ID and node ID involved. By examining what's happening here closely, you can see quite a few characteristics of the way Azure Batch works. Tasks get automatically scheduled onto nodes with available capacity. However, the environment variables we're using (AZ_BATCH_TASK_ID and AZ_BATCH_NODE_ID), to show information about which task is being run on which node, are only available in the context of the task user, a default account created on each node in order to run tasks. Also, the --command-line argument doesn't directly run under a shell, so you have to explicitly invoke the shell using /bin/bash -c in order to use commands like printenv and grep.

   This loop completes the Azure Batch job implementation. We'll leave the Azure Batch account and pool of worker nodes in place for use in the next module, but we'll delete the Batch job. Because the Batch job acts as a container for its Batch tasks, the tasks are also deleted.

9. Run the following command in the Cloud Shell in order to delete the Batch job:

   az batch job delete --job-id myjob -y
   

Continue