How to force docker to pull image with every batch task execution?

Nick Holt 5

The documentation for the container_run_options input for the TaskContainerSettings class [link] describe it as

These additional options are supplied as arguments to the "docker create" command, in addition to those controlled by the Batch Service.

However, when I provide the --pull=always option, I get the following error message from my batch compute node:

ContainerInvalidSettings: At least one value of specified task container settings is invalid

Message: Some supplied docker options are not recognized or are unsupported: --pull, always

As far as I'm aware, the --pull always option has been available for several years as a docker run/create option.

I've tried reaching out to support for this issue and I did not get much of an explanation as to why this setting does not work. What's even more confusing is if I SSH directly on to the compute node, I can use the --pull=always option.

How can I ensure that the latest docker image is pulled before the execution of of my command on the batch compute node? This is a concern for our development workflow as we tend to use the latest tag, and if we have a long running batch compute node there is a risk it will use a stale image when creating the container.

Prrudram-MSFT 21,886 Reputation points

2023-03-11T09:34:13.86+00:00
Hello @Nick Holt

As per the information in the documents, container_run_options input for the TaskContainerSettings class is used to provide additional options to the docker create command. The --pull=always option is not supported by the Azure Batch service.

To ensure that the latest Docker image is pulled before the execution of your command on the Batch compute node, you can use the image_name input for the TaskContainerSettings class to specify the latest image. The image_name input is used to specify the Docker image to use for the task.

For example, if you want to use the latest version of the tensorflow/tensorflow image, you can specify the following:

task_container_settings = batch.models.TaskContainerSettings( image_name='tensorflow/tensorflow:latest' )

This will ensure that the latest version of the tensorflow/tensorflow image is used for the task.

You can also use the import_py2package_from_pypi runbook to import the latest version of a Python 2 package from PyPI into your Azure Automation account. The runbook requires a Run As account for the Automation account to work.

The runbook requires the following parameters:

-s <subscriptionId>

-g <resourceGroup>

-a <automationAccount>

-m <modulePackage>

You can use the imported package in a runbook by adding the following code to list all the resource groups in an Azure subscription:

import azure.mgmt.resource import automationassets from azure_automation_utility import get_automation_runas_credential # Authenticate to Azure using the Azure Automation RunAs service principal runas_connection = automationassets.get_automation_connection("AzureRunAsConnection") azure_credential = get_automation_runas_credential() # Intialize the resource manager client resource_client = azure.mgmt.resource.ResourceManagementClient( azure_cred

Adding the document sources that may help.

https://azure.microsoft.com/en-us/blog/running-docker-container-on-azure-batch/

https://learn.microsoft.com/en-us/azure/container-registry/container-registry-tutorial-quick-task

Do let me know if this information is helpful.
Nick Holt 5 Reputation points

2023-03-13T15:24:37.75+00:00

The help article about container workloads on batch specifically mentions the docker create documentation for additional options. How can we know which options are available and which are not without trying?

https://learn.microsoft.com/en-us/azure/batch/batch-docker-container-workloads#container-task-command-line

I will do some testing with the latest tag on my image to report on a few behaviours based on your response.
Prrudram-MSFT 21,886 Reputation points

2023-03-13T18:39:01.17+00:00

@Nick Holt Sure. Thanks for the update.
Nick Holt 5 Reputation points

2023-05-09T13:48:37.2266667+00:00

In case this is useful for anyone in the future, the --pull=always option is not supported by Azure Batch. My solution was to create a special batch pool for development jobs that is allowed to scale down to zero. When I update my re-usable docker tag, "latest", I just need to ensure that the pool has scaled back down to zero since my last development test.
Timmy Schweer 0 Reputation points

2024-02-28T13:36:10.93+00:00

What a boomer.. So when testing a docker Image for Azure Batch I do not only have to wait until my docker image is build and pushed into the container registry I also have to stop and start the virtual machine. That is a feedback time of ~10 Minutes which really bad.