Image processing with batch model deployments
APPLIES TO:
Azure CLI ml extension v2 (current)
Python SDK azure-ai-ml v2 (current)
Batch model deployments can be used for processing tabular data, but also any other file type like images. Those deployments are supported in both MLflow and custom models. In this tutorial, we will learn how to deploy a model that classifies images according to the ImageNet taxonomy.
About this sample
The model we are going to work with was built using TensorFlow along with the RestNet architecture (Identity Mappings in Deep Residual Networks). A sample of this model can be downloaded from here. The model has the following constrains that is important to keep in mind for deployment:
- It works with images of size 244x244 (tensors of
(224, 224, 3)). - It requires inputs to be scaled to the range
[0,1].
The information in this article is based on code samples contained in the azureml-examples repository. To run the commands locally without having to copy/paste YAML and other files, clone the repo, and then change directories to the cli/endpoints/batch/deploy-models/imagenet-classifier if you are using the Azure CLI or sdk/python/endpoints/batch/deploy-models/imagenet-classifier if you are using our SDK for Python.
git clone https://github.com/Azure/azureml-examples --depth 1
cd azureml-examples/cli/endpoints/batch/deploy-models/imagenet-classifier
Follow along in Jupyter Notebooks
You can follow along this sample in a Jupyter Notebook. In the cloned repository, open the notebook: imagenet-classifier-batch.ipynb.
Prerequisites
Before following the steps in this article, make sure you have the following prerequisites:
An Azure subscription. If you don't have an Azure subscription, create a free account before you begin. Try the free or paid version of Azure Machine Learning.
An Azure Machine Learning workspace. If you don't have one, use the steps in the How to manage workspaces article to create one.
Ensure you have the following permissions in the workspace:
Create/manage batch endpoints and deployments: Use roles Owner, contributor, or custom role allowing
Microsoft.MachineLearningServices/workspaces/batchEndpoints/*.Create ARM deployments in the workspace resource group: Use roles Owner, contributor, or custom role allowing
Microsoft.Resources/deployments/writein the resource group where the workspace is deployed.
You will need to install the following software to work with Azure Machine Learning:
The Azure CLI and the
mlextension for Azure Machine Learning.az extension add -n mlNote
Pipeline component deployments for Batch Endpoints were introduced in version 2.7 of the
mlextension for Azure CLI. Useaz extension update --name mlto get the last version of it.
Connect to your workspace
The workspace is the top-level resource for Azure Machine Learning, providing a centralized place to work with all the artifacts you create when you use Azure Machine Learning. In this section, we'll connect to the workspace in which you'll perform deployment tasks.
Pass in the values for your subscription ID, workspace, location, and resource group in the following code:
az account set --subscription <subscription>
az configure --defaults workspace=<workspace> group=<resource-group> location=<location>
Image classification with batch deployments
In this example, we are going to learn how to deploy a deep learning model that can classify a given image according to the taxonomy of ImageNet.
Create the endpoint
First, let's create the endpoint that will host the model:
Decide on the name of the endpoint:
ENDPOINT_NAME="imagenet-classifier-batch"
The following YAML file defines a batch endpoint:
endpoint.yml
$schema: https://azuremlschemas.azureedge.net/latest/batchEndpoint.schema.json
name: imagenet-classifier-batch
description: A batch endpoint for performing image classification using a TFHub model ImageNet model.
auth_mode: aad_token
Run the following code to create the endpoint.
az ml batch-endpoint create --file endpoint.yml --name $ENDPOINT_NAME
Registering the model
Model deployments can only deploy registered models so we need to register it. You can skip this step if the model you are trying to deploy is already registered.
Downloading a copy of the model:
Register the model:
Creating a scoring script
We need to create a scoring script that can read the images provided by the batch deployment and return the scores of the model. The following script:
- Indicates an
initfunction that load the model usingkerasmodule intensorflow. - Indicates a
runfunction that is executed for each mini-batch the batch deployment provides. - The
runfunction read one image of the file at a time - The
runmethod resizes the images to the expected sizes for the model. - The
runmethod rescales the images to the range[0,1]domain, which is what the model expects. - It returns the classes and the probabilities associated with the predictions.
code/score-by-file/batch_driver.py
import os
import numpy as np
import pandas as pd
import tensorflow as tf
from os.path import basename
from PIL import Image
from tensorflow.keras.models import load_model
def init():
global model
global input_width
global input_height
# AZUREML_MODEL_DIR is an environment variable created during deployment
model_path = os.path.join(os.environ["AZUREML_MODEL_DIR"], "model")
# load the model
model = load_model(model_path)
input_width = 244
input_height = 244
def run(mini_batch):
results = []
for image in mini_batch:
data = Image.open(image).resize(
(input_width, input_height)
) # Read and resize the image
data = np.array(data) / 255.0 # Normalize
data_batch = tf.expand_dims(
data, axis=0
) # create a batch of size (1, 244, 244, 3)
# perform inference
pred = model.predict(data_batch)
# Compute probabilities, classes and labels
pred_prob = tf.math.reduce_max(tf.math.softmax(pred, axis=-1)).numpy()
pred_class = tf.math.argmax(pred, axis=-1).numpy()
results.append([basename(image), pred_class[0], pred_prob])
return pd.DataFrame(results)
Tip
Although images are provided in mini-batches by the deployment, this scoring script processes one image at a time. This is a common pattern as trying to load the entire batch and send it to the model at once may result in high-memory pressure on the batch executor (OOM exeptions). However, there are certain cases where doing so enables high throughput in the scoring task. This is the case for instance of batch deployments over a GPU hardware where we want to achieve high GPU utilization. See High throughput deployments for an example of a scoring script that takes advantage of it.
Note
If you are trying to deploy a generative model (one that generates files), please read how to author a scoring script as explained at Deployment of models that produces multiple files.
Creating the deployment
One the scoring script is created, it's time to create a batch deployment for it. Follow the following steps to create it:
Ensure you have a compute cluster created where we can create the deployment. In this example we are going to use a compute cluster named
gpu-cluster. Although it's not required, we use GPUs to speed up the processing.We need to indicate over which environment we are going to run the deployment. In our case, our model runs on
TensorFlow. Azure Machine Learning already has an environment with the required software installed, so we can reutilize this environment. We are just going to add a couple of dependencies in aconda.ymlfile.Now, let create the deployment.
To create a new deployment under the created endpoint, create a
YAMLconfiguration like the following. You can check the full batch endpoint YAML schema for extra properties.$schema: https://azuremlschemas.azureedge.net/latest/batchDeployment.schema.json endpoint_name: imagenet-classifier-batch name: imagenet-classifier-resnetv2 description: A ResNetV2 model architecture for performing ImageNet classification in batch type: model model: azureml:imagenet-classifier@latest compute: azureml:gpu-cluster environment: name: tensorflow27-cuda11-gpu image: mcr.microsoft.com/azureml/curated/tensorflow-2.7-ubuntu20.04-py38-cuda11-gpu:latest conda_file: environment/conda.yaml code_configuration: code: code/score-by-file scoring_script: batch_driver.py resources: instance_count: 2 settings: max_concurrency_per_instance: 1 mini_batch_size: 5 output_action: append_row output_file_name: predictions.csv retry_settings: max_retries: 3 timeout: 300 error_threshold: -1 logging_level: infoThen, create the deployment with the following command:
az ml batch-deployment create --file deployment-by-file.yml --endpoint-name $ENDPOINT_NAME --set-defaultAlthough you can invoke a specific deployment inside of an endpoint, you will usually want to invoke the endpoint itself, and let the endpoint decide which deployment to use. Such deployment is named the "default" deployment. This gives you the possibility of changing the default deployment - and hence changing the model serving the deployment - without changing the contract with the user invoking the endpoint. Use the following instruction to update the default deployment:
az ml batch-endpoint update --name $ENDPOINT_NAME --set defaults.deployment_name=$DEPLOYMENT_NAMEAt this point, our batch endpoint is ready to be used.
Testing out the deployment
For testing our endpoint, we are going to use a sample of 1000 images from the original ImageNet dataset. Batch endpoints can only process data that is located in the cloud and that is accessible from the Azure Machine Learning workspace. In this example, we are going to upload it to an Azure Machine Learning data store. Particularly, we are going to create a data asset that can be used to invoke the endpoint for scoring. However, notice that batch endpoints accept data that can be placed in multiple type of locations.
Let's download the associated sample data:
Now, let's create the data asset from the data just downloaded
Create a data asset definition in
YAML:imagenet-sample-unlabeled.yml
$schema: https://azuremlschemas.azureedge.net/latest/data.schema.json name: imagenet-sample-unlabeled description: A sample of 1000 images from the original ImageNet dataset. Download content from https://azuremlexampledata.blob.core.windows.net/data/imagenet-1000.zip. type: uri_folder path: dataThen, create the data asset:
az ml data create -f imagenet-sample-unlabeled.ymlNow that the data is uploaded and ready to be used, let's invoke the endpoint:
JOB_NAME=$(az ml batch-endpoint invoke --name $ENDPOINT_NAME --input azureml:imagenet-sample-unlabeled@latest --query name -o tsv)Note
The utility
jqmay not be installed on every installation. You can get instructions in this link.Tip
Notice how we are not indicating the deployment name in the invoke operation. That's because the endpoint automatically routes the job to the default deployment. Since our endpoint only has one deployment, then that one is the default one. You can target an specific deployment by indicating the argument/parameter
deployment_name.A batch job is started as soon as the command returns. You can monitor the status of the job until it finishes:
Once the deployment is finished, we can download the predictions:
The output predictions will look like the following. Notice that the predictions have been combined with the labels for the convenience of the reader. To know more about how to achieve this see the associated notebook.
import pandas as pd score = pd.read_csv("named-outputs/score/predictions.csv", header=None, names=['file', 'class', 'probabilities'], sep=' ') score['label'] = score['class'].apply(lambda pred: imagenet_labels[pred]) scorefile class probabilities label n02088094_Afghan_hound.JPEG 161 0.994745 Afghan hound n02088238_basset 162 0.999397 basset n02088364_beagle.JPEG 165 0.366914 bluetick n02088466_bloodhound.JPEG 164 0.926464 bloodhound ... ... ... ...
High throughput deployments
As mentioned before, the deployment we just created processes one image a time, even when the batch deployment is providing a batch of them. In most cases this is the best approach as it simplifies how the models execute and avoids any possible out-of-memory problems. However, in certain others we may want to saturate as much as possible the utilization of the underlying hardware. This is the case GPUs for instance.
On those cases, we may want to perform inference on the entire batch of data. That implies loading the entire set of images to memory and sending them directly to the model. The following example uses TensorFlow to read batch of images and score them all at once. It also uses TensorFlow ops to do any data preprocessing so the entire pipeline will happen on the same device being used (CPU/GPU).
Warning
Some models have a non-linear relationship with the size of the inputs in terms of the memory consumption. Batch again (as done in this example) or decrease the size of the batches created by the batch deployment to avoid out-of-memory exceptions.
Creating the scoring script:
code/score-by-batch/batch_driver.py
import os import numpy as np import pandas as pd import tensorflow as tf from tensorflow.keras.models import load_model def init(): global model global input_width global input_height # AZUREML_MODEL_DIR is an environment variable created during deployment model_path = os.path.join(os.environ["AZUREML_MODEL_DIR"], "model") # load the model model = load_model(model_path) input_width = 244 input_height = 244 def decode_img(file_path): file = tf.io.read_file(file_path) img = tf.io.decode_jpeg(file, channels=3) img = tf.image.resize(img, [input_width, input_height]) return img / 255.0 def run(mini_batch): images_ds = tf.data.Dataset.from_tensor_slices(mini_batch) images_ds = images_ds.map(decode_img).batch(64) # perform inference pred = model.predict(images_ds) # Compute probabilities, classes and labels pred_prob = tf.math.reduce_max(tf.math.softmax(pred, axis=-1)).numpy() pred_class = tf.math.argmax(pred, axis=-1).numpy() return pd.DataFrame( [mini_batch, pred_prob, pred_class], columns=["file", "probability", "class"] )Tip
- Notice that this script is constructing a tensor dataset from the mini-batch sent by the batch deployment. This dataset is preprocessed to obtain the expected tensors for the model using the
mapoperation with the functiondecode_img. - The dataset is batched again (16) send the data to the model. Use this parameter to control how much information you can load into memory and send to the model at once. If running on a GPU, you will need to carefully tune this parameter to achieve the maximum utilization of the GPU just before getting an OOM exception.
- Once predictions are computed, the tensors are converted to
numpy.ndarray.
- Notice that this script is constructing a tensor dataset from the mini-batch sent by the batch deployment. This dataset is preprocessed to obtain the expected tensors for the model using the
Now, let create the deployment.
To create a new deployment under the created endpoint, create a
YAMLconfiguration like the following. You can check the full batch endpoint YAML schema for extra properties.$schema: https://azuremlschemas.azureedge.net/latest/batchDeployment.schema.json endpoint_name: imagenet-classifier-batch name: imagenet-classifier-resnetv2 description: A ResNetV2 model architecture for performing ImageNet classification in batch type: model model: azureml:imagenet-classifier@latest compute: azureml:gpu-cluster environment: name: tensorflow27-cuda11-gpu image: mcr.microsoft.com/azureml/curated/tensorflow-2.7-ubuntu20.04-py38-cuda11-gpu:latest conda_file: environment/conda.yaml code_configuration: code: code/score-by-batch scoring_script: batch_driver.py resources: instance_count: 2 tags: device_acceleration: CUDA device_batching: 16 settings: max_concurrency_per_instance: 1 mini_batch_size: 5 output_action: append_row output_file_name: predictions.csv retry_settings: max_retries: 3 timeout: 300 error_threshold: -1 logging_level: infoThen, create the deployment with the following command:
az ml batch-deployment create --file deployment-by-batch.yml --endpoint-name $ENDPOINT_NAME --set-defaultYou can use this new deployment with the sample data shown before. Remember that to invoke this deployment you should either indicate the name of the deployment in the invocation method or set it as the default one.
Considerations for MLflow models processing images
MLflow models in Batch Endpoints support reading images as input data. Since MLflow deployments don't require a scoring script, have the following considerations when using them:
- Image files supported includes:
.png,.jpg,.jpeg,.tiff,.bmpand.gif. - MLflow models should expect to recieve a
np.ndarrayas input that will match the dimensions of the input image. In order to support multiple image sizes on each batch, the batch executor will invoke the MLflow model once per image file. - MLflow models are highly encouraged to include a signature, and if they do it must be of type
TensorSpec. Inputs are reshaped to match tensor's shape if available. If no signature is available, tensors of typenp.uint8are inferred. - For models that include a signature and are expected to handle variable size of images, then include a signature that can guarantee it. For instance, the following signature example will allow batches of 3 channeled images.
import numpy as np
import mlflow
from mlflow.models.signature import ModelSignature
from mlflow.types.schema import Schema, TensorSpec
input_schema = Schema([
TensorSpec(np.dtype(np.uint8), (-1, -1, -1, 3)),
])
signature = ModelSignature(inputs=input_schema)
(...)
mlflow.<flavor>.log_model(..., signature=signature)
You can find a working example in the Jupyter notebook imagenet-classifier-mlflow.ipynb. For more information about how to use MLflow models in batch deployments read Using MLflow models in batch deployments.
Next steps
Feedback
Submit and view feedback for