Share via

Batch scoring

  • Article

Scoring is done for a set of records simultaneously and happens on a schedule. Response time varies from minutes to hours or even days. Useful for periodic scoring.

Azure ML Pipelines

Azure ML supports the ParallelRunStep class that is useful for parallel data preprocessing and for batch inferencing. In general, a batch inferencing pipeline is no different from a training pipeline, but it uses existing model(s) rather than training a new model. Let’s look at some advantages/disadvantages of batch inferencing:

Advantages:

  • Azure ML Compute(s) can be utilized with CPU or GPU nodes when needed.
  • Ability to use different node types for different steps.
  • Training code for data preprocessing can be reused.
  • MLOps (Machine Learning development and Operation process) is the same as for training pipeline (except moving code into production).
  • Potentially, it can be executed on-premises (via Azure Arc).

Disadvantages:

  • Azure ML is required to be in the Production environment (including related services).
  • Data should be located in a supported data storage that might require more services like Azure Data Factory to orchestrate the pipeline\n-

For more information:

Batch scoring endpoints

Batch scoring endpoints are based on the ParallelRunStep and Azure ML Compute, and they bring some simplicity to the Azure ML Pipeline approach. The main idea is to allow for the ability to deploy a batch scoring pipeline using just one line of code (CLI), model, and scoring script. Thus, all the code that creates a pipeline is not needed. At the same time, it assumes that inferencing pipeline is a single-step flow. So, it’s unusable for complex pipelines.

Advantages:

  • Allows developers to focus their efforts on scoring script only.
  • Azure ML Compute(s) can be utilized with CPU/GPU nodes when needed.
  • MLOps can be implemented as a series of bash scripts thanks to Azure ML CLI 2.0.

Disadvantages:

  • Single step pipelines only (a single step based on ParallelRunStep).
  • Azure Blob should be used as for now.<!--- should this be "...be used as is for now? --->
  • Azure Machine Learning Workspace is required in the Production environment (including related services).

For more information:

Batch Scoring using Databricks

Databricks can be used for Batch inferencing if Azure Machine Learning is not allowed in the Production environment. This decision must be made carefully because it introduces complexity to the scoring pipeline and to the MLOps process. The typical MLOps pattern in this scenario uses MLFlow projects, but access to the code repository is required to execute the scoring flow.

Advantages:

  • Ability to use CPU or GPU automated clusters by request.
  • Integration with MLFlow and some integration with Azure.

Disadvantages:

  • Databricks MLOps is different and it adds further complexity to the project.
  • Deep Spark knowledge is required to implement batch scoring on a cluster rather than on a single node.
  • No clear/simple ability to use different clusters (node types) for different steps.
  • Hard to reuse existing code from Azure ML to implement pipelines.

For more information: