Share via

AI model deployment

  • Article

Model deployment involves taking a trained ML model, packaging it in a reusable asset (be it a container image or a pipeline), and deploying that asset to be used for inference. It also includes packaging data preparation code alongside the model to manipulate incoming data to match the expected inputs for the trained model.

💡Key Outcomes of Deployment:

  • Performance and parallel processing improvements are implemented against the optimum compute environment.
  • Monitoring and observability consistent with the entire solution have been implemented.
  • The target deployment endpoint is available to other services within the solution. Security, authentication and access controls are implemented.
  • Integration and system tests are in place.
  • A release pipeline per environment, bound to source control is implemented.
  • Batch or runtime scoring monitoring is in place to track the model’s performance over time.
  • User interactions with the model are logged for future improvement

Model Flighting

Details on Model Flighting can be found in the Azure ML official docs

Model Release

Details on Model Release can be found in Model Release

Deployment Infrastructure

There are several ways to deploy ML solutions in the cloud or on the edge.

Cloud-based deployment

Azure offers many different options for hosting ML solutions in the cloud.

Azure Kubernetes Service

Azure Kubernetes Service (AKS) makes it simple to deploy a managed Kubernetes cluster in Azure. AKS reduces the complexity and operational overhead of managing Kubernetes by offloading much of that responsibility to Azure. As a hosted Kubernetes service, Azure handles critical tasks like health monitoring and maintenance for you.

Azure Machine Learning

Azure Machine Learning service is a cloud service used to train, deploy, automate, and manage machine learning models, all at a broad scale.

Azure Logic Apps

Connect your business-critical apps and services with Azure Logic Apps, automating your workflows without writing a single line of code.

Azure Functions

Azure Functions is a computing service that allows you to run your code in response to events happening in Azure, third-party, or on-premises systems. It enhances the Azure application platform by providing an event-driven, compute-on-demand experience.

Azure AI Search is a managed service that provides an enrichment pipeline. This service can extract information, invoke custom APIs along with Azure AI Services and store the output to a datastore.

Edge-based deployment

In addition to the cloud deployment, the models created in Azure ML can also be deployed to "the edge".

In this case, the model is used on computes that reside outside the cloud - and either on premise, or on an isolated device.

The most common reasons for edge deployment are:

  • Performance the time taken to have the inference data travel up and down the cloud can be eliminated, reducing the total inference time

  • Security the user may be concerned about sending inference data to a cloud service. This may be due either to some regulations preventing them from doing so, or for some safety or security concerns.

The most common mechanism for edge-deployment is normally done by creating a docker container image with the ML model, and deploying this image to an on-premise or disconnected compute.

When that is the case, some mechanisms are used to deploy the container(s).

Azure IoT Edge Modules

Azure IoT Edge is a mechanism to deploy custom business logic to edge devices with centralized management via Azure IoT Hub.

More information on it can be found in What is IoT Edge?.

Kubernetes clusters and Azure ARC

Azure Arc simplifies governance and management of on-premise and multi-cloud infrastructure by delivering a consistent multi-cloud and on-premises management platform.

For instance, using Azure ARC the user can deploy containers directly to on-premise infrastructure (such as edge-devices).

More information on Azure ARC can be found in Azure ARC.

Others

Whilst there are other mechanisms to deploy Docker containers to edge devices, we will focus on the two mechanisms mentioned above.

Additional Deployment Information