AI model deployment
Model deployment involves taking a trained ML model, packaging it in a reusable asset (be it a container image or a pipeline), and deploying that asset to be used for inference. It also includes packaging data preparation code alongside the model to manipulate incoming data to match the expected inputs for the trained model.
💡Key Outcomes of Deployment:
- Performance and parallel processing improvements are implemented against the optimum compute environment.
- Monitoring and observability consistent with the entire solution have been implemented.
- The target deployment endpoint is available to other services within the solution. Security, authentication and access controls are implemented.
- Integration and system tests are in place.
- A release pipeline per environment, bound to source control is implemented.
- Batch or runtime scoring monitoring is in place to track the model’s performance over time.
- User interactions with the model are logged for future improvement
Model Flighting
Details on Model Flighting can be found in the Azure ML official docs
Model Release
Details on Model Release can be found in Model Release
Deployment Infrastructure
There are several ways to deploy ML solutions in the cloud or on the edge.
Cloud-based deployment
Azure offers many different options for hosting ML solutions in the cloud.
Azure Kubernetes Service
Azure Kubernetes Service (AKS) makes it simple to deploy a managed Kubernetes cluster in Azure. AKS reduces the complexity and operational overhead of managing Kubernetes by offloading much of that responsibility to Azure. As a hosted Kubernetes service, Azure handles critical tasks like health monitoring and maintenance for you.
Azure Machine Learning
Azure Machine Learning service is a cloud service used to train, deploy, automate, and manage machine learning models, all at a broad scale.
Azure Logic Apps
Connect your business-critical apps and services with Azure Logic Apps, automating your workflows without writing a single line of code.
Azure Functions
Azure Functions is a computing service that allows you to run your code in response to events happening in Azure, third-party, or on-premises systems. It enhances the Azure application platform by providing an event-driven, compute-on-demand experience.
Azure AI Search
Azure AI Search is a managed service that provides an enrichment pipeline. This service can extract information, invoke custom APIs along with Azure AI Services and store the output to a datastore.
Edge-based deployment
In addition to the cloud deployment, the models created in Azure ML can also be deployed to "the edge".
In this case, the model is used on computes that reside outside the cloud - and either on premise, or on an isolated device.
The most common reasons for edge deployment are:
Performance the time taken to have the inference data travel up and down the cloud can be eliminated, reducing the total inference time
Security the user may be concerned about sending inference data to a cloud service. This may be due either to some regulations preventing them from doing so, or for some safety or security concerns.
The most common mechanism for edge-deployment is normally done by creating a docker container image with the ML model, and deploying this image to an on-premise or disconnected compute.
When that is the case, some mechanisms are used to deploy the container(s).
Azure IoT Edge Modules
Azure IoT Edge is a mechanism to deploy custom business logic to edge devices with centralized management via Azure IoT Hub.
More information on it can be found in What is IoT Edge?.
Kubernetes clusters and Azure ARC
Azure Arc simplifies governance and management of on-premise and multi-cloud infrastructure by delivering a consistent multi-cloud and on-premises management platform.
For instance, using Azure ARC the user can deploy containers directly to on-premise infrastructure (such as edge-devices).
More information on Azure ARC can be found in Azure ARC.
Others
Whilst there are other mechanisms to deploy Docker containers to edge devices, we will focus on the two mechanisms mentioned above.
Additional Deployment Information
- For an in-depth exploration of using Azure Machine Learning to deploy models, see the Azure ML Deployment Trade Study document in this repository.
- Official Azure ML docs on deployment: