Share via

Real-time scoring

  • Article

Scoring implementation is on a per-record basis, and response time is not more than 60 seconds. Below are brief discussions of some different deployment options. For a brief comparison of each type of scoring refer to the Scoring type table.

Managed Azure Kubernetes Service (AKS)/Arc Kubernetes

In the case when a customer prefers to manage infrastructure themselves and connect components from different subscriptions, AKS is the best option. Azure ML Workspace can be linked to AKS and deployment can happen automatically using standard or custom images. This choice provides the highest flexibility in managing a cluster, which enables many enterprise grade cluster services to be utilized.

If the Kubernetes is outside the Azure cloud, it can still be connected to Azure using Azure Arc (let's use the Arc Kubernetes name for such a compute target). With an Arc Kubernetes cluster, it's possible to build, train, and deploy models into any infrastructure on-premises and across multi-cloud using Kubernetes.

Advantages*

  • Self-managed infrastructure (advantage and disadvantage).
  • Proven technology that supports auto-scaling, monitoring, VNET, and GPU nodes.
  • AKS and Azure ML can be in different subscriptions.
  • External Kubernetes cluster can be used thanks to Azure Arc.

Disadvantages:

  • User responsibility for the infrastructure.
  • Complexity.

For more information:

Managed Online Endpoints

This is the recommended approach by the Azure ML team.

Advantages:

  • Azure ML managed infrastructure based on Azure ML Compute.
  • Logging and monitoring are supported (out-of-the-box).
  • Native blue/green deployment.

Disadvantages:

  • No ability to publish several endpoints to the same compute cluster.
  • VNET support still is in Public Preview.

For more information:

Azure Functions

Azure Functions is a serverless compute service in Azure. The Azure Functions service is effective if it is possible to implement an inferencing script as a single thread process, particularly where a GPU is not required. In most cases, the model is likely to be small and a data preprocessing flow is not needed or doesn’t require many resources. Typically event-driven, short-lived and low compute workloads are recommended. However, Azure Functions may not be the optimal choice for high compute workloads.

Advantages:

  • Supports a staging deployment Platform as a Service approach if deployed to Azure.
  • Can be built in many different programming languages like Python and C#.
  • Supports deployment in containers, if needed, and can be deployed locally.

Disadvantages:

  • Doesn’t support GPU.
  • Latency in ML solutions can be a problem.
  • Azure ML doesn’t support this way to deploy.

For more information:

Azure Container Instances (ACI)

Azure Container Instance (ACI) is an effective way to deploy images and containers in an environment that is less complex than a full orchestration cluster/system such as Azure Kubernetes Service (AKS). Containers can be deployed via images stored in a public or private registry. Runtimes and environments are customizable per image and both Windows and Linux are supported. Deployments of approved images increase version control, testing, and consistency across containers.

NOTE: Using ACI in production isn't recommended since there are limitations to ACI availability and scaling. Refer to the AKS solutions for a containerized deployment approach.

Advantages:

  • Supports deployment in containers and can be deployed locally.
  • Images can be stored in an enterprise Azure Container Registry (ACR) with versioning.
  • ACI, ACR and Azure ML can be in different subscriptions.

Disadvantages:

  • Cannot control orchestration of a cluster.
  • Since ACI is a manged service, certain network restrictions apply, including dynamic IPs.
  • Limitations for scalability and high availability.

For more information:

Unmanaged Kubernetes/Azure container Apps/IoT Edge

All the compute targets in this option have one common property: they are not supported by Azure ML. Therefore, if there is a requirement to support one of these targets, it's your responsibility to build a custom image and deploy it to the compute. This option makes sense if all other options are not available.

Advantages:

  • Supports non-standard scenarios.

Disadvantages:

  • User responsibility for the infrastructure.
  • User responsibility for building the right image and deployment.
  • Complexity

For more information:

Azure Container Apps documentation