Working on deployment of 170 ml models using ML studio and azure Kubernetes service which is referred on the below doc link "https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/machine-learning/how-to-deploy-azure-kubernetes-service.md". We are training the model using python script with the custom environment and we are registering the ml model on the Azure ML services. Once we register the mode we are deploying it on the AKS by using the container images. While deploying the ML model we are able to deploy up 10 to 11 models per pods for each Node in AKS. When we try to deploy the model on the same node we are getting deployment timeout error and we are getting the below error message.

Hi @suvedharan The number of models to be deployed is limited to 1,000 models per deployment (per container). Autoscaling for Azure ML model deployments is azureml-fe, which is a smart request router. Since all inference requests go through it, it has the necessary data to automatically scale the deployed model(s). more details If the Answer is helpful, please click Accept Answer and up-vote , so that it can help others in the community looking for help on similar topics.

how many models can be deployed in single node in azure kubernetes service?

Accepted answer

SUNOJ KUMAR YELURU 14,021 Reputation points MVP

2021-09-04T12:32:05.4+00:00

Hi @suvedharan

The number of models to be deployed is limited to 1,000 models per deployment (per container).

Autoscaling for Azure ML model deployments is azureml-fe, which is a smart request router. Since all inference requests go through it, it has the necessary data to automatically scale the deployed model(s).
more details

If the Answer is helpful, please click Accept Answer and up-vote, so that it can help others in the community looking for help on similar topics.
Please sign in to rate this answer.
suvedharan 21 Reputation points

2021-09-04T13:35:36.017+00:00

For deploying the model in Azure Kubernetes Service using python language with below deployment configuration (auth_enabled = Flase, autoscale_enabled = Flase, cpu_cores = 0.6, memory_gb = 1, cpu_cores_limit = 2, memory_gb_limit = 2).

We also checked on the azure documentation and we could able to find any configuration or deployment setup for aks nodes.

Can you please provide us more clarification regarding "The number of models to be deployed is limited to 1,000 models per deployment (per container)" and Can you please give insight/feedback on how to increase the number of a ml model deployed in each node Azure Kubernetes Service. Thanks!

SUNOJ KUMAR YELURU 14,021 Reputation points MVP

2021-09-04T15:12:31.98+00:00

To deploy a model to Azure Kubernetes Service, create a deployment configuration that describes the compute resources needed. For example, number of cores and memory. You also need an inference configuration, which describes the environment needed to host the model and web service. refer https://learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-azure-kubernetes-service?tabs=python#deploy-to-aks

from azureml.core.webservice import AksWebservice, Webservice from azureml.core.model import Model from azureml.core.compute import AksCompute aks_target = AksCompute(ws,"myaks") # If deploying to a cluster configured for dev/test, ensure that it was created with enough # cores and memory to handle this deployment configuration. Note that memory is also used by # things such as dependencies and AML components. deployment_config = AksWebservice.deploy_configuration(cpu_cores = 1, memory_gb = 1) service = Model.deploy(ws, "myservice", [model], inference_config, deployment_config, aks_target) service.wait_for_deployment(show_output = True) print(service.state) print(service.get_logs())

Deploy your machine learning or deep learning model as a web service in the Azure cloud.
Deploy machine learning models to Azure
Sign in to comment

Share via

how many models can be deployed in single node in azure kubernetes service?

0 additional answers