how many models can be deployed in single node in azure kubernetes service?

suvedharan 21 Reputation points
2021-09-04T09:07:52.547+00:00

Working on deployment of 170 ml models using ML studio and azure Kubernetes service which is referred on the below doc link "https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/machine-learning/how-to-deploy-azure-kubernetes-service.md".

We are training the model using python script with the custom environment and we are registering the ml model on the Azure ML services. Once we register the mode we are deploying it on the AKS by using the container images.

While deploying the ML model we are able to deploy up 10 to 11 models per pods for each Node in AKS. When we try to deploy the model on the same node we are getting deployment timeout error and we are getting the below error message.
129300-image-2021-09-04t13-25-12-512z.png

Azure Machine Learning
Azure Machine Learning
An Azure machine learning service for building and deploying models.
2,657 questions
Azure Kubernetes Service (AKS)
Azure Kubernetes Service (AKS)
An Azure service that provides serverless Kubernetes, an integrated continuous integration and continuous delivery experience, and enterprise-grade security and governance.
1,931 questions
0 comments No comments
{count} votes

Accepted answer
  1. SUNOJ KUMAR YELURU 13,996 Reputation points MVP
    2021-09-04T12:32:05.4+00:00

    Hi @suvedharan

    The number of models to be deployed is limited to 1,000 models per deployment (per container).

    Autoscaling for Azure ML model deployments is azureml-fe, which is a smart request router. Since all inference requests go through it, it has the necessary data to automatically scale the deployed model(s).
    more details

    If the Answer is helpful, please click Accept Answer and up-vote, so that it can help others in the community looking for help on similar topics.


0 additional answers

Sort by: Most helpful