Azure ML - AKS Service deployment unable to handle concurrent requests despite auto scaling

Subhankar Nayak 1

I have deployed around 23 models (amounting to 1.57 GB) in a Azure ML workspace using Azure Kubernetes Service. For the AKS cluster, I have used 3 D8sv3 nodes, and enabled cluster auto scaling for the cluster up to 6 nodes.
The AksWebService is configured with 4.4 cores, 16 GB memory. I have enabled pod auto scaling for the Web service, having set autoscale_max_replicas at 40:

aks_config = AksWebservice.deploy_configuration(cpu_cores = 4.4, memory_gb = 16, autoscale_enabled = True,  
                                                description = 'TEST - Configuration for Kubernetes Compute Target',  
                                                enable_app_insights = True, max_request_wait_time = 25000,  
                                                autoscale_target_utilization = 0.6, autoscale_max_replicas = 40)

I tried running load tests with 10 concurrent users (using JMeter). And I monitored the cluster application insights:

I can see the nodes and pods scaling. However, there is no spike in CPU/memory utilization. For 10 concurrent requests, only 5 to 6 requests pass, the rest fail. When I send an individual request to the deployed endpoint, the response is generated in 7 to 9 seconds. However, in the load test logs, there are plenty requests taking more than 15 seconds to generate a response. And the requests taking more than 25 seconds, fail with status code 503. I increased the max_request_wait_time due to this reason, however, I don't understand why it would take so much time despite such amount of compute, and the dashboard shows that memory isn't even 30% utilized. Should I be changing the replica_max_concurrent_requests param? Or should I be increasing the autoscale_max_replicas even more? Concurrent requests load may sometimes reach 100 in production, is there any solution to this?

Will be grateful for any advice. Thanks.

romungi-MSFT 41,861 Microsoft Employee

@Subhankar Nayak A response time of 7 to 25 seconds sounds quite high. Did you get a chance to look at the guidance from the trouble shooting document for AKS deployments mentioned here, since you are seeing 503 status codes? The guidance to calculate the replicas is

from math import ceil  
# target requests per second  
targetRps = 20  
# time to process the request (in seconds)  
reqTime = 10  
# Maximum requests per container  
maxReqPerContainer = 1  
# target_utilization. 70% in this example  
targetUtilization = .7  
  
concurrentRequests = targetRps * reqTime / targetUtilization  
  
# Number of container replicas  
replicas = ceil(concurrentRequests / maxReqPerContainer)

Subhankar Nayak 1 Reputation point

2020-10-01T13:48:25.637+00:00

Hi, Yes I did go through this doc. Since I expect around 200 concurrent requests, it will require 200 replicas. But is this feasible? Because, I am not sure about how many containers each replica deploys. Will the current compute of the cluster be sufficient for 200 replicas simultaneously? Also, keeping autoscale max replica at 40, test for 10 concurrent users fail. So am not sure if increasing replicas limit will solve this. That is why I was wondering if I can use the replica_max_concurrent_requests param to better utilize the replicas (the docs mention that this param should be changed only if advised by someone from the Azure Technical support or ML Team)
romungi-MSFT 41,861 Reputation points Microsoft Employee

2020-10-05T10:21:10.95+00:00

@Subhankar Nayak Yes, the documentation advises not to change this setting unless reviewed by our ML team. Since your requirement is to use 200 concurrent requests or 200 replicas which might need a change in replica_max_concurrent_requests parameter you can request a review of your setup with our ML team by providing feedback using the smiley icon from the top right hand side of the ml.azure.com page.

Please ensure to check the option to allow MS to contact you over email while providing the details in this request so our team can get back to you. If you need further help or if this no response from this request we can connect you directly with our technical support team by providing a one time support request for this issue.