How to scale out App Service Plan based on HTTP status 503 errors?

Question

When our App Service Plan is overloaded, many clients get HTTP status 503: Service Unavailable errors. I haven't been able to find any data in available metrics that correlates closely enough to 503 errors to effectively react with scale out. One part of the service's client base uploads error logs to the service that are persisted in the database. Enough of them are getting through during these periods that this data would be an appropriate scale out factor if I could figure out how to implement it. I looked at Logic Apps that can query for the persisted error logs but I can't find any way for a Logic App to implement App Service Plan scale out function. I am far from being an expert in all Azure services. Can you give me any ideas to implement scale out in this scenario? Thank you.

Update: I observed a spate of 503's one night lasting from 00:00 UTC through 07:00 UTC. These are off-peak hours and all resource demands were low. There was no maintenance of any kind on our behalf occurring. This spate of 503's was a singular, random occurrence and follows no observable pattern in any resource metric.
Conclusion: 503's were possibly a result of some transient, unknown issue with Azure infrastructure. As long as it doesn't become common, scaling out for 503's is probably not a productive use of time. Whether or not scaling out would have done any good in this scenario is unknown.

Answer 1

Hi @RJBreneman ,

The best practice is to configure auto-scale for App Service based on a threshold e.g. 70%-80% CPU/Memory utilization before the App Service Plan is overloaded. You can arrive at a threshold by monitoring the App Service plan,

But to answer you question on scale out using custom metrics like request/service failure, below are few options:

One way to scale out using custom metrics is by enabling Application Insights for the App Service as described here https://learn.microsoft.com/en-us/azure/azure-monitor/autoscale/autoscale-custom-metric
Application Insights also has standard metrics that include count of failed requests https://learn.microsoft.com/en-us/azure/azure-monitor/app/standard-metrics#failure-metrics

To scale out using Azure Functions, you can try to write a powershell function as per this link https://learn.microsoft.com/en-us/azure/app-service/scripts/powershell-scale-manual

Answer 2

Azure App Service environments support autoscaling. You can autoscale individual worker pools based on metrics or schedule. Autoscaling optimizes your resource utilization by automatically growing and shrinking an App Service environment to fit your budget and or load profile.

You scale up by changing the pricing tier of the App Service plan that your app belongs to. Scale out: Increase the number of VM instances that run your app. You can scale out to as many as 30 instances, depending on your pricing tier.