Performance Severly Degraded

Question

We are seeing severely degraded performance on our app service. The app seems to be crashing, http errors(5xx), and SNAT port exhaustion in the browser and in our logs.

Can someone help provide next steps for troubleshooting?

Answer 1

Hello @brtrach-MSFT

The major cause for intermittent connection issues is hitting a limit while making new outbound connections. The limits you can hit include:
TCP Connections: There is a limit on the number of outbound connections that can be made. The limit on outbound connections is associated with the size of the worker used.
SNAT ports: Each instance on Azure App service is initially given a pre-allocated number of 128 SNAT ports. The SNAT port limit affects opening connections to the same address and port combination. If your app creates connections to a mix of address and port combinations, you will not use up your SNAT ports. The SNAT ports are used up when you have repeated calls to the same address and port combination. Once a port has been released, the port is available for reuse as needed. The Azure Network load balancer reclaims SNAT port from closed connections only after waiting for 4 minutes.

I would also recommend you to increase instance size of your AppService and check Diagnose and solve problems option

Answer 2

There are a few recommendations here that can help with gathering additional logging to scope the issue.

First, we recommend that you

• Enable Application Insights
• Check Eventlog.xml file
• Enable Crash Monitoring

These logs should help get the information needed for debugging the code.
Another notable point in regards to SNAT Port Exhaustion however is to please ensure that you are reusing SNAT Connections. Here is an article that really helps with that issue in particular- https://devblogs.microsoft.com/premier-developer/reducing-snat-port-consumption-in-azure-app-services/

In addition to the above, we have a few best practices which can be configured for your application in order to ensure maximum uptime / best performance while running on the Cloud.
For example, configuring auto scale rules, disabling ARR affinity, and configuring Health Check to name some. All of these features and other best practices for your App Service are outlined at the following- https://azure.github.io/AppService/2020/05/15/Robust-Apps-for-the-cloud.html