Unknown Azure Web App Recycling Causing Websocket Issues

Question

We have an Azure Web App running on a single Instance which is being recycled at random times. It takes about 15 mins each time for what appears to be a new instance of the Web App to be created and the old instance to be disabled. This is causing problems as the Web App is a web socket server and whenever the app is recycled the web sockets are still connected to the old instance until that 15 mins is completed at which time our code detects that the connection has been ended and restarts the web sockets.

Another interesting point is that we have 7 identical Web Apps sharing a single App Service Plan. 5 of those 7 Web Apps are getting recycled at the same times, but 2 are not.
In the Web App Metrics the Count of Total App Domains is one of the indicators I'm looking at to know when each of these 15 min recycles is happening (See Image Below).

The Web App Application Event Logs also show the following three MsiInstaller Warnings each time the app is recycled.

These instances do have Proactive Auto-Heal enabled, but the Auto-Heal history doesn't show any events in this 24 hour window.

My question is what could be causing this recycling? What else should I look at on the Web App to research this? We do have a plan to make our web socket implementation handle this type of occurrence, so we don't need help there, only to understand why this recycling is happening.

Answer 1

@Lowell Burns , 1/2

Thanks for posting this question. Apologies for any inconvenience with this issue and for a long post, I'm highlighting a few ways to identify the issue and make some considerations for your use-case.

  Based on the information you provided, I understand that your Azure Web App is being recycled at random times, which is causing problems for your web socket server.  

Just to clarify, what App Service Plan (ASP) are you leveraging?

I understand 5/7 WebApps are getting restarted, are all those Windows or Linux Apps?  

1. One possible reason for the issue could be that the App Service Plan (ASP) is running out of resources, such as memory or CPU. You can check the resource usage of your Web App by going to the "Metrics" section in the Azure portal and selecting the appropriate metrics, such as CPU usage or memory usage. If you find that the resource usage is consistently high. (If feasible), I understand you’re looking to understand the cause of the recycle problem and plan to handle this in socket, the following would help isolate the issue. you can consider moving the Web Apps to separate ASP or scaling up the existing App Service Plan to a higher tier.
2. If the Always On setting is off, please enable it from Azure Portal –---Your WebApp > Configuration blade (> General settings).
3. You may leverage App Service diagnostics from Azure Portal> Navigate to your App Service app in the Azure Portal. (screenshot below) ---- In the left navigation, click on Diagnose and solve problems - Review - “Configuration and Management" and “Availability and Performance” options.

I hope this information helps point you in the right direction. If you have any further questions or concerns, please let us know.

Answer 2

@Lowell Burns , 2/2

Regarding the issue with 5 of your 7 Web Apps getting recycled at the same times, it could be due to the fact that they are sharing the same ASP. When multiple Web Apps are running on the same ASP, they share the same resources, which can lead to resource contention and recycling of the worker process.  

Some consideration depending on your scenario:

1. Please checkout the App Service limits for each ASPs ( resources – compute, scale-out, CPU time, Web sockets per instance) -- App Service limits -For example  - If you scale an app in the Basic tier to two instances, you have 350 concurrent connections for each of the two instances. For Standard tier and above, there are no theoretical limits to web sockets, but other factors can limit the number of web sockets. For example, maximum concurrent requests allowed (defined by maxConcurrentRequestsPerCpu) are: 7,500 per small VM, 15,000 per medium VM (7,500 x 2 cores), and 75,000 per large VM (18,750 x 4 cores).
2. Why should you care about TCP Connections?
   Let's say that you have created two web apps on the ASP, and both are breaking. An example of when TCP Connections can cause this behavior in your web apps could be that one app is leaking a lot of socket handles and ends up hitting the machine wide TCP Connection limit. App Service enforces limits on the number of outbound connections that can be outstanding at any given point in time. When web apps run into these connection limits, they will start intermittently failing because calls to those remote endpoints will fail, causing downtime.
3. Please check this FAQ: The app is resource-intensive. 
   Should I put an app in a new plan or an existing plan?