How to get notified when Diagnostic events occurs
Last night, we had an event where all of our app services worker processes were recycled. This lead to some strange errors and also an MySql-in-app service not restarting as well (or because of?).
Initially we just saw some strange error logs while we could see that MySql-in-app was not running (we use it for a small production setup). Finally, after alot of digging in Application Insights and on-premise event logs (we have an SQL Server networked in via VPN), we noticed this thing here:
The full message is:
... your application was recycled due to an intermittent Azure infrastructure issue while accessing remote file storage. This can happen due to multiple reasons like platform instances getting upgraded or instance(s) experiencing high latencies accessing the remote storage. In case the instance(s) where your application is running is experiencing high latencies accessing remote storage, the platform tries to heal your application by switching to different remote storage which is having low latency. This can also cause the application process to restart. If this is a one off occurrence, then no further action plan is needed from your side. But if you see it occur frequently, then enabling local cache feature can help reduce dependency on storage file servers to some extent.
This message seems to appear in all App Services, running on our App Service Plan. The timestamp is also the same. Our ASP is a Windows based P3v2 with a single instance (I know). It's hosted in North Europe, and from the Service Health app, I don't see any issues that would have impacted us.
So now I have a few questions:
- Is there any way to find out the REAL cause here? Was the storage at issue something we've setup, or something internal to Azure?
- Is this message/recycling feature related to Proactive Auto-Heal? We do run a Windows plan and the feature is enabled.
- And perhaps most importantly, is there any way to get notified when this occurs? I've been looking through Azure Monitor, but I cannot find anything. Possibly I need to setup diagnotics settings for the App Service Plan, or something else?