Azure Application Gateway showing unusual Healthy Host Count

Paul Richards 0 Reputation points

A few days ago we started receiving unhealthy host greater than 0 alerts for a little cluster of three custom auth services. It's not unusual for this to happen from time to time and typically restarting the container(s) is enough to correct the issue. In this case however we haven't been able to remedy the issue even after restarting all the containers and refreshing the backend pools entries. Upon further investigation into the Application Gateway metrics we noticed a very unusual trend in the Healthy Host Count where it grew beyond the typical 3 hosts in a linear fashion to 10s of thousands???. I have also attached the Unhealthy Host Count chart for the same period which shows the a number of unhealthy hosts popping up in stark contrast to the beginning of the chart where activity was normal. We have not changed these services recently and have ruled out a db issue but our main confusion is how so many hosts are being reported as healthy when the max should be 3 and what to try next? User's image

User's image

Azure Application Gateway
Azure Application Gateway
An Azure service that provides a platform-managed, scalable, and highly available application delivery controller as a service.
978 questions
Azure Database for PostgreSQL
{count} votes