Exposing /healthz route explicitly on cluster v1.23 caused outage

Question

Our team still has clusters running on version 1.23.x that we're planning to upgrade to 1.24.x in the near future. We are aware of the issue with 1.24 where the load balancer always performs a health probe check and / is used by default, even if the azure-load-balancer-health-probe-request-path annotation isn't specified. Since Nginx ingress exposes /healthz instead of /, the load balancer probe check fails, and that resulted in requests not being sent to the backend.

To work around this, we explicitly specify in the annotation to expose /healthz for our v1.24 clusters, and this works. However, when we attempt to do the same for our v1.23 cluster, this still somehow caused requests to not get send to our backend. We were under the assumption that since Nginx ingress always exposes /healthz regardless of AKS version, the load balancer health probe check should not be failing with the added annotation. Are we missing something?

The annotation that was added is: service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /heathz

Answer 1

@Thomas Phan Thanks for posting your query on Microsoft Q&A.

I was able to check with the internal team. Prior to 1.24+, it uses TCP as default health probe protocol but from 1.24+ onwards it uses HTTP/HTTPS as health probe protocol and requires /healthz path to check backend health check.

If you put /healthz path in earlier versions than 1.24+ it will not work as this is the behavioral change from Kubernetes version 1.24+ onwards.

I hope this helps. If you have any questions at all, please let me know in the "comments" and I would be happy to help you. Comment is the fastest way of notifying the experts.

Please don’t forget to Accept Answer and hit Yes for "was this answer helpful" wherever the information provided helps you. This can be beneficial to other community members for remediation for similar issues.