Share via

How to avoid terminate issues on AKS when scaling in

Mesquita, Walter 96 Reputation points
2021-05-13T14:23:10.057+00:00

I have an internal service (rest APIs) in AKS with HPA configured to it. It works fine when it needs to scale-out to handle a higher traffic period. Although when AKS understands that it can scale-in I start having issues. I got a lot of 503 errors in the clients, even after the pod starts its termination process the pod continues receiving requests.

I found a very similar 2-years-age issue with ingress: https://github.com/Azure/application-gateway-kubernetes-ingress/issues/641

Anyone else with the same issue? Anyone with an idea on how to solve it?

Azure Kubernetes Service
Azure Kubernetes Service
An Azure service that provides serverless Kubernetes, an integrated continuous integration and continuous delivery experience, and enterprise-grade security and governance.
2,447 questions
Accepted answer
  1. Mesquita, Walter 96 Reputation points
    2021-06-07T11:45:59.293+00:00

    I am using SpringBoot in my REST services and I needed to set the spring property server.shutdown=graceful. I left the management of pod shutdown with K8s default behavior.

    0 comments

1 additional answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. prmanhas-MSFT 17,946 Reputation points Microsoft Employee Moderator
    2021-05-17T07:17:03.24+00:00

    @Mesquita, Walter Apologies for the delay in response and all the inconvenience caused because of the issue.

    The mentioned Git issue has explained the solution well. This comment has explained it in details.
    The key here is preStop hook and terminationGracePeriodSeconds. With Kubernetes v1.21, a probe level terminationGracePeriodSeconds has also been introduced in alpha as mentioned here.

    You can read more about it here.

    Hope it helps!!!

    Please "Accept as Answer" if it helped so it can help others in community looking for help on similar topics.

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.