HPA is not working properly in AKS

Somiya 246 Reputation points
2022-05-11T07:16:04.457+00:00

Hi team,
I have deployed an HPA and defined the metrics for the CPU and memory, in case of high load the pod as well as the cluster is scaling but when the load reduces they are not scaling down. Please find the below details
200853-image.png

200827-image.png

error on the HPA

 Warning  FailedGetResourceMetric       8m2s (x22 over 111m)  horizontal-pod-autoscaler  failed to get cpu utilization: did not receive metrics for any ready pods     
  Warning  FailedComputeMetricsReplicas  8m2s (x8 over 54m)    horizontal-pod-autoscaler  invalid metrics (1 invalid out of 2), first error is: failed to get cpu utilization: did not receive metrics for any ready pods     

HPA deployment file

apiVersion: autoscaling/v2beta2     
kind: HorizontalPodAutoscaler     
metadata:     
  name: dev-hpa     
  namespace: dev-iapi     
spec:     
  maxReplicas: 5 # define max replica count     
  minReplicas: 2  # define min replica count     
  scaleTargetRef:     
    apiVersion: apps/v1     
    kind: Deployment     
    name: dev-deployment     
  metrics:     
    - type: Resource     
      resource:     
        name: cpu     
        target:     
          type: Utilization     
          averageUtilization: 70     
    - type: Resource     
      resource:     
        name: memory     
        target:     
          type: Utilization     
          averageUtilization: 65     
Azure Kubernetes Service (AKS)
Azure Kubernetes Service (AKS)
An Azure service that provides serverless Kubernetes, an integrated continuous integration and continuous delivery experience, and enterprise-grade security and governance.
1,992 questions
{count} votes

7 answers

Sort by: Most helpful
  1. Jochem van Eldijk 6 Reputation points
    2022-06-15T15:20:43.057+00:00

    i am facing the same issue. Would be interested in any response you could give. @KarishmaTiwari-MSFT

    1 person found this answer helpful.
    0 comments No comments

  2. Bandhaalli, KISHORE 1 Reputation point
    2022-08-16T11:17:05.84+00:00

    i am also seeing similar behavior of error messages. Is the root cause identified ??

    0 comments No comments

  3. Lopes Gomes, Eduardo 21 Reputation points
    2022-08-19T11:57:08.317+00:00

    I'm having this same problem, I'll follow this post to see if anyone found the solution.

    0 comments No comments

  4. GERVAIS Hugo 1 Reputation point
    2022-09-02T14:33:32.027+00:00

    i am also seeing similar behavior of error messages. It starting to become a big problem for your organisation , I had to static number of pods because of this (autoscaling not possible anymore)


  5. GERVAIS Hugo 1 Reputation point
    2022-09-05T16:10:54.293+00:00

    Hi , im using aks version 1.24.3

    0 comments No comments