HPA is not working properly in AKS

Question

Hi team,
I have deployed an HPA and defined the metrics for the CPU and memory, in case of high load the pod as well as the cluster is scaling but when the load reduces they are not scaling down. Please find the below details

error on the HPA

 Warning  FailedGetResourceMetric       8m2s (x22 over 111m)  horizontal-pod-autoscaler  failed to get cpu utilization: did not receive metrics for any ready pods     
  Warning  FailedComputeMetricsReplicas  8m2s (x8 over 54m)    horizontal-pod-autoscaler  invalid metrics (1 invalid out of 2), first error is: failed to get cpu utilization: did not receive metrics for any ready pods     

HPA deployment file

apiVersion: autoscaling/v2beta2     
kind: HorizontalPodAutoscaler     
metadata:     
  name: dev-hpa     
  namespace: dev-iapi     
spec:     
  maxReplicas: 5 # define max replica count     
  minReplicas: 2  # define min replica count     
  scaleTargetRef:     
    apiVersion: apps/v1     
    kind: Deployment     
    name: dev-deployment     
  metrics:     
    - type: Resource     
      resource:     
        name: cpu     
        target:     
          type: Utilization     
          averageUtilization: 70     
    - type: Resource     
      resource:     
        name: memory     
        target:     
          type: Utilization     
          averageUtilization: 65     

Answer 1

i am facing the same issue. Would be interested in any response you could give. @KarishmaTiwari-MSFT

Answer 2

i am also seeing similar behavior of error messages. Is the root cause identified ??

Answer 3

I'm having this same problem, I'll follow this post to see if anyone found the solution.

Answer 4

i am also seeing similar behavior of error messages. It starting to become a big problem for your organisation , I had to static number of pods because of this (autoscaling not possible anymore)

Answer 5

Hi , im using aks version 1.24.3